dataflow/ERR/2023_003
Dataflow job does not fail during execution due to incorrect specification of subnet
Product: Dataflow
Rule class: ERR - Something that is very likely to be wrong
Description
A dataflow job runs successfully if subnet is properly specified while launching the job, otherwise it fails with Invalid subnetwork specified error.
You can search in the Logs Explorer for such jobs with the logging query:
resource.type="dataflow_step"
log_id("dataflow.googleapis.com/job-message")
severity=ERROR
textPayload=~"Workflow failed. Causes: Invalid subnetwork specified: .* should match regions/REGION/subnetworks/SUBNETWORK."
Remediation
- If the subnetwork is located in a Shared VPC network, you must use the complete URL. When specifying the subnetwork URL for Shared VPC, ensure that HOST_PROJECT_ID is the project in which the VPC is hosted. Please make sure specified subnet exists in your project if you are not using shared VPC or exists in host project if you are using shared VPC. For best practices, please follow this doc.