dataflow/ERR/2023_004

Dataflow job does not fail during execution due to violating an organization policy constraint in project

Product: Dataflow
Rule class: ERR - Something that is very likely to be wrong

Description

A Dataflow job might fail if there are organization policy constraints (eg: constraints/compute.vmExternalIpAccess, constraints/compute.requireShieldedVm etc.) that might limit VM instance creation in their project.

You can search in the Logs Explorer for such jobs with the logging query:

resource_type='dataflow_step',
severity=ERROR
log_id("dataflow.googleapis.com/job-message")
textPayload=~"Constraint constraints/.* violated for project"

Remediation

The error occurs because your organization has constraints in place that prohibit some aspect of the VM instance creation process, like the account being used, or the zone being targeted. A possible resolution can be found in doc.

Recommendations for specific constraints violated are as follows:

  • constraints/compute.vmExternalIpAccess - This indicates that you have a policy that restricts the use of external IP addresses. To fix this error, you can set the no_use_public_ips flag to use internal IP addresses for all communication within Dataflow workers.

  • constraints/compute.requireShieldedVm - This indicates that you have a policy that requires all new Compute Engine VM instances use Shielded disk images with Secure Boot, vTPM, and Integrity Monitoring options enabled. To fix this error, we can specify enable_secure_boot in dataflow_service_options configuration, or through experiment –experiments=enable_secure_boot to be compliant with constraints/compute.requireShieldedVm org policy.

For jobs affected by other organization policy constraints, please refer the documentation to identify how resource is being restricted and work with your organization admin to overcome these errors.

Further information