dataflow/ERR/2024_001

Dataflow job is not facing GCE resource constraints.

Product: Dataflow
Rule class: ERR - Something that is very likely to be wrong

Description

Dataflow jobs may fail due to GCE resource quotas which can inadversedly cause jobs to fail to launch.

You can search in the Log Explorer for such jobs with the below mentioned logging query:

resource_type='dataflow_step',
severity>=WARNING
log_id("dataflow.googleapis.com/job-message")
textPayload~="has insufficient quota(s) to execute this workflow" OR "The quota check has failed" OR "Quota exceeded" OR "Compute Engine API has not fully initialized" OR "Throttling logger worker" OR "This workload is drawing too much egress bandwidth from Cloud Storage" OR "Per-customer shuffle size limit exceeded" OR "RESOURCE_EXHAUSTED: Exceeds 'AppendRows throughput' quota" OR "RESOURCE_EXHAUSTED: Exceeds 'Concurrent connections'" OR "RESOURCE_EXHAUSTED: Exceeds 'CreateWriteStream requests'"

Remediation

Below are the remediation for different types of errors:

  1. ‘has insufficient quota(s) to execute this workflow’, ‘The quota check has failed’, ‘Quota exceeded’, Please refer to the following documentation to request for a quota increase.

  2. “Compute Engine API has not fully initialized” Please reach out to Cloud Support Team to help increase quota.

  3. “Throttling logger worker” Dataflow job is running into Dataflow abusive logs quota. This limit is hardcoded and cannot be increased. Possible mitigations include logging only on errors or writing these logs to storage sink (BigQuery, GCS, etc.). Reference

  4. “This workload is drawing too much egress bandwidth from Cloud Storage” Dataflow job is running into Cloud Storage egress bandwidth quota errors. This quota depends on bucket settings and may be different for single or multi-region buckets.

Please check the documentation on how to increase the limits for the project. Reference

  1. “Per-customer shuffle size limit exceeded” OR “RESOURCE_EXHAUSTED: Exceeds ‘AppendRows throughput’ quota” OR “RESOURCE_EXHAUSTED: Exceeds ‘Concurrent connections’ quota” OR “RESOURCE_EXHAUSTED: Exceeds ‘CreateWriteStream requests’ quota” Please reach out to Cloud Support Team to help increase quota.

Further information