dataflow/WARN/2024_001

Dataflow job is not returning Operation ongoing or Processing Stuck logs.

Product: Dataflow
Rule class: WARN - Something that is possibly wrong

Description

The Dataflow job will return this warning when your DoFn code is slow, or waiting for some slow external operation to complete or when your DoFn code might be stuck, deadlocked, or abnormally slow to finish processing.

You can search this in Logs Explorer for such jobs with the below mentioned Logging Query:

  """
  resource.type="dataflow_step"
  resource.labels.job_id="%dataflowJobID%"
  "Operation ongoing in step" OR "Processing stuck in step" OR "Operation ongoing for over" OR "Operation ongoing in transform" OR "Operation ongoing in bundle"
  severity>="WARNING"
  """

Remediation

To determine which is the case, expand the Cloud Monitoring log entry to see a stack trace by running the above log query. Look for messages that indicate that the DoFn code is stuck or otherwise encountering issues. If no messages are present, the issue might be the execution speed of the DoFn code. Consider using Cloud Profiler or other tool to investigate the performance of your code.

Further information