dataflow/WARN/2024_002

Dataflow job using Streaming Appliance is not getting stuck due to Commit failed: computation doesn’t have the state family.

Product: Dataflow
Rule class: WARN - Something that is possibly wrong

Description

A Dataflow job running with Streaming Appliance may get stuck and return the below warnings during its execution:

Commit failed: computation doesn’t have the state family

You can search this in Logs Explorer for such jobs with the below mentioned Logging Query:

  """
  resource.type="dataflow_step"
  resource.labels.job_id="%dataflowJobID%"
  "Commit failed: computation doesn't have the state family"
  severity>="WARNING"
  """

This warning will not be visible for jobs using Streaming Engine as we proactively reject such problematic pipelines, however in Appliance jobs, it may get into a bad state and become stuck.

Remediation

This is a known issue when updating pipelines that have a flatten operation followed by a ParDo that uses a side input. The workaround is to change the pipeline shape to avoid this issue. For example if the pipeline looks something like:

ParDo ParDo \ / \ / A flatten operation | | ParDo using side input | Output

The recommended work-around is to change this to something like:

ParDo ParDo \ / \ / ParDo using side input ParDo using side input \ / \ / A flatten operation | | Output

Further information