dataflow/ERR/2024_004

The Dataflow job has the necessary GCS permissions for the temporary bucket.

Product: Dataflow
Rule class: ERR - Something that is very likely to be wrong

Description

Two primary reasons cause Dataflow jobs to fail when writing to a storage bucket: either the specified bucket does not exist within the targeted Google Cloud project, or the associated service account lacks the necessary permissions to write to it.

Remediation

  • If the bucket is missing, create it in the target project
  • If this is permissions error, assign the role storage.objectAdmin to the service account. The service account is the same which is managing the Compute Engine instance.

Further information