dataflow/ERR/2023_005
Dataflow job does not fail during execution due to credential or permission issue
Product: Dataflow
Rule class: ERR - Something that is very likely to be wrong
Description
A dataflow job runs successfully if dataflow api is enabled and dataflow service account and controller service account have sufficient permissiont.
You can search in the Logs Explorer for such jobs with the logging query:
resource_type='dataflow_step',
severity=ERROR
log_id("dataflow.googleapis.com/job-message")
textPayload=~"Workflow failed. Causes: There was a problem refreshing your credentials."
Remediation
Complete error message is:
Workflow failed. Causes: There was a problem refreshing your credentials.
Please check: 1. The Dataflow API is enabled for your project.
2. Make sure both the Dataflow service account and the controller service account have sufficient permissions.
If you are not specifying a controller service account, ensure the default Compute Engine service account PROJECT_NUMBER-compute@developer.gserviceaccount.com exists and has sufficient permissions.
If you have deleted the default Compute Engine service account, you must specify a controller service account
To resolve this error:
- Dataflow API should be enabled
- Worker Service Account should have the roles/dataflow.worker role
- Dataflow service account of type service-@dataflow-service-producer-prod.iam.gserviceaccount.com should have roles/dataflow.serviceAgent role