dataflow/WARN/2023_002

Dataflow worker logs are Throttled

Product: Dataflow
Rule class: WARN - Something that is possibly wrong

Description

Worker log messages are limited to 15,000 messages every 30 seconds, per worker. If this limit is reached, a single worker log message is added saying that logging is throttled.

Remediation

  1. Limit the log messages by logging only errors using try/catch blocks within ParDo’s.
  2. Write the required logs as Strings, make it a second output PCollection that can be written to a storage sink like BigQuery, Cloud Storage.

Further information

Dataflow Worker Log Limits