dataproc/WARN/2023_001
Concurrent Job limit was not exceeded
Product: Cloud Dataproc
Rule class: WARN - Something that is possibly wrong
Description
If the Dataproc agent reach the concurrent job submission limit, Dataproc job scheduling delays can be observed.
Remediation
The maximum number of concurrent jobs based on master VM memory is exceeded (the job driver runs on the Dataproc cluster master VM). By default, Dataproc reserves 3.5GB of memory for applications, and allows 1 job per GB.Set the dataproc:dataproc.scheduler.max-concurrent-jobs cluster property to a value suited to your job requirements