composer/BP/2023_002
Cloud Composer’s worker concurrency is not limited by parallelism parameter
Product: Cloud Composer
Rule class: BP - Best practice, opinionated recommendation
Description
The parallelism defines the maximum number of task instances that can run concurrently in Airflow. Generally, the parameter should be equal or higher than a product of maximum number of workers and worker_concurrency. Otherwise, resources in workers could not be fully-utilized.
Remediation
Observe your environment’s performance and adjust parallelism or
worker_concurrency parameter properly.