composer/WARN/2022_003
Composer scheduler parses all DAG files without overloading
Product: Cloud Composer
Rule class: WARN - Something that is possibly wrong
Description
The schedulers parse DAGs before scheduling DAG runs. If DAGs take a long time to parse, this consumes scheduler’s capacity and might reduce the performance of DAG runs.
If the total DAG parse time exceeds about 10 seconds, the schedulers might be overloaded with DAG parsing and cannot run DAGs effectively. The default DAG parsing frequency in Airflow is 30 seconds; if DAG parsing time exceeds this threshold, parsing cycles start to overlap, which then exhausts scheduler’s capacity.
To check on the Monitoring dashboard, observe graphs for the total DAG parse time in the Composer Monitoring - DAG Statistics section.
Remediation
According to your observations, you might want to:
- Simplify your DAGs, including their Python dependencies.
- Increase the DAG file parsing interval and increase the DAG directory listing interval.
- Increase the number of schedulers.
- Increase the CPU of schedulers.