composer/WARN/2023_003
Cloud Composer tasks are not failed due to resource pressure
Product: Cloud Composer
Rule class: WARN - Something that is possibly wrong
Description
During execution of a task, Airflow worker’s subprocess responsible for Airflow task execution could be interrupted abruptly due to resource pressure. In this case, the task would be failed without emitting logs.
Remediation
Possible solutions:
- In Cloud Composer 1, create a new environment with a larger machine type than the current machine type. Consider adding more nodes to your environment and lower [celery]worker_concurrency for your workers.
- In Cloud Composer 2, increase memory limits for Airflow workers.