composer/WARN/2023_002
Cloud Composer tasks are not interrupted by SIGKILL
Product: Cloud Composer
Rule class: WARN - Something that is possibly wrong
Description
Sometimes your task might be using more memory than Airflow worker is allocated. In such a situation it might be interrupted by Negsignal.SIGKILL. The system sends this signal to avoid further memory consumption which might impact the execution of other Airflow tasks.
You can find the issue from Cloud Logging with the filter below:
resource.type="cloud_composer_environment"
log_id("airflow-worker")
textPayload="Task exited with return code Negsignal.SIGKILL"
resource.labels.environment_name="<COMPOSER_ENV_NAME>"
severity=INFO
Remediation
Possible solutions:
- Lower worker_concurrency of Airflow worker
- In the case of Cloud Composer 2, increase memory of Airflow workers
- In the case of Cloud Composer 1, upgrade to bigger machine type used in Composer cluster
- Optimize your tasks to use less memory