gce/WARN/2023_002

Serial logs don’t contain out-of-memory message due to Airflow task run

Product: Compute Engine
Rule class: WARN - Something that is possibly wrong

Description

Sometimes Composer Airflow task might be using more memory and no proper logs will be seen in task log. In such cases we can observe out of memory messages in the k8s node log in the following way: “Memory cgroup out of memory: Killed process 123456 (airflow task ru)”.

Remediation

  1. Consider reducing the worker_concurrency and observe the worker performance.
  2. Based on the task usage requirement, consider increasing the airflow-worker memory.

Further information

You need to scale the Composer environment based on observation: https://cloud.google.com/composer/docs/composer-2/optimize-environments