composer/Runbook Airflow Worker Restart

Runbook addresses the root causes of Airflow worker instability.

Product: Cloud Composer Kind: Debugging Tree

Description

This runbook checks for the following:

  • Verify Composer environment exists and find the associated GKE cluster
  • Check for OOMKilled Events in Logs
  • Check for SIGKILL / Zombie Tasks
  • Check for Ephemeral Storage Exhaustion
  • Check for gcs-syncd Ephemeral Storage Limit Issue
  • Check for Worker Liveness Probe Logs.
  • Check for Unschedulable Pods

Executing this runbook

gcpdiag runbook composer/runbook-airflow-worker-restart \
  -p project_id=value \
  -p name=value \
  -p start_time=value \
  -p end_time=value

Parameters

Name Required Default Type Help
project_id True None str The Project ID of the resource under investigation
name True None str The name of the Composer environment
start_time True None datetime The start time of the investigation
end_time True None datetime The end time of the investigation

Get help on available commands

gcpdiag runbook --help

Potential Steps