gke/WARN/2021_007

GKE nodes have enough free space on the boot disk.

Product: Google Kubernetes Engine
Rule class: WARN - Something that is possibly wrong

Description

GKE nodes need free space on their boot disks to be able to function properly. If /var is getting full, it might be because logs are not being rotated correctly, or maybe a container is creating too much data in the overlayfs.

Remediation

You can use the following Cloud Monitoring query to determine the nodes with full disk:

fetch gce_instance
  | metric 'compute.googleapis.com/guest/disk/bytes_used'
  | filter metric.device_name == 'sda1'
  | {within_str}
  | next_older 5m
  | filter_ratio_by [resource.instance_id], metric.state == 'free'
  | every 5m

Further information