dataproc/ERR/2023_008

Disk space of YARN NodeManagers is okay.

Product: Cloud Dataproc
Rule class: ERR - Something that is very likely to be wrong

Description

YARN ResourceManager has reported UNHEALTHY YARN NodeManagers due to exceeding the maximum percentage of disk space utilization allowed.

Remediation

The user cache is stored in the directory specified by the yarn.nodemanager.local-dirs property in the yarn-site.xml file. This file is located at /etc/hadoop/conf/yarn-site.xml. You can check the free space in the /hadoop/yarn/nm-local-dir path, and free up space by deleting the /hadoop/yarn/nm-local-dir/usercache user cache folder.
Recreate your cluster with larger disk space, which will increase the throughput cap.

Further information

Check the following documentation to address the issue: https://cloud.google.com/dataproc/docs/support/spark-job-tuning#yarn_nodes_are_unhealthy.
YARN properties: yarn-default.xml.