dataproc/WARN/2024_002

HDFS can write file(s) to DataNode(s).

Product: Cloud Dataproc
Rule class: WARN - Something that is possibly wrong

Description

HDFS had issues writing file(s) to DataNode(s). The HDFS NameNode logs show the following message:

java.io.IOException: File [...] could only be [written/replicated] to x nodes instead of minReplication (=n). There are y datanode(s) running and z node(s) are excluded in this operation.

This can be caused by different reasons:

  • The HDFS filesystem has no available space.

  • The DataNodes may be unavailable or there is no communication between NameNode and Datanode(s).

  • The reserved space for non-DFS use is not sufficient (dfs.datanode.du.reserved).

Remediation

  • Check the HDFS remaining capacity: dfs/FSNamesystem/CapacityRemainingGB.

    • If low, free up some HDFS space or add primary worker nodes to the cluster.

  • Check the status of NameNode and DataNodes services and the related logs.

    • Address the issue based on the findings, e.g. network related issues related to the communication between NameNode and Datanodes.