dataproc/ERR/2023_004
Dataproc cluster firewall rules for connectivity between master and worker nodes established!
Product: Cloud Dataproc
Rule class: ERR - Something that is very likely to be wrong
Description
The master node needs to communicate with the worker nodes during cluster creation. Sometimes VM to VM communications are blocked by firewall rules.
Remediation
- You can see the log from dataproc agent if the yarn app became orphaned and
killed by dataproc agent.
Log query example :
resource.type=“cloud_dataproc_cluster”
resource.labels.cluster_uuid="<Datproc_cluster_uuid>"
“<YARN_app_id>”
logName: “projects//logs/google.dataproc.agent” - Check for firewall rule warnings. Make sure that the user has the correct firewall rule in place. Refer to the public documentation for the correct Dataproc Cluster Network Configuration. To see what is blocking the master node from communicating with the worker nodes, a connectivity test can be performed by using the Ad Hoc Connectivity Test found under “Network Intelligence” in your GCP console.