dataproc/Check Preemptible

Product: Cloud Dataproc
Step Type: AUTOMATED STEP

Description

None

Failure Reason

Cluster {cluster.name} uses preemptible workers and their count exceeds 50% of the total worker count leading to shuffle fetch failures.

Failure Remediation

To improve stability:

• Reduce the number of preemptible workers.
• Use non-preemptible workers.
• Explore Enhanced Flexibility Mode (EFM) for better control over preemptible instances.

Success Reason

Cluster {cluster.name} uses preemptible workers. While within the recommended limit, preemptions might still lead to FetchFailedExceptions.