bigquery/ERR/2022_004

BigQuery jobs are not failing due to shuffle operation resources exceeded

Product: BigQuery
Rule class: ERR - Something that is very likely to be wrong

Description

The query job failed because the maximum disk and memory limit available for shuffle operations was exceeded for the project or organization.

Remediation

The limit cannot be increased for users - it is linearly correlated to the number of slots available in the project. Provision more slots for this to be increased.

Further recommendations that can help:

  • Reduce query concurrency
  • Optimize the query to use less shuffle resources
  • Query less data within a single query job
  • Materialize intermediate results

Further information