bigquery/ERR/2022_001

BigQuery does not exceed rate limits

Product: BigQuery
Rule class: ERR - Something that is very likely to be wrong

Description

BigQuery has various quotas that limit the rate and volume of incoming requests. These quotas exist both to protect the backend systems, and to help guard against unexpected billing if you submit large jobs.

Remediation

Slow down the request rate. To resolve these limit issues, retry the operation after few seconds. Use exponential backoff between retry attempts. If you consistently reach one of these longer-term quota limits, you should analyze your workload for ways to mitigate the issue. Mitigations can include optimizing your workload or requesting a quota increase.

Further information

To diagnose issues, do the following:

  • Use INFORMATION_SCHEMA views to analyze the underlying issue. These views contain metadata about your BigQuery resources, including jobs, reservations, and streaming inserts. For example, the following query uses the INFORMATION_SCHEMA.JOBS view to list all quota-related errors within the past day:
SELECT
 job_id,
 creation_time,
 error_result
FROM `region-us`.INFORMATION_SCHEMA.JOBS
WHERE creation_time > TIMESTAMP_SUB(CURRENT_TIMESTAMP, INTERVAL 1 DAY) AND
      error_result.reason IN ('rateLimitExceeded', 'quotaExceeded')
  • View errors in Cloud Audit Logs Using Logs Explorer, the following query returns errors with either Quota exceeded or limit in the message string:
resource.type = ("bigquery_project" OR "bigquery_dataset")
protoPayload.status.code ="7"
protoPayload.status.message: ("Quota exceeded" OR "limit")