bigquery/WARN/2024_006

BigQuery project does not exceed maximum tabledata.list bytes per second per project

Product: BigQuery
Rule class: WARN - Something that is possibly wrong

Description

BigQuery returns this error when the project number mentioned in the error message reaches the maximum size of data that can be read through the tabledata.list API call in a project per second. Your project can return a maximum of 7.5 GB of table row data per minute via tabledata.list in the us and eu multi-regions, and 3.7 GB of table row data per minute in all other regions. This quota applies to the project that contains the table being read. Other APIs including jobs.getQueryResults and fetching results from jobs.query and jobs.insert can also consume this quota.

Remediation

To resolve this error, do the following:

  • In general, we recommend trying to stay below this limit. For example, by spacing out requests over a longer period with delays. If the error doesn’t happen frequently, implementing retries with exponential backoff solves this issue.

  • If the use case expects fast and frequent reading of large amount of data from a table, we recommend using BigQuery Storage Read API instead of the tabledata.list API.

  • If the preceding suggestions do not work, you can request a quota increase from Google Cloud console API dashboard by doing the following:

    1. Go to the Google Cloud console API dashboard.
    2. In the dashboard, filter for Quota: Tabledata list bytes per minute (default quota).
    3. Select the quota and follow the instruction in Requesting higher quota limit.

It might take several days to review and process the request.

Further information

The BigQuery Storage Read API can sustain significantly higher throughput than tabledata.list. If you need more throughput than allowed under this quota, consider using the BigQuery Storage Read API.