bigquery/WARN/2024_005
BigQuery table does not exceed quota for Number of partition modifications to a column partitioned table
Product: BigQuery
Rule class: WARN - Something that is possibly wrong
Description
BigQuery returns this error when your column-partitioned table reaches the quota of the number of partition modifications permitted per day. Partition modifications include the total of all load jobs, copy jobs, and query jobs that append or overwrite a destination partition. To see the value of the Number of partition modifications per column-partitioned table per day limit, see Partitioned tables.
Remediation
This quota cannot be increased. To resolve this quota error, do the following:
- Change the partitioning on the table to have more data in each partition, in order to decrease the total number of partitions. For example, change from partitioning by day to partitioning by month or change how you partition the table.
- Use clustering instead of partitioning.
- If you frequently load data from multiple small files stored in Cloud Storage that uses a job per file, then combine multiple load jobs into a single job. You can load from multiple Cloud Storage URIs with a comma-separated list (for example, gs://my_path/file_1,gs://my_path/file_2), or by using wildcards (for example, gs://my_path/*). For more information, see Batch loading data.
- If you use load, select or copy jobs to append single rows of data to a table, for example, then you should consider batching multiple jobs into one job. BigQuery doesn’t perform well when used as a relational database. As a best practice, avoid running frequent, single-row append actions.
- To append data at a high rate, consider using BigQuery Storage Write API. It is a recommended solution for high-performance data ingestion. The BigQuery Storage Write API has robust features, including exactly-once delivery semantics. To learn about limits and quotas, see Storage Write API and to see costs of using this API, see BigQuery data ingestion pricing.