You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
bigquery: benchmark arrow + jobs.query, Storage Read API and define routing thresholds #7041
This issue tracks benchmarking for BigQuery query result reading across:
Standard REST JSON pagination (jobs.query + jobs.getQueryResults)
Arrow results in jobs.query (QueryResultsFormat::Arrow, #[cfg(google_cloud_unstable_bigquery_arrow)])
Storage Read API acceleration (#[cfg(google_cloud_unstable_bigquery_storage_read)]) — both single-stream (jobs/{job_id}/streams/_default) and multi-stream (ReadSession on the destination table)
Goals:
Measure performance & memory improvements:
Track end-to-end query time, row read/deserialization time, and throughput (rows/sec).
Track heap memory usage (stats_alloc: total bytes allocated, bytes/row, total allocation count, allocations/row) for both raw Row iteration and typed #[derive(FromRow)] deserialization.
Determine routing thresholds:
Find the crossover threshold where switching from inline/paginated REST results to opening a Storage Read API stream becomes faster.
Evaluate whether and at what threshold the extra RPC overhead of jobs.get + CreateReadSession + parallel gRPC streams outperforms reading directly from the single _default job stream (projects/{p}/locations/{l}/jobs/{j}/streams/_default).
Activity