Skip to content

bigquery: Multi-stream ReadSession acceleration for large query results #7039

Description

@alvarowolfx

Extend StorageReader to automatically upgrade large query result reads (above the multi-stream row threshold) to a parallel multi-stream ReadSession against the query job's destination table.

  • Fetch the query job's destination_table via JobService::get_job when total_rows / total_estimated_row_count exceeds the multi-stream threshold.
  • Create a ReadSession with DataFormat::Arrow and CompressionCodec::Lz4Frame on the destination table.
  • Spawn concurrent stream reader tasks across session.streams sending decoded RecordBatches over a bounded channel to StorageReader::next_batch() (ensuring proper backpressure and task cancellation on drop).
  • Unit and integration tests for multi-stream session creation and fallback behavior when no destination table or only a single stream is available.

Metadata

Metadata

Assignees

Labels

api: bigqueryIssues related to the BigQuery API.type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions