You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit bfa1a93
Browse filesBrowse the repository at this point in the historyBrowse files
feat(parquet): row-group morselization for sibling FileStream stealing
When a parquet file is scanned inside a shared sibling-stream pool (the
`SharedWorkSource` introduced by #21351), the first stream to open the
file now donates its remaining row groups back to the shared queue so
idle sibling partitions can steal them. A single large parquet file no
longer bottlenecks on one worker.
Implementation:
- `ParquetOpenState` gains a `SplitAndDonate` state between
`LoadMetadata` and `PrepareFilters`. The donor keeps the first
eligible row group and pushes each remaining one onto the front of
the shared queue as a `PartitionedFile` clone whose `range` is a
one-byte `FileRange` at the row group's starting offset. The
existing `prune_by_range` path matches that offset and scopes the
stealer to just that row group — no new extension types, no metadata
carriage, no access-plan donation.
- If the caller pre-narrowed the scan with a `file_range` that still
covers multiple row groups (byte-range file partitioning), splitting
stays inside that range: donated ranges remain subsets of the
caller's.
- Caller-supplied `ParquetAccessPlan` in `extensions` and single-row-
group scopes suppress donation.
- `SharedWorkSource` is `pub` and gets `push_front` / `pop_front` /
`Default`. `row_group_start_offset` is extracted so it's shared with
`prune_by_range`.
Stealers re-load the parquet footer; object stores typically cache the
range so this is cheap. Sharing loaded metadata across siblings is left
for a follow-up.
5 new tests cover: basic donation + stealer round-trip, single-RG
files, caller access-plan suppression, splitting inside a caller
`file_range`, and single-RG caller ranges.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
0 commit comments