Skip to content

Commit bfa1a93

Browse files
Dandandanclaude
andcommitted
feat(parquet): row-group morselization for sibling FileStream stealing
When a parquet file is scanned inside a shared sibling-stream pool (the `SharedWorkSource` introduced by #21351), the first stream to open the file now donates its remaining row groups back to the shared queue so idle sibling partitions can steal them. A single large parquet file no longer bottlenecks on one worker. Implementation: - `ParquetOpenState` gains a `SplitAndDonate` state between `LoadMetadata` and `PrepareFilters`. The donor keeps the first eligible row group and pushes each remaining one onto the front of the shared queue as a `PartitionedFile` clone whose `range` is a one-byte `FileRange` at the row group's starting offset. The existing `prune_by_range` path matches that offset and scopes the stealer to just that row group — no new extension types, no metadata carriage, no access-plan donation. - If the caller pre-narrowed the scan with a `file_range` that still covers multiple row groups (byte-range file partitioning), splitting stays inside that range: donated ranges remain subsets of the caller's. - Caller-supplied `ParquetAccessPlan` in `extensions` and single-row- group scopes suppress donation. - `SharedWorkSource` is `pub` and gets `push_front` / `pop_front` / `Default`. `row_group_start_offset` is extracted so it's shared with `prune_by_range`. Stealers re-load the parquet footer; object stores typically cache the range so this is cheap. Sharing loaded metadata across siblings is left for a follow-up. 5 new tests cover: basic donation + stealer round-trip, single-RG files, caller access-plan suppression, splitting inside a caller `file_range`, and single-RG caller ranges. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
1 parent 9a1ed57 commit bfa1a93

7 files changed

Lines changed: 383 additions & 20 deletions

File tree

‎datafusion/datasource-parquet/src/opener.rs‎

Lines changed: 328 additions & 2 deletions
Large diffs are not rendered by default.

‎datafusion/datasource-parquet/src/row_group_filter.rs‎

Lines changed: 12 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,17 @@ use parquet::data_type::Decimal;
3333
use parquet::schema::types::SchemaDescriptor;
3434
use parquet::{bloom_filter::Sbbf, file::metadata::RowGroupMetaData};
3535

36+
/// Starting byte offset of a row group in its parquet file.
37+
///
38+
/// Uses the first column's dictionary page offset when present, otherwise its
39+
/// data page offset — intentionally *not* the metadata location, per
40+
/// <https://github.com/apache/datafusion/issues/5995>.
41+
pub fn row_group_start_offset(metadata: &RowGroupMetaData) -> i64 {
42+
let col = metadata.column(0);
43+
col.dictionary_page_offset()
44+
.unwrap_or_else(|| col.data_page_offset())
45+
}
46+
3647
/// Reduces the [`ParquetAccessPlan`] based on row group level metadata.
3748
///
3849
/// This struct implements the various types of pruning that are applied to a
@@ -224,17 +235,7 @@ impl RowGroupAccessPlanFilter {
224235
if !self.access_plan.should_scan(idx) {
225236
continue;
226237
}
227-
228-
// Skip the row group if the first dictionary/data page are not
229-
// within the range.
230-
//
231-
// note don't use the location of metadata
232-
// <https://github.com/apache/datafusion/issues/5995>
233-
let col = metadata.column(0);
234-
let offset = col
235-
.dictionary_page_offset()
236-
.unwrap_or_else(|| col.data_page_offset());
237-
if !range.contains(offset) {
238+
if !range.contains(row_group_start_offset(metadata)) {
238239
self.access_plan.skip(idx);
239240
}
240241
}

‎datafusion/datasource-parquet/src/source.rs‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ use datafusion_common::config::ConfigOptions;
2929
#[cfg(feature = "parquet_encryption")]
3030
use datafusion_common::config::EncryptionFactoryOptions;
3131
use datafusion_datasource::as_file_source;
32-
use datafusion_datasource::file_stream::FileOpener;
32+
use datafusion_datasource::file_stream::{FileOpener, SharedWorkSource};
3333
use datafusion_datasource::morsel::Morselizer;
3434

3535
use arrow::datatypes::TimeUnit;
@@ -526,6 +526,7 @@ impl FileSource for ParquetSource {
526526
object_store: Arc<dyn ObjectStore>,
527527
base_config: &FileScanConfig,
528528
partition: usize,
529+
shared_work_source: Option<SharedWorkSource>,
529530
) -> datafusion_common::Result<Box<dyn Morselizer>> {
530531
let expr_adapter_factory = base_config
531532
.expr_adapter_factory
@@ -580,6 +581,7 @@ impl FileSource for ParquetSource {
580581
encryption_factory: self.get_encryption_factory_with_config(),
581582
max_predicate_cache_size: self.max_predicate_cache_size(),
582583
reverse_row_groups: self.reverse_row_groups,
584+
shared_work_source,
583585
}))
584586
}
585587

‎datafusion/datasource/src/file.rs‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ use std::sync::Arc;
2424

2525
use crate::file_groups::FileGroupPartitioner;
2626
use crate::file_scan_config::FileScanConfig;
27-
use crate::file_stream::FileOpener;
27+
use crate::file_stream::{FileOpener, SharedWorkSource};
2828
use crate::morsel::{FileOpenerMorselizer, Morselizer};
2929
#[expect(deprecated)]
3030
use crate::schema_adapter::SchemaAdapterFactory;
@@ -82,11 +82,18 @@ pub trait FileSource: Any + Send + Sync {
8282
///
8383
/// It is preferred to implement the [`Morselizer`] API directly by
8484
/// implementing this method.
85+
///
86+
/// `shared_work_source`, when `Some`, is the queue of unopened files
87+
/// shared across sibling streams. File sources that can sub-divide a
88+
/// single file into smaller stealable work units (e.g. parquet row-group
89+
/// splitting) may push donated chunks onto it; sources that cannot simply
90+
/// ignore the parameter.
8591
fn create_morselizer(
8692
&self,
8793
object_store: Arc<dyn ObjectStore>,
8894
base_config: &FileScanConfig,
8995
partition: usize,
96+
_shared_work_source: Option<SharedWorkSource>,
9097
) -> Result<Box<dyn Morselizer>> {
9198
let opener = self.create_file_opener(object_store, base_config, partition)?;
9299
Ok(Box::new(FileOpenerMorselizer::new(opener)))

‎datafusion/datasource/src/file_scan_config/mod.rs‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -597,8 +597,6 @@ impl DataSource for FileScanConfig {
597597

598598
let source = self.file_source.with_batch_size(batch_size);
599599

600-
let morselizer = source.create_morselizer(object_store, self, partition)?;
601-
602600
// Extract the shared work source from the sibling state if it exists.
603601
// This allows multiple sibling streams to steal work from a single
604602
// shared queue of unopened files.
@@ -607,6 +605,13 @@ impl DataSource for FileScanConfig {
607605
.and_then(|state| state.downcast_ref::<SharedWorkSource>())
608606
.cloned();
609607

608+
let morselizer = source.create_morselizer(
609+
object_store,
610+
self,
611+
partition,
612+
shared_work_source.clone(),
613+
)?;
614+
610615
let stream = FileStreamBuilder::new(self)
611616
.with_partition(partition)
612617
.with_shared_work_source(shared_work_source)

‎datafusion/datasource/src/file_stream/mod.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ use self::scan_state::{ScanAndReturn, ScanState};
4747

4848
pub use builder::FileStreamBuilder;
4949
pub use metrics::{FileStreamMetrics, StartableTime};
50+
pub use work_source::SharedWorkSource;
5051

5152
/// A stream that iterates record batch by record batch, file over file.
5253
pub struct FileStream {

‎datafusion/datasource/src/file_stream/work_source.rs‎

Lines changed: 24 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ impl WorkSource {
6464
/// It uses a [`Mutex`] internally to provide thread-safe access
6565
/// to the shared file queue.
6666
#[derive(Debug, Clone)]
67-
pub(crate) struct SharedWorkSource {
67+
pub struct SharedWorkSource {
6868
inner: Arc<SharedWorkSourceInner>,
6969
}
7070

@@ -73,6 +73,12 @@ pub(super) struct SharedWorkSourceInner {
7373
files: Mutex<VecDeque<PartitionedFile>>,
7474
}
7575

76+
impl Default for SharedWorkSource {
77+
fn default() -> Self {
78+
Self::new(std::iter::empty())
79+
}
80+
}
81+
7682
impl SharedWorkSource {
7783
/// Create a shared work source containing the provided unopened files.
7884
pub(crate) fn new(files: impl IntoIterator<Item = PartitionedFile>) -> Self {
@@ -91,8 +97,23 @@ impl SharedWorkSource {
9197

9298
/// Pop the next file from the shared work queue.
9399
///
94-
/// Returns `None` if the queue is empty
95-
fn pop_front(&self) -> Option<PartitionedFile> {
100+
/// Returns `None` if the queue is empty.
101+
pub fn pop_front(&self) -> Option<PartitionedFile> {
96102
self.inner.files.lock().pop_front()
97103
}
104+
105+
/// Push files to the front of the shared work queue.
106+
///
107+
/// Used when an in-flight file is sub-divided (e.g. into row-group-sized
108+
/// chunks): the donor keeps one chunk and pushes the rest to the front so
109+
/// any idle sibling picks them up before starting work on an unrelated
110+
/// whole file. Items preserve their order: the first element of `items`
111+
/// ends up at the very front of the queue.
112+
pub fn push_front(&self, items: impl IntoIterator<Item = PartitionedFile>) {
113+
let items: Vec<PartitionedFile> = items.into_iter().collect();
114+
let mut queue = self.inner.files.lock();
115+
for file in items.into_iter().rev() {
116+
queue.push_front(file);
117+
}
118+
}
98119
}

0 commit comments

Comments
 (0)