Skip to content

Commit b4f813b

Browse files
authored
refactor: use row-group-local selections for Parquet scans (apache#25608)
## Which issue does this PR close? - Related to apache#24358 - Related to apache#23883 - Related to apache#24186 (comment) ## Rationale for this change Parquet access plans already describe selections per row group, but the scan path flattens them into one selection and reconstructs group boundaries when reversing the scan. Keeping selections attached to their row groups preserves local coordinates and bitmap representations while simplifying ordering and decoder rebuilds. ## What changes are included in this PR? - Pass row-group-local selections to the Parquet push decoder and keep each selection together with its static match status. - Validate selection lengths and remove empty selections during preparation; reverse and reorder whole row-group entries. - Remove global-selection conversion from the scan path and simplify reversal. Deprecate `ParquetAccessPlan::into_overall_row_selection` while retaining its existing behavior. - Reuse the decoder's remaining selections when only the row filter changes, avoiding redundant selector copies. - Preserve the current policy that disables statistics reordering and runtime pruning for scans with selections. ## What is the testing strategy for this PR? The new integration test covers mixed bitmap and selector selections across row groups. Unit tests cover selection validation, empty selections, match-status alignment, and reversal of non-contiguous groups with unequal sizes. Passed targeted tests: - `cargo test -p datafusion-datasource-parquet --lib access_plan::test --offline` - `cargo test -p datafusion-datasource-parquet --lib sort::tests --offline` - `cargo test -p datafusion-datasource-parquet --lib push_decoder::tests --offline` - `cargo test -p datafusion-datasource-parquet --lib statistics_order_tests --offline` - `cargo test -p datafusion --test parquet_integration external_access_plan --offline` - `cargo test -p datafusion --test parquet_integration dynamic_row_group_pruning --offline` Pre-submission checks: `cargo fmt --all`, `cargo clippy --all-targets --all-features -- -D warnings`, and `uv run ./dev/rust_lint.sh`. ## Are there any user-facing changes? `ParquetAccessPlan::into_overall_row_selection` is deprecated starting in 56.0.0 but remains available with its existing behavior. The upgrade guide describes migration for direct callers. Users who attach a `ParquetAccessPlan` to a `PartitionedFile` need no changes.
1 parent 975c0f0 commit b4f813b

8 files changed

Lines changed: 522 additions & 1338 deletions

File tree

‎datafusion/core/tests/parquet/external_access_plan.rs‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -310,6 +310,36 @@ async fn two_selections() {
310310
}
311311
}
312312

313+
#[tokio::test]
314+
async fn mixed_bitmap_and_selector_selections() {
315+
// RG 0 selects a/c with a bitmap, RG 1 selects f/h with selectors.
316+
// Each representation must keep coordinates local to its row group.
317+
TestFull {
318+
access_plan: Some(ParquetAccessPlan::new(vec![
319+
RowGroupAccess::Selection(RowSelection::from(
320+
arrow::buffer::BooleanBuffer::from(vec![true, false, true, false, false]),
321+
)),
322+
RowGroupAccess::Selection(select_two_rows()),
323+
])),
324+
row_selection: None,
325+
expected_rows: 4,
326+
expected_output: Some(&[
327+
"+------+------------+",
328+
"| utf8 | large_utf8 |",
329+
"+------+------------+",
330+
"| a | a |",
331+
"| c | c |",
332+
"| f | f |",
333+
"| h | h |",
334+
"+------+------------+",
335+
]),
336+
predicate: None,
337+
}
338+
.run()
339+
.await
340+
.unwrap();
341+
}
342+
313343
#[tokio::test]
314344
async fn bad_row_groups() {
315345
let err = TestFull {

‎datafusion/datasource-parquet/src/access_plan.rs‎

Lines changed: 294 additions & 324 deletions
Large diffs are not rendered by default.

‎datafusion/datasource-parquet/src/opener/mod.rs‎

Lines changed: 27 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -1665,8 +1665,8 @@ impl RowGroupsPrunedParquetOpen {
16651665
//
16661666
// 2. `reverse`: flip the iteration order for DESC requests, applied
16671667
// AFTER any reorder so the reversed order is correct whether or
1668-
// not reorder changed anything. Also handles `row_selection`
1669-
// remapping.
1668+
// not reorder changed anything. Local selections move with their
1669+
// row groups.
16701670
//
16711671
// For sorted data: reorder is a no-op, reverse gives perfect DESC.
16721672
// For unsorted data: reorder fixes the order, reverse flips for DESC.
@@ -1685,7 +1685,7 @@ impl RowGroupsPrunedParquetOpen {
16851685
)?;
16861686
}
16871687
if prepared.reverse_row_groups {
1688-
prepared_plan = prepared_plan.reverse(file_metadata.as_ref())?;
1688+
prepared_plan = prepared_plan.reverse();
16891689
}
16901690
Ok(prepared_plan)
16911691
};
@@ -1720,9 +1720,9 @@ impl RowGroupsPrunedParquetOpen {
17201720
} = {
17211721
// Build the prepared access plan first — `prepare_access_plan` may
17221722
// call `reorder_by_statistics` (for `sort_order_for_reorder`) and
1723-
// `reverse` (for `reverse_row_groups`), both of which mutate
1724-
// `row_group_indexes` to the physical scan order the decoder will
1725-
// actually read. We MUST build our `rg_plan` from this reordered
1723+
// `reverse` (for `reverse_row_groups`), both of which arrange row
1724+
// groups in the physical scan order the decoder will read.
1725+
// We MUST build our `rg_plan` from this reordered
17261726
// list, otherwise our per-RG pruner check would consult the
17271727
// metadata of a different RG than the decoder is about to yield.
17281728
let decoder_config = DecoderBuilderConfig {
@@ -1734,35 +1734,23 @@ impl RowGroupsPrunedParquetOpen {
17341734
};
17351735

17361736
let prepared_access_plan = prepare_access_plan(access_plan)?;
1737-
// #24355: a row selection (from page-index pruning, or an externally
1738-
// supplied `ParquetRowSelection`) is carried by the decoder as one
1739-
// flat selection over the concatenation of the remaining row groups.
1740-
// The runtime pruner's `into_builder().with_row_groups(...)` rebuild
1741-
// drops row groups without slicing that selection to match, so record
1742-
// whether a selection is present and disable runtime pruning below
1743-
// when it is (mirroring `reorder_by_statistics`, which also bails when
1744-
// a row selection is present). The proper fix that keeps pruning
1745-
// under a live selection is tracked in
1746-
// https://github.com/apache/arrow-rs/issues/10624 /
1747-
// https://github.com/apache/datafusion/issues/24358.
1748-
let has_row_selection = prepared_access_plan.row_selection.is_some();
1749-
// Build `rg_plan` parallel to the decoder's view: the
1750-
// `prepared_access_plan` has already had its empty-selection
1751-
// row groups stripped, so 1:1 correspondence with the readers
1752-
// arrow-rs will hand back is restored. We zip with the
1753-
// `fully_matched` flag so the stream can toggle the per-row
1754-
// `RowFilter` per RG.
1755-
let rg_plan: VecDeque<RgPlanEntry> = prepared_access_plan
1756-
.row_group_indexes
1757-
.iter()
1758-
.copied()
1759-
.zip(prepared_access_plan.fully_matched.iter().copied())
1760-
.map(|(rg_index, fully_matched)| RgPlanEntry {
1761-
rg_index,
1762-
fully_matched,
1763-
bytes: row_group_bytes(&rg_metadata[rg_index]),
1764-
})
1765-
.collect();
1737+
let has_row_selection = prepared_access_plan.has_row_selection;
1738+
// Move selections into the decoder, retaining only the metadata
1739+
// needed for pruning, filter toggles, and byte accounting.
1740+
let (selections, rg_plan): (Vec<_>, VecDeque<RgPlanEntry>) =
1741+
prepared_access_plan
1742+
.row_groups
1743+
.into_iter()
1744+
.map(|rg| {
1745+
let rg_index = rg.selection.row_group_index();
1746+
let entry = RgPlanEntry {
1747+
rg_index,
1748+
bytes: row_group_bytes(&rg_metadata[rg_index]),
1749+
fully_matched: rg.fully_matched,
1750+
};
1751+
(rg.selection, entry)
1752+
})
1753+
.unzip();
17661754

17671755
// Decide the initial row filter state based on the first RG to
17681756
// read. If that RG is `fully_matched` the per-row predicate is
@@ -1778,8 +1766,7 @@ impl RowGroupsPrunedParquetOpen {
17781766
let first_rg_fully_matched = rg_plan.front().is_some_and(|e| e.fully_matched);
17791767
let row_filter_context = precomputed_context;
17801768

1781-
let mut builder =
1782-
decoder_config.build(prepared_access_plan, reader_metadata.clone());
1769+
let mut builder = decoder_config.build(selections, reader_metadata.clone());
17831770
let mut filter_installed = false;
17841771
if let Some(ctx) = row_filter_context.as_ref() {
17851772
if first_rg_fully_matched {
@@ -1857,11 +1844,9 @@ impl RowGroupsPrunedParquetOpen {
18571844
// via the `DynamicFilterTracker` watch channel (#22460), so detecting
18581845
// a threshold change is a single atomic load — not a tree walk per
18591846
// RG check.
1860-
// Also disabled when a row selection is live (#24355) — page-index
1861-
// pruning is the common source: the pruner rebuilds the decoder via
1862-
// `with_row_groups(...)`, which drops row groups without slicing the
1863-
// carried selection to match, so pruning under a live selection returns
1864-
// wrong results. Decline to prune in that case.
1847+
// Preserve the current policy for scans with selections. Enabling
1848+
// runtime pruning for them is separate from adopting the local API
1849+
// and is tracked in https://github.com/apache/datafusion/issues/24358.
18651850
let row_group_pruner =
18661851
match (&prepared.predicate, rg_plan.len() > 1, has_row_selection) {
18671852
(Some(predicate), true, false)

‎datafusion/datasource-parquet/src/push_decoder.rs‎

Lines changed: 22 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@
2929
//! [`RowGroupPruner`] is consulted; row groups it proves unwinnable are
3030
//! dropped from the head of `rg_plan` and the decoder is rebuilt via
3131
//! [`ParquetPushDecoder::into_builder`] +
32-
//! [`ParquetPushDecoderBuilder::with_row_groups`] so the skipped RGs are
32+
//! [`ParquetPushDecoderBuilder::with_row_group_selections`] so the skipped RGs are
3333
//! bypassed entirely — no decode, no row-filter eval.
3434
//!
3535
//! The opener constructs both halves and hands the state off to
@@ -50,7 +50,9 @@ use parquet::arrow::arrow_reader::{
5050
ArrowReaderMetadata, ParquetRecordBatchReader, RowFilter, RowSelectionPolicy,
5151
};
5252
use parquet::arrow::async_reader::AsyncFileReader;
53-
use parquet::arrow::push_decoder::{ParquetPushDecoder, ParquetPushDecoderBuilder};
53+
use parquet::arrow::push_decoder::{
54+
ParquetPushDecoder, ParquetPushDecoderBuilder, RowGroupSelection,
55+
};
5456
use parquet::file::metadata::ParquetMetaData;
5557

5658
use datafusion_common::{DataFusionError, Result, internal_err};
@@ -60,7 +62,6 @@ use datafusion_physical_plan::metrics::{BaselineMetrics, Count, Gauge};
6062
use datafusion_pruning::{PruningPredicate, PruningPredicateBuilder};
6163

6264
use crate::ParquetFileMetrics;
63-
use crate::access_plan::PreparedAccessPlan;
6465
use crate::decoder_projection::DecoderProjection;
6566
use crate::metrics::{ByteProgress, RowFilterSkippedFullyMatchedMetric};
6667
use crate::row_filter::{
@@ -83,14 +84,14 @@ pub(crate) struct DecoderBuilderConfig<'a> {
8384
}
8485

8586
impl DecoderBuilderConfig<'_> {
86-
/// Build a [`ParquetPushDecoderBuilder`] from a prepared access plan.
87+
/// Build a [`ParquetPushDecoderBuilder`] from row-group-local selections.
8788
///
8889
/// The caller is expected to attach the
8990
/// [`RowFilter`] and predicate
9091
/// cache size on the returned builder.
9192
pub(crate) fn build(
9293
&self,
93-
prepared_access_plan: PreparedAccessPlan,
94+
row_group_selections: Vec<RowGroupSelection>,
9495
metadata: ArrowReaderMetadata,
9596
) -> ParquetPushDecoderBuilder {
9697
let mut builder = ParquetPushDecoderBuilder::new_with_metadata(metadata)
@@ -100,10 +101,7 @@ impl DecoderBuilderConfig<'_> {
100101
if self.force_filter_selections {
101102
builder = builder.with_row_selection_policy(RowSelectionPolicy::Selectors);
102103
}
103-
if let Some(row_selection) = prepared_access_plan.row_selection {
104-
builder = builder.with_row_selection(row_selection);
105-
}
106-
builder = builder.with_row_groups(prepared_access_plan.row_group_indexes);
104+
builder = builder.with_row_group_selections(row_group_selections);
107105
if let Some(limit) = self.decoder_limit {
108106
builder = builder.with_limit(limit);
109107
}
@@ -301,7 +299,7 @@ pub(crate) struct PushDecoderStreamState {
301299
/// statistics and drop RGs the current threshold proves cannot
302300
/// contribute. The decoder is rebuilt via
303301
/// [`ParquetPushDecoder::into_builder`] +
304-
/// [`ParquetPushDecoderBuilder::with_row_groups`] so the skipped RGs are
302+
/// [`ParquetPushDecoderBuilder::with_row_group_selections`] so the skipped RGs are
305303
/// bypassed entirely. `None` when the scan has no watching dynamic
306304
/// predicate or only one row group remains.
307305
pub(crate) row_group_pruner: Option<RowGroupPruner>,
@@ -565,6 +563,9 @@ impl PushDecoderStreamState {
565563
/// removed every page — which would otherwise leave `rg_plan` trailing the
566564
/// decoder by one: a later prune/rebuild would then re-include an
567565
/// already-delivered row group (#24352) or toggle the filter for the wrong RG.
566+
/// Row-group-local selections keep selections and match status aligned when
567+
/// preparing or reordering the plan, but do not prevent this decoder-side
568+
/// advancement, so frontier synchronization is still required.
568569
fn sync_rg_plan_to_decoder_frontier(&mut self) -> Result<()> {
569570
match self
570571
.decoder
@@ -669,9 +670,18 @@ impl PushDecoderStreamState {
669670
}
670671

671672
let decoder = self.decoder.take().expect("decoder present");
672-
let new_indices: Vec<usize> = self.rg_plan.iter().map(|e| e.rg_index).collect();
673673
let mut builder = decoder.into_builder().map_err(DataFusionError::from)?;
674-
builder = builder.with_row_groups(new_indices);
674+
// Filter-only rebuilds preserve the decoder's remaining selections.
675+
// Runtime pruning is disabled for scans with selections, so pruned
676+
// plans can be rebuilt from row-group indexes alone.
677+
if pruned_count > 0 {
678+
let selections = self
679+
.rg_plan
680+
.iter()
681+
.map(|e| RowGroupSelection::new(e.rg_index, None))
682+
.collect();
683+
builder = builder.with_row_group_selections(selections);
684+
}
675685
if filter_needs_toggle {
676686
let want_filter = desired_filter.expect("filter_needs_toggle ⇒ desired Some");
677687
if want_filter {

0 commit comments

Comments
 (0)