Skip to content

Commit 9ba77e7

Browse files
committed
test: update coexistence test for apache#24355 pruner-disable; clarify row_selection source
The pre-existing dynamic_rg_pruning_coexists_with_page_index_row_selection test asserted the runtime pruner stays active (pruned >= 1) alongside a page-index selection. apache#24355 deliberately disables the pruner when a row selection is present, so rename it to dynamic_rg_pruning_disabled_when_page_index_row_selection_present and assert row_groups_pruned_dynamic_filter == 0 while results stay correct and page-index pruning still runs. Also reword the opener comments: a row selection comes from page-index pruning or an externally supplied ParquetRowSelection (not stats/bloom, which only skip/scan whole row groups).
1 parent 7dd27a6 commit 9ba77e7

2 files changed

Lines changed: 31 additions & 28 deletions

File tree

‎datafusion/core/tests/parquet/dynamic_row_group_pruning.rs‎

Lines changed: 24 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -297,9 +297,12 @@ fn build_five_thousand_row_rgs(schema: &Arc<Schema>) -> Vec<RecordBatch> {
297297
.collect()
298298
}
299299

300-
/// Co-existence test for **page-index `RowSelection`** + dynamic RG
301-
/// pruning. Tests that the `into_builder` rebuild preserves the
302-
/// `RowSelection` derived from page-index pruning across RG drops.
300+
/// Regression test for #24355: when a page-index `RowSelection` is live,
301+
/// the runtime dynamic row-group pruner is intentionally **not built**, so
302+
/// its `into_builder` rebuild can never drop a row group without slicing the
303+
/// carried selection (which would silently return wrong rows). Correctness is
304+
/// bought at the cost of the dynamic-pruning optimization for this scan; the
305+
/// proper fix that keeps both is tracked upstream in arrow-rs #10624 / #24358.
303306
///
304307
/// Layout: 5 RGs × 1000 rows, with `data_page_row_count_limit=100` so
305308
/// each RG has 10 pages of 100 rows.
@@ -309,17 +312,14 @@ fn build_five_thousand_row_rgs(schema: &Arc<Schema>) -> Vec<RecordBatch> {
309312
/// first 5 pages (values 0..500) are pruned, the last 5 (500..1000)
310313
/// are scanned. RGs 1..4 keep all their pages (every page has
311314
/// `max >= 500`). The decoder receives a `RowSelection` that masks
312-
/// out those first 5 pages of RG 0.
313-
/// - `ORDER BY v DESC LIMIT 5` fills the TopK heap from RG 4
314-
/// (`max=4999`); the tightened threshold (≥ 4995) then proves RGs
315-
/// 0..3 unreachable and the runtime pruner drops them in one
316-
/// `into_builder` rebuild.
317-
///
318-
/// If `into_builder` did **not** preserve the row selection (or
319-
/// truncated / shifted it incorrectly), either the result rows would
320-
/// drift or the count of pruned pages would drop to zero.
315+
/// out those first 5 pages of RG 0 — its presence is what suppresses
316+
/// the runtime pruner.
317+
/// - `ORDER BY v DESC LIMIT 5` would let the tightened TopK threshold
318+
/// (≥ 4995) prune RGs 0..3, but because a row selection is present the
319+
/// runtime pruner is never created, so `row_groups_pruned_dynamic_filter`
320+
/// stays 0. Results are still correct and page-index pruning still runs.
321321
#[tokio::test]
322-
async fn dynamic_rg_pruning_coexists_with_page_index_row_selection() {
322+
async fn dynamic_rg_pruning_disabled_when_page_index_row_selection_present() {
323323
let schema = Arc::new(Schema::new(vec![Field::new("v", DataType::Int64, false)]));
324324
let batches = build_five_thousand_row_rgs(&schema);
325325

@@ -348,12 +348,9 @@ async fn dynamic_rg_pruning_coexists_with_page_index_row_selection() {
348348
);
349349
}
350350

351-
// Page-index pruning must have engaged: RG 0's first 5 pages are
352-
// entirely < 500. If `into_builder` dropped the row-selection state,
353-
// this metric would still report the original count (it is captured
354-
// at file open). Combined with the dynamic-pruner assertion below it
355-
// proves both mechanisms were active and that the rebuild left the
356-
// selection coherent — otherwise the result rows above would drift.
351+
// Page-index pruning still engages: RG 0's first 5 pages are entirely
352+
// < 500. #24355 only suppresses the *runtime* row-group pruner, not
353+
// page-index pruning, so this must remain non-zero.
357354
let pages_pruned = output.metric_value("page_index_pages_pruned").unwrap_or(0);
358355
assert!(
359356
pages_pruned >= 5,
@@ -362,13 +359,17 @@ async fn dynamic_rg_pruning_coexists_with_page_index_row_selection() {
362359
output.description(),
363360
);
364361

362+
// The runtime dynamic pruner must be disabled while a page-index row
363+
// selection is live (#24355): with no pruner there is no rebuild that
364+
// could misapply the carried selection. Before the fix the pruner ran
365+
// and this metric was >= 1.
365366
let pruned = output
366367
.row_groups_pruned_dynamic_filter()
367368
.expect("`row_groups_pruned_dynamic_filter` metric must be registered");
368-
assert!(
369-
pruned >= 1,
370-
"with TopK + tight threshold the runtime pruner must skip at least \
371-
one row group; pruned={pruned}\n{}",
369+
assert_eq!(
370+
pruned, 0,
371+
"runtime row-group pruning must be skipped when a page-index row \
372+
selection is present; pruned={pruned}\n{}",
372373
output.description(),
373374
);
374375
}

‎datafusion/datasource-parquet/src/opener/mod.rs‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1464,7 +1464,8 @@ impl RowGroupsPrunedParquetOpen {
14641464
};
14651465

14661466
let prepared_access_plan = prepare_access_plan(access_plan)?;
1467-
// #24355: a page-index row selection is carried by the decoder as one
1467+
// #24355: a row selection (from page-index pruning, or an externally
1468+
// supplied `ParquetRowSelection`) is carried by the decoder as one
14681469
// flat selection over the concatenation of the remaining row groups.
14691470
// The runtime pruner's `into_builder().with_row_groups(...)` rebuild
14701471
// drops row groups without slicing that selection to match, so record
@@ -1512,10 +1513,11 @@ impl RowGroupsPrunedParquetOpen {
15121513
// via the `DynamicFilterTracker` watch channel (#22460), so detecting
15131514
// a threshold change is a single atomic load — not a tree walk per
15141515
// RG check.
1515-
// Also disabled when a page-index row selection is live (#24355): the
1516-
// pruner rebuilds the decoder via `with_row_groups(...)`, which drops row
1517-
// groups without slicing the carried selection to match, so pruning under
1518-
// a live selection returns wrong results. Decline to prune in that case.
1516+
// Also disabled when a row selection is live (#24355) — page-index
1517+
// pruning is the common source: the pruner rebuilds the decoder via
1518+
// `with_row_groups(...)`, which drops row groups without slicing the
1519+
// carried selection to match, so pruning under a live selection returns
1520+
// wrong results. Decline to prune in that case.
15191521
let row_group_pruner =
15201522
match (&prepared.predicate, rg_plan.len() > 1, has_row_selection) {
15211523
(Some(predicate), true, false)

0 commit comments

Comments
 (0)