Skip to content

Commit 23339c5

Browse files
committed
fix(parquet): gate runtime RG-prune rebuild on is_at_row_group_boundary
ClickBench Q24 / Q26 (`SELECT … WHERE x <> '' ORDER BY ts LIMIT 10`) were failing with: Parquet error: into_builder called mid-row-group; check is_at_row_group_boundary() first `transition()` in `push_decoder.rs` re-enters Step 2 on every loop iteration — including the iterations where Step 3 returned `NeedsData` and we pushed byte ranges but the decoder has not yet handed back a reader. At those moments the decoder sits in `ReadingRowGroup` state but `is_at_row_group_boundary()` is `false`, and arrow-rs's `into_builder` errors out (it can only rebuild at a clean RG boundary). Step 2's prune-and-rebuild block now skips when the decoder is mid-RG. Step 3 still drives the decoder forward in that iteration, and the next boundary re-enters Step 2 with the pruner in the same state (the pruner is stateful but idempotent — re-evaluating costs one cached `pp.prune` per RG, no rebuild). No semantic change for queries that were already passing. Add `dynamic_rg_pruner_does_not_call_into_builder_mid_row_group`: a 20-RG × 50-row file plus `ORDER BY v ASC LIMIT 10` gives the pruner multiple boundaries to attempt rebuilds and reliably trips the failure mode on the pre-fix code.
1 parent 1b55225 commit 23339c5

2 files changed

Lines changed: 94 additions & 8 deletions

File tree

‎datafusion/core/tests/parquet/dynamic_row_group_pruning.rs‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -207,6 +207,79 @@ async fn dynamic_rg_pruning_metric_quiet_without_topk() {
207207
);
208208
}
209209

210+
/// Regression for "into_builder called mid-row-group" — surfaced by
211+
/// ClickBench Q24 / Q26 (`SELECT … WHERE x <> '' ORDER BY ts LIMIT 10`).
212+
///
213+
/// The push-decoder state machine re-enters Step 2 on every iteration of
214+
/// the `transition` loop, including iterations where Step 3 returned
215+
/// `NeedsData` and pushed byte ranges but has not yet produced a reader
216+
/// for the upcoming row group. At those moments the decoder is in
217+
/// `ReadingRowGroup` state but `is_at_row_group_boundary()` is `false`,
218+
/// and the runtime row-group pruner's `into_builder()` rebuild path
219+
/// errored out with:
220+
///
221+
/// ```text
222+
/// Parquet error: into_builder called mid-row-group;
223+
/// check is_at_row_group_boundary() first
224+
/// ```
225+
///
226+
/// The fix in `push_decoder.rs::Step 2` gates the prune-and-rebuild on
227+
/// `is_at_row_group_boundary()`. This test reproduces the trigger: a
228+
/// many-RG file (so the pruner has work to do) plus an `ORDER BY` query
229+
/// whose TopK threshold tightens enough to make the pruner want to
230+
/// rebuild more than once during the scan. Before the fix the query
231+
/// returned an `Execution` / `Parquet` error; after the fix it returns
232+
/// the expected ten rows and the pruner fires.
233+
#[tokio::test]
234+
async fn dynamic_rg_pruner_does_not_call_into_builder_mid_row_group() {
235+
let schema = Arc::new(Schema::new(vec![Field::new("v", DataType::Int64, false)]));
236+
// 20 disjoint row groups of 50 values each. With 20 RGs the pruner
237+
// gets multiple boundaries to attempt rebuilds, so any path that
238+
// calls `into_builder` outside a boundary is hit reliably.
239+
let batches: Vec<RecordBatch> = (0..20i64)
240+
.map(|rg| {
241+
let base = rg * 50;
242+
let values: Vec<i64> = (base..base + 50).collect();
243+
let col: ArrayRef = Arc::new(Int64Array::from(values));
244+
RecordBatch::try_new(Arc::clone(&schema), vec![col]).unwrap()
245+
})
246+
.collect();
247+
248+
let mut ctx = ContextWithParquet::with_custom_data(
249+
Scenario::Int,
250+
RowGroup(50),
251+
Arc::clone(&schema),
252+
batches,
253+
)
254+
.await;
255+
256+
let output = ctx.query("SELECT v FROM t ORDER BY v ASC LIMIT 10").await;
257+
258+
// Correctness: smallest ten values are 0..=9.
259+
assert_eq!(output.result_rows, 10, "query must return LIMIT rows");
260+
let formatted = output.pretty_results();
261+
for v in 0..=9i64 {
262+
assert!(
263+
formatted.contains(&format!("| {v} ")),
264+
"output must contain smallest value {v}; got:\n{formatted}",
265+
);
266+
}
267+
268+
// Behavior: with 20 disjoint RGs and a tight TopK, the dynamic
269+
// pruner must skip a meaningful share of them. We don't pin the
270+
// exact count — what matters is that the scan *completed* without
271+
// the mid-row-group rebuild error.
272+
let pruned = output
273+
.row_groups_pruned_dynamic_filter()
274+
.expect("`row_groups_pruned_dynamic_filter` metric must be registered");
275+
assert!(
276+
pruned >= 1,
277+
"dynamic RG pruner must skip at least one row group; \
278+
pruned={pruned}\n{}",
279+
output.description(),
280+
);
281+
}
282+
210283
/// Build five sorted `RecordBatch`es with 1000 values each so that, when
211284
/// the writer is configured with `row_per_group=1000` and
212285
/// `data_page_row_count_limit=100`, every row group ends up with **ten

‎datafusion/datasource-parquet/src/push_decoder.rs‎

Lines changed: 21 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -315,14 +315,27 @@ impl PushDecoderStreamState {
315315
}
316316
}
317317

318-
// Step 2: at RG boundary, scan the entire `rg_plan` and drop
319-
// every RG the pruner proves cannot contribute — head, interior,
320-
// and tail alike. Evaluating per-RG stats against the cached
321-
// `PruningPredicate` is cheap; the expensive part is the
322-
// `into_builder` rebuild, so we do at most one rebuild per
323-
// boundary regardless of how many RGs were dropped. Buffered
324-
// bytes for already-fetched RGs carry across the rebuild.
325-
if !self.rg_plan.is_empty() {
318+
// Step 2: when the decoder is sitting on a row-group boundary,
319+
// scan the entire `rg_plan` and drop every RG the pruner proves
320+
// cannot contribute — head, interior, and tail alike. Evaluating
321+
// per-RG stats against the cached `PruningPredicate` is cheap;
322+
// the expensive part is the `into_builder` rebuild, so we do at
323+
// most one rebuild per boundary regardless of how many RGs were
324+
// dropped. Buffered bytes for already-fetched RGs carry across
325+
// the rebuild.
326+
//
327+
// `into_builder` errors out mid-row-group, so we gate the prune
328+
// pass on `is_at_row_group_boundary()`. When the decoder is
329+
// mid-RG (e.g. byte ranges have been pushed but no reader has
330+
// been handed back yet), step 3 drives it forward and we get
331+
// another chance at the next boundary — the pruner is stateful
332+
// and idempotent, so deferring loses nothing.
333+
let at_boundary = self
334+
.decoder
335+
.as_ref()
336+
.expect("decoder present")
337+
.is_at_row_group_boundary();
338+
if at_boundary && !self.rg_plan.is_empty() {
326339
let mut pruned_count = 0usize;
327340
if let Some(pruner) = self.row_group_pruner.as_mut() {
328341
let mut kept = VecDeque::with_capacity(self.rg_plan.len());

0 commit comments

Comments
 (0)