Skip to content

Commit 8a166f4

Browse files
adriangbclaude
andcommitted
feat(parquet): carry row selections across placement rebuilds
The opener froze the adaptive filter placement of a file when a row selection was live (apache#24355). Page index pruning installs a selection on every row group, also a select-all one, thus the placement of a file with a page index never changed (`filter_placement_changes=0` on TPC-H). apache#24355 was about runtime row group pruning, which drops row groups. The placement rebuild only changes the `RowFilter` and the projection mask: `ParquetPushDecoder::into_builder` returns the remaining row group plan with the selection of each row group that is not read yet, and the rebuild does not replace it. Thus the placement can change with a live selection. Remove the freeze. The freeze for a change of the decoded batch schema (nested columns) stays: the batch coalescer holds batches with the current schema. The new opener test uses a page index selection that skips rows in the first three row groups. It fails with the freeze (no placement change) and when the rebuild drops the remaining selections (the scan fetches the pruned pages). Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 19dc188 commit 8a166f4

3 files changed

Lines changed: 147 additions & 24 deletions

File tree

‎datafusion/datasource-parquet/src/filter_placement/mod.rs‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -35,8 +35,14 @@
3535
//! [`FilePlacement`] decides at file open and again at each row group
3636
//! boundary. The stream then rebuilds the decoder
3737
//! (`ParquetPushDecoder::into_builder`) with the new `RowFilter` and
38-
//! projection mask, and rebuilds the post-scan filter. The measurements are
39-
//! pooled over all files and partitions of the scan ([`PlacementSites`]).
38+
//! projection mask, and rebuilds the post-scan filter. The rebuilt decoder
39+
//! keeps the row selections of the remaining row groups (for example from
40+
//! the page index). The measurements are pooled over all files and
41+
//! partitions of the scan ([`PlacementSites`]).
42+
//!
43+
//! The placement of a file stops changing only when a change would change
44+
//! the schema of the decoded batches (possible with nested columns): the
45+
//! batch coalescer holds batches with the current schema.
4046
//!
4147
//! Conjuncts that `build_row_filter` rejects for a file always run in the
4248
//! post-scan filter (required) or are not used (optional). Optional
@@ -288,11 +294,6 @@ impl FilePlacement {
288294
self.frozen = true;
289295
}
290296

291-
/// Stops all later changes.
292-
pub(crate) fn freeze(&mut self) {
293-
self.frozen = true;
294-
}
295-
296297
/// True if the candidate at `index` is a predicate of the `RowFilter`.
297298
pub(crate) fn in_row_filter(&self, index: usize) -> bool {
298299
self.conjuncts

‎datafusion/datasource-parquet/src/opener/mod.rs‎

Lines changed: 132 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1927,17 +1927,11 @@ impl RowGroupsPrunedParquetOpen {
19271927
// installed filter is owned by the decoder and is not
19281928
// recoverable once replaced.
19291929
let first_rg_fully_matched = rg_plan.front().is_some_and(|e| e.fully_matched);
1930-
let mut row_filter_context = precomputed_context;
1931-
// A rebuild at a row group boundary cannot keep a live row
1932-
// selection (#24355), thus the adaptive filter placement keeps
1933-
// its first decision for this file.
1934-
if has_row_selection
1935-
&& let Some(placement) = row_filter_context
1936-
.as_mut()
1937-
.and_then(|ctx| ctx.placement.as_mut())
1938-
{
1939-
placement.freeze();
1940-
}
1930+
// The adaptive filter placement can change at a row group
1931+
// boundary also when a row selection is live: the rebuild keeps
1932+
// the selections of the remaining row groups (see
1933+
// `PushDecoderStreamState::rebuild_decoder_at_boundary`).
1934+
let row_filter_context = precomputed_context;
19411935

19421936
let mut builder = decoder_config.build(selections, reader_metadata.clone());
19431937
let mut filter_installed = false;
@@ -5704,6 +5698,16 @@ mod test {
57045698
async fn write_file(
57055699
store: &Arc<dyn ObjectStore>,
57065700
a: impl Fn(usize) -> i32,
5701+
) -> (SchemaRef, PartitionedFile) {
5702+
write_file_with_pages(store, a, None).await
5703+
}
5704+
5705+
/// Like [`write_file`], with at most `page_rows` rows in each data
5706+
/// page if it is `Some`.
5707+
async fn write_file_with_pages(
5708+
store: &Arc<dyn ObjectStore>,
5709+
a: impl Fn(usize) -> i32,
5710+
page_rows: Option<usize>,
57075711
) -> (SchemaRef, PartitionedFile) {
57085712
let schema = Arc::new(Schema::new(vec![
57095713
Field::new("a", DataType::Int32, false),
@@ -5717,9 +5721,14 @@ mod test {
57175721
],
57185722
)
57195723
.unwrap();
5720-
let props = WriterProperties::builder()
5721-
.set_max_row_group_row_count(Some(ROWS_PER_ROW_GROUP))
5722-
.build();
5724+
let mut props = WriterProperties::builder()
5725+
.set_max_row_group_row_count(Some(ROWS_PER_ROW_GROUP));
5726+
if let Some(page_rows) = page_rows {
5727+
props = props
5728+
.set_data_page_row_count_limit(page_rows)
5729+
.set_write_batch_size(page_rows);
5730+
}
5731+
let props = props.build();
57235732
let size = write_parquet_batches(
57245733
Arc::clone(store),
57255734
"placement.parquet",
@@ -5822,6 +5831,115 @@ mod test {
58225831
assert_eq!(post_scan.post_scan_rows(), TOTAL_ROWS);
58235832
}
58245833

5834+
/// The values of `v` that a scan with the page index returns, and
5835+
/// its metrics.
5836+
async fn scan_values_with_page_index(
5837+
store: &Arc<dyn ObjectStore>,
5838+
schema: &SchemaRef,
5839+
file: &PartitionedFile,
5840+
predicate: &Arc<dyn PhysicalExpr>,
5841+
pushdown_filters: bool,
5842+
filter_placement: bool,
5843+
) -> (Vec<i64>, ExecutionPlanMetricsSet) {
5844+
let metrics = ExecutionPlanMetricsSet::new();
5845+
let morselizer = ParquetMorselizerBuilder::new()
5846+
.with_store(Arc::clone(store))
5847+
.with_schema(Arc::clone(schema))
5848+
.with_predicate(Arc::clone(predicate))
5849+
.with_pushdown_filters(pushdown_filters)
5850+
.with_filter_placement(filter_placement)
5851+
.with_enable_page_index(true)
5852+
.with_metrics(metrics.clone())
5853+
.build();
5854+
let stream = open_file(&morselizer, file.clone()).await.unwrap();
5855+
let batches = stream.try_collect::<Vec<_>>().await.unwrap();
5856+
let values = batches
5857+
.iter()
5858+
.flat_map(|batch| {
5859+
batch
5860+
.column(1)
5861+
.as_any()
5862+
.downcast_ref::<Int64Array>()
5863+
.unwrap()
5864+
.values()
5865+
.to_vec()
5866+
})
5867+
.collect();
5868+
(values, metrics)
5869+
}
5870+
5871+
/// The page index skips the first pages of the first row group
5872+
/// (`v >= 1000`) and pages of the second and third row groups
5873+
/// (`v < 20000 OR v >= 40000`), thus the scan has a live row
5874+
/// selection. The placement still changes at the first row group
5875+
/// boundary, the rebuilt decoder keeps the selections of the
5876+
/// remaining row groups, and the result is the same as with a row
5877+
/// filter only and with a post-scan filter only.
5878+
#[tokio::test]
5879+
async fn placement_changes_with_page_index_selection() {
5880+
use datafusion_physical_plan::metrics::MetricValue;
5881+
5882+
let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
5883+
let (schema, file) =
5884+
write_file_with_pages(&store, |i| (i % 2) as i32, Some(100)).await;
5885+
let predicate = logical2physical(
5886+
&col("a").eq(lit(0)).and(col("v").gt_eq(lit(1000_i64))).and(
5887+
col("v")
5888+
.lt(lit(20_000_i64))
5889+
.or(col("v").gt_eq(lit(40_000_i64))),
5890+
),
5891+
&schema,
5892+
);
5893+
5894+
let (adaptive, adaptive_metrics) = scan_values_with_page_index(
5895+
&store, &schema, &file, &predicate, true, true,
5896+
)
5897+
.await;
5898+
let (row_filter, row_filter_metrics) = scan_values_with_page_index(
5899+
&store, &schema, &file, &predicate, true, false,
5900+
)
5901+
.await;
5902+
let (post_scan, post_scan_metrics) = scan_values_with_page_index(
5903+
&store, &schema, &file, &predicate, false, false,
5904+
)
5905+
.await;
5906+
5907+
let expected: Vec<i64> = (1000..20_000)
5908+
.chain(40_000..TOTAL_ROWS as i64)
5909+
.step_by(2)
5910+
.collect();
5911+
assert_eq!(adaptive, expected);
5912+
assert_eq!(row_filter, expected);
5913+
assert_eq!(post_scan, expected);
5914+
5915+
// The page index skipped rows: the selection is not select-all.
5916+
let MetricValue::PruningMetrics {
5917+
pruning_metrics, ..
5918+
} = adaptive_metrics
5919+
.clone_inner()
5920+
.sum_by_name("page_index_rows_pruned")
5921+
.unwrap()
5922+
else {
5923+
panic!("expected page pruning metrics");
5924+
};
5925+
// Whole pages only: a page that has a row in the result is kept.
5926+
assert!(pruning_metrics.pruned() > 20_000);
5927+
assert!(
5928+
counter_metric_value(&adaptive_metrics, "filter_placement_changes") > 0
5929+
);
5930+
// The rebuilt decoder fetches only the pages that the selections
5931+
// keep, as the scans that do not change the placement. A rebuild
5932+
// that dropped the selections of the remaining row groups fetches
5933+
// the pruned pages of the second and third row groups.
5934+
let bytes = |metrics: &ExecutionPlanMetricsSet| {
5935+
counter_metric_value(metrics, "bytes_scanned")
5936+
};
5937+
assert!(
5938+
bytes(&adaptive_metrics)
5939+
<= bytes(&row_filter_metrics).max(bytes(&post_scan_metrics))
5940+
);
5941+
}
5942+
58255943
/// A filter that removes long runs of rows saves decode time in a row
58265944
/// filter, thus it stays a row filter.
58275945
#[tokio::test]

‎datafusion/datasource-parquet/src/push_decoder.rs‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -902,9 +902,13 @@ impl PushDecoderStreamState {
902902

903903
let decoder = self.decoder.take().expect("decoder present");
904904
let mut builder = decoder.into_builder().map_err(DataFusionError::from)?;
905-
// Filter-only rebuilds preserve the decoder's remaining selections.
906-
// Runtime pruning is disabled for scans with selections, so pruned
907-
// plans can be rebuilt from row-group indexes alone.
905+
// `into_builder` returns the decoder's remaining row group plan with
906+
// the selection of each row group that is not read yet. Thus a
907+
// rebuild that changes only the filter or the projection (the
908+
// fully-matched toggle, the adaptive filter placement) keeps a live
909+
// row selection (for example from the page index). Runtime pruning
910+
// is disabled for scans with selections, so pruned plans can be
911+
// rebuilt from row-group indexes alone.
908912
if pruned_count > 0 {
909913
let selections = self
910914
.rg_plan

0 commit comments

Comments
 (0)