Skip to content

Commit fc15e25

Browse files
authored
refactor(parquet): extract InitialDecoderState from the opener decoder-setup block (apache#24572)
## Which issue does this PR close? - Closes apache#24286. ## Rationale for this change Split out from apache#23696 (per @adriangb's [decomposition proposal](apache#23696 (comment)), PR 3). The decoder-setup block in the opener returns a bare `(decoder, rg_plan, has_row_selection)` tuple. Giving it a named `InitialDecoderState` struct keeps the values carried out of the block self-documenting, and means the follow-up feature PR (apache#23696) adds fields to a named struct instead of reshaping a positional tuple. ## What changes are included in this PR? - `push_decoder.rs`: new `pub(crate) struct InitialDecoderState { decoder, rg_plan, has_row_selection }`. - `opener/mod.rs`: the decoder-setup block returns `InitialDecoderState { .. }` and is destructured back into the same locals. Pure refactor, no behavior change. ## Are these changes tested? Covered by the existing `dynamic_row_group_pruning` integration tests (they drive the opener path); all pass. `clippy --all-targets --all-features -D warnings` is clean. ## Are there any user-facing changes? No.
1 parent 933710d commit fc15e25

2 files changed

Lines changed: 27 additions & 3 deletions

File tree

‎datafusion/datasource-parquet/src/opener/mod.rs‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,8 @@ use crate::decoder_projection::DecoderProjection;
2828
use crate::metrics::ByteProgress;
2929
use crate::page_filter::PagePruningAccessPlanFilter;
3030
use crate::push_decoder::{
31-
DecoderBuilderConfig, PushDecoderStreamState, RgPlanEntry, RowGroupPruner,
31+
DecoderBuilderConfig, InitialDecoderState, PushDecoderStreamState, RgPlanEntry,
32+
RowGroupPruner,
3233
};
3334
use crate::row_filter::RowFilterGenerator;
3435
use crate::row_group_filter::{RowGroupAccessPlanFilter, row_group_in_range};
@@ -1458,7 +1459,11 @@ impl RowGroupsPrunedParquetOpen {
14581459
prepared.virtual_state.as_deref(),
14591460
)?;
14601461

1461-
let (decoder, rg_plan, has_row_selection) = {
1462+
let InitialDecoderState {
1463+
decoder,
1464+
rg_plan,
1465+
has_row_selection,
1466+
} = {
14621467
let pushdown_predicate = prepared
14631468
.pushdown_filters
14641469
.then_some(prepared.predicate.as_ref())
@@ -1520,7 +1525,11 @@ impl RowGroupsPrunedParquetOpen {
15201525
}
15211526
}
15221527

1523-
(builder.build()?, rg_plan, has_row_selection)
1528+
InitialDecoderState {
1529+
decoder: builder.build()?,
1530+
rg_plan,
1531+
has_row_selection,
1532+
}
15241533
};
15251534

15261535
// Track how much of this file range the scan has finished with. Credit

‎datafusion/datasource-parquet/src/push_decoder.rs‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -117,6 +117,21 @@ pub(crate) struct RgPlanEntry {
117117
pub(crate) bytes: u64,
118118
}
119119

120+
/// The initial per-file decoder state the opener builds and hands off to the
121+
/// [`PushDecoderStreamState`] stream driver.
122+
///
123+
/// Named rather than a bare tuple so the fields carried out of the decoder
124+
/// setup block stay self-documenting as more are added.
125+
pub(crate) struct InitialDecoderState {
126+
/// The freshly built push decoder for this file.
127+
pub(crate) decoder: ParquetPushDecoder,
128+
/// The per-row-group plan, in the physical scan order the decoder reads.
129+
pub(crate) rg_plan: VecDeque<RgPlanEntry>,
130+
/// Whether a row selection is live for this scan. Runtime row-group
131+
/// pruning is disabled when it is (see the opener for why).
132+
pub(crate) has_row_selection: bool,
133+
}
134+
120135
/// Runtime row-group pruner driven by a dynamic predicate (e.g. the
121136
/// threshold expression a `TopK` operator pushes down).
122137
///

0 commit comments

Comments
 (0)