Skip to content

Commit aba5ea0

Browse files
cool9850311claude
andcommitted
fix: let leaf expressions reach the scan past a RepartitionExec
`ProjectionExec::benefits_from_input_partitioning` is true as soon as any expression in the projection has `KeepInPlace` placement. That single flag does two opposing things: `EnsureRequirements` inserts a `RepartitionExec` beneath the projection, and `RepartitionExec::try_swapping_with_projection` then refuses to let the same projection descend. A `MoveTowardsLeafNodes` expression sharing that projection can therefore never reach the source. `file_row_index()` and `input_file_name()` are then evaluated directly and fail with "is source dependent and cannot be evaluated directly", while `get_field` silently reads the whole struct instead of pruning to the field. The leaf call does not have to be wrapped in anything: an unrelated computed column in the same projection is enough, because the flag is a property of the whole projection. `ProjectionPushdown` now splits such a projection. The leaf expressions move into a lower projection that the source absorbs, and the computation stays above the repartition where it is still parallelised, so the intent of both guards is kept rather than one traded for the other. The split is committed only when the child really absorbs the leaf projection, and only `RoundRobinBatch` repartitioning is handled, because hash and range partitioning reference columns by index and would need remapping. Only the columns the outer expressions still reference outside the extracted leaves are passed through, so `s['a'] + 1` no longer reads the whole struct alongside the extracted field. Regression tests cover both symptom classes. Two traps are documented in the test files because either one makes the tests pass against broken code: the table must exceed `batch_size` for a repartition to be inserted at all, and a bare `LIMIT n` is pushed into the scan and hides the bug again. Closes apache#27131 Co-Authored-By: Claude Opus 5 <[email protected]>
1 parent 3f4a37f commit aba5ea0

3 files changed

Lines changed: 457 additions & 1 deletion

File tree

‎datafusion/physical-optimizer/src/projection_pushdown.rs‎

Lines changed: 203 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,14 +18,19 @@ use datafusion_common::tree_node::{
1818
Transformed, TransformedResult, TreeNode, TreeNodeRecursion,
1919
};
2020
use datafusion_common::{JoinSide, JoinType, Result, internal_datafusion_err};
21+
use datafusion_expr::ExpressionPlacement;
2122
use datafusion_physical_expr::expressions::{Column, LambdaExpr, LambdaVariable};
23+
use datafusion_physical_expr::projection::ProjectionExpr;
2224
use datafusion_physical_expr_common::physical_expr::{PhysicalExpr, is_volatile};
23-
use datafusion_physical_plan::ExecutionPlan;
2425
use datafusion_physical_plan::joins::NestedLoopJoinExec;
2526
use datafusion_physical_plan::joins::utils::{ColumnIndex, JoinFilter};
2627
use datafusion_physical_plan::projection::{
2728
ProjectionExec, remove_unnecessary_projections,
2829
};
30+
use datafusion_physical_plan::repartition::RepartitionExec;
31+
use datafusion_physical_plan::{
32+
ChildrenPropertiesMode, ExecutionPlan, Partitioning, ReplaceChildrenOptions,
33+
};
2934

3035
/// This rule inspects `ProjectionExec`'s in the given physical plan and tries to
3136
/// remove or swap with its child.
@@ -61,6 +66,18 @@ impl PhysicalOptimizerRule for ProjectionPushdown {
6166
})
6267
.map(|t| t.data)?;
6368

69+
// Push what can already descend, then split the projections that are
70+
// stuck above a `RepartitionExec`, then clean up once more: the split
71+
// can leave an outer projection that is now only column references.
72+
let plan = plan.transform_down(remove_unnecessary_projections).data()?;
73+
let plan = plan
74+
.transform_down(|plan| {
75+
Ok(match try_split_leaf_projection(&plan)? {
76+
Some(new_plan) => Transformed::yes(new_plan),
77+
None => Transformed::no(plan),
78+
})
79+
})
80+
.data()?;
6481
plan.transform_down(remove_unnecessary_projections).data()
6582
}
6683

@@ -449,6 +466,191 @@ fn remap_filter_expr(
449466
.data()
450467
}
451468

469+
/// Prefix for the columns this rule adds when it splits a projection.
470+
const LEAF_COLUMN_PREFIX: &str = "__datafusion_leaf";
471+
472+
/// Name of the column the split adds for the leaf expression at `index`.
473+
///
474+
/// Single source of truth: the lower projection's alias and the reference the
475+
/// outer projection is rewritten to must agree, or the rewrite silently points
476+
/// at nothing.
477+
fn leaf_column_name(index: usize) -> String {
478+
format!("{LEAF_COLUMN_PREFIX}_{index}")
479+
}
480+
481+
/// Splits a `ProjectionExec` that mixes leaf expressions with ones that must
482+
/// stay in place, moving the leaves into a projection the source absorbs.
483+
///
484+
/// `benefits_from_input_partitioning` is true as soon as any expression is
485+
/// `KeepInPlace`, which both inserts a `RepartitionExec` below the projection
486+
/// and stops that projection descending past it, so a `MoveTowardsLeafNodes`
487+
/// expression sharing it never reaches the source. Splitting leaves the
488+
/// computation above the repartition, still parallelised.
489+
fn try_split_leaf_projection(
490+
plan: &Arc<dyn ExecutionPlan>,
491+
) -> Result<Option<Arc<dyn ExecutionPlan>>> {
492+
let Some(projection) = plan.downcast_ref::<ProjectionExec>() else {
493+
return Ok(None);
494+
};
495+
let Some(repartition) = projection.input().downcast_ref::<RepartitionExec>() else {
496+
return Ok(None);
497+
};
498+
// Hash and Range partitioning reference columns by index, which would have
499+
// to be remapped against the rewritten child schema. Round robin carries no
500+
// column references, and is what `EnsureRequirements` inserts here.
501+
if !matches!(repartition.partitioning(), Partitioning::RoundRobinBatch(_)) {
502+
return Ok(None);
503+
}
504+
505+
let leaves = collect_leaf_exprs(projection.expr())?;
506+
if leaves.is_empty() {
507+
return Ok(None);
508+
}
509+
510+
// Pass through only the child columns the outer expressions still need. A
511+
// column used *only* inside an extracted leaf must be dropped, otherwise the
512+
// source keeps reading it: for `s['a'] + 1` that would read the whole struct
513+
// alongside the extracted field, which is worse than not splitting at all.
514+
let child = repartition.input();
515+
let child_schema = child.schema();
516+
let passthrough_indices = collect_outer_columns(projection.expr(), &leaves)?;
517+
let mut lower_exprs = passthrough_indices
518+
.iter()
519+
.map(|&idx| {
520+
let field = child_schema.field(idx);
521+
ProjectionExpr {
522+
expr: Arc::new(Column::new(field.name(), idx)),
523+
alias: field.name().to_string(),
524+
}
525+
})
526+
.collect::<Vec<_>>();
527+
for (index, leaf) in leaves.iter().enumerate() {
528+
lower_exprs.push(ProjectionExpr {
529+
expr: Arc::clone(leaf),
530+
alias: leaf_column_name(index),
531+
});
532+
}
533+
let lower_projection = ProjectionExec::try_new(lower_exprs, Arc::clone(child))?;
534+
535+
// Only commit the split if the child really absorbs the leaf projection.
536+
// Otherwise this would just add a projection that helps nobody.
537+
let Some(new_child) = child.try_swapping_with_projection(&lower_projection)? else {
538+
return Ok(None);
539+
};
540+
541+
let new_exprs = projection
542+
.expr()
543+
.iter()
544+
.map(|proj_expr| {
545+
Ok(ProjectionExpr {
546+
expr: rewrite_outer_expr(
547+
Arc::clone(&proj_expr.expr),
548+
&leaves,
549+
&passthrough_indices,
550+
)?,
551+
alias: proj_expr.alias.clone(),
552+
})
553+
})
554+
.collect::<Result<Vec<_>>>()?;
555+
556+
// The child gained the leaf columns, so the repartition's properties have to
557+
// be recomputed rather than carried over.
558+
let new_repartition = Arc::clone(projection.input()).replace_children(
559+
vec![new_child],
560+
ReplaceChildrenOptions::new(ChildrenPropertiesMode::Recompute),
561+
)?;
562+
Ok(Some(Arc::new(ProjectionExec::try_new(
563+
new_exprs,
564+
new_repartition,
565+
)?)))
566+
}
567+
568+
/// Collects the distinct outermost [`ExpressionPlacement::MoveTowardsLeafNodes`]
569+
/// sub-expressions of `exprs`.
570+
///
571+
/// Nested leaf expressions are not collected separately: `s['a']['b']` is one
572+
/// unit to hand to the source, not two.
573+
fn collect_leaf_exprs(exprs: &[ProjectionExpr]) -> Result<Vec<Arc<dyn PhysicalExpr>>> {
574+
let mut leaves: Vec<Arc<dyn PhysicalExpr>> = vec![];
575+
for proj_expr in exprs {
576+
proj_expr.expr.apply(|node| {
577+
if node.placement() == ExpressionPlacement::MoveTowardsLeafNodes {
578+
if !leaves.iter().any(|leaf| leaf == node) {
579+
leaves.push(Arc::clone(node));
580+
}
581+
return Ok(TreeNodeRecursion::Jump);
582+
}
583+
Ok(TreeNodeRecursion::Continue)
584+
})?;
585+
}
586+
Ok(leaves)
587+
}
588+
589+
/// Collects the child column indices that `exprs` still reference *outside* the
590+
/// extracted leaf sub-expressions, sorted and deduplicated.
591+
///
592+
/// Recursion stops at a leaf: columns used only inside one are supplied by the
593+
/// lower projection itself and must not be passed through.
594+
fn collect_outer_columns(
595+
exprs: &[ProjectionExpr],
596+
leaves: &[Arc<dyn PhysicalExpr>],
597+
) -> Result<Vec<usize>> {
598+
let mut indices: Vec<usize> = vec![];
599+
for proj_expr in exprs {
600+
proj_expr.expr.apply(|node| {
601+
if leaves.iter().any(|leaf| leaf == node) {
602+
return Ok(TreeNodeRecursion::Jump);
603+
}
604+
if let Some(column) = node.downcast_ref::<Column>() {
605+
indices.push(column.index());
606+
}
607+
Ok(TreeNodeRecursion::Continue)
608+
})?;
609+
}
610+
indices.sort_unstable();
611+
indices.dedup();
612+
Ok(indices)
613+
}
614+
615+
/// Rewrites an outer projection expression against the split child schema:
616+
/// every leaf sub-expression becomes a reference to the column the split added
617+
/// for it, and every surviving column is remapped to its new index.
618+
fn rewrite_outer_expr(
619+
expr: Arc<dyn PhysicalExpr>,
620+
leaves: &[Arc<dyn PhysicalExpr>],
621+
passthrough_indices: &[usize],
622+
) -> Result<Arc<dyn PhysicalExpr>> {
623+
// The leaf columns were appended after the passthrough ones.
624+
let first_leaf = passthrough_indices.len();
625+
expr.transform_down(|node| {
626+
if let Some(pos) = leaves.iter().position(|leaf| leaf == &node) {
627+
let column = Column::new(&leaf_column_name(pos), first_leaf + pos);
628+
return Ok(Transformed::new(
629+
Arc::new(column) as Arc<dyn PhysicalExpr>,
630+
true,
631+
TreeNodeRecursion::Jump,
632+
));
633+
}
634+
if let Some(column) = node.downcast_ref::<Column>() {
635+
let new_index = passthrough_indices
636+
.iter()
637+
.position(|&idx| idx == column.index())
638+
.ok_or_else(|| {
639+
internal_datafusion_err!(
640+
"projection split dropped column {} that is still referenced",
641+
column.index()
642+
)
643+
})?;
644+
if new_index != column.index() {
645+
let remapped = Column::new(column.name(), new_index);
646+
return Ok(Transformed::yes(Arc::new(remapped) as Arc<dyn PhysicalExpr>));
647+
}
648+
}
649+
Ok(Transformed::no(node))
650+
})
651+
.data()
652+
}
653+
452654
#[cfg(test)]
453655
mod test {
454656
use super::*;

‎datafusion/sqllogictest/test_files/file_row_index.slt‎

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -151,6 +151,145 @@ ORDER BY column1
151151
statement ok
152152
DROP TABLE parquet_two_files;
153153

154+
# A leaf expression must still reach the scan when its projection also holds a
155+
# `KeepInPlace` one; see `try_split_leaf_projection` in
156+
# `datafusion/physical-optimizer/src/projection_pushdown.rs`. These need a
157+
# repartition, so `batch_size` is below the row count: do not raise it and do
158+
# not add a bare `LIMIT`, or the queries pass against broken code.
159+
160+
statement ok
161+
SET datafusion.execution.batch_size = 2;
162+
163+
# The marker wrapped in arithmetic.
164+
query II
165+
SELECT file_row_index() + 1, column1 FROM parquet_table ORDER BY column1
166+
----
167+
1 10
168+
2 20
169+
3 30
170+
4 40
171+
5 50
172+
173+
# The marker is BARE here. An unrelated computed column in the same projection
174+
# is enough, because the flag is a property of the whole projection.
175+
query II
176+
SELECT file_row_index(), column1 + 1 FROM parquet_table ORDER BY column1
177+
----
178+
0 11
179+
1 21
180+
2 31
181+
3 41
182+
4 51
183+
184+
# Not specific to arithmetic: any enclosing expression has the same effect.
185+
query TI
186+
SELECT arrow_typeof(file_row_index()), column1 FROM parquet_table ORDER BY column1
187+
----
188+
Int64 10
189+
Int64 20
190+
Int64 30
191+
Int64 40
192+
Int64 50
193+
194+
# The sibling marker behaves the same way. Compared against a predicate rather
195+
# than printed, so the expectation does not depend on the absolute file path.
196+
query BI
197+
SELECT length(input_file_name()) > 0, column1 FROM parquet_table ORDER BY column1
198+
----
199+
true 10
200+
true 20
201+
true 30
202+
true 40
203+
true 50
204+
205+
# Both markers in one projection, each inside an enclosing expression.
206+
query BI
207+
SELECT input_file_name() IS NOT NULL AND file_row_index() >= 0, column1
208+
FROM parquet_table ORDER BY column1
209+
----
210+
true 10
211+
true 20
212+
true 30
213+
true 40
214+
true 50
215+
216+
# A bare `input_file_name()` beside a computed column: the same trigger as the
217+
# `file_row_index()` case above. Asserted through the plan rather than the
218+
# values, because the value is an absolute path.
219+
query TT
220+
EXPLAIN SELECT input_file_name(), column1 + 1 FROM parquet_table
221+
----
222+
logical_plan
223+
01)Projection: input_file_name(), CAST(parquet_table.column1 AS Int64) + Int64(1)
224+
02)--TableScan: parquet_table projection=[column1]
225+
physical_plan
226+
01)ProjectionExec: expr=[__datafusion_leaf_0@1 as input_file_name(), CAST(column1@0 AS Int64) + 1 as parquet_table.column1 + Int64(1)]
227+
02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
228+
03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/file_row_index/parquet_table/data.parquet]]}, projection=[column1, input_file_name() as __datafusion_leaf_0], file_type=parquet
229+
230+
# The same marker twice. Both calls are one expression, so the split gives them
231+
# a single column and the marker is evaluated once per row, not twice. Both
232+
# outputs must still agree with each other.
233+
query III
234+
SELECT file_row_index(), file_row_index() + 1, column1
235+
FROM parquet_table ORDER BY column1
236+
----
237+
0 1 10
238+
1 2 20
239+
2 3 30
240+
3 4 40
241+
4 5 50
242+
243+
# Shapes that already worked must keep working: the leaf rules extract from
244+
# `Filter`, `Sort` and `Aggregate`, so those never depended on this fix.
245+
query I
246+
SELECT column1 FROM parquet_table WHERE file_row_index() + 1 = 3
247+
----
248+
30
249+
250+
query I
251+
SELECT column1 FROM parquet_table ORDER BY file_row_index() + 1
252+
----
253+
10
254+
20
255+
30
256+
40
257+
50
258+
259+
query I
260+
SELECT sum(file_row_index() + 1) FROM parquet_table
261+
----
262+
15
263+
264+
# A bare marker on its own is unaffected: no `KeepInPlace` expression shares
265+
# the projection, so nothing is inserted beneath it.
266+
query II
267+
SELECT file_row_index(), column1 FROM parquet_table ORDER BY column1
268+
----
269+
0 10
270+
1 20
271+
2 30
272+
3 40
273+
4 50
274+
275+
# Raising `batch_size` above the row count removes the repartition. This must
276+
# keep working, and it is also the control that proves trap 1 above is real.
277+
statement ok
278+
SET datafusion.execution.batch_size = 8192;
279+
280+
query II
281+
SELECT file_row_index() + 1, column1 FROM parquet_table ORDER BY column1
282+
----
283+
1 10
284+
2 20
285+
3 30
286+
4 40
287+
5 50
288+
289+
# `target_partitions = 1` also removes the repartition, but is not covered here:
290+
# `RESET` restores the global default rather than the harness value, which the
291+
# harness rejects as leftover configuration.
292+
154293
statement ok
155294
DROP TABLE parquet_table;
156295

0 commit comments

Comments
 (0)