Skip to content

Commit 4ee8c64

Browse files
cool9850311claude
andcommitted
test: pin the leaf projection split's refusal path and the shapes it fixes
Review follow-up. Nothing here changes what the split computes. The split is only committed when the child really absorbs the leaf projection, but no test held that: removing the check left all 545 SLT files, 50 physical-optimizer unit tests and 669 `core_integration` ones green. A memory table refuses the lower projection, because `all_alias_free_columns` rejects an aliased `get_field`, and that is also what any node that does not override `try_swapping_with_projection` does. An `EXPLAIN` on such a table now pins the plan; with the check removed it fails on the extra `ProjectionExec`. Collecting each distinct leaf once was likewise unpinned, and the values cannot show it: two calls to the same marker agree with each other whether they share a column or not. The duplicated-marker case gains an `EXPLAIN`, which fails with a second `__datafusion_leaf_1` in the scan projection if the de-duplication goes. Three shapes the split fixes had no test either. Each reaches a different path: a TopK sorts on a column the split produced and runs above the repartition with `preserve_partitioning=[true]`; a marker inside a lambda body has to be found through `LambdaExpr`; and CSV is a second scan implementation, while every other test here uses Parquet. All three error with `is source dependent and cannot be evaluated directly` without the split. The cleanup walk this PR had added after the split is removed rather than made conditional, because it cannot do anything. `remove_unnecessary_projections` has three paths for a projection: it is not removable, since the split renames and keeps the original output schema; `overrides_metadata` changes nothing; and `RepartitionExec` declines it either for not narrowing, when every expression stayed a plain column, or for holding a computed one, when an expression consumed more than one leaf and the projection did narrow. Nothing else in the tree moved, as the split preserves the projection's schema. Instrumenting the walk confirmed it changed no plan in any of the 19 splits the suite fires. Handling only `RoundRobinBatch` was unpinned the same way, and no SQL reaches that shape: the guard is hit 1326 times across the suite, every one of them round robin. The rule is a plain plan-to-plan function though, so the plan can be built by hand, which `split_declines_hash_repartition` does, after `ensure_requirements.rs` for the hash repartition and `test_utils.rs` for the scan. The scan has to be Parquet rather than `EmptyExec`: only a source that absorbs the leaf projection lets the split commit once the check is gone. Also drops a comment reference to a trap number that only exists in the pull request description, and rewrites the one on `collect_leaf_exprs` that used `s['a']['b']` as its example: the logical optimizer folds that into a single call, so the branch it describes never sees it. It now says what stopping at a leaf avoids instead. Part of apache#27131 Co-Authored-By: Claude Opus 5 <[email protected]>
1 parent 322129a commit 4ee8c64

5 files changed

Lines changed: 176 additions & 14 deletions

File tree

‎datafusion/core/tests/physical_optimizer/projection_pushdown.rs‎

Lines changed: 79 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ use arrow::compute::SortOptions;
99
use arrow::datatypes::{DataType, Field, Schema, SchemaRef};
1010
use datafusion::datasource::listing::PartitionedFile;
1111
use datafusion::datasource::memory::MemorySourceConfig;
12-
use datafusion::datasource::physical_plan::CsvSource;
12+
use datafusion::datasource::physical_plan::{CsvSource, ParquetSource};
1313
use datafusion::datasource::source::DataSourceExec;
1414
use datafusion::prelude::{ParquetReadOptions, SessionContext};
1515
use datafusion_common::config::{ConfigOptions, CsvOptions};
@@ -1957,3 +1957,81 @@ async fn split_keeps_field_metadata() -> Result<()> {
19571957

19581958
Ok(())
19591959
}
1960+
1961+
/// The split only handles round robin partitioning, because hash and range keys
1962+
/// reference columns by index and would have to be remapped against the
1963+
/// rewritten child schema. No SQL reaches this shape, so the test builds it:
1964+
/// above a join, the leaf is already extracted into its own projection below the
1965+
/// join, and nothing else puts a projection on a hash repartition.
1966+
#[tokio::test]
1967+
async fn split_declines_hash_repartition() -> Result<()> {
1968+
let schema: SchemaRef = Arc::new(Schema::new(vec![
1969+
Field::new("id", DataType::Int64, true),
1970+
Field::new(
1971+
"s",
1972+
DataType::Struct(vec![Field::new("value", DataType::Int64, true)].into()),
1973+
true,
1974+
),
1975+
]));
1976+
let config = FileScanConfigBuilder::new(
1977+
ObjectStoreUrl::parse("test:///").unwrap(),
1978+
Arc::new(ParquetSource::new(Arc::clone(&schema))),
1979+
)
1980+
.with_file(PartitionedFile::new("x".to_string(), 100))
1981+
.build();
1982+
// Parquet absorbs a leaf projection, so the split would commit here if the
1983+
// partitioning check did not stop it first.
1984+
let hash = Arc::new(RepartitionExec::try_new(
1985+
DataSourceExec::from_data_source(config),
1986+
Partitioning::Hash(vec![col("id", &schema)?], 4),
1987+
)?);
1988+
1989+
let leaf: Arc<dyn PhysicalExpr> = Arc::new(ScalarFunctionExpr::new(
1990+
"get_field",
1991+
datafusion_functions::core::get_field(),
1992+
vec![
1993+
col("s", &schema)?,
1994+
Arc::new(Literal::new(ScalarValue::from("value"))),
1995+
],
1996+
Arc::new(Field::new("v", DataType::Int64, true)),
1997+
Arc::new(ConfigOptions::default()),
1998+
));
1999+
let projection: Arc<dyn ExecutionPlan> = Arc::new(ProjectionExec::try_new(
2000+
vec![
2001+
ProjectionExpr {
2002+
expr: leaf,
2003+
alias: "v".to_string(),
2004+
},
2005+
ProjectionExpr {
2006+
expr: binary(
2007+
col("id", &schema)?,
2008+
Operator::Plus,
2009+
Arc::new(Literal::new(ScalarValue::Int64(Some(1)))),
2010+
&schema,
2011+
)?,
2012+
alias: "w".to_string(),
2013+
},
2014+
],
2015+
hash,
2016+
)?);
2017+
2018+
let optimized = ProjectionPushdown::new()
2019+
.optimize(Arc::clone(&projection), &ConfigOptions::default())?;
2020+
let actual = displayable(optimized.as_ref())
2021+
.indent(true)
2022+
.to_string()
2023+
.trim()
2024+
.to_string();
2025+
2026+
// Unchanged. Without the check the scan absorbs the leaf and the hash keys
2027+
// are left indexing the old schema.
2028+
assert_snapshot!(
2029+
actual,
2030+
@r"
2031+
ProjectionExec: expr=[get_field(s@1, value) as v, id@0 + 1 as w]
2032+
RepartitionExec: partitioning=Hash([id@0], 4), input_partitions=1
2033+
DataSourceExec: file_groups={1 group: [[x]]}, projection=[id, s], file_type=parquet
2034+
"
2035+
);
2036+
Ok(())
2037+
}

‎datafusion/physical-optimizer/src/projection_pushdown.rs‎

Lines changed: 15 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -67,18 +67,19 @@ impl PhysicalOptimizerRule for ProjectionPushdown {
6767
.map(|t| t.data)?;
6868

6969
// Push what can already descend, then split the projections that are
70-
// stuck above a `RepartitionExec`, then clean up once more: the split
71-
// can leave an outer projection that is now only column references.
70+
// stuck above a `RepartitionExec`. The split needs no cleanup pass of
71+
// its own: the outer projection it leaves either keeps every expression
72+
// a plain column, and is then no narrower than the widened child, or it
73+
// narrows and holds a computed one. `RepartitionExec` declines both, it
74+
// is not removable because it renames, and nothing else moved.
7275
let plan = plan.transform_down(remove_unnecessary_projections).data()?;
73-
let plan = plan
74-
.transform_down(|plan| {
75-
Ok(match try_split_leaf_projection(&plan)? {
76-
Some(new_plan) => Transformed::yes(new_plan),
77-
None => Transformed::no(plan),
78-
})
76+
plan.transform_down(|plan| {
77+
Ok(match try_split_leaf_projection(&plan)? {
78+
Some(new_plan) => Transformed::yes(new_plan),
79+
None => Transformed::no(plan),
7980
})
80-
.data()?;
81-
plan.transform_down(remove_unnecessary_projections).data()
81+
})
82+
.data()
8283
}
8384

8485
fn name(&self) -> &str {
@@ -585,8 +586,10 @@ fn try_split_leaf_projection(
585586
/// Collects the distinct outermost [`ExpressionPlacement::MoveTowardsLeafNodes`]
586587
/// sub-expressions of `exprs`.
587588
///
588-
/// Nested leaf expressions are not collected separately: `s['a']['b']` is one
589-
/// unit to hand to the source, not two.
589+
/// Recursion stops at a leaf, so one nested inside another is not collected too:
590+
/// that would add a column no expression reads, and read more of the struct than
591+
/// the outer leaf needs. Plain SQL does not reach it, because the logical
592+
/// optimizer folds nested field access into a single call.
590593
fn collect_leaf_exprs(exprs: &[ProjectionExpr]) -> Result<Vec<Arc<dyn PhysicalExpr>>> {
591594
let mut leaves: Vec<Arc<dyn PhysicalExpr>> = vec![];
592595
for proj_expr in exprs {

‎datafusion/sqllogictest/test_files/file_row_index.slt‎

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -240,6 +240,29 @@ FROM parquet_table ORDER BY column1
240240
3 4 40
241241
4 5 50
242242

243+
# The single column is what the values above cannot show: both outputs would
244+
# also agree if the marker were read twice.
245+
query TT
246+
EXPLAIN SELECT file_row_index(), file_row_index() + 1, column1 FROM parquet_table
247+
----
248+
logical_plan
249+
01)Projection: file_row_index(), file_row_index() + Int64(1), parquet_table.column1
250+
02)--TableScan: parquet_table projection=[column1]
251+
physical_plan
252+
01)ProjectionExec: expr=[__datafusion_leaf_0@1 as file_row_index(), __datafusion_leaf_0@1 + 1 as file_row_index() + Int64(1), column1@0 as column1]
253+
02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
254+
03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/file_row_index/parquet_table/data.parquet]]}, projection=[column1, CAST(__datafusion_file_row_index@1 AS Int64;N) as __datafusion_leaf_0], file_type=parquet
255+
256+
# `ORDER BY ... LIMIT` is a TopK that sorts on a column the split produced,
257+
# with `preserve_partitioning=[true]`, so it runs above the repartition on the
258+
# rewritten schema. No dynamic filter reaches the scan here: that needs the
259+
# sort key to be the computed sibling rather than the rewritten marker.
260+
query II
261+
SELECT file_row_index() AS r, column1 + 1 FROM parquet_table ORDER BY r DESC LIMIT 2
262+
----
263+
4 51
264+
3 41
265+
243266
# Shapes that already worked must keep working: the leaf rules extract from
244267
# `Filter`, `Sort` and `Aggregate`, so those never depended on this fix.
245268
query I
@@ -273,7 +296,8 @@ SELECT file_row_index(), column1 FROM parquet_table ORDER BY column1
273296
4 50
274297

275298
# Raising `batch_size` above the row count removes the repartition. This must
276-
# keep working, and it is also the control that proves trap 1 above is real.
299+
# keep working, and it is also the control showing that `batch_size` is what
300+
# triggers the repartition.
277301
statement ok
278302
SET datafusion.execution.batch_size = 8192;
279303

‎datafusion/sqllogictest/test_files/input_file_name.slt‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,27 @@ physical_plan
6464
query error Execution error: input_file_name\(\) is source dependent and cannot be evaluated directly
6565
SELECT input_file_name() FROM (VALUES (1)) v(x);
6666

67+
# CSV has its own scan implementation, so the leaf split has to reach it too.
68+
# Needs a `KeepInPlace` sibling and a repartition, hence `column1 + 1` and the
69+
# `batch_size` below the row count.
70+
statement ok
71+
SET datafusion.execution.batch_size = 2;
72+
73+
query BI
74+
SELECT length(input_file_name()) > 0, column1 + 1
75+
FROM csv_table
76+
ORDER BY 2
77+
----
78+
true 11
79+
true 21
80+
true 31
81+
true 41
82+
true 51
83+
true 61
84+
85+
statement ok
86+
SET datafusion.execution.batch_size = 8192;
87+
6788
statement ok
6889
DROP TABLE csv_table;
6990

‎datafusion/sqllogictest/test_files/projection_pushdown.slt‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2825,9 +2825,45 @@ FROM leaf_split ORDER BY v;
28252825
[h!, i!] 250 epsilon 5
28262826
[g!] 300 delta 4
28272827

2828+
# A leaf expression inside the lambda body, rather than beside it: the split has
2829+
# to find it through `LambdaExpr` before the parameter rebase even applies.
2830+
query ?IT
2831+
SELECT array_transform(arr, x -> x + file_row_index()) AS a, s['value'] AS v, s['label'] AS l
2832+
FROM leaf_split ORDER BY v;
2833+
----
2834+
[1, 2, 3] 100 alpha
2835+
[8] 150 gamma
2836+
[5, 6] 200 beta
2837+
[14] 250 epsilon
2838+
[10, 11, 12] 300 delta
2839+
28282840
statement ok
28292841
SET datafusion.sql_parser.dialect = generic;
28302842

2843+
# A source that does not absorb the projection keeps the plan unchanged: the
2844+
# split must not add a projection nobody can use. A memory table refuses
2845+
# because `all_alias_free_columns` rejects the aliased `get_field`, and that is
2846+
# the default for any node that does not override `try_swapping_with_projection`.
2847+
statement ok
2848+
CREATE TABLE leaf_split_mem AS
2849+
SELECT column1 AS id, column2 AS s FROM VALUES
2850+
(1, {value: 100}), (2, {value: 200}), (3, {value: 150}),
2851+
(4, {value: 300}), (5, {value: 250});
2852+
2853+
query TT
2854+
EXPLAIN SELECT s['value'] AS v, id + 1 AS w FROM leaf_split_mem;
2855+
----
2856+
logical_plan
2857+
01)Projection: get_field(leaf_split_mem.s, Utf8("value")) AS v, leaf_split_mem.id + Int64(1) AS w
2858+
02)--TableScan: leaf_split_mem projection=[id, s]
2859+
physical_plan
2860+
01)ProjectionExec: expr=[get_field(s@1, value) as v, id@0 + 1 as w]
2861+
02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
2862+
03)----DataSourceExec: partitions=1, partition_sizes=[3]
2863+
2864+
statement ok
2865+
DROP TABLE leaf_split_mem;
2866+
28312867
statement ok
28322868
SET datafusion.execution.batch_size = 8192;
28332869

0 commit comments

Comments
 (0)