Skip to content

Commit 233aeab

Browse files
adriangbclaude
andauthored
fix: keep renamed columns intact in leaf projection pushdown (apache#25478)
## Which issue does this PR close? - Closes apache#25446. Part of the leaf-pushdown EPIC: apache#25459 ## Rationale for this change A valid statement fails to plan. A sub-query projection renames a column to the name of a different column of the same input, and the query reads a struct field above it. `datafusion.optimizer.enable_leaf_expression_pushdown` is `true` by default, so this is a plain planning failure. ```sql create table t(a int, b int, s struct<x varchar>) as values (1, 10, {x: 'p'}), (2, 20, {x: 'q'}); -- 1. a rename beside a same-name alias, under a filter select b, a, s['x'] from (select t.a as b, t.b as a, s from t) where a > 0; -- 2. the same shape under a limit select b, a, s['x'] from (select t.a as b, t.b as a, s from t) limit 10; -- 3. a swap of two column names select a, c, s['x'] from (select t.b as a, t.a as c, s from t) where a > 0; ``` Observed on `main`, for all three: ```text Optimizer rule 'push_down_leaf_projections' failed caused by Schema error: Schema contains qualified field name t.a and unqualified field name a which would be ambiguous ``` Expected, and what you get with `set datafusion.optimizer.enable_leaf_expression_pushdown = false;`: ```text +---+----+--------+ +----+---+--------+ | b | a | t.s[x] | | a | c | t.s[x] | +---+----+--------+ +----+---+--------+ | 1 | 10 | p | | 10 | 1 | p | | 2 | 20 | q | | 20 | 2 | q | +---+----+--------+ +----+---+--------+ statements 1 and 2 statement 3 ``` `order by` and `group by` above the rename fail the same way. Query generators emit this shape. ## What changes are included in this PR? Both changes are in `datafusion/optimizer/src/extract_leaf_expressions.rs`. **1. The merge of the extraction projection.** `build_extraction_projection_impl` merges the extraction projection into the projection below it. For every column the parent needs, it resolved the name through the projection's rename map and then added the *input* column that the name resolves to. For `select t.a as b, t.b as a, s from t` the parent's `b` resolves to `t.a`, and `t.a` lands beside the output field `a` of the other rename. `Projection::try_new` rejects that schema. The merge keeps every expression of the projection below it, so the parent can still read each name that projection produces. Those names are now skipped instead of resolved, and a pass-through column whose name is one of the projection's output names is never added. A rename is a computed output, and the parent refers to it by its output name only. **2. The recovery check.** With the merge fixed, the shape reaches `split_and_push_projection`, which decided if it needs a recovery projection from the set of unqualified field names alone. Equal name sets do not prove equal values: the pushed plan exposes the table column `t.a` where the projection computed `t.a AS b`. Without this hunk the statements above return the columns swapped instead of an error. The recovery projection now also stays when a recovery expression is not a pass-through of a column. That second hunk is the same change as apache#25445, down to the text, so the two merge without a conflict. If that PR lands first, the hunk falls out of the rebase. **How this composes with the other open PRs on this function.** apache#25412 resolves both sides of the pass-through comparison against the input schema, and apache#25456 replaces the rename map with `ProjectionInliner`. This change sits above both: it decides which columns to add before either one resolves a name. I merged this branch into a local branch that holds both PRs. The single conflict is the loop header, which takes the guards of this PR around the body of the other two. The optimizer tests, the `struct`, `cse` and `push_down_filter` sqllogictest files, and the differential fuzz all pass there with the swap-alias shape of the fuzz generator turned on: 250 short cases, 125 short Parquet cases, 5000 extended cases and 1000 extended Parquet cases, 0 failures and 0 skips. ## What is the testing strategy for this PR? - `datafusion/sqllogictest/test_files/struct.slt`: the three statements above, plus the `order by` and `group by` variants, plus a swap where the struct field name is also a column name of the table, plus one `EXPLAIN` that shows the renames above the extraction projection and the struct field still read at the scan. - `datafusion/optimizer/src/extract_leaf_expressions.rs`: `test_extract_above_projection_that_swaps_column_names` covers the ambiguous schema. `test_extract_above_projection_that_redefines_column_name` covers the recovery projection that must keep a rename alive. - Three plan snapshots of neighbouring tests lose a duplicate pass-through column in the intermediate stage. The optimized plans of those three tests do not change. Commands and results: ```text cargo test --profile ci -p datafusion-optimizer 884, 26, 5 passed; 0 failed cargo test --profile ci -p datafusion-sqllogictest --test sqllogictests 520/520 files completed, 0 failures cargo clippy --profile ci -p datafusion-optimizer --all-targets -- -D warnings clean ``` ## Are there any user-facing changes? Statements of this shape plan and run. There is no API change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.1 <[email protected]>
1 parent 6cab074 commit 233aeab

2 files changed

Lines changed: 178 additions & 15 deletions

File tree

‎datafusion/optimizer/src/extract_leaf_expressions.rs‎

Lines changed: 104 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -764,12 +764,32 @@ fn build_extraction_projection_impl(
764764
.filter_map(|e| resolve_against(input_schema, passthrough_column(e)?))
765765
.collect();
766766

767+
// The names the merged projection already produces. The merge keeps
768+
// every expression of `existing`, so the parent can still read each of
769+
// these names, and a pass-through column that carries one of them makes
770+
// the output schema ambiguous.
771+
let output_names: std::collections::HashSet<&str> = existing
772+
.schema
773+
.fields()
774+
.iter()
775+
.map(|f| f.name().as_str())
776+
.collect();
777+
767778
for col in columns_needed {
779+
// The projection produces this name, so the parent reads it from the
780+
// projection's own output. Do not resolve the name through the rename
781+
// map: a rename such as `t.a AS b` is a computed output, and its
782+
// input column `t.a` beside the output field `a` of a second rename
783+
// gives an ambiguous schema (issue #25446).
784+
if output_names.contains(col.name.as_str()) {
785+
continue;
786+
}
768787
let col_expr = Expr::Column(col.clone());
769788
let resolved = replace_cols_by_name(col_expr, &replace_map)?;
770789
if let Expr::Column(resolved_col) = &resolved
771790
&& let Some(input_col) = resolve_against(input_schema, resolved_col)
772791
&& !existing_cols.contains(&input_col)
792+
&& !output_names.contains(input_col.name.as_str())
773793
{
774794
proj_exprs.push(Expr::Column(input_col));
775795
}
@@ -2955,14 +2975,11 @@ mod tests {
29552975
## After Pushdown
29562976
Projection: __datafusion_extracted_1 AS leaf_udf(x,Utf8("a"))
29572977
Filter: x IS NOT NULL
2958-
Projection: test.user AS x, leaf_udf(test.user, Utf8("a")) AS __datafusion_extracted_1, test.user
2978+
Projection: test.user AS x, leaf_udf(test.user, Utf8("a")) AS __datafusion_extracted_1
29592979
TableScan: test projection=[user]
29602980
29612981
## Optimized
2962-
Projection: __datafusion_extracted_1 AS leaf_udf(x,Utf8("a"))
2963-
Filter: x IS NOT NULL
2964-
Projection: test.user AS x, leaf_udf(test.user, Utf8("a")) AS __datafusion_extracted_1
2965-
TableScan: test projection=[user]
2982+
(same as after pushdown)
29662983
"#)
29672984
}
29682985

@@ -2989,14 +3006,11 @@ mod tests {
29893006
## After Pushdown
29903007
Projection: __datafusion_extracted_1 IS NOT NULL AS leaf_udf(x,Utf8("a")) IS NOT NULL
29913008
Filter: x IS NOT NULL
2992-
Projection: test.user AS x, leaf_udf(test.user, Utf8("a")) AS __datafusion_extracted_1, test.user
3009+
Projection: test.user AS x, leaf_udf(test.user, Utf8("a")) AS __datafusion_extracted_1
29933010
TableScan: test projection=[user]
29943011
29953012
## Optimized
2996-
Projection: __datafusion_extracted_1 IS NOT NULL AS leaf_udf(x,Utf8("a")) IS NOT NULL
2997-
Filter: x IS NOT NULL
2998-
Projection: test.user AS x, leaf_udf(test.user, Utf8("a")) AS __datafusion_extracted_1
2999-
TableScan: test projection=[user]
3013+
(same as after pushdown)
30003014
"#)
30013015
}
30023016

@@ -3018,17 +3032,92 @@ mod tests {
30183032
## After Extraction
30193033
Projection: x
30203034
Filter: __datafusion_extracted_1 = Utf8("active")
3021-
Projection: test.user AS x, leaf_udf(test.user, Utf8("a")) AS __datafusion_extracted_1, test.user
3035+
Projection: test.user AS x, leaf_udf(test.user, Utf8("a")) AS __datafusion_extracted_1
30223036
TableScan: test projection=[user]
30233037
30243038
## After Pushdown
30253039
(same as after extraction)
30263040
30273041
## Optimized
3028-
Projection: x
3029-
Filter: __datafusion_extracted_1 = Utf8("active")
3030-
Projection: test.user AS x, leaf_udf(test.user, Utf8("a")) AS __datafusion_extracted_1
3031-
TableScan: test projection=[user]
3042+
(same as after pushdown)
3043+
"#)
3044+
}
3045+
3046+
/// A projection that swaps two column names, below a filter and a struct
3047+
/// field read. The merge must not resolve the parent's column references
3048+
/// through the rename map: the input column `test.user` beside the output
3049+
/// field `user` of the other rename gives an ambiguous schema (#25446).
3050+
#[test]
3051+
fn test_extract_above_projection_that_swaps_column_names() -> Result<()> {
3052+
let table_scan = test_table_scan_with_struct()?;
3053+
let plan = LogicalPlanBuilder::from(table_scan)
3054+
.project(vec![col("user").alias("id"), col("id").alias("user")])?
3055+
.filter(col("user").gt(lit(0u32)))?
3056+
.project(vec![col("id"), col("user"), leaf_udf(col("id"), "name")])?
3057+
.build()?;
3058+
3059+
assert_stages!(plan, @r#"
3060+
## Original Plan
3061+
Projection: id, user, leaf_udf(id, Utf8("name"))
3062+
Filter: user > UInt32(0)
3063+
Projection: test.user AS id, test.id AS user
3064+
TableScan: test projection=[id, user]
3065+
3066+
## After Extraction
3067+
(same as original)
3068+
3069+
## After Pushdown
3070+
Projection: id, user, __datafusion_extracted_1 AS leaf_udf(id,Utf8("name"))
3071+
Filter: user > UInt32(0)
3072+
Projection: test.user AS id, test.id AS user, leaf_udf(test.user, Utf8("name")) AS __datafusion_extracted_1
3073+
TableScan: test projection=[id, user]
3074+
3075+
## Optimized
3076+
(same as after pushdown)
3077+
"#)
3078+
}
3079+
3080+
/// A projection that gives a computed column the name of its own input
3081+
/// column. The extraction projection goes below the filter, which puts a
3082+
/// column named `id` under the rename, so the two plans have the same set
3083+
/// of field names. The recovery projection must stay, or the rename is lost
3084+
/// and the query gives wrong results.
3085+
#[test]
3086+
fn test_extract_above_projection_that_redefines_column_name() -> Result<()> {
3087+
let table_scan = test_table_scan_with_struct()?;
3088+
let plan = LogicalPlanBuilder::from(table_scan)
3089+
.filter(col("id").gt(lit(0u32)))?
3090+
.project(vec![(col("id") * lit(10u32)).alias("id"), col("user")])?
3091+
.alias("sub")?
3092+
.project(vec![col("sub.id"), leaf_udf(col("sub.user"), "name")])?
3093+
.build()?;
3094+
3095+
assert_stages!(plan, @r#"
3096+
## Original Plan
3097+
Projection: sub.id, leaf_udf(sub.user, Utf8("name"))
3098+
SubqueryAlias: sub
3099+
Projection: test.id * UInt32(10) AS id, test.user
3100+
Filter: test.id > UInt32(0)
3101+
TableScan: test projection=[id, user]
3102+
3103+
## After Extraction
3104+
(same as original)
3105+
3106+
## After Pushdown
3107+
Projection: sub.id, __datafusion_extracted_1 AS leaf_udf(sub.user,Utf8("name"))
3108+
SubqueryAlias: sub
3109+
Projection: test.id * UInt32(10) AS id, test.user, __datafusion_extracted_1
3110+
Filter: test.id > UInt32(0)
3111+
Projection: leaf_udf(test.user, Utf8("name")) AS __datafusion_extracted_1, test.id, test.user
3112+
TableScan: test projection=[id, user]
3113+
3114+
## Optimized
3115+
Projection: sub.id, __datafusion_extracted_1 AS leaf_udf(sub.user,Utf8("name"))
3116+
SubqueryAlias: sub
3117+
Projection: test.id * UInt32(10) AS id, __datafusion_extracted_1
3118+
Filter: test.id > UInt32(0)
3119+
Projection: leaf_udf(test.user, Utf8("name")) AS __datafusion_extracted_1, test.id
3120+
TableScan: test projection=[id, user]
30323121
"#)
30333122
}
30343123

‎datafusion/sqllogictest/test_files/struct.slt‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1907,6 +1907,80 @@ from (values (1),(2),(1)) group by 1;
19071907
{a: 1, z: NULL} 2
19081908
{a: 2, z: NULL} 1
19091909

1910+
# A sub-query projection that renames a column to the name of a different
1911+
# column of the same input, with a struct field read above it. Leaf projection
1912+
# pushdown resolved the parent's column references through the rename map and
1913+
# added the renamed input column a second time, which made the output schema
1914+
# ambiguous (https://github.com/apache/datafusion/issues/25446).
1915+
statement ok
1916+
create table rename_swap_struct(a int, b int, s struct<x varchar>) as values (1, 10, {x: 'p'}), (2, 20, {x: 'q'});
1917+
1918+
# a rename beside a same-name alias, under a filter
1919+
query IIT
1920+
select b, a, s['x'] from (select rename_swap_struct.a as b, rename_swap_struct.b as a, s from rename_swap_struct) where a > 15 order by b;
1921+
----
1922+
2 20 q
1923+
1924+
# the same shape under a limit
1925+
query IIT rowsort
1926+
select b, a, s['x'] from (select rename_swap_struct.a as b, rename_swap_struct.b as a, s from rename_swap_struct) limit 10;
1927+
----
1928+
1 10 p
1929+
2 20 q
1930+
1931+
# a swap of two column names
1932+
query IIT
1933+
select a, c, s['x'] from (select rename_swap_struct.b as a, rename_swap_struct.a as c, s from rename_swap_struct) where a > 0 order by a;
1934+
----
1935+
10 1 p
1936+
20 2 q
1937+
1938+
# the same shape under an order by
1939+
query IIT
1940+
select b, a, s['x'] from (select rename_swap_struct.a as b, rename_swap_struct.b as a, s from rename_swap_struct) order by b;
1941+
----
1942+
1 10 p
1943+
2 20 q
1944+
1945+
# the same shape under a group by
1946+
query III
1947+
select b, a, count(s['x']) from (select rename_swap_struct.a as b, rename_swap_struct.b as a, s from rename_swap_struct) group by b, a order by b;
1948+
----
1949+
1 10 1
1950+
2 20 1
1951+
1952+
# the rename stays above the extraction projection, and the struct field is
1953+
# still read at the scan
1954+
query TT
1955+
explain select b, a, s['x'] from (select rename_swap_struct.a as b, rename_swap_struct.b as a, s from rename_swap_struct) where a > 15;
1956+
----
1957+
logical_plan
1958+
01)Projection: rename_swap_struct.a AS b, rename_swap_struct.b AS a, __datafusion_extracted_1 AS rename_swap_struct.s[x]
1959+
02)--Filter: rename_swap_struct.b > Int32(15)
1960+
03)----Projection: get_field(rename_swap_struct.s, Utf8("x")) AS __datafusion_extracted_1, rename_swap_struct.a, rename_swap_struct.b
1961+
04)------TableScan: rename_swap_struct projection=[a, b, s]
1962+
physical_plan
1963+
01)ProjectionExec: expr=[a@1 as b, b@2 as a, __datafusion_extracted_1@0 as rename_swap_struct.s[x]]
1964+
02)--FilterExec: b@2 > 15
1965+
03)----ProjectionExec: expr=[get_field(s@2, x) as __datafusion_extracted_1, a@0 as a, b@1 as b]
1966+
04)------DataSourceExec: partitions=1, partition_sizes=[1]
1967+
1968+
statement ok
1969+
drop table rename_swap_struct;
1970+
1971+
# The same swap where the struct field name is also a column name of the table.
1972+
statement ok
1973+
create table rename_swap_struct_field(a int, b int, s struct<b varchar>) as values (1, 10, {b: 'p'}), (2, 20, {b: 'q'});
1974+
1975+
query IIT
1976+
select a, b, s['b'] from (select b as a, a as b, s from rename_swap_struct_field limit 100) order by a;
1977+
----
1978+
10 1 p
1979+
20 2 q
1980+
1981+
statement ok
1982+
drop table rename_swap_struct_field;
1983+
19101984
# Merging an extraction projection into a projection whose output lost its
19111985
# qualifier. Eliminating the empty side of the union leaves a projection of
19121986
# bare column names over a qualified input, and the merge used to add the

0 commit comments

Comments
 (0)