Skip to content

Commit 02da2e9

Browse files
adriangbcetra3
authored andcommitted
fix: keep renamed columns intact in leaf projection pushdown (apache#25478)
## Which issue does this PR close? - Closes apache#25446. Part of the leaf-pushdown EPIC: apache#25459 ## Rationale for this change A valid statement fails to plan. A sub-query projection renames a column to the name of a different column of the same input, and the query reads a struct field above it. `datafusion.optimizer.enable_leaf_expression_pushdown` is `true` by default, so this is a plain planning failure. ```sql create table t(a int, b int, s struct<x varchar>) as values (1, 10, {x: 'p'}), (2, 20, {x: 'q'}); -- 1. a rename beside a same-name alias, under a filter select b, a, s['x'] from (select t.a as b, t.b as a, s from t) where a > 0; -- 2. the same shape under a limit select b, a, s['x'] from (select t.a as b, t.b as a, s from t) limit 10; -- 3. a swap of two column names select a, c, s['x'] from (select t.b as a, t.a as c, s from t) where a > 0; ``` Observed on `main`, for all three: ```text Optimizer rule 'push_down_leaf_projections' failed caused by Schema error: Schema contains qualified field name t.a and unqualified field name a which would be ambiguous ``` Expected, and what you get with `set datafusion.optimizer.enable_leaf_expression_pushdown = false;`: ```text +---+----+--------+ +----+---+--------+ | b | a | t.s[x] | | a | c | t.s[x] | +---+----+--------+ +----+---+--------+ | 1 | 10 | p | | 10 | 1 | p | | 2 | 20 | q | | 20 | 2 | q | +---+----+--------+ +----+---+--------+ statements 1 and 2 statement 3 ``` `order by` and `group by` above the rename fail the same way. Query generators emit this shape. ## What changes are included in this PR? Both changes are in `datafusion/optimizer/src/extract_leaf_expressions.rs`. **1. The merge of the extraction projection.** `build_extraction_projection_impl` merges the extraction projection into the projection below it. For every column the parent needs, it resolved the name through the projection's rename map and then added the *input* column that the name resolves to. For `select t.a as b, t.b as a, s from t` the parent's `b` resolves to `t.a`, and `t.a` lands beside the output field `a` of the other rename. `Projection::try_new` rejects that schema. The merge keeps every expression of the projection below it, so the parent can still read each name that projection produces. Those names are now skipped instead of resolved, and a pass-through column whose name is one of the projection's output names is never added. A rename is a computed output, and the parent refers to it by its output name only. **2. The recovery check.** With the merge fixed, the shape reaches `split_and_push_projection`, which decided if it needs a recovery projection from the set of unqualified field names alone. Equal name sets do not prove equal values: the pushed plan exposes the table column `t.a` where the projection computed `t.a AS b`. Without this hunk the statements above return the columns swapped instead of an error. The recovery projection now also stays when a recovery expression is not a pass-through of a column. That second hunk is the same change as apache#25445, down to the text, so the two merge without a conflict. If that PR lands first, the hunk falls out of the rebase. **How this composes with the other open PRs on this function.** apache#25412 resolves both sides of the pass-through comparison against the input schema, and apache#25456 replaces the rename map with `ProjectionInliner`. This change sits above both: it decides which columns to add before either one resolves a name. I merged this branch into a local branch that holds both PRs. The single conflict is the loop header, which takes the guards of this PR around the body of the other two. The optimizer tests, the `struct`, `cse` and `push_down_filter` sqllogictest files, and the differential fuzz all pass there with the swap-alias shape of the fuzz generator turned on: 250 short cases, 125 short Parquet cases, 5000 extended cases and 1000 extended Parquet cases, 0 failures and 0 skips. ## What is the testing strategy for this PR? - `datafusion/sqllogictest/test_files/struct.slt`: the three statements above, plus the `order by` and `group by` variants, plus a swap where the struct field name is also a column name of the table, plus one `EXPLAIN` that shows the renames above the extraction projection and the struct field still read at the scan. - `datafusion/optimizer/src/extract_leaf_expressions.rs`: `test_extract_above_projection_that_swaps_column_names` covers the ambiguous schema. `test_extract_above_projection_that_redefines_column_name` covers the recovery projection that must keep a rename alive. - Three plan snapshots of neighbouring tests lose a duplicate pass-through column in the intermediate stage. The optimized plans of those three tests do not change. Commands and results: ```text cargo test --profile ci -p datafusion-optimizer 884, 26, 5 passed; 0 failed cargo test --profile ci -p datafusion-sqllogictest --test sqllogictests 520/520 files completed, 0 failures cargo clippy --profile ci -p datafusion-optimizer --all-targets -- -D warnings clean ``` ## Are there any user-facing changes? Statements of this shape plan and run. There is no API change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.1 <[email protected]> Backport of apache#25478 (merged upstream as 233aeab).
1 parent 86f1f26 commit 02da2e9

2 files changed

Lines changed: 178 additions & 15 deletions

File tree

‎datafusion/optimizer/src/extract_leaf_expressions.rs‎

Lines changed: 104 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -764,12 +764,32 @@ fn build_extraction_projection_impl(
764764
.filter_map(|e| resolve_against(input_schema, passthrough_column(e)?))
765765
.collect();
766766

767+
// The names the merged projection already produces. The merge keeps
768+
// every expression of `existing`, so the parent can still read each of
769+
// these names, and a pass-through column that carries one of them makes
770+
// the output schema ambiguous.
771+
let output_names: std::collections::HashSet<&str> = existing
772+
.schema
773+
.fields()
774+
.iter()
775+
.map(|f| f.name().as_str())
776+
.collect();
777+
767778
for col in columns_needed {
779+
// The projection produces this name, so the parent reads it from the
780+
// projection's own output. Do not resolve the name through the rename
781+
// map: a rename such as `t.a AS b` is a computed output, and its
782+
// input column `t.a` beside the output field `a` of a second rename
783+
// gives an ambiguous schema (issue #25446).
784+
if output_names.contains(col.name.as_str()) {
785+
continue;
786+
}
768787
let col_expr = Expr::Column(col.clone());
769788
let resolved = replace_cols_by_name(col_expr, &replace_map)?;
770789
if let Expr::Column(resolved_col) = &resolved
771790
&& let Some(input_col) = resolve_against(input_schema, resolved_col)
772791
&& !existing_cols.contains(&input_col)
792+
&& !output_names.contains(input_col.name.as_str())
773793
{
774794
proj_exprs.push(Expr::Column(input_col));
775795
}
@@ -2981,14 +3001,11 @@ mod tests {
29813001
## After Pushdown
29823002
Projection: __datafusion_extracted_1 AS leaf_udf(x,Utf8("a"))
29833003
Filter: x IS NOT NULL
2984-
Projection: test.user AS x, leaf_udf(test.user, Utf8("a")) AS __datafusion_extracted_1, test.user
3004+
Projection: test.user AS x, leaf_udf(test.user, Utf8("a")) AS __datafusion_extracted_1
29853005
TableScan: test projection=[user]
29863006
29873007
## Optimized
2988-
Projection: __datafusion_extracted_1 AS leaf_udf(x,Utf8("a"))
2989-
Filter: x IS NOT NULL
2990-
Projection: test.user AS x, leaf_udf(test.user, Utf8("a")) AS __datafusion_extracted_1
2991-
TableScan: test projection=[user]
3008+
(same as after pushdown)
29923009
"#)
29933010
}
29943011

@@ -3015,14 +3032,11 @@ mod tests {
30153032
## After Pushdown
30163033
Projection: __datafusion_extracted_1 IS NOT NULL AS leaf_udf(x,Utf8("a")) IS NOT NULL
30173034
Filter: x IS NOT NULL
3018-
Projection: test.user AS x, leaf_udf(test.user, Utf8("a")) AS __datafusion_extracted_1, test.user
3035+
Projection: test.user AS x, leaf_udf(test.user, Utf8("a")) AS __datafusion_extracted_1
30193036
TableScan: test projection=[user]
30203037
30213038
## Optimized
3022-
Projection: __datafusion_extracted_1 IS NOT NULL AS leaf_udf(x,Utf8("a")) IS NOT NULL
3023-
Filter: x IS NOT NULL
3024-
Projection: test.user AS x, leaf_udf(test.user, Utf8("a")) AS __datafusion_extracted_1
3025-
TableScan: test projection=[user]
3039+
(same as after pushdown)
30263040
"#)
30273041
}
30283042

@@ -3044,17 +3058,92 @@ mod tests {
30443058
## After Extraction
30453059
Projection: x
30463060
Filter: __datafusion_extracted_1 = Utf8("active")
3047-
Projection: test.user AS x, leaf_udf(test.user, Utf8("a")) AS __datafusion_extracted_1, test.user
3061+
Projection: test.user AS x, leaf_udf(test.user, Utf8("a")) AS __datafusion_extracted_1
30483062
TableScan: test projection=[user]
30493063
30503064
## After Pushdown
30513065
(same as after extraction)
30523066
30533067
## Optimized
3054-
Projection: x
3055-
Filter: __datafusion_extracted_1 = Utf8("active")
3056-
Projection: test.user AS x, leaf_udf(test.user, Utf8("a")) AS __datafusion_extracted_1
3057-
TableScan: test projection=[user]
3068+
(same as after pushdown)
3069+
"#)
3070+
}
3071+
3072+
/// A projection that swaps two column names, below a filter and a struct
3073+
/// field read. The merge must not resolve the parent's column references
3074+
/// through the rename map: the input column `test.user` beside the output
3075+
/// field `user` of the other rename gives an ambiguous schema (#25446).
3076+
#[test]
3077+
fn test_extract_above_projection_that_swaps_column_names() -> Result<()> {
3078+
let table_scan = test_table_scan_with_struct()?;
3079+
let plan = LogicalPlanBuilder::from(table_scan)
3080+
.project(vec![col("user").alias("id"), col("id").alias("user")])?
3081+
.filter(col("user").gt(lit(0u32)))?
3082+
.project(vec![col("id"), col("user"), leaf_udf(col("id"), "name")])?
3083+
.build()?;
3084+
3085+
assert_stages!(plan, @r#"
3086+
## Original Plan
3087+
Projection: id, user, leaf_udf(id, Utf8("name"))
3088+
Filter: user > UInt32(0)
3089+
Projection: test.user AS id, test.id AS user
3090+
TableScan: test projection=[id, user]
3091+
3092+
## After Extraction
3093+
(same as original)
3094+
3095+
## After Pushdown
3096+
Projection: id, user, __datafusion_extracted_1 AS leaf_udf(id,Utf8("name"))
3097+
Filter: user > UInt32(0)
3098+
Projection: test.user AS id, test.id AS user, leaf_udf(test.user, Utf8("name")) AS __datafusion_extracted_1
3099+
TableScan: test projection=[id, user]
3100+
3101+
## Optimized
3102+
(same as after pushdown)
3103+
"#)
3104+
}
3105+
3106+
/// A projection that gives a computed column the name of its own input
3107+
/// column. The extraction projection goes below the filter, which puts a
3108+
/// column named `id` under the rename, so the two plans have the same set
3109+
/// of field names. The recovery projection must stay, or the rename is lost
3110+
/// and the query gives wrong results.
3111+
#[test]
3112+
fn test_extract_above_projection_that_redefines_column_name() -> Result<()> {
3113+
let table_scan = test_table_scan_with_struct()?;
3114+
let plan = LogicalPlanBuilder::from(table_scan)
3115+
.filter(col("id").gt(lit(0u32)))?
3116+
.project(vec![(col("id") * lit(10u32)).alias("id"), col("user")])?
3117+
.alias("sub")?
3118+
.project(vec![col("sub.id"), leaf_udf(col("sub.user"), "name")])?
3119+
.build()?;
3120+
3121+
assert_stages!(plan, @r#"
3122+
## Original Plan
3123+
Projection: sub.id, leaf_udf(sub.user, Utf8("name"))
3124+
SubqueryAlias: sub
3125+
Projection: test.id * UInt32(10) AS id, test.user
3126+
Filter: test.id > UInt32(0)
3127+
TableScan: test projection=[id, user]
3128+
3129+
## After Extraction
3130+
(same as original)
3131+
3132+
## After Pushdown
3133+
Projection: sub.id, __datafusion_extracted_1 AS leaf_udf(sub.user,Utf8("name"))
3134+
SubqueryAlias: sub
3135+
Projection: test.id * UInt32(10) AS id, test.user, __datafusion_extracted_1
3136+
Filter: test.id > UInt32(0)
3137+
Projection: leaf_udf(test.user, Utf8("name")) AS __datafusion_extracted_1, test.id, test.user
3138+
TableScan: test projection=[id, user]
3139+
3140+
## Optimized
3141+
Projection: sub.id, __datafusion_extracted_1 AS leaf_udf(sub.user,Utf8("name"))
3142+
SubqueryAlias: sub
3143+
Projection: test.id * UInt32(10) AS id, __datafusion_extracted_1
3144+
Filter: test.id > UInt32(0)
3145+
Projection: leaf_udf(test.user, Utf8("name")) AS __datafusion_extracted_1, test.id
3146+
TableScan: test projection=[id, user]
30583147
"#)
30593148
}
30603149

‎datafusion/sqllogictest/test_files/struct.slt‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1907,6 +1907,80 @@ from (values (1),(2),(1)) group by 1;
19071907
{a: 1, z: NULL} 2
19081908
{a: 2, z: NULL} 1
19091909

1910+
# A sub-query projection that renames a column to the name of a different
1911+
# column of the same input, with a struct field read above it. Leaf projection
1912+
# pushdown resolved the parent's column references through the rename map and
1913+
# added the renamed input column a second time, which made the output schema
1914+
# ambiguous (https://github.com/apache/datafusion/issues/25446).
1915+
statement ok
1916+
create table rename_swap_struct(a int, b int, s struct<x varchar>) as values (1, 10, {x: 'p'}), (2, 20, {x: 'q'});
1917+
1918+
# a rename beside a same-name alias, under a filter
1919+
query IIT
1920+
select b, a, s['x'] from (select rename_swap_struct.a as b, rename_swap_struct.b as a, s from rename_swap_struct) where a > 15 order by b;
1921+
----
1922+
2 20 q
1923+
1924+
# the same shape under a limit
1925+
query IIT rowsort
1926+
select b, a, s['x'] from (select rename_swap_struct.a as b, rename_swap_struct.b as a, s from rename_swap_struct) limit 10;
1927+
----
1928+
1 10 p
1929+
2 20 q
1930+
1931+
# a swap of two column names
1932+
query IIT
1933+
select a, c, s['x'] from (select rename_swap_struct.b as a, rename_swap_struct.a as c, s from rename_swap_struct) where a > 0 order by a;
1934+
----
1935+
10 1 p
1936+
20 2 q
1937+
1938+
# the same shape under an order by
1939+
query IIT
1940+
select b, a, s['x'] from (select rename_swap_struct.a as b, rename_swap_struct.b as a, s from rename_swap_struct) order by b;
1941+
----
1942+
1 10 p
1943+
2 20 q
1944+
1945+
# the same shape under a group by
1946+
query III
1947+
select b, a, count(s['x']) from (select rename_swap_struct.a as b, rename_swap_struct.b as a, s from rename_swap_struct) group by b, a order by b;
1948+
----
1949+
1 10 1
1950+
2 20 1
1951+
1952+
# the rename stays above the extraction projection, and the struct field is
1953+
# still read at the scan
1954+
query TT
1955+
explain select b, a, s['x'] from (select rename_swap_struct.a as b, rename_swap_struct.b as a, s from rename_swap_struct) where a > 15;
1956+
----
1957+
logical_plan
1958+
01)Projection: rename_swap_struct.a AS b, rename_swap_struct.b AS a, __datafusion_extracted_1 AS rename_swap_struct.s[x]
1959+
02)--Filter: rename_swap_struct.b > Int32(15)
1960+
03)----Projection: get_field(rename_swap_struct.s, Utf8("x")) AS __datafusion_extracted_1, rename_swap_struct.a, rename_swap_struct.b
1961+
04)------TableScan: rename_swap_struct projection=[a, b, s]
1962+
physical_plan
1963+
01)ProjectionExec: expr=[a@1 as b, b@2 as a, __datafusion_extracted_1@0 as rename_swap_struct.s[x]]
1964+
02)--FilterExec: b@2 > 15
1965+
03)----ProjectionExec: expr=[get_field(s@2, x) as __datafusion_extracted_1, a@0 as a, b@1 as b]
1966+
04)------DataSourceExec: partitions=1, partition_sizes=[1]
1967+
1968+
statement ok
1969+
drop table rename_swap_struct;
1970+
1971+
# The same swap where the struct field name is also a column name of the table.
1972+
statement ok
1973+
create table rename_swap_struct_field(a int, b int, s struct<b varchar>) as values (1, 10, {b: 'p'}), (2, 20, {b: 'q'});
1974+
1975+
query IIT
1976+
select a, b, s['b'] from (select b as a, a as b, s from rename_swap_struct_field limit 100) order by a;
1977+
----
1978+
10 1 p
1979+
20 2 q
1980+
1981+
statement ok
1982+
drop table rename_swap_struct_field;
1983+
19101984
# Merging an extraction projection into a projection whose output lost its
19111985
# qualifier. Eliminating the empty side of the union leaves a projection of
19121986
# bare column names over a qualified input, and the merge used to add the

0 commit comments

Comments
 (0)