Skip to content

Commit 47135be

Browse files
adriangbclaude
andauthored
fix: resolve pass-through columns against the input when merging an extraction projection (apache#25412)
## Which issue does this PR close? None filed. ## Rationale for this change This query fails to plan on `main`: ```sql create table t(v int, s struct<a int>, env varchar) as values (1, {a: 10}, 'prod'), (2, {a: 20}, 'dev'); with samples as ( select v, s, env from t ), expanded as ( select v, s, env from samples union all select v, s, env from samples where 1 = 2 ) select env, sum(s['a']) from expanded group by env; ``` ``` Optimizer rule 'push_down_leaf_projections' failed caused by Schema error: Schema contains qualified field name samples.env and unqualified field name env which would be ambiguous ``` `enable_leaf_expression_pushdown` is on by default, so this is a plain planning failure for a valid statement. A `union all` whose one side is provably empty is what produces the shape, and query generators emit that: a dashboard panel adds a second branch behind a comparison of two constants and the panel stops working. ## What changes are included in this PR? `build_extraction_projection_impl` merges an extraction projection into the projection below it, and then adds the pass-through columns the merged projection does not already carry. It compared the columns it was about to add against the projection's own expressions without putting the two in the same space. A projection can list bare column names over a qualified input. Removing the empty side of the union leaves exactly that. The comparison then misses, the column is added a second time under its bare name, and `Projection::try_new` rejects a schema that holds `samples.env` and a bare `env` together. This PR resolves both sides against the input schema before the comparison, and pushes the column under the name the input gives it. A needed column that is not in the input, or that is ambiguous there, is skipped. Before this change such a column was added and `Projection::try_new` failed at the merge. A same-name alias such as `t.c AS c` counts as a pass-through on the existing side too, the same as a bare `t.c`. ## Are these changes tested? Yes. The statement above is added to `struct.slt`, with a result check and an `EXPLAIN` plan pin. It fails on `main` with the error above and passes with this change. Three unit tests in `extract_leaf_expressions.rs` cover the merge directly. All three fail on `main`: - `test_merge_bare_column_into_qualified_projection`: a filter names `a` over a projection that outputs `test.a`. - `test_merge_into_projection_with_same_name_alias`: the projection spells the pass-through as `test.a AS a`. - `test_merge_appends_qualified_column_for_bare_request`: the projection keeps only `test.a` and the merge needs a bare `b`. The appended column must be `test.b`. On `main` it is a bare `b`. `cargo test --workspace --exclude datafusion-sqllogictest` passes: 126 suites, 12479 tests, 0 failures. The sqllogictest suite passes too: 520 files, 0 failures. `cargo clippy -p datafusion-optimizer --all-targets` and `cargo fmt --all -- --check` are clean. ## Are there any user-facing changes? A statement of this shape plans instead of failing. No API change. ## Note on overlapping work apache#25388 (draft) edits the same arm of `build_extraction_projection_impl`, for a different problem: duplicated evaluation of `KeepInPlace` expressions, apache#25329. It does not touch the qualifier comparison this PR changes, so the two are independent in behaviour, but whichever lands second will need a small rebase at the tail of the `columns_needed` loop. apache#25282 also appends to the end of `struct.slt`, as this PR does. Keeping both blocks is the whole resolution. Part of the leaf-pushdown EPIC: apache#25459 --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 28cefb7 commit 47135be

2 files changed

Lines changed: 186 additions & 12 deletions

File tree

‎datafusion/optimizer/src/extract_leaf_expressions.rs‎

Lines changed: 135 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -672,6 +672,20 @@ impl<'a> LeafExpressionExtractor<'a> {
672672
}
673673
}
674674

675+
/// Resolves `col` against `schema` and returns it with the qualifier that
676+
/// `schema` gives the field. Returns `None` if `schema` does not hold `col`,
677+
/// or if the name is ambiguous in `schema`.
678+
///
679+
/// Two spellings of the same input column (a bare `c` and a qualified `t.c`)
680+
/// resolve to the same `Column`, so they compare equal. A column pushed into
681+
/// a projection this way uses the name that the input gives it.
682+
fn resolve_against(schema: &DFSchema, col: &Column) -> Option<Column> {
683+
schema
684+
.qualified_field_from_column(col)
685+
.ok()
686+
.map(Column::from)
687+
}
688+
675689
/// Build an extraction projection above the target node (shared by both passes).
676690
///
677691
/// If the target is an existing projection, merges into it. This requires
@@ -737,29 +751,31 @@ fn build_extraction_projection_impl(
737751
// than target_schema (the projection's output) because columns produced
738752
// by alias expressions (e.g., CSE's __common_expr_N) exist in the output but
739753
// not the input, and cannot be added as pass-through Column references.
754+
//
755+
// Resolve the pass-through columns of `existing` (including same-name
756+
// aliases such as `t.c AS c`) and the columns in `columns_needed`
757+
// against the input schema before comparing them. Otherwise a bare `c`
758+
// and a qualified `t.c` do not match, and the merged projection holds
759+
// both, which `Projection::try_new` rejects as ambiguous.
760+
let input_schema = existing.input.schema();
740761
let existing_cols: IndexSet<Column> = existing
741762
.expr
742763
.iter()
743-
.filter_map(|e| {
744-
if let Expr::Column(c) = e {
745-
Some(c.clone())
746-
} else {
747-
None
748-
}
749-
})
764+
.filter_map(|e| resolve_against(input_schema, passthrough_column(e)?))
750765
.collect();
751766

752-
let input_schema = existing.input.schema();
753767
for col in columns_needed {
754768
let col_expr = Expr::Column(col.clone());
755769
let resolved = replace_cols_by_name(col_expr, &replace_map)?;
756770
if let Expr::Column(resolved_col) = &resolved
757-
&& !existing_cols.contains(resolved_col)
758-
&& input_schema.has_column(resolved_col)
771+
&& let Some(input_col) = resolve_against(input_schema, resolved_col)
772+
&& !existing_cols.contains(&input_col)
759773
{
760-
proj_exprs.push(Expr::Column(resolved_col.clone()));
774+
proj_exprs.push(Expr::Column(input_col));
761775
}
762-
// If resolved to non-column expr, it's already computed by existing projection
776+
// Skip a column that resolves to a non-column expr: the existing
777+
// projection already computes it. Also skip a column that is not in
778+
// the input or is ambiguous there: no single spelling is correct.
763779
}
764780

765781
Projection::try_new(proj_exprs, Arc::clone(&existing.input))
@@ -2405,6 +2421,113 @@ mod tests {
24052421
"#)
24062422
}
24072423

2424+
/// A filter can name a column bare (`a`) while the projection below it
2425+
/// outputs the qualified `test.a`. The optimizer leaves this shape when it
2426+
/// removes the empty input of a `UNION ALL` (for example
2427+
/// `... UNION ALL SELECT ... WHERE 1 = 2`). The merge must match the two, and not add a bare `a`
2428+
/// beside `test.a`, which is an ambiguous schema.
2429+
#[test]
2430+
fn test_merge_bare_column_into_qualified_projection() -> Result<()> {
2431+
let table_scan = test_table_scan()?;
2432+
let projection = LogicalPlanBuilder::from(table_scan)
2433+
.project(vec![
2434+
col("test.a"),
2435+
col("test.b"),
2436+
(col("test.c") + lit(1)).alias("d"),
2437+
])?
2438+
.build()?;
2439+
let predicate =
2440+
leaf_udf(Expr::Column(Column::new_unqualified("a")), "x").eq(lit(1));
2441+
let plan = LogicalPlan::Filter(datafusion_expr::Filter::try_new(
2442+
predicate,
2443+
Arc::new(projection),
2444+
)?);
2445+
2446+
assert_stages!(plan, @r#"
2447+
## Original Plan
2448+
Filter: leaf_udf(a, Utf8("x")) = Int32(1)
2449+
Projection: test.a, test.b, test.c + Int32(1) AS d
2450+
TableScan: test projection=[a, b, c]
2451+
2452+
## After Extraction
2453+
Projection: test.a, test.b, d
2454+
Filter: __datafusion_extracted_1 = Int32(1)
2455+
Projection: test.a, test.b, test.c + Int32(1) AS d, leaf_udf(a, Utf8("x")) AS __datafusion_extracted_1
2456+
TableScan: test projection=[a, b, c]
2457+
2458+
## After Pushdown
2459+
(same as after extraction)
2460+
2461+
## Optimized
2462+
(same as after pushdown)
2463+
"#)
2464+
}
2465+
2466+
/// When the merge appends a needed column that the projection does not
2467+
/// keep, it must append the input's qualified column (`test.b`), not the
2468+
/// bare `b` that the caller asked for.
2469+
#[test]
2470+
fn test_merge_appends_qualified_column_for_bare_request() -> Result<()> {
2471+
let projection = LogicalPlanBuilder::from(test_table_scan()?)
2472+
.project(vec![col("test.a")])?
2473+
.build()?;
2474+
let target_schema = Arc::clone(projection.schema());
2475+
let columns_needed: IndexSet<Column> =
2476+
IndexSet::from([Column::new_unqualified("b")]);
2477+
2478+
let merged = build_extraction_projection_impl(
2479+
&[],
2480+
&columns_needed,
2481+
&Arc::new(projection),
2482+
&target_schema,
2483+
)?;
2484+
2485+
assert_eq!(
2486+
merged.expr,
2487+
vec![
2488+
Expr::Column(Column::new(Some("test"), "a")),
2489+
Expr::Column(Column::new(Some("test"), "b")),
2490+
]
2491+
);
2492+
Ok(())
2493+
}
2494+
2495+
/// A projection can spell a pass-through column as a same-name alias
2496+
/// (`test.a AS a`). Merging an extraction into it must treat that alias as
2497+
/// the pass-through it is, and not add `test.a` beside the `a` it outputs,
2498+
/// which is an ambiguous schema.
2499+
#[test]
2500+
fn test_merge_into_projection_with_same_name_alias() -> Result<()> {
2501+
let table_scan = test_table_scan()?;
2502+
let plan = LogicalPlanBuilder::from(table_scan)
2503+
.project(vec![
2504+
col("test.a").alias("a"),
2505+
col("test.b").alias("b"),
2506+
col("test.c").alias("c"),
2507+
])?
2508+
.filter(leaf_udf(col("a"), "x").eq(lit(1)))?
2509+
.build()?;
2510+
2511+
assert_stages!(plan, @r#"
2512+
## Original Plan
2513+
Filter: leaf_udf(a, Utf8("x")) = Int32(1)
2514+
Projection: test.a AS a, test.b AS b, test.c AS c
2515+
TableScan: test projection=[a, b, c]
2516+
2517+
## After Extraction
2518+
Projection: a, b, c
2519+
Filter: __datafusion_extracted_1 = Int32(1)
2520+
Projection: test.a AS a, test.b AS b, test.c AS c, leaf_udf(test.a, Utf8("x")) AS __datafusion_extracted_1
2521+
TableScan: test projection=[a, b, c]
2522+
2523+
## After Pushdown
2524+
(same as after extraction)
2525+
2526+
## Optimized
2527+
(same as after pushdown)
2528+
"#)
2529+
}
2530+
24082531
// =========================================================================
24092532
// Join extraction tests
24102533
// =========================================================================

‎datafusion/sqllogictest/test_files/struct.slt‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1906,3 +1906,54 @@ from (values (1),(2),(1)) group by 1;
19061906
----
19071907
{a: 1, z: NULL} 2
19081908
{a: 2, z: NULL} 1
1909+
1910+
# Merging an extraction projection into a projection whose output lost its
1911+
# qualifier. Eliminating the empty side of the union leaves a projection of
1912+
# bare column names over a qualified input, and the merge used to add the
1913+
# pass-through columns under those bare names beside the qualified ones the
1914+
# projection already carried, which is an ambiguous schema.
1915+
statement ok
1916+
create table leaf_merge_source(v int, s struct<a int>, env varchar) as values (1, {a: 10}, 'prod'), (2, {a: 20}, 'dev');
1917+
1918+
statement ok
1919+
set datafusion.explain.logical_plan_only = true;
1920+
1921+
query TT
1922+
explain with samples as (
1923+
select v, s, env from leaf_merge_source
1924+
),
1925+
expanded as (
1926+
select v, s, env from samples
1927+
union all
1928+
select v, s, env from samples where 1 = 2
1929+
)
1930+
select env, sum(s['a']) from expanded group by env order by env;
1931+
----
1932+
logical_plan
1933+
01)Sort: expanded.env ASC NULLS LAST
1934+
02)--Projection: expanded.env, sum(__datafusion_extracted_1) AS sum(expanded.s[a])
1935+
03)----Aggregate: groupBy=[[expanded.env]], aggr=[[sum(CAST(__datafusion_extracted_1 AS Int64))]]
1936+
04)------SubqueryAlias: expanded
1937+
05)--------SubqueryAlias: samples
1938+
06)----------Projection: leaf_merge_source.env, get_field(s, Utf8("a")) AS __datafusion_extracted_1
1939+
07)------------TableScan: leaf_merge_source projection=[s, env]
1940+
1941+
statement ok
1942+
set datafusion.explain.logical_plan_only = false;
1943+
1944+
query TI
1945+
with samples as (
1946+
select v, s, env from leaf_merge_source
1947+
),
1948+
expanded as (
1949+
select v, s, env from samples
1950+
union all
1951+
select v, s, env from samples where 1 = 2
1952+
)
1953+
select env, sum(s['a']) from expanded group by env order by env;
1954+
----
1955+
dev 20
1956+
prod 10
1957+
1958+
statement ok
1959+
drop table leaf_merge_source;

0 commit comments

Comments
 (0)