Skip to content

Commit 80e9788

Browse files
moraistejerinaclaudegabotechs
authored
fix: map remaining union input by position in propagate_empty_relation (apache#26557)
## Which issue does this PR close? - Closes apache#26516. ## Rationale for this change A `UNION ALL` fails to plan with `Schema error: No field named ...` when the optimizer removes its first input and a later input remains. For example: ```sql SELECT 1 WHERE false UNION ALL (SELECT 2 LIMIT 1); ``` `propagate_empty_relation` puts a projection over the remaining input with `Projection::new_from_schema`. That function makes column references from the union's field names, which come from the first input. When the remaining input has other names, the projection refers to columns that do not exist, and `optimize_projections` fails. ## What changes are included in this PR? - When one input remains and its schema is different from the union's schema, map its columns to the union's output fields **by position**, alias each one to the union's name, and build the projection with `Projection::try_new` (which also checks the result). - Three existing snapshots change their text from `Projection: a, b, c` to `Projection: test.a AS a, test.b AS b, test.c AS c`. In those tests the remaining input is the first input, so the old by-name references worked. The output schema is the same. - This also changes the `EXPLAIN` output of the `struct.slt` test added in apache#25412. That PR made `push_down_leaf_projections` handle the bare-name projection that `propagate_empty_relation` left after it removed an empty union input. With this change, the projection aliases the qualified input columns (`leaf_merge_source.s AS s`). The test still plans and returns the same rows. apache#25412 also handles this same-name alias form, and its unit tests still build the bare-name shape directly. ## What is the testing strategy for this PR? - New unit test `propagate_union_remaining_input_with_other_names`: the first input is empty and the second input has other column names. Without the fix, the rule produces `Projection: a, b, c` over an input that only has `x, y, z`. - New `union.slt` cases with the two queries from the issue. ## Are there any user-facing changes? No. The queries that failed now return results. There are no public API changes. Co-authored-by: Jorge Morais <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Gabriel <[email protected]>
1 parent 209d852 commit 80e9788

3 files changed

Lines changed: 62 additions & 10 deletions

File tree

‎datafusion/optimizer/src/propagate_empty_relation.rs‎

Lines changed: 45 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -210,11 +210,21 @@ impl OptimizerRule for PropagateEmptyRelation {
210210
if child.schema().eq(plan.schema()) {
211211
Ok(Transformed::yes(child))
212212
} else {
213+
// The union takes its field names from its first
214+
// input, which may be gone. Map the remaining
215+
// input's columns by position.
216+
let exprs = child
217+
.schema()
218+
.columns()
219+
.into_iter()
220+
.zip(plan.schema().iter())
221+
.map(|(col, (qualifier, field))| {
222+
Expr::Column(col)
223+
.alias_qualified(qualifier.cloned(), field.name())
224+
})
225+
.collect::<Vec<_>>();
213226
Ok(Transformed::yes(LogicalPlan::Projection(
214-
Projection::new_from_schema(
215-
Arc::new(child),
216-
Arc::clone(plan.schema()),
217-
),
227+
Projection::try_new(exprs, Arc::new(child))?,
218228
)))
219229
}
220230
} else {
@@ -439,7 +449,8 @@ mod tests {
439449

440450
let plan = LogicalPlanBuilder::from(left).union(right)?.build()?;
441451

442-
let expected = "Projection: a, b, c\n TableScan: test";
452+
let expected =
453+
"Projection: test.a AS a, test.b AS b, test.c AS c\n TableScan: test";
443454
assert_together_optimized_plan(plan, expected, true)
444455
}
445456

@@ -529,7 +540,34 @@ mod tests {
529540

530541
let plan = LogicalPlanBuilder::from(left).union(right)?.build()?;
531542

532-
let expected = "Projection: a, b, c\n TableScan: test";
543+
let expected =
544+
"Projection: test.a AS a, test.b AS b, test.c AS c\n TableScan: test";
545+
assert_together_optimized_plan(plan, expected, true)
546+
}
547+
548+
#[test]
549+
fn propagate_union_remaining_input_with_other_names() -> Result<()> {
550+
// Union
551+
// Filter: false
552+
// TableScan: test
553+
// Projection: test2.a AS x, test2.b AS y, test2.c AS z
554+
// TableScan: test2
555+
let left = LogicalPlanBuilder::from(test_table_scan()?)
556+
.filter(lit(false))?
557+
.build()?;
558+
let right = LogicalPlanBuilder::from(test_table_scan_with_name("test2")?)
559+
.project(vec![
560+
col("a").alias("x"),
561+
col("b").alias("y"),
562+
col("c").alias("z"),
563+
])?
564+
.build()?;
565+
566+
let plan = LogicalPlanBuilder::from(left).union(right)?.build()?;
567+
568+
let expected = "Projection: x AS a, y AS b, z AS c\
569+
\n Projection: test2.a AS x, test2.b AS y, test2.c AS z\
570+
\n TableScan: test2";
533571
assert_together_optimized_plan(plan, expected, true)
534572
}
535573

@@ -813,7 +851,7 @@ mod tests {
813851
.union(three)?
814852
.build()?;
815853

816-
let expected = "Projection: a, b, c\
854+
let expected = "Projection: test.a AS a, test.b AS b, test.c AS c\
817855
\n TableScan: test";
818856

819857
assert_together_optimized_plan(plan, expected, true)

‎datafusion/sqllogictest/test_files/struct.slt‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1982,8 +1982,8 @@ statement ok
19821982
drop table rename_swap_struct_field;
19831983

19841984
# Merging an extraction projection into a projection whose output lost its
1985-
# qualifier. Eliminating the empty side of the union leaves a projection of
1986-
# bare column names over a qualified input, and the merge used to add the
1985+
# qualifier. Eliminating the empty side of the union leaves a projection that
1986+
# renames qualified input columns to bare names, and the merge used to add the
19871987
# pass-through columns under those bare names beside the qualified ones the
19881988
# projection already carried, which is an ambiguous schema.
19891989
statement ok
@@ -2009,7 +2009,7 @@ logical_plan
20092009
03)----Aggregate: groupBy=[[expanded.env]], aggr=[[sum(CAST(__datafusion_extracted_1 AS Int64))]]
20102010
04)------SubqueryAlias: expanded
20112011
05)--------SubqueryAlias: samples
2012-
06)----------Projection: leaf_merge_source.env, get_field(s, Utf8("a")) AS __datafusion_extracted_1
2012+
06)----------Projection: leaf_merge_source.env, get_field(leaf_merge_source.s, Utf8("a")) AS __datafusion_extracted_1
20132013
07)------------TableScan: leaf_merge_source projection=[s, env]
20142014

20152015
statement ok

‎datafusion/sqllogictest/test_files/union.slt‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1144,6 +1144,20 @@ query TITT
11441144
----
11451145
ns3 4 t3 m
11461146

1147+
# repro for https://github.com/apache/datafusion/issues/26516
1148+
# should not error when only a later union input remains
1149+
query I
1150+
SELECT 1 WHERE false UNION ALL (SELECT 2 LIMIT 1);
1151+
----
1152+
2
1153+
1154+
query I
1155+
SELECT a FROM (VALUES (1)) t(a) WHERE false
1156+
UNION ALL
1157+
(SELECT b FROM (VALUES (2)) s(b) LIMIT 1);
1158+
----
1159+
2
1160+
11471161
# Config reset
11481162
statement ok
11491163
RESET datafusion.execution.batch_size;

0 commit comments

Comments
 (0)