Skip to content

Commit e6f3b56

Browse files
adriangbclaude
authored andcommitted
fix: keep the recovery projection when leaf pushdown would drop a computed same-name column (apache#25445)
## Which issue does this PR close? - Closes apache#25414. ## Rationale for this change A subquery that computes a column under the same name as its input column gives wrong results. This happens with the default configuration. ```sql create table t(a int, s struct<b varchar>) as values (1, {b: 'x'}), (2, {b: 'y'}); select a, s['b'] from (select -a as a, s from t where a > 0); ``` | Query | DataFusion on `main` | DuckDB and PostgreSQL | |---|---|---| | `select a, s['b'] from (select -a as a, s from t where a > 0)` | `1 x` / `2 y` | `-1 x` / `-2 y` | | `select a, s['b'] from (select -a as a, s from t limit 10)` | `1 x` / `2 y` | `-1 x` / `-2 y` | | `select a, s['b'] from (select a * 10 as a, s from t limit 10)` | `Internal error` | `10 x` / `20 y` | | `select a, s['b'] from (select -a as a, s from t) where a < 0` | `1 x` / `2 y` | `-1 x` / `-2 y` | | `select a, count(s['b']) from (select -a as a, s from t where a > 0) group by a` | groups `1` and `2` | groups `-1` and `-2` | The fourth query returns rows that fail its own filter. The third query stops the plan with this message: ``` Internal error: Assertion failed: compatible: Failed due to a difference in schemas: original schema: ... "a", data_type: Int64 ... new schema: ... "a", data_type: Int32 ... ``` The results are correct with `set datafusion.optimizer.enable_leaf_expression_pushdown = false;`. The cause is in `split_and_push_projection` in `datafusion/optimizer/src/extract_leaf_expressions.rs`. The rule pushes the extraction of `s['b']` below the projection that computes `-a AS a`. It then decides if it must keep a recovery projection, and it decides from the set of unqualified field names of the pushed plan. A name says nothing about the value behind it. For this projection: ```text Projection: (- t.a) AS a, t.s, get_field(t.s, "b") AS __datafusion_extracted_1 Filter: t.a > Int32(0) TableScan: t ``` the pushed plan holds the names `a`, `s` and `__datafusion_extracted_1` again, but its `a` is the table column `t.a`, not `- t.a`. The two sets are equal, the rule drops the recovery projection, and the computed column becomes its own input column: ```text Projection: a, __datafusion_extracted_1 AS t.s[b] Filter: t.a > Int32(0) Projection: get_field(t.s, "b") AS __datafusion_extracted_1, t.a, t.s TableScan: t ``` The same comparison ignores data types. That is why `a * 10 AS a` fails the optimizer schema check instead of giving wrong data. ## What changes are included in this PR? The recovery projection now also stays when a recovery expression computes a value, that is, when it is not a pass-through of a column. `passthrough_column` gives that answer, and it already accepts a requalification such as `t.a AS a`. The name comparison stays as it is. It catches the leaked-column case that it was written for, which the expression check does not see. The comparison still ignores qualifiers, so the `SubqueryAlias` requalification behaviour does not change. The change is inside `split_and_push_projection` only. apache#25412 touches `build_extraction_projection_impl` in the same file. The two changes are independent and fix different bugs. The only overlap is that both PRs append a block to the end of `datafusion/sqllogictest/test_files/struct.slt`, so the second one to merge needs a trivial rebase there. ## What is the testing strategy for this PR? Six SQL shapes go in `datafusion/sqllogictest/test_files/struct.slt`: through a `Filter`, through a `Limit`, a computed column with a different data type, an outer filter on the computed column, a group key, and through a `Sort`. One `EXPLAIN` shows that the projection that computes `-a` stays in the plan. A counterfactual run confirms each one. With the fix reverted, the new block gives 6 errors: the `EXPLAIN`, four wrong results and the internal error. The `Sort` shape was already correct and guards it. A unit test, `test_recovery_kept_for_same_name_computed_column` in `extract_leaf_expressions.rs`, builds the plan shape that loses the computed column and runs the two leaf rules alone, in their production order. All 883 `datafusion-optimizer` tests pass. No existing insta snapshot changes. All 520 sqllogictest files pass. ## Are there any user-facing changes? Queries of this shape now give correct results. There are no API changes. Part of the leaf-pushdown EPIC: apache#25459 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.1 <[email protected]>
1 parent de01d0d commit e6f3b56

2 files changed

Lines changed: 123 additions & 1 deletion

File tree

‎datafusion/optimizer/src/extract_leaf_expressions.rs‎

Lines changed: 54 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1111,6 +1111,18 @@ fn split_and_push_projection(
11111111
// `SubqueryAlias` re-qualification (`sub.__datafusion_extracted_1` vs
11121112
// `__datafusion_extracted_1`) that a qualified/ordered comparison would
11131113
// spuriously treat as drift, stacking redundant recovery projections.
1114+
//
1115+
// A name comparison alone is not sufficient. A name says nothing about the
1116+
// *value* behind it. Take the projection
1117+
// `(- t.a) AS a, t.s, get_field(t.s, "b") AS __datafusion_extracted_1`. When
1118+
// the extraction goes below it, the pushed plan keeps every name, but it
1119+
// exposes the table column `t.a` where the projection computed `- t.a`. If
1120+
// the recovery projection goes away, the computed column becomes its own
1121+
// input column and the query gives wrong results. See
1122+
// <https://github.com/apache/datafusion/issues/25414>.
1123+
//
1124+
// So the recovery projection also stays when a recovery expression computes
1125+
// a value, that is, when it is not a pass-through of a column.
11141126
let base_names: BTreeSet<&str> = base_plan
11151127
.schema()
11161128
.fields()
@@ -1122,7 +1134,10 @@ fn split_and_push_projection(
11221134
.iter()
11231135
.map(|f| f.name().as_str())
11241136
.collect();
1125-
let needs_recovery = base_names != original_names;
1137+
let computes_a_value = recovery_exprs
1138+
.iter()
1139+
.any(|expr| passthrough_column(expr).is_none());
1140+
let needs_recovery = base_names != original_names || computes_a_value;
11261141

11271142
// Wrap with recovery projection if the output schema changed
11281143
if needs_recovery {
@@ -3759,7 +3774,45 @@ mod tests {
37593774
&no_columns,
37603775
&child
37613776
));
3777+
Ok(())
3778+
}
37623779

3780+
/// Regression test for <https://github.com/apache/datafusion/issues/25414>.
3781+
///
3782+
/// `(- test.id) AS id` computes a new value under the same name as its input
3783+
/// column `test.id`. Pushing the extraction below that projection makes
3784+
/// `test.id` visible again under the name `id`. The recovery projection must
3785+
/// stay, or the computed column is silently replaced by the table column.
3786+
///
3787+
/// The two leaf rules run alone here, in their production order.
3788+
/// `optimize_projections` merges the two projections into one and hides the
3789+
/// shape, and it only runs after both leaf rules.
3790+
#[test]
3791+
fn test_recovery_kept_for_same_name_computed_column() -> Result<()> {
3792+
let table_scan = test_table_scan_with_struct()?;
3793+
let plan = LogicalPlanBuilder::from(table_scan)
3794+
.filter(col("id").gt(lit(0u32)))?
3795+
.project(vec![
3796+
Expr::Negative(Box::new(col("id"))).alias("id"),
3797+
col("user"),
3798+
])?
3799+
.project(vec![col("id"), leaf_udf(col("user"), "name")])?
3800+
.build()?;
3801+
3802+
let ctx = OptimizerContext::new().with_max_passes(1);
3803+
let optimizer = Optimizer::with_rules(vec![
3804+
Arc::new(ExtractLeafExpressions::new()),
3805+
Arc::new(PushDownLeafProjections::new()),
3806+
]);
3807+
let optimized = optimizer.optimize(plan, &ctx, |_, _| {})?;
3808+
3809+
insta::assert_snapshot!(format!("{optimized}"), @r#"
3810+
Projection: id, __datafusion_extracted_1 AS leaf_udf(test.user,Utf8("name"))
3811+
Projection: (- test.id) AS id, test.user, __datafusion_extracted_1
3812+
Filter: test.id > UInt32(0)
3813+
Projection: leaf_udf(test.user, Utf8("name")) AS __datafusion_extracted_1, test.id, test.user
3814+
TableScan: test
3815+
"#);
37633816
Ok(())
37643817
}
37653818
}

‎datafusion/sqllogictest/test_files/struct.slt‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1803,3 +1803,72 @@ drop view struct_ctor_view;
18031803

18041804
statement ok
18051805
drop table struct_ctor_null;
1806+
1807+
# Regression test for https://github.com/apache/datafusion/issues/25414.
1808+
# `-a AS a` computes a new value under the same name as its input column.
1809+
# Leaf expression pushdown moves `s['b']` below that projection. The projection
1810+
# that computes `-a` must stay, or the query returns the input column `a`.
1811+
1812+
statement ok
1813+
create table leaf_same_name(a int, s struct<b varchar>) as values (1, {b: 'x'}), (2, {b: 'y'});
1814+
1815+
# The computed column must survive the pushdown. Line 02 holds `(- t.a) AS a`.
1816+
query TT
1817+
explain select a, s['b'] from (select -a as a, s from leaf_same_name where a > 0);
1818+
----
1819+
logical_plan
1820+
01)Projection: (- leaf_same_name.a) AS a, __datafusion_extracted_1 AS leaf_same_name.s[b]
1821+
02)--Filter: leaf_same_name.a > Int32(0)
1822+
03)----Projection: get_field(leaf_same_name.s, Utf8("b")) AS __datafusion_extracted_1, leaf_same_name.a
1823+
04)------TableScan: leaf_same_name projection=[a, s]
1824+
physical_plan
1825+
01)ProjectionExec: expr=[(- a@1) as a, __datafusion_extracted_1@0 as leaf_same_name.s[b]]
1826+
02)--RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
1827+
03)----FilterExec: a@1 > 0
1828+
04)------ProjectionExec: expr=[get_field(s@1, b) as __datafusion_extracted_1, a@0 as a]
1829+
05)--------DataSourceExec: partitions=1, partition_sizes=[1]
1830+
1831+
# Through a Filter
1832+
query IT rowsort
1833+
select a, s['b'] from (select -a as a, s from leaf_same_name where a > 0);
1834+
----
1835+
-1 x
1836+
-2 y
1837+
1838+
# Through a Limit
1839+
query IT rowsort
1840+
select a, s['b'] from (select -a as a, s from leaf_same_name limit 10);
1841+
----
1842+
-1 x
1843+
-2 y
1844+
1845+
# A computed column whose type differs from its input column
1846+
query IT rowsort
1847+
select a, s['b'] from (select a * 10 as a, s from leaf_same_name limit 10);
1848+
----
1849+
10 x
1850+
20 y
1851+
1852+
# The outer filter must see the computed value
1853+
query IT rowsort
1854+
select a, s['b'] from (select -a as a, s from leaf_same_name) where a < 0;
1855+
----
1856+
-1 x
1857+
-2 y
1858+
1859+
# The group key must be the computed value
1860+
query II rowsort
1861+
select a, count(s['b']) from (select -a as a, s from leaf_same_name where a > 0) group by a;
1862+
----
1863+
-1 1
1864+
-2 1
1865+
1866+
# Through a Sort
1867+
query IT rowsort
1868+
select a, s['b'] from (select -a as a, s from leaf_same_name order by a);
1869+
----
1870+
-1 x
1871+
-2 y
1872+
1873+
statement ok
1874+
drop table leaf_same_name;

0 commit comments

Comments
 (0)