Repository navigation
Commit e6f3b56
fix: keep the recovery projection when leaf pushdown would drop a computed same-name column (apache#25445)
## Which issue does this PR close?
- Closes apache#25414.
## Rationale for this change
A subquery that computes a column under the same name as its input
column gives wrong results. This happens with the default configuration.
```sql
create table t(a int, s struct<b varchar>) as values (1, {b: 'x'}), (2, {b: 'y'});
select a, s['b'] from (select -a as a, s from t where a > 0);
```
| Query | DataFusion on `main` | DuckDB and PostgreSQL |
|---|---|---|
| `select a, s['b'] from (select -a as a, s from t where a > 0)` | `1 x`
/ `2 y` | `-1 x` / `-2 y` |
| `select a, s['b'] from (select -a as a, s from t limit 10)` | `1 x` /
`2 y` | `-1 x` / `-2 y` |
| `select a, s['b'] from (select a * 10 as a, s from t limit 10)` |
`Internal error` | `10 x` / `20 y` |
| `select a, s['b'] from (select -a as a, s from t) where a < 0` | `1 x`
/ `2 y` | `-1 x` / `-2 y` |
| `select a, count(s['b']) from (select -a as a, s from t where a > 0)
group by a` | groups `1` and `2` | groups `-1` and `-2` |
The fourth query returns rows that fail its own filter. The third query
stops the plan with this message:
```
Internal error: Assertion failed: compatible: Failed due to a difference in schemas:
original schema: ... "a", data_type: Int64 ...
new schema: ... "a", data_type: Int32 ...
```
The results are correct with `set
datafusion.optimizer.enable_leaf_expression_pushdown = false;`.
The cause is in `split_and_push_projection` in
`datafusion/optimizer/src/extract_leaf_expressions.rs`. The rule pushes
the extraction of `s['b']` below the projection that computes `-a AS a`.
It then decides if it must keep a recovery projection, and it decides
from the set of unqualified field names of the pushed plan.
A name says nothing about the value behind it. For this projection:
```text
Projection: (- t.a) AS a, t.s, get_field(t.s, "b") AS __datafusion_extracted_1
Filter: t.a > Int32(0)
TableScan: t
```
the pushed plan holds the names `a`, `s` and `__datafusion_extracted_1`
again, but its `a` is the table column `t.a`, not `- t.a`. The two sets
are equal, the rule drops the recovery projection, and the computed
column becomes its own input column:
```text
Projection: a, __datafusion_extracted_1 AS t.s[b]
Filter: t.a > Int32(0)
Projection: get_field(t.s, "b") AS __datafusion_extracted_1, t.a, t.s
TableScan: t
```
The same comparison ignores data types. That is why `a * 10 AS a` fails
the optimizer schema check instead of giving wrong data.
## What changes are included in this PR?
The recovery projection now also stays when a recovery expression
computes a value, that is, when it is not a pass-through of a column.
`passthrough_column` gives that answer, and it already accepts a
requalification such as `t.a AS a`.
The name comparison stays as it is. It catches the leaked-column case
that it was written for, which the expression check does not see. The
comparison still ignores qualifiers, so the `SubqueryAlias`
requalification behaviour does not change.
The change is inside `split_and_push_projection` only.
apache#25412 touches
`build_extraction_projection_impl` in the same file. The two changes are
independent and fix different bugs. The only overlap is that both PRs
append a block to the end of
`datafusion/sqllogictest/test_files/struct.slt`, so the second one to
merge needs a trivial rebase there.
## What is the testing strategy for this PR?
Six SQL shapes go in `datafusion/sqllogictest/test_files/struct.slt`:
through a `Filter`, through a `Limit`, a computed column with a
different data type, an outer filter on the computed column, a group
key, and through a `Sort`. One `EXPLAIN` shows that the projection that
computes `-a` stays in the plan.
A counterfactual run confirms each one. With the fix reverted, the new
block gives 6 errors: the `EXPLAIN`, four wrong results and the internal
error. The `Sort` shape was already correct and guards it.
A unit test, `test_recovery_kept_for_same_name_computed_column` in
`extract_leaf_expressions.rs`, builds the plan shape that loses the
computed column and runs the two leaf rules alone, in their production
order.
All 883 `datafusion-optimizer` tests pass. No existing insta snapshot
changes. All 520 sqllogictest files pass.
## Are there any user-facing changes?
Queries of this shape now give correct results. There are no API
changes.
Part of the leaf-pushdown EPIC:
apache#25459
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.1 <[email protected]>1 parent de01d0d commit e6f3b56
2 files changed
Lines changed: 123 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1111 | 1111 | | |
1112 | 1112 | | |
1113 | 1113 | | |
| 1114 | + | |
| 1115 | + | |
| 1116 | + | |
| 1117 | + | |
| 1118 | + | |
| 1119 | + | |
| 1120 | + | |
| 1121 | + | |
| 1122 | + | |
| 1123 | + | |
| 1124 | + | |
| 1125 | + | |
1114 | 1126 | | |
1115 | 1127 | | |
1116 | 1128 | | |
| |||
1122 | 1134 | | |
1123 | 1135 | | |
1124 | 1136 | | |
1125 | | - | |
| 1137 | + | |
| 1138 | + | |
| 1139 | + | |
| 1140 | + | |
1126 | 1141 | | |
1127 | 1142 | | |
1128 | 1143 | | |
| |||
3759 | 3774 | | |
3760 | 3775 | | |
3761 | 3776 | | |
| 3777 | + | |
| 3778 | + | |
3762 | 3779 | | |
| 3780 | + | |
| 3781 | + | |
| 3782 | + | |
| 3783 | + | |
| 3784 | + | |
| 3785 | + | |
| 3786 | + | |
| 3787 | + | |
| 3788 | + | |
| 3789 | + | |
| 3790 | + | |
| 3791 | + | |
| 3792 | + | |
| 3793 | + | |
| 3794 | + | |
| 3795 | + | |
| 3796 | + | |
| 3797 | + | |
| 3798 | + | |
| 3799 | + | |
| 3800 | + | |
| 3801 | + | |
| 3802 | + | |
| 3803 | + | |
| 3804 | + | |
| 3805 | + | |
| 3806 | + | |
| 3807 | + | |
| 3808 | + | |
| 3809 | + | |
| 3810 | + | |
| 3811 | + | |
| 3812 | + | |
| 3813 | + | |
| 3814 | + | |
| 3815 | + | |
3763 | 3816 | | |
3764 | 3817 | | |
3765 | 3818 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1803 | 1803 | | |
1804 | 1804 | | |
1805 | 1805 | | |
| 1806 | + | |
| 1807 | + | |
| 1808 | + | |
| 1809 | + | |
| 1810 | + | |
| 1811 | + | |
| 1812 | + | |
| 1813 | + | |
| 1814 | + | |
| 1815 | + | |
| 1816 | + | |
| 1817 | + | |
| 1818 | + | |
| 1819 | + | |
| 1820 | + | |
| 1821 | + | |
| 1822 | + | |
| 1823 | + | |
| 1824 | + | |
| 1825 | + | |
| 1826 | + | |
| 1827 | + | |
| 1828 | + | |
| 1829 | + | |
| 1830 | + | |
| 1831 | + | |
| 1832 | + | |
| 1833 | + | |
| 1834 | + | |
| 1835 | + | |
| 1836 | + | |
| 1837 | + | |
| 1838 | + | |
| 1839 | + | |
| 1840 | + | |
| 1841 | + | |
| 1842 | + | |
| 1843 | + | |
| 1844 | + | |
| 1845 | + | |
| 1846 | + | |
| 1847 | + | |
| 1848 | + | |
| 1849 | + | |
| 1850 | + | |
| 1851 | + | |
| 1852 | + | |
| 1853 | + | |
| 1854 | + | |
| 1855 | + | |
| 1856 | + | |
| 1857 | + | |
| 1858 | + | |
| 1859 | + | |
| 1860 | + | |
| 1861 | + | |
| 1862 | + | |
| 1863 | + | |
| 1864 | + | |
| 1865 | + | |
| 1866 | + | |
| 1867 | + | |
| 1868 | + | |
| 1869 | + | |
| 1870 | + | |
| 1871 | + | |
| 1872 | + | |
| 1873 | + | |
| 1874 | + | |
0 commit comments