Repository navigation
Commit 02da2e9
fix: keep renamed columns intact in leaf projection pushdown (apache#25478)
## Which issue does this PR close?
- Closes apache#25446.
Part of the leaf-pushdown EPIC:
apache#25459
## Rationale for this change
A valid statement fails to plan. A sub-query projection renames a column
to the name of a different column of the same input, and the query reads
a struct field above it.
`datafusion.optimizer.enable_leaf_expression_pushdown` is `true` by
default, so this is a plain planning failure.
```sql
create table t(a int, b int, s struct<x varchar>) as values (1, 10, {x: 'p'}), (2, 20, {x: 'q'});
-- 1. a rename beside a same-name alias, under a filter
select b, a, s['x'] from (select t.a as b, t.b as a, s from t) where a > 0;
-- 2. the same shape under a limit
select b, a, s['x'] from (select t.a as b, t.b as a, s from t) limit 10;
-- 3. a swap of two column names
select a, c, s['x'] from (select t.b as a, t.a as c, s from t) where a > 0;
```
Observed on `main`, for all three:
```text
Optimizer rule 'push_down_leaf_projections' failed
caused by
Schema error: Schema contains qualified field name t.a and unqualified field name a which would be ambiguous
```
Expected, and what you get with `set
datafusion.optimizer.enable_leaf_expression_pushdown = false;`:
```text
+---+----+--------+ +----+---+--------+
| b | a | t.s[x] | | a | c | t.s[x] |
+---+----+--------+ +----+---+--------+
| 1 | 10 | p | | 10 | 1 | p |
| 2 | 20 | q | | 20 | 2 | q |
+---+----+--------+ +----+---+--------+
statements 1 and 2 statement 3
```
`order by` and `group by` above the rename fail the same way. Query
generators emit this shape.
## What changes are included in this PR?
Both changes are in
`datafusion/optimizer/src/extract_leaf_expressions.rs`.
**1. The merge of the extraction projection.**
`build_extraction_projection_impl` merges the extraction projection into
the projection below it. For every column the parent needs, it resolved
the name through the projection's rename map and then added the *input*
column that the name resolves to. For `select t.a as b, t.b as a, s from
t` the parent's `b` resolves to `t.a`, and `t.a` lands beside the output
field `a` of the other rename. `Projection::try_new` rejects that
schema.
The merge keeps every expression of the projection below it, so the
parent can still read each name that projection produces. Those names
are now skipped instead of resolved, and a pass-through column whose
name is one of the projection's output names is never added. A rename is
a computed output, and the parent refers to it by its output name only.
**2. The recovery check.** With the merge fixed, the shape reaches
`split_and_push_projection`, which decided if it needs a recovery
projection from the set of unqualified field names alone. Equal name
sets do not prove equal values: the pushed plan exposes the table column
`t.a` where the projection computed `t.a AS b`. Without this hunk the
statements above return the columns swapped instead of an error. The
recovery projection now also stays when a recovery expression is not a
pass-through of a column.
That second hunk is the same change as
apache#25445, down to the text, so
the two merge without a conflict. If that PR lands first, the hunk falls
out of the rebase.
**How this composes with the other open PRs on this function.**
apache#25412 resolves both sides of
the pass-through comparison against the input schema, and
apache#25456 replaces the rename map
with `ProjectionInliner`. This change sits above both: it decides which
columns to add before either one resolves a name. I merged this branch
into a local branch that holds both PRs. The single conflict is the loop
header, which takes the guards of this PR around the body of the other
two. The optimizer tests, the `struct`, `cse` and `push_down_filter`
sqllogictest files, and the differential fuzz all pass there with the
swap-alias shape of the fuzz generator turned on: 250 short cases, 125
short Parquet cases, 5000 extended cases and 1000 extended Parquet
cases, 0 failures and 0 skips.
## What is the testing strategy for this PR?
- `datafusion/sqllogictest/test_files/struct.slt`: the three statements
above, plus the `order by` and `group by` variants, plus a swap where
the struct field name is also a column name of the table, plus one
`EXPLAIN` that shows the renames above the extraction projection and the
struct field still read at the scan.
- `datafusion/optimizer/src/extract_leaf_expressions.rs`:
`test_extract_above_projection_that_swaps_column_names` covers the
ambiguous schema.
`test_extract_above_projection_that_redefines_column_name` covers the
recovery projection that must keep a rename alive.
- Three plan snapshots of neighbouring tests lose a duplicate
pass-through column in the intermediate stage. The optimized plans of
those three tests do not change.
Commands and results:
```text
cargo test --profile ci -p datafusion-optimizer
884, 26, 5 passed; 0 failed
cargo test --profile ci -p datafusion-sqllogictest --test sqllogictests
520/520 files completed, 0 failures
cargo clippy --profile ci -p datafusion-optimizer --all-targets -- -D warnings
clean
```
## Are there any user-facing changes?
Statements of this shape plan and run. There is no API change.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.1 <[email protected]>
Backport of apache#25478 (merged upstream as 233aeab).1 parent 86f1f26 commit 02da2e9
2 files changed
Lines changed: 178 additions & 15 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
764 | 764 | | |
765 | 765 | | |
766 | 766 | | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
767 | 778 | | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
| 784 | + | |
| 785 | + | |
| 786 | + | |
768 | 787 | | |
769 | 788 | | |
770 | 789 | | |
771 | 790 | | |
772 | 791 | | |
| 792 | + | |
773 | 793 | | |
774 | 794 | | |
775 | 795 | | |
| |||
2981 | 3001 | | |
2982 | 3002 | | |
2983 | 3003 | | |
2984 | | - | |
| 3004 | + | |
2985 | 3005 | | |
2986 | 3006 | | |
2987 | 3007 | | |
2988 | | - | |
2989 | | - | |
2990 | | - | |
2991 | | - | |
| 3008 | + | |
2992 | 3009 | | |
2993 | 3010 | | |
2994 | 3011 | | |
| |||
3015 | 3032 | | |
3016 | 3033 | | |
3017 | 3034 | | |
3018 | | - | |
| 3035 | + | |
3019 | 3036 | | |
3020 | 3037 | | |
3021 | 3038 | | |
3022 | | - | |
3023 | | - | |
3024 | | - | |
3025 | | - | |
| 3039 | + | |
3026 | 3040 | | |
3027 | 3041 | | |
3028 | 3042 | | |
| |||
3044 | 3058 | | |
3045 | 3059 | | |
3046 | 3060 | | |
3047 | | - | |
| 3061 | + | |
3048 | 3062 | | |
3049 | 3063 | | |
3050 | 3064 | | |
3051 | 3065 | | |
3052 | 3066 | | |
3053 | 3067 | | |
3054 | | - | |
3055 | | - | |
3056 | | - | |
3057 | | - | |
| 3068 | + | |
| 3069 | + | |
| 3070 | + | |
| 3071 | + | |
| 3072 | + | |
| 3073 | + | |
| 3074 | + | |
| 3075 | + | |
| 3076 | + | |
| 3077 | + | |
| 3078 | + | |
| 3079 | + | |
| 3080 | + | |
| 3081 | + | |
| 3082 | + | |
| 3083 | + | |
| 3084 | + | |
| 3085 | + | |
| 3086 | + | |
| 3087 | + | |
| 3088 | + | |
| 3089 | + | |
| 3090 | + | |
| 3091 | + | |
| 3092 | + | |
| 3093 | + | |
| 3094 | + | |
| 3095 | + | |
| 3096 | + | |
| 3097 | + | |
| 3098 | + | |
| 3099 | + | |
| 3100 | + | |
| 3101 | + | |
| 3102 | + | |
| 3103 | + | |
| 3104 | + | |
| 3105 | + | |
| 3106 | + | |
| 3107 | + | |
| 3108 | + | |
| 3109 | + | |
| 3110 | + | |
| 3111 | + | |
| 3112 | + | |
| 3113 | + | |
| 3114 | + | |
| 3115 | + | |
| 3116 | + | |
| 3117 | + | |
| 3118 | + | |
| 3119 | + | |
| 3120 | + | |
| 3121 | + | |
| 3122 | + | |
| 3123 | + | |
| 3124 | + | |
| 3125 | + | |
| 3126 | + | |
| 3127 | + | |
| 3128 | + | |
| 3129 | + | |
| 3130 | + | |
| 3131 | + | |
| 3132 | + | |
| 3133 | + | |
| 3134 | + | |
| 3135 | + | |
| 3136 | + | |
| 3137 | + | |
| 3138 | + | |
| 3139 | + | |
| 3140 | + | |
| 3141 | + | |
| 3142 | + | |
| 3143 | + | |
| 3144 | + | |
| 3145 | + | |
| 3146 | + | |
3058 | 3147 | | |
3059 | 3148 | | |
3060 | 3149 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1907 | 1907 | | |
1908 | 1908 | | |
1909 | 1909 | | |
| 1910 | + | |
| 1911 | + | |
| 1912 | + | |
| 1913 | + | |
| 1914 | + | |
| 1915 | + | |
| 1916 | + | |
| 1917 | + | |
| 1918 | + | |
| 1919 | + | |
| 1920 | + | |
| 1921 | + | |
| 1922 | + | |
| 1923 | + | |
| 1924 | + | |
| 1925 | + | |
| 1926 | + | |
| 1927 | + | |
| 1928 | + | |
| 1929 | + | |
| 1930 | + | |
| 1931 | + | |
| 1932 | + | |
| 1933 | + | |
| 1934 | + | |
| 1935 | + | |
| 1936 | + | |
| 1937 | + | |
| 1938 | + | |
| 1939 | + | |
| 1940 | + | |
| 1941 | + | |
| 1942 | + | |
| 1943 | + | |
| 1944 | + | |
| 1945 | + | |
| 1946 | + | |
| 1947 | + | |
| 1948 | + | |
| 1949 | + | |
| 1950 | + | |
| 1951 | + | |
| 1952 | + | |
| 1953 | + | |
| 1954 | + | |
| 1955 | + | |
| 1956 | + | |
| 1957 | + | |
| 1958 | + | |
| 1959 | + | |
| 1960 | + | |
| 1961 | + | |
| 1962 | + | |
| 1963 | + | |
| 1964 | + | |
| 1965 | + | |
| 1966 | + | |
| 1967 | + | |
| 1968 | + | |
| 1969 | + | |
| 1970 | + | |
| 1971 | + | |
| 1972 | + | |
| 1973 | + | |
| 1974 | + | |
| 1975 | + | |
| 1976 | + | |
| 1977 | + | |
| 1978 | + | |
| 1979 | + | |
| 1980 | + | |
| 1981 | + | |
| 1982 | + | |
| 1983 | + | |
1910 | 1984 | | |
1911 | 1985 | | |
1912 | 1986 | | |
| |||
0 commit comments