Repository navigation
Commit 7a1f468
authored
fix: preserve nested projection expressions (apache#24686)
## Which issue does this PR close?
- No issue has been filed.
## Rationale for this change
Anonymous nested projections that reuse the same output name can return
wrong results. For example, each `i + 1 AS i` layer must be evaluated
independently, but the projection optimizer could drop one layer when
two consecutive projection expression vectors were structurally equal.
Structural equality does not imply that a projection is safe to elide:
repeated computations such as `i + 1 AS i` have the same expression
shape but must still be evaluated twice.
## What changes are included in this PR?
- Remove the structural-equality fast path that directly elided one of
two consecutive projections.
- Keep the existing iterative whole-chain merge, so deep projection
chains still collapse within one optimizer rule invocation.
- Continue using the normal projection rewrite path, which composes
repeated expressions and preserves aliases and field metadata.
- Add focused regressions for repeated non-idempotent projections,
one-pass collapse of a 12-level chain, and metadata-bearing aliases.
- Add an execution-level SQLLogicTest with six anonymous `i + 1 AS i`
layers under both `max_passes = 1` and the default optimizer
configuration.
## What is the testing strategy for this PR?
The focused unit tests verify that:
- two structurally equal `i + 1 AS i` projections retain both additions;
- a 12-level chain preserves all 12 additions and collapses to one
`Projection` with `max_passes = 1`;
- a metadata-bearing `Alias(Column)` is merged without losing field
metadata.
The SQLLogicTest executes a six-level anonymous projection chain against
a temporary table with both one optimizer pass and the default pass
count.
Verified with:
```text
cargo fmt --all --check
cargo test -p datafusion-optimizer optimize_projections
# 58 passed; 0 failed
cargo test -p datafusion-optimizer --test optimizer_integration
# 26 passed; 0 failed
cargo test --profile ci -p datafusion-sqllogictest --test sqllogictests -- projection.slt
# 1/1 files completed; 0 failures
cargo clippy -p datafusion-optimizer --all-targets --all-features -- -D warnings
# passed
git diff --check
# passed
```
## Are there any user-facing changes?
Yes. Deep anonymous nested projections that reuse an output name now
preserve every projection expression and return the correct result.
There are no public API or configuration changes.
---------
Signed-off-by: discord9 <[email protected]>
Signed-off-by: discord9 <[email protected]>1 parent d02bd78 commit 7a1f468
2 files changed
Lines changed: 131 additions & 11 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
556 | 556 | | |
557 | 557 | | |
558 | 558 | | |
559 | | - | |
560 | | - | |
561 | | - | |
562 | | - | |
563 | | - | |
564 | | - | |
565 | | - | |
566 | | - | |
567 | | - | |
568 | | - | |
569 | | - | |
570 | 559 | | |
571 | 560 | | |
572 | 561 | | |
| |||
1210 | 1199 | | |
1211 | 1200 | | |
1212 | 1201 | | |
| 1202 | + | |
| 1203 | + | |
| 1204 | + | |
| 1205 | + | |
| 1206 | + | |
| 1207 | + | |
| 1208 | + | |
| 1209 | + | |
| 1210 | + | |
| 1211 | + | |
| 1212 | + | |
| 1213 | + | |
| 1214 | + | |
| 1215 | + | |
| 1216 | + | |
| 1217 | + | |
| 1218 | + | |
| 1219 | + | |
| 1220 | + | |
| 1221 | + | |
| 1222 | + | |
| 1223 | + | |
| 1224 | + | |
| 1225 | + | |
| 1226 | + | |
| 1227 | + | |
| 1228 | + | |
| 1229 | + | |
| 1230 | + | |
| 1231 | + | |
| 1232 | + | |
| 1233 | + | |
| 1234 | + | |
| 1235 | + | |
| 1236 | + | |
| 1237 | + | |
| 1238 | + | |
| 1239 | + | |
| 1240 | + | |
| 1241 | + | |
| 1242 | + | |
| 1243 | + | |
| 1244 | + | |
| 1245 | + | |
| 1246 | + | |
| 1247 | + | |
| 1248 | + | |
| 1249 | + | |
| 1250 | + | |
| 1251 | + | |
| 1252 | + | |
| 1253 | + | |
| 1254 | + | |
| 1255 | + | |
| 1256 | + | |
| 1257 | + | |
| 1258 | + | |
| 1259 | + | |
| 1260 | + | |
| 1261 | + | |
| 1262 | + | |
| 1263 | + | |
| 1264 | + | |
1213 | 1265 | | |
1214 | 1266 | | |
1215 | 1267 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
217 | 217 | | |
218 | 218 | | |
219 | 219 | | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
220 | 288 | | |
221 | 289 | | |
222 | 290 | | |
| |||
0 commit comments