Repository navigation
Commit cbc04dd
fix: preserve nested projection expressions (apache#24686)
## Which issue does this PR close?
- No issue has been filed.
## Rationale for this change
Anonymous nested projections that reuse the same output name can return
wrong results. For example, each `i + 1 AS i` layer must be evaluated
independently, but the projection optimizer could drop one layer when
two consecutive projection expression vectors were structurally equal.
Structural equality does not imply that a projection is safe to elide:
repeated computations such as `i + 1 AS i` have the same expression
shape but must still be evaluated twice.
## What changes are included in this PR?
- Remove the structural-equality fast path that directly elided one of
two consecutive projections.
- Keep the existing iterative whole-chain merge, so deep projection
chains still collapse within one optimizer rule invocation.
- Continue using the normal projection rewrite path, which composes
repeated expressions and preserves aliases and field metadata.
- Add focused regressions for repeated non-idempotent projections,
one-pass collapse of a 12-level chain, and metadata-bearing aliases.
- Add an execution-level SQLLogicTest with six anonymous `i + 1 AS i`
layers under both `max_passes = 1` and the default optimizer
configuration.
## What is the testing strategy for this PR?
The focused unit tests verify that:
- two structurally equal `i + 1 AS i` projections retain both additions;
- a 12-level chain preserves all 12 additions and collapses to one
`Projection` with `max_passes = 1`;
- a metadata-bearing `Alias(Column)` is merged without losing field
metadata.
The SQLLogicTest executes a six-level anonymous projection chain against
a temporary table with both one optimizer pass and the default pass
count.
Verified with:
```text
cargo fmt --all --check
cargo test -p datafusion-optimizer optimize_projections
# 58 passed; 0 failed
cargo test -p datafusion-optimizer --test optimizer_integration
# 26 passed; 0 failed
cargo test --profile ci -p datafusion-sqllogictest --test sqllogictests -- projection.slt
# 1/1 files completed; 0 failures
cargo clippy -p datafusion-optimizer --all-targets --all-features -- -D warnings
# passed
git diff --check
# passed
```
## Are there any user-facing changes?
Yes. Deep anonymous nested projections that reuse an output name now
preserve every projection expression and return the correct result.
There are no public API or configuration changes.
---------
Signed-off-by: discord9 <[email protected]>
Signed-off-by: discord9 <[email protected]>
(cherry picked from commit 7a1f468)1 parent 53e60cd commit cbc04dd
2 files changed
Lines changed: 131 additions & 11 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
556 | 556 | | |
557 | 557 | | |
558 | 558 | | |
559 | | - | |
560 | | - | |
561 | | - | |
562 | | - | |
563 | | - | |
564 | | - | |
565 | | - | |
566 | | - | |
567 | | - | |
568 | | - | |
569 | | - | |
570 | 559 | | |
571 | 560 | | |
572 | 561 | | |
| |||
1267 | 1256 | | |
1268 | 1257 | | |
1269 | 1258 | | |
| 1259 | + | |
| 1260 | + | |
| 1261 | + | |
| 1262 | + | |
| 1263 | + | |
| 1264 | + | |
| 1265 | + | |
| 1266 | + | |
| 1267 | + | |
| 1268 | + | |
| 1269 | + | |
| 1270 | + | |
| 1271 | + | |
| 1272 | + | |
| 1273 | + | |
| 1274 | + | |
| 1275 | + | |
| 1276 | + | |
| 1277 | + | |
| 1278 | + | |
| 1279 | + | |
| 1280 | + | |
| 1281 | + | |
| 1282 | + | |
| 1283 | + | |
| 1284 | + | |
| 1285 | + | |
| 1286 | + | |
| 1287 | + | |
| 1288 | + | |
| 1289 | + | |
| 1290 | + | |
| 1291 | + | |
| 1292 | + | |
| 1293 | + | |
| 1294 | + | |
| 1295 | + | |
| 1296 | + | |
| 1297 | + | |
| 1298 | + | |
| 1299 | + | |
| 1300 | + | |
| 1301 | + | |
| 1302 | + | |
| 1303 | + | |
| 1304 | + | |
| 1305 | + | |
| 1306 | + | |
| 1307 | + | |
| 1308 | + | |
| 1309 | + | |
| 1310 | + | |
| 1311 | + | |
| 1312 | + | |
| 1313 | + | |
| 1314 | + | |
| 1315 | + | |
| 1316 | + | |
| 1317 | + | |
| 1318 | + | |
| 1319 | + | |
| 1320 | + | |
| 1321 | + | |
1270 | 1322 | | |
1271 | 1323 | | |
1272 | 1324 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
217 | 217 | | |
218 | 218 | | |
219 | 219 | | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
220 | 288 | | |
221 | 289 | | |
222 | 290 | | |
| |||
0 commit comments