Repository navigation
Commit 56eca4a
fix: keep a constant sort key when a projection reads a duplicate alias (apache#26397)
## Which issue does this PR close?
- Closes apache#26391.
## Rationale for this change
A query with a filter and a sort on an expression over a struct field
can fail planning with a `SanityCheckPlan` error. `EnsureRequirements`
removes a sort key that the filter makes constant, and after
`ProjectionPushdown` merges two projections the plan no longer knows
that the key is constant. See apache#26391 for a `datafusion-cli` reproduction
and a step-by-step trace.
The cause is in `EquivalenceGroup::project`. When a projection names one
expression twice, under two aliases, an equivalence class is recorded
with the first alias only. A projection source that reads the second
alias is equal to a member of that class only after normalization
through the equivalence classes, but `augment_projection_mapping` looked
up the class of the source as written. The class, and the constant it
carries, was then dropped.
## What changes are included in this PR?
`augment_projection_mapping`
(`datafusion/physical-expr/src/equivalence/class.rs`) falls back to the
class of the normalized source when the source as written is in no
class.
- It still tries the source as written first. `normalize_expr` rewrites
bottom-up, so an expression that is itself a class member can normalize
to a form that is not, and replacing the first lookup could lose matches
that work today.
- It normalizes only expressions with children. A leaf that is in no
class normalizes to itself, so most mapping entries (plain columns) cost
nothing extra.
apache#26391 also names the `self.is_expr_constant(source)` lookup in
`project`. This PR leaves it unchanged. The constant in apache#26391 is an
equivalence class that holds a literal, and the change above carries
that class through the projection. A fallback in `is_expr_constant`
would also cover a constant without a literal value, but it made
`physical_plan_tpcds_all` about 4% slower, and no query I found needs
it.
## What is the testing strategy for this PR?
- `projection_pushdown.slt`: the query from apache#26391, with a result
assertion. It fails with the `SanityCheckPlan` error without this
change.
- `test_project_constant_of_equivalent_source` and
`test_project_class_of_equivalent_source` (`class.rs`): a constant class
and a non-constant class that a projection source reaches only through
`a == b`. Both fail without this change.
`sql_planner` benchmarks against `main`:
| Benchmark | Change |
|---|---|
| `physical_select_one_from_700` | −0.7% |
| `physical_select_all_from_1000` | −0.6% |
| `physical_plan_struct_join_agg_sort` | +0.8% |
| `physical_plan_tpch_all` | +2.4% |
| `physical_plan_tpcds_all` | +0.8% |
Two runs of `main` on the same machine differed by up to 2%, so I read
these as noise.
## Are there any user-facing changes?
The query shape in apache#26391 plans and runs. Some plans can drop sort keys
that are now known to be constant. No public API changes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <[email protected]>1 parent 390622a commit 56eca4a
2 files changed
Lines changed: 146 additions & 1 deletion
File tree
- datafusion
- physical-expr/src/equivalence
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
670 | 670 | | |
671 | 671 | | |
672 | 672 | | |
673 | | - | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
674 | 684 | | |
675 | 685 | | |
676 | 686 | | |
| |||
1276 | 1286 | | |
1277 | 1287 | | |
1278 | 1288 | | |
| 1289 | + | |
| 1290 | + | |
| 1291 | + | |
| 1292 | + | |
| 1293 | + | |
| 1294 | + | |
| 1295 | + | |
| 1296 | + | |
| 1297 | + | |
| 1298 | + | |
| 1299 | + | |
| 1300 | + | |
| 1301 | + | |
| 1302 | + | |
| 1303 | + | |
| 1304 | + | |
| 1305 | + | |
| 1306 | + | |
| 1307 | + | |
| 1308 | + | |
| 1309 | + | |
| 1310 | + | |
| 1311 | + | |
| 1312 | + | |
| 1313 | + | |
| 1314 | + | |
| 1315 | + | |
| 1316 | + | |
| 1317 | + | |
| 1318 | + | |
| 1319 | + | |
| 1320 | + | |
| 1321 | + | |
| 1322 | + | |
| 1323 | + | |
| 1324 | + | |
| 1325 | + | |
| 1326 | + | |
| 1327 | + | |
| 1328 | + | |
| 1329 | + | |
| 1330 | + | |
| 1331 | + | |
| 1332 | + | |
| 1333 | + | |
| 1334 | + | |
| 1335 | + | |
| 1336 | + | |
| 1337 | + | |
| 1338 | + | |
| 1339 | + | |
| 1340 | + | |
| 1341 | + | |
| 1342 | + | |
| 1343 | + | |
| 1344 | + | |
| 1345 | + | |
| 1346 | + | |
| 1347 | + | |
| 1348 | + | |
| 1349 | + | |
| 1350 | + | |
| 1351 | + | |
| 1352 | + | |
| 1353 | + | |
| 1354 | + | |
| 1355 | + | |
| 1356 | + | |
| 1357 | + | |
| 1358 | + | |
| 1359 | + | |
| 1360 | + | |
| 1361 | + | |
| 1362 | + | |
| 1363 | + | |
| 1364 | + | |
| 1365 | + | |
| 1366 | + | |
| 1367 | + | |
| 1368 | + | |
| 1369 | + | |
| 1370 | + | |
| 1371 | + | |
| 1372 | + | |
| 1373 | + | |
| 1374 | + | |
| 1375 | + | |
| 1376 | + | |
| 1377 | + | |
| 1378 | + | |
| 1379 | + | |
| 1380 | + | |
| 1381 | + | |
| 1382 | + | |
| 1383 | + | |
| 1384 | + | |
| 1385 | + | |
| 1386 | + | |
| 1387 | + | |
| 1388 | + | |
1279 | 1389 | | |
1280 | 1390 | | |
1281 | 1391 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2622 | 2622 | | |
2623 | 2623 | | |
2624 | 2624 | | |
| 2625 | + | |
| 2626 | + | |
| 2627 | + | |
| 2628 | + | |
| 2629 | + | |
| 2630 | + | |
| 2631 | + | |
| 2632 | + | |
| 2633 | + | |
| 2634 | + | |
| 2635 | + | |
| 2636 | + | |
| 2637 | + | |
| 2638 | + | |
| 2639 | + | |
| 2640 | + | |
| 2641 | + | |
| 2642 | + | |
| 2643 | + | |
| 2644 | + | |
| 2645 | + | |
| 2646 | + | |
| 2647 | + | |
| 2648 | + | |
| 2649 | + | |
| 2650 | + | |
| 2651 | + | |
| 2652 | + | |
| 2653 | + | |
| 2654 | + | |
| 2655 | + | |
| 2656 | + | |
| 2657 | + | |
| 2658 | + | |
| 2659 | + | |
0 commit comments