Repository navigation
Commit 2273e33
feat: report statistics, ordering and partitioning of materialized CTE scans
A `MaterializedCteScanExec` is a leaf, and the CTE body is a child of `MaterializedCteExec`, so the scan reported unknown statistics, no ordering and unknown partitioning.
`JoinSelection` could then build a hash table on a large table and not on a small CTE, and the continuation could sort or repartition rows that the body already sorted or partitioned.
The planner now copies the properties of the body to the scans in `optimize_physical_plan`:
- Statistics: each scan gets the statistics of the body of the initial plan, before the optimizer runs.
- Ordering and hash partitioning: after the plan is optimized, each scan of the initial plan gets the partition count, the output ordering and the hash partitioning of the optimized body, and the plan is optimized again.
The second plan is used only if each body in it has the properties that its scans report.
If not, the first plan is used, in which the scans report no ordering and no partitioning.
A scan with the properties of the body replays one body partition per partition, and fails if the partition counts are not the same.
Closes apache#26230.
Co-Authored-By: Claude Opus 5.5 <[email protected]>1 parent 1a43b6c commit 2273e33
4 files changed
Lines changed: 1101 additions & 86 deletions
File tree
- datafusion
- core/src
- datasource
- physical-plan/src
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
527 | 527 | | |
528 | 528 | | |
529 | 529 | | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
530 | 584 | | |
0 commit comments