Repository navigation
Commit 063402b
feat: decide automatically when to materialize a CTE
Add the option `datafusion.optimizer.enable_automatic_cte_materialization`, which is off by default.
With the default, each query plans as before.
When the option is on, the SQL planner plans each non-recursive CTE that has no directive as a `MaterializedCte` with `automatic = true`, in each dialect.
The new optimizer rule `InlineAutomaticCtes` replaces an automatic node with `MaterializedCte::inline` when the CTE has zero or one reference, or when its body is volatile, has an outer reference, or is only projections and filters on one table, or when a `LIMIT` with a fetch is above all references in the chain of one-input nodes from the root of the query.
All other automatic CTEs stay materialized and are computed one time.
`AS MATERIALIZED` and `AS NOT MATERIALIZED` override the rule.
This commit completes the feature: the physical planner inlines an automatic CTE that has an unbounded body, because such a body cannot be buffered and the query worked before.
Planning such a body returns an internal `UnboundedAutomaticCte` error, and `create_initial_plan` catches it, inlines that CTE and plans again.
An explicit `AS MATERIALIZED` CTE with an unbounded body still returns a planning error.
The diff of `cte_auto_materialize.slt` shows the unbounded cases that now plan and run.
Closes apache#26237.
Co-Authored-By: Claude Opus 5.5 <[email protected]>1 parent abe5f5d commit 063402b
2 files changed
Lines changed: 119 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
316 | 316 | | |
317 | 317 | | |
318 | 318 | | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
319 | 360 | | |
320 | 361 | | |
321 | 362 | | |
| |||
574 | 615 | | |
575 | 616 | | |
576 | 617 | | |
| 618 | + | |
| 619 | + | |
| 620 | + | |
| 621 | + | |
| 622 | + | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
577 | 650 | | |
578 | 651 | | |
579 | 652 | | |
| |||
1941 | 2014 | | |
1942 | 2015 | | |
1943 | 2016 | | |
| 2017 | + | |
| 2018 | + | |
| 2019 | + | |
| 2020 | + | |
| 2021 | + | |
| 2022 | + | |
| 2023 | + | |
1944 | 2024 | | |
1945 | 2025 | | |
1946 | 2026 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
638 | 638 | | |
639 | 639 | | |
640 | 640 | | |
641 | | - | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
642 | 657 | | |
643 | 658 | | |
644 | 659 | | |
| |||
658 | 673 | | |
659 | 674 | | |
660 | 675 | | |
661 | | - | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
| 696 | + | |
662 | 697 | | |
663 | 698 | | |
664 | | - | |
| 699 | + | |
665 | 700 | | |
666 | 701 | | |
| 702 | + | |
667 | 703 | | |
668 | 704 | | |
669 | 705 | | |
| |||
0 commit comments