Repository navigation
Commit b69dd03
fix: rebase lambda parameters and keep metadata when splitting a projection
Two regressions in the projection split, both reported in review. Each query
below works on `main` and failed with the split.
Lambda parameters are indexed after the input columns, so widening the input
leaves them pointing at the wrong column:
SELECT array_transform(arr, x -> x + 1), s['value'], s['label'] FROM t;
`remap_filter_expr` in this file already rebases parameters for the join filter
pushdown, so the node-level shift moves into `rebase_lambda_variable` and both
callers use it rather than carrying two copies.
The symptom depends on where the stale index lands. Out of range it is dropped
and the query happens to be correct; in range it fails `LambdaVariable`'s field
check; in range with a matching name and type it reads the wrong column and
returns wrong values with no error. The last case is covered, since the field
check is a backstop rather than the guarantee.
`ProjectionExec::try_new` derives the output schema from the expressions alone
and drops the field metadata the planner attached through
`try_new_with_schema_metadata`, which the rule's own `schema_check` then
rejects:
PhysicalOptimizer rule 'ProjectionPushdown' failed. Schema mismatch.
The split now rebuilds the projection with that constructor, taking the
metadata from the projection it replaces.
Tests. The value queries live in `projection_pushdown.slt`; the fixture gains
`arr` and `tags`, appended after `s` so the existing `get_field(s@1, ...)`
assertions do not move. A nested lambda needs two leaf expressions: with one
the split is width-neutral, the parameter does not move, and the query passes
against unfixed code. The metadata test is a Rust test because no SQL syntax
attaches field metadata; it asserts on the physical plan's schema, not the
DataFrame's, and uses Parquet because only a source that absorbs the leaf
projection makes the split fire.
Part of apache#27131
Co-Authored-By: Claude Opus 5 <[email protected]>1 parent 3545ba6 commit b69dd03
3 files changed
Lines changed: 149 additions & 20 deletions
File tree
- datafusion
- core/tests/physical_optimizer
- physical-optimizer/src
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| 5 | + | |
5 | 6 | | |
6 | 7 | | |
7 | 8 | | |
| |||
10 | 11 | | |
11 | 12 | | |
12 | 13 | | |
| 14 | + | |
13 | 15 | | |
| 16 | + | |
14 | 17 | | |
15 | 18 | | |
16 | 19 | | |
| |||
20 | 23 | | |
21 | 24 | | |
22 | 25 | | |
| 26 | + | |
23 | 27 | | |
24 | 28 | | |
25 | 29 | | |
| |||
1905 | 1909 | | |
1906 | 1910 | | |
1907 | 1911 | | |
| 1912 | + | |
| 1913 | + | |
| 1914 | + | |
| 1915 | + | |
| 1916 | + | |
| 1917 | + | |
| 1918 | + | |
| 1919 | + | |
| 1920 | + | |
| 1921 | + | |
| 1922 | + | |
| 1923 | + | |
| 1924 | + | |
| 1925 | + | |
| 1926 | + | |
| 1927 | + | |
| 1928 | + | |
| 1929 | + | |
| 1930 | + | |
| 1931 | + | |
| 1932 | + | |
| 1933 | + | |
| 1934 | + | |
| 1935 | + | |
| 1936 | + | |
| 1937 | + | |
| 1938 | + | |
| 1939 | + | |
| 1940 | + | |
| 1941 | + | |
| 1942 | + | |
| 1943 | + | |
| 1944 | + | |
| 1945 | + | |
| 1946 | + | |
| 1947 | + | |
| 1948 | + | |
| 1949 | + | |
| 1950 | + | |
| 1951 | + | |
| 1952 | + | |
| 1953 | + | |
| 1954 | + | |
| 1955 | + | |
| 1956 | + | |
| 1957 | + | |
| 1958 | + | |
| 1959 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
451 | 451 | | |
452 | 452 | | |
453 | 453 | | |
454 | | - | |
455 | | - | |
456 | | - | |
457 | | - | |
458 | | - | |
459 | | - | |
460 | | - | |
461 | | - | |
462 | | - | |
| 454 | + | |
| 455 | + | |
463 | 456 | | |
464 | 457 | | |
465 | 458 | | |
466 | 459 | | |
467 | 460 | | |
468 | 461 | | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
469 | 480 | | |
470 | 481 | | |
471 | 482 | | |
| |||
547 | 558 | | |
548 | 559 | | |
549 | 560 | | |
| 561 | + | |
550 | 562 | | |
551 | 563 | | |
552 | 564 | | |
| |||
559 | 571 | | |
560 | 572 | | |
561 | 573 | | |
562 | | - | |
563 | | - | |
564 | | - | |
565 | | - | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
566 | 583 | | |
567 | 584 | | |
568 | 585 | | |
| |||
614 | 631 | | |
615 | 632 | | |
616 | 633 | | |
617 | | - | |
| 634 | + | |
| 635 | + | |
618 | 636 | | |
619 | 637 | | |
620 | 638 | | |
621 | 639 | | |
| 640 | + | |
622 | 641 | | |
623 | 642 | | |
624 | 643 | | |
| 644 | + | |
625 | 645 | | |
626 | 646 | | |
627 | 647 | | |
| |||
646 | 666 | | |
647 | 667 | | |
648 | 668 | | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
649 | 677 | | |
650 | 678 | | |
651 | 679 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2689 | 2689 | | |
2690 | 2690 | | |
2691 | 2691 | | |
| 2692 | + | |
| 2693 | + | |
2692 | 2694 | | |
2693 | 2695 | | |
2694 | | - | |
| 2696 | + | |
2695 | 2697 | | |
2696 | | - | |
2697 | | - | |
2698 | | - | |
2699 | | - | |
2700 | | - | |
| 2698 | + | |
| 2699 | + | |
| 2700 | + | |
| 2701 | + | |
| 2702 | + | |
2701 | 2703 | | |
2702 | 2704 | | |
2703 | 2705 | | |
| |||
2779 | 2781 | | |
2780 | 2782 | | |
2781 | 2783 | | |
| 2784 | + | |
| 2785 | + | |
| 2786 | + | |
| 2787 | + | |
| 2788 | + | |
| 2789 | + | |
| 2790 | + | |
| 2791 | + | |
| 2792 | + | |
| 2793 | + | |
| 2794 | + | |
| 2795 | + | |
| 2796 | + | |
| 2797 | + | |
| 2798 | + | |
| 2799 | + | |
| 2800 | + | |
| 2801 | + | |
| 2802 | + | |
| 2803 | + | |
| 2804 | + | |
| 2805 | + | |
| 2806 | + | |
| 2807 | + | |
| 2808 | + | |
| 2809 | + | |
| 2810 | + | |
| 2811 | + | |
| 2812 | + | |
| 2813 | + | |
| 2814 | + | |
| 2815 | + | |
| 2816 | + | |
| 2817 | + | |
| 2818 | + | |
| 2819 | + | |
| 2820 | + | |
| 2821 | + | |
| 2822 | + | |
| 2823 | + | |
| 2824 | + | |
| 2825 | + | |
| 2826 | + | |
| 2827 | + | |
| 2828 | + | |
| 2829 | + | |
| 2830 | + | |
2782 | 2831 | | |
2783 | 2832 | | |
2784 | 2833 | | |
| |||
0 commit comments