Repository navigation
Commit 5c0e73f
authored
fix: preserve ProjectionExec metadata during serialization (apache#25009)
## Which issue does this PR close?
- Closes apache#24695.
## Rationale for this change
Physical-plan serialization can lose projection field and schema
metadata. The projection payload should preserve its output metadata,
including inherited metadata, without relying on the child to
reconstruct it.
## What changes are included in this PR?
- Add an optional schema to `ProjectionExecNode`, emitted when the
projection has schema or output-field metadata, or explicitly clears
input metadata.
- Restore metadata through `try_new_with_schema_metadata`, while
deriving field names, data types, and nullability from the expressions.
- Keep older payloads without the optional field readable.
- Use explicit metadata maps in tests for compatibility with both Arrow
59 and Arrow 60.
This preserves the projection's metadata independently of child
metadata. It does not make the entire execution plan independent of its
children. Omitting duplicated inherited metadata is left for a follow-up
once the serialization contract is agreed.
## What is the testing strategy for this PR?
Binary and JSON round trips cover inherited and overridden metadata,
schema-only and field-only metadata, extension metadata, explicit
clearing, metadata-free projections, and older payloads.
Child-independent reconstruction and field-count validation remain
covered.
- Reproduced the Arrow 60 metadata type-inference compilation failure
before the test fix. All 272 protobuf integration tests pass with the
fix applied to the current-main integration.
- The exact PR branch passes formatting, all-target/all-feature Clippy
with warnings denied, and `./dev/rust_lint.sh`.
- The exact PR branch passes the extended workspace suite and all 518
SQL logic-test files.
## Are there any user-facing changes?
Projection schema and output-field metadata survive physical-plan
serialization. Nested field definitions remain part of the
expression-derived data types. The generated `ProjectionExecNode` Rust
struct gains an optional `schema` field, affecting exhaustive struct
literals. New readers accept older payloads. This targets main, not a
55.1 backport.
---------
Signed-off-by: goutamadwant <[email protected]>1 parent 3ed377a commit 5c0e73f
5 files changed
Lines changed: 350 additions & 4 deletions
File tree
- datafusion
- physical-plan/src
- proto-models
- proto
- src/generated
- proto/tests/cases/plans
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
652 | 652 | | |
653 | 653 | | |
654 | 654 | | |
655 | | - | |
656 | | - | |
| 655 | + | |
657 | 656 | | |
658 | 657 | | |
659 | 658 | | |
660 | 659 | | |
661 | 660 | | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
662 | 674 | | |
663 | 675 | | |
664 | 676 | | |
665 | 677 | | |
666 | 678 | | |
667 | 679 | | |
668 | 680 | | |
| 681 | + | |
669 | 682 | | |
670 | 683 | | |
671 | 684 | | |
| |||
701 | 714 | | |
702 | 715 | | |
703 | 716 | | |
| 717 | + | |
704 | 718 | | |
705 | 719 | | |
706 | 720 | | |
| |||
715 | 729 | | |
716 | 730 | | |
717 | 731 | | |
718 | | - | |
| 732 | + | |
| 733 | + | |
| 734 | + | |
| 735 | + | |
| 736 | + | |
| 737 | + | |
| 738 | + | |
| 739 | + | |
| 740 | + | |
719 | 741 | | |
720 | 742 | | |
721 | 743 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1526 | 1526 | | |
1527 | 1527 | | |
1528 | 1528 | | |
| 1529 | + | |
| 1530 | + | |
| 1531 | + | |
| 1532 | + | |
| 1533 | + | |
1529 | 1534 | | |
1530 | 1535 | | |
1531 | 1536 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24658 | 24658 | | |
24659 | 24659 | | |
24660 | 24660 | | |
| 24661 | + | |
| 24662 | + | |
| 24663 | + | |
24661 | 24664 | | |
24662 | 24665 | | |
24663 | 24666 | | |
| |||
24668 | 24671 | | |
24669 | 24672 | | |
24670 | 24673 | | |
| 24674 | + | |
| 24675 | + | |
| 24676 | + | |
24671 | 24677 | | |
24672 | 24678 | | |
24673 | 24679 | | |
| |||
24682 | 24688 | | |
24683 | 24689 | | |
24684 | 24690 | | |
| 24691 | + | |
24685 | 24692 | | |
24686 | 24693 | | |
24687 | 24694 | | |
24688 | 24695 | | |
24689 | 24696 | | |
24690 | 24697 | | |
24691 | 24698 | | |
| 24699 | + | |
24692 | 24700 | | |
24693 | 24701 | | |
24694 | 24702 | | |
| |||
24713 | 24721 | | |
24714 | 24722 | | |
24715 | 24723 | | |
| 24724 | + | |
24716 | 24725 | | |
24717 | 24726 | | |
24718 | 24727 | | |
| |||
24735 | 24744 | | |
24736 | 24745 | | |
24737 | 24746 | | |
| 24747 | + | |
24738 | 24748 | | |
24739 | 24749 | | |
24740 | 24750 | | |
| |||
24755 | 24765 | | |
24756 | 24766 | | |
24757 | 24767 | | |
| 24768 | + | |
| 24769 | + | |
| 24770 | + | |
| 24771 | + | |
| 24772 | + | |
| 24773 | + | |
24758 | 24774 | | |
24759 | 24775 | | |
24760 | 24776 | | |
24761 | 24777 | | |
24762 | 24778 | | |
24763 | 24779 | | |
| 24780 | + | |
24764 | 24781 | | |
24765 | 24782 | | |
24766 | 24783 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2302 | 2302 | | |
2303 | 2303 | | |
2304 | 2304 | | |
| 2305 | + | |
| 2306 | + | |
| 2307 | + | |
| 2308 | + | |
| 2309 | + | |
| 2310 | + | |
2305 | 2311 | | |
2306 | 2312 | | |
2307 | 2313 | | |
| |||
0 commit comments