Repository navigation
Commit b16bd3d
Add ExecutionPlan try_to_proto / try_from_proto hooks + ProjectionExec reference (apache#23495)
## Which issue does this PR close?
- Part of apache#23494.
## Rationale for this change
This is the foundation for porting `ExecutionPlan` serialization off the
central `downcast_ref` chain in
`datafusion/proto/src/physical_plan/mod.rs`, mirroring what apache#21929 did
for `PhysicalExpr` (tracked in apache#22418). Everything else in apache#23494
depends on this PR; it can't be parallelized because it defines the ctx
API and wiring that every per-plan follow-up uses.
## What changes are included in this PR?
In `datafusion-physical-plan` (feature-gated `#[cfg(feature =
"proto")]`):
- `ExecutionPlan::try_to_proto(&self, ctx: &ExecutionPlanEncodeCtx) ->
Result<Option<PhysicalPlanNode>>` — a new trait method defaulting to
`Ok(None)` (so un-migrated plans are unchanged).
- `ExecutionPlanEncodeCtx` / `ExecutionPlanDecodeCtx` plus the internal
`ExecutionPlanEncode` / `ExecutionPlanDecode` dispatch traits (defined
here, implemented in `datafusion-proto`) — the same dependency-inversion
the `PhysicalExpr` ctx uses. The ctx exposes
`encode_child`/`decode_child` (+ `_children`),
`encode_expr`/`decode_expr`, and typed **bytes-only**
`encode_udaf`/`decode_udaf` (+ udf/udwf) for function-carrying plans. No
proto type is ever named by physical-plan beyond the pure prost types in
`datafusion-proto-models`.
- An `expect_plan_variant!` macro mirroring `expect_expr_variant!`.
In `datafusion-proto`:
- `ConverterPlanEncoder` / `ConverterPlanDecoder` implement the dispatch
traits over the existing `PhysicalExtensionCodec` + converter (the
function serde reuses today's `fun_definition` byte semantics exactly).
The encode dispatch resolves `downcast_delegate()` first (so wrapper
plans serialize as their delegate, exactly like the `downcast_ref` chain
sees them), then calls the hook, then falls back to the existing
downcast chain.
`ProjectionExec` is migrated as the reference implementation: its old
encode arm is deleted and its decode arm reduced to
`ProjectionExec::try_from_proto`, so a green roundtrip proves the hook
is the only path.
## Are these changes tested?
Yes — the existing `proto_integration` roundtrip suite (including TPC-H)
passes unchanged, which exercises the `ProjectionExec` migration
end-to-end, plus a new test asserting that a wrapper plan delegating its
downcast identity to `ProjectionExec` still serializes as a projection
through the hook (it fails without the `downcast_delegate()`
resolution). The bytes-only function serde and the required-field decode
helpers — which have no in-tree plan caller until the function-carrying
plans migrate — are pinned by dedicated unit tests (payload semantics,
decode lookup order, error paths), addressing the patch-coverage gap.
`cargo clippy --all-targets -- -D warnings`, `cargo fmt`, and
`RUSTDOCFLAGS="-D warnings" cargo doc` are clean.
## Are there any user-facing changes?
New public API: the `try_to_proto` trait method (with a default, so
existing `ExecutionPlan` impls are unaffected) and the ctx types. No
wire-format change.
No API is removed: per the [API health
policy](https://datafusion.apache.org/contributor-guide/api-health.html),
the two per-plan `PhysicalPlanNodeExt` methods this migration replaces
(`try_into_projection_physical_plan` / `try_from_projection_exec`) are
kept with their original bodies under `#[deprecated(since = "55.0.0")]`.
(They were also never released: `PhysicalPlanNodeExt` was introduced by
apache#21929 after 54.0.0 shipped.) The per-plan follow-ups in apache#23494 will
deprecate their corresponding methods the same way, with removal after
the policy window.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>1 parent d416011 commit b16bd3d
6 files changed
Lines changed: 1008 additions & 11 deletions
File tree
- datafusion
- physical-plan/src
- proto
- src/physical_plan
- tests/cases
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
823 | 823 | | |
824 | 824 | | |
825 | 825 | | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
| 844 | + | |
| 845 | + | |
| 846 | + | |
826 | 847 | | |
827 | 848 | | |
828 | 849 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
88 | 88 | | |
89 | 89 | | |
90 | 90 | | |
| 91 | + | |
| 92 | + | |
91 | 93 | | |
92 | 94 | | |
93 | 95 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
500 | 500 | | |
501 | 501 | | |
502 | 502 | | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
503 | 568 | | |
504 | 569 | | |
505 | 570 | | |
| |||
0 commit comments