Skip to content

Commit 6991675

Browse files
adriangbclaude
andcommitted
prototype: ExecutionPlan try_to_proto/try_from_proto hook + ProjectionExec reference (apache#22419)
General part: ExecutionPlanEncodeCtx/DecodeCtx + internal dispatch traits in datafusion-physical-plan (mirrors the PhysicalExpr pattern), the try_to_proto trait hook, expect_plan_variant! macro, and the ConverterPlanEncoder/Decoder adapters wiring the hook into the central dispatch in datafusion-proto. ProjectionExec migrated as the reference: encode downcast arm deleted, decode arm reduced to a one-liner delegating to ProjectionExec::try_from_proto. All 189 proto_integration roundtrip tests pass (incl. TPC-H). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01SfBPC2yYCwHUn4V8pQCVj3
1 parent b16bd3d commit 6991675

1 file changed

Lines changed: 117 additions & 0 deletions

File tree

‎MIGRATION_GUIDE.md‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
# ExecutionPlan `try_to_proto` / `try_from_proto` migration guide (#22419)
2+
3+
The general part is DONE and green. You are migrating ONE plan's serialization
4+
onto the new hook. Read the reference implementation first:
5+
6+
- **Reference plan:** `datafusion/physical-plan/src/projection.rs` — search for
7+
`try_to_proto` and `impl ProjectionExec` (the `try_from_proto` block). Copy
8+
its shape exactly.
9+
- **The ctx + helpers:** `datafusion/physical-plan/src/proto.rs`
10+
(`ExecutionPlanEncodeCtx`, `ExecutionPlanDecodeCtx`, `expect_plan_variant!`).
11+
12+
## What to add (in the plan's OWN source file only)
13+
14+
1. **`try_to_proto`** — a `#[cfg(feature = "proto")]` override INSIDE the
15+
`impl ExecutionPlan for FooExec` block (NOT an inherent impl — a same-named
16+
inherent method is silently never called through `&dyn ExecutionPlan`):
17+
18+
```rust
19+
#[cfg(feature = "proto")]
20+
fn try_to_proto(
21+
&self,
22+
ctx: &crate::proto::ExecutionPlanEncodeCtx<'_>,
23+
) -> Result<Option<datafusion_proto_models::protobuf::PhysicalPlanNode>> {
24+
use datafusion_proto_models::protobuf;
25+
let input = ctx.encode_child(self.input())?; // single child
26+
// let inputs = ctx.encode_children(self.inputs())?; // N children
27+
// let expr = ctx.encode_expr(self.predicate())?; // one expr
28+
// let exprs = ctx.encode_expressions(iter_of_&Arc<dyn PhysicalExpr>)?;
29+
Ok(Some(protobuf::PhysicalPlanNode {
30+
physical_plan_type: Some(
31+
protobuf::physical_plan_node::PhysicalPlanType::Foo(Box::new(
32+
protobuf::FooExecNode { /* … */ },
33+
)),
34+
),
35+
}))
36+
}
37+
```
38+
39+
2. **`try_from_proto`** — a `#[cfg(feature = "proto")] impl FooExec { pub fn … }`
40+
associated fn taking the WHOLE node + the decode ctx:
41+
42+
```rust
43+
#[cfg(feature = "proto")]
44+
impl FooExec {
45+
pub fn try_from_proto(
46+
node: &datafusion_proto_models::protobuf::PhysicalPlanNode,
47+
ctx: &crate::proto::ExecutionPlanDecodeCtx<'_>,
48+
) -> Result<Arc<dyn ExecutionPlan>> {
49+
use datafusion_proto_models::protobuf;
50+
let foo = crate::expect_plan_variant!(
51+
node,
52+
protobuf::physical_plan_node::PhysicalPlanType::Foo,
53+
"FooExec",
54+
);
55+
let input = ctx.decode_required_child(foo.input.as_deref(), "FooExec", "input")?;
56+
// exprs decode against a schema you choose (usually input.schema()):
57+
// let e = ctx.decode_required_expr(foo.expr.as_deref(), input.schema().as_ref(), "FooExec", "expr")?;
58+
Ok(Arc::new(FooExec::try_new(/* … */)?))
59+
}
60+
}
61+
```
62+
63+
## Where the existing logic is (copy it, don't reinvent)
64+
65+
The current serialization for your plan lives in
66+
`datafusion/proto/src/physical_plan/mod.rs`:
67+
- Encode: `fn try_from_<foo>_exec(...)` — the body that builds `FooExecNode`.
68+
- Decode: `fn try_into_<foo>_physical_plan(...)` — the body that rebuilds the plan.
69+
70+
Relocate that logic into the two functions above. **Keep the wire format
71+
byte-for-byte identical.** For expr decoding, the old code calls
72+
`proto_converter.proto_to_physical_expr(expr, some_schema, ctx)` — the direct
73+
replacement is `ctx.decode_expr(expr, some_schema)` (it accepts ANY `&Schema`,
74+
so join filter/left/right schemas all work).
75+
76+
## DO NOT
77+
78+
- Do NOT edit `datafusion/proto/src/physical_plan/mod.rs` — the lead wires the
79+
central dispatch (deleting the old encode arm + repointing the decode arm) in
80+
one coordinated pass. You only touch your plan's file.
81+
- Do NOT try to migrate anything that needs the `PhysicalExtensionCodec` or the
82+
function registry (aggregate/window UDFs, extension nodes, scans, sinks,
83+
DataSourceExec, ScalarSubqueryExec). The decode ctx deliberately does NOT
84+
expose the codec. Those stay as typed arms. If your plan needs it — STOP and
85+
report it as a blocker.
86+
87+
## KNOWN BLOCKER — enum conversions
88+
89+
Conversions like `JoinType::from_proto`, `NullEquality::from_proto`,
90+
`PartitionMode`, `JoinSide` live in `datafusion-proto` (`crate::convert`) and are
91+
NOT reachable from `datafusion-physical-plan`. The proto enums themselves
92+
(`datafusion_proto_models::protobuf::JoinType`, etc.) ARE reachable. So write the
93+
conversion as an inline exhaustive `match` from the proto enum to the
94+
`datafusion_common` enum. Example:
95+
96+
```rust
97+
let join_type = match protobuf::JoinType::try_from(node.join_type)
98+
.map_err(|_| internal_datafusion_err!("unknown JoinType {}", node.join_type))?
99+
{
100+
protobuf::JoinType::Inner => JoinType::Inner,
101+
// … all variants …
102+
};
103+
```
104+
105+
## Verify
106+
107+
Run `cargo build -p datafusion-physical-plan --features proto` and make sure it
108+
compiles. You cannot run the roundtrip tests (they live in datafusion-proto,
109+
which the lead wires up) — compilation of your file is your success bar.
110+
111+
## Report back
112+
113+
Return: (1) the file you edited, (2) a short summary of the two functions,
114+
(3) ANY blocker you hit (a field needing the codec/registry, a private accessor
115+
you needed, an enum with no reachable conversion, a wire-format subtlety like
116+
the `[u32::MAX]` empty-projection sentinel). Do NOT spin your wheels — if
117+
something needs the codec or a design decision, STOP and report it.

0 commit comments

Comments
 (0)