|
| 1 | +# ExecutionPlan `try_to_proto` / `try_from_proto` migration guide (#22419) |
| 2 | + |
| 3 | +The general part is DONE and green. You are migrating ONE plan's serialization |
| 4 | +onto the new hook. Read the reference implementation first: |
| 5 | + |
| 6 | +- **Reference plan:** `datafusion/physical-plan/src/projection.rs` — search for |
| 7 | + `try_to_proto` and `impl ProjectionExec` (the `try_from_proto` block). Copy |
| 8 | + its shape exactly. |
| 9 | +- **The ctx + helpers:** `datafusion/physical-plan/src/proto.rs` |
| 10 | + (`ExecutionPlanEncodeCtx`, `ExecutionPlanDecodeCtx`, `expect_plan_variant!`). |
| 11 | + |
| 12 | +## What to add (in the plan's OWN source file only) |
| 13 | + |
| 14 | +1. **`try_to_proto`** — a `#[cfg(feature = "proto")]` override INSIDE the |
| 15 | + `impl ExecutionPlan for FooExec` block (NOT an inherent impl — a same-named |
| 16 | + inherent method is silently never called through `&dyn ExecutionPlan`): |
| 17 | + |
| 18 | + ```rust |
| 19 | + #[cfg(feature = "proto")] |
| 20 | + fn try_to_proto( |
| 21 | + &self, |
| 22 | + ctx: &crate::proto::ExecutionPlanEncodeCtx<'_>, |
| 23 | + ) -> Result<Option<datafusion_proto_models::protobuf::PhysicalPlanNode>> { |
| 24 | + use datafusion_proto_models::protobuf; |
| 25 | + let input = ctx.encode_child(self.input())?; // single child |
| 26 | + // let inputs = ctx.encode_children(self.inputs())?; // N children |
| 27 | + // let expr = ctx.encode_expr(self.predicate())?; // one expr |
| 28 | + // let exprs = ctx.encode_expressions(iter_of_&Arc<dyn PhysicalExpr>)?; |
| 29 | + Ok(Some(protobuf::PhysicalPlanNode { |
| 30 | + physical_plan_type: Some( |
| 31 | + protobuf::physical_plan_node::PhysicalPlanType::Foo(Box::new( |
| 32 | + protobuf::FooExecNode { /* … */ }, |
| 33 | + )), |
| 34 | + ), |
| 35 | + })) |
| 36 | + } |
| 37 | + ``` |
| 38 | + |
| 39 | +2. **`try_from_proto`** — a `#[cfg(feature = "proto")] impl FooExec { pub fn … }` |
| 40 | + associated fn taking the WHOLE node + the decode ctx: |
| 41 | + |
| 42 | + ```rust |
| 43 | + #[cfg(feature = "proto")] |
| 44 | + impl FooExec { |
| 45 | + pub fn try_from_proto( |
| 46 | + node: &datafusion_proto_models::protobuf::PhysicalPlanNode, |
| 47 | + ctx: &crate::proto::ExecutionPlanDecodeCtx<'_>, |
| 48 | + ) -> Result<Arc<dyn ExecutionPlan>> { |
| 49 | + use datafusion_proto_models::protobuf; |
| 50 | + let foo = crate::expect_plan_variant!( |
| 51 | + node, |
| 52 | + protobuf::physical_plan_node::PhysicalPlanType::Foo, |
| 53 | + "FooExec", |
| 54 | + ); |
| 55 | + let input = ctx.decode_required_child(foo.input.as_deref(), "FooExec", "input")?; |
| 56 | + // exprs decode against a schema you choose (usually input.schema()): |
| 57 | + // let e = ctx.decode_required_expr(foo.expr.as_deref(), input.schema().as_ref(), "FooExec", "expr")?; |
| 58 | + Ok(Arc::new(FooExec::try_new(/* … */)?)) |
| 59 | + } |
| 60 | + } |
| 61 | + ``` |
| 62 | + |
| 63 | +## Where the existing logic is (copy it, don't reinvent) |
| 64 | + |
| 65 | +The current serialization for your plan lives in |
| 66 | +`datafusion/proto/src/physical_plan/mod.rs`: |
| 67 | +- Encode: `fn try_from_<foo>_exec(...)` — the body that builds `FooExecNode`. |
| 68 | +- Decode: `fn try_into_<foo>_physical_plan(...)` — the body that rebuilds the plan. |
| 69 | + |
| 70 | +Relocate that logic into the two functions above. **Keep the wire format |
| 71 | +byte-for-byte identical.** For expr decoding, the old code calls |
| 72 | +`proto_converter.proto_to_physical_expr(expr, some_schema, ctx)` — the direct |
| 73 | +replacement is `ctx.decode_expr(expr, some_schema)` (it accepts ANY `&Schema`, |
| 74 | +so join filter/left/right schemas all work). |
| 75 | + |
| 76 | +## DO NOT |
| 77 | + |
| 78 | +- Do NOT edit `datafusion/proto/src/physical_plan/mod.rs` — the lead wires the |
| 79 | + central dispatch (deleting the old encode arm + repointing the decode arm) in |
| 80 | + one coordinated pass. You only touch your plan's file. |
| 81 | +- Do NOT try to migrate anything that needs the `PhysicalExtensionCodec` or the |
| 82 | + function registry (aggregate/window UDFs, extension nodes, scans, sinks, |
| 83 | + DataSourceExec, ScalarSubqueryExec). The decode ctx deliberately does NOT |
| 84 | + expose the codec. Those stay as typed arms. If your plan needs it — STOP and |
| 85 | + report it as a blocker. |
| 86 | + |
| 87 | +## KNOWN BLOCKER — enum conversions |
| 88 | + |
| 89 | +Conversions like `JoinType::from_proto`, `NullEquality::from_proto`, |
| 90 | +`PartitionMode`, `JoinSide` live in `datafusion-proto` (`crate::convert`) and are |
| 91 | +NOT reachable from `datafusion-physical-plan`. The proto enums themselves |
| 92 | +(`datafusion_proto_models::protobuf::JoinType`, etc.) ARE reachable. So write the |
| 93 | +conversion as an inline exhaustive `match` from the proto enum to the |
| 94 | +`datafusion_common` enum. Example: |
| 95 | + |
| 96 | +```rust |
| 97 | +let join_type = match protobuf::JoinType::try_from(node.join_type) |
| 98 | + .map_err(|_| internal_datafusion_err!("unknown JoinType {}", node.join_type))? |
| 99 | +{ |
| 100 | + protobuf::JoinType::Inner => JoinType::Inner, |
| 101 | + // … all variants … |
| 102 | +}; |
| 103 | +``` |
| 104 | + |
| 105 | +## Verify |
| 106 | + |
| 107 | +Run `cargo build -p datafusion-physical-plan --features proto` and make sure it |
| 108 | +compiles. You cannot run the roundtrip tests (they live in datafusion-proto, |
| 109 | +which the lead wires up) — compilation of your file is your success bar. |
| 110 | + |
| 111 | +## Report back |
| 112 | + |
| 113 | +Return: (1) the file you edited, (2) a short summary of the two functions, |
| 114 | +(3) ANY blocker you hit (a field needing the codec/registry, a private accessor |
| 115 | +you needed, an enum with no reachable conversion, a wire-format subtlety like |
| 116 | +the `[u32::MAX]` empty-projection sentinel). Do NOT spin your wheels — if |
| 117 | +something needs the codec or a design decision, STOP and report it. |
0 commit comments