Skip to content

Commit ba61dab

Browse files
adriangbclaude
andcommitted
docs: document the extension ExecutionPlan decode registry
Adds the 55.0.0 upgrade-guide entry for the new opt-in path: what the codec-as- discriminator problem was, how a plan names and registers itself, and the one thing that is not optional — `PhysicalExtensionNode` gained a `plan_name` field, so exhaustive struct literals must set it. Part of apache#24625. Co-Authored-By: Claude Opus 5 <[email protected]>
1 parent 585e597 commit ba61dab

1 file changed

Lines changed: 80 additions & 0 deletions

File tree

  • docs/source/library-user-guide/upgrading

‎docs/source/library-user-guide/upgrading/55.0.0.md‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1493,3 +1493,83 @@ gained support for those bounds in this release, having previously failed with
14931493
use `UNBOUNDED PRECEDING` or `UNBOUNDED FOLLOWING`.
14941494

14951495
See [issue #24327](https://github.com/apache/datafusion/issues/24327) for details.
1496+
1497+
### Extension `ExecutionPlan`s can decode without a `PhysicalExtensionCodec`
1498+
1499+
`ExecutionPlan::try_to_proto` let a plan serialize itself, but there was no way
1500+
back: decoding an extension node always routed through
1501+
`PhysicalExtensionCodec::try_decode`. `PhysicalExtensionNode` carried no type
1502+
discriminator, so the codec _was_ the discriminator —
1503+
`ComposedPhysicalExtensionCodec` had to try each registered codec in turn and
1504+
read a decode error as "not mine", which made resolution depend on registration
1505+
order and on error strings, and made a name collision between two independent
1506+
crates undetectable.
1507+
1508+
`PhysicalExtensionNode` now has an optional `plan_name`, and a session can map
1509+
that name to a decoder. An extension plan implements `ExtensionExecutionPlan`
1510+
and is registered on the session that will decode it:
1511+
1512+
```rust,ignore
1513+
use datafusion::physical_plan::proto::{
1514+
ExecutionPlanDecodeCtx, ExecutionPlanEncodeCtx, ExecutionPlanRegistryExt,
1515+
ExtensionExecutionPlan,
1516+
};
1517+
1518+
impl ExecutionPlan for MyExec {
1519+
// ...
1520+
fn try_to_proto(
1521+
&self,
1522+
ctx: &ExecutionPlanEncodeCtx<'_>,
1523+
) -> Result<Option<PhysicalPlanNode>> {
1524+
// The name comes from `Self::PLAN_NAME`, so it cannot drift from the
1525+
// name you register.
1526+
Ok(Some(ctx.encode_extension::<Self, _>(
1527+
my_payload_bytes(self)?,
1528+
self.children(),
1529+
)?))
1530+
}
1531+
}
1532+
1533+
impl ExtensionExecutionPlan for MyExec {
1534+
// Namespace the name so a collision with another crate is an error at
1535+
// registration rather than a mis-decode.
1536+
const PLAN_NAME: &'static str = "my-crate.MyExec";
1537+
1538+
fn try_from_proto(
1539+
node: &PhysicalPlanNode,
1540+
ctx: &ExecutionPlanDecodeCtx<'_>,
1541+
) -> Result<Arc<dyn ExecutionPlan>> {
1542+
let (payload, children) = ctx.decode_extension(node, Self::PLAN_NAME)?;
1543+
// `ctx.task_ctx()` is available here, so a plan that rebuilds session
1544+
// state at decode time (a connection pool, say) can do so.
1545+
my_plan_from_bytes(payload, children)
1546+
}
1547+
}
1548+
1549+
// On every session that decodes the plan — for a distributed engine, the
1550+
// workers as well as the coordinator.
1551+
let mut config = SessionConfig::new();
1552+
config.register_execution_plan::<MyExec>()?;
1553+
```
1554+
1555+
A registered plan needs no `PhysicalExtensionCodec` at all, and two crates'
1556+
plans coexist on one session without a `ComposedPhysicalExtensionCodec`.
1557+
1558+
**Who is affected:**
1559+
1560+
- Nobody is required to change anything. This is opt-in per plan type: a
1561+
`PhysicalExtensionNode` with no name, or with a name the decoding session does
1562+
not know, takes the `PhysicalExtensionCodec` chain exactly as before, and the
1563+
wire format stays compatible in both directions.
1564+
- Users constructing `PhysicalExtensionNode` with an exhaustive struct literal
1565+
must set the new `plan_name` field (`None` reproduces the previous behavior).
1566+
- Replacing a `PhysicalExtensionCodec` with a registration changes only _how_
1567+
the payload is dispatched, not what the payload is. Readers running an older
1568+
DataFusion ignore `plan_name` and hand the payload to their codec, so a mixed
1569+
fleet keeps working only while the bytes your `try_to_proto` writes stay
1570+
decodable by the codec you removed.
1571+
1572+
`PhysicalExtensionCodec` is not deprecated: extension `PhysicalExpr`s and UDF
1573+
payloads still need it, and it remains the fallback for unmigrated plans.
1574+
1575+
See [issue #24625](https://github.com/apache/datafusion/issues/24625) for details.

0 commit comments

Comments
 (0)