Skip to content

Commit 2fb65e9

Browse files
adriangbclaude
andcommitted
refactor: make the decoder storage private and ExtensionPlanParts a struct
Addresses review feedback on apache#24631. `ExtensionPlanParts` was a public tuple alias; it is now a struct with named `payload` / `children` fields. `ExecutionPlanDecoder` was a public `fn`-pointer alias — a trait with one method spelled as a type. Its doc claimed `try_from_proto` is an inherent function "so it can return `Arc<dyn ExecutionPlan>`", which is simply wrong: `ExecutionPlan::with_new_children` returns exactly that from an object-safe trait. The real constraint is that `try_from_proto` is a *constructor* — a receiver-less associated function has no `self` to dispatch on, so it cannot be reached through a trait object, and a fn pointer is what a registry can hold. So the fn pointer stays, but as a private storage detail: * `ExecutionPlanDecoder` is no longer public, and its doc now states the actual reason. * `ExecutionPlanRegistry::decoder` (which handed the pointer out) is replaced by `decode(name, node, ctx) -> Option<Result<..>>`, which also makes the "not mine" / "mine and it failed" distinction explicit at the type level, and by `contains(name)`. * `register_decoder` and its partner `encode_extension_named` are dropped. They existed only to register a decoder that is not a single type's constant, which no caller needs yet, and they were the only reason the decoder type had to be public. Re-adding them later — over a `dyn` decoder object, which would also admit stateful and closure decoders — is additive. `ExtensionExecutionPlan` is now the whole public contract for authoring one, and the registry's storage can change without a breaking change. `RegisteredPlan` loses its `Option<TypeId>`, since every entry now has a type. Test coverage moves with the API: the by-name-alias dispatch test goes away with the feature, and the empty-name check now runs through `register::<T>()` with a plan whose `PLAN_NAME` is empty. Part of apache#24625. Co-Authored-By: Claude Opus 5 <[email protected]>
1 parent 64d15ce commit 2fb65e9

6 files changed

Lines changed: 91 additions & 156 deletions

File tree

‎datafusion/physical-plan/src/proto/mod.rs‎

Lines changed: 19 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -93,8 +93,7 @@ use datafusion_proto_models::protobuf::{
9393
use crate::ExecutionPlan;
9494

9595
pub use registry::{
96-
ExecutionPlanDecoder, ExecutionPlanRegistry, ExecutionPlanRegistryExt,
97-
ExtensionExecutionPlan,
96+
ExecutionPlanRegistry, ExecutionPlanRegistryExt, ExtensionExecutionPlan,
9897
};
9998

10099
/// Internal dispatch trait backing [`ExecutionPlanEncodeCtx`].
@@ -176,11 +175,15 @@ pub trait ExecutionPlanDecode {
176175
fn decode_udwf(&self, name: &str, payload: Option<&[u8]>) -> Result<Arc<WindowUDF>>;
177176
}
178177

179-
/// The wire parts of an extension plan: its opaque payload, borrowed from the
180-
/// node it was read from, and its already-decoded children.
181-
///
182-
/// Returned by [`ExecutionPlanDecodeCtx::decode_extension`].
183-
pub type ExtensionPlanParts<'n> = (&'n [u8], Vec<Arc<dyn ExecutionPlan>>);
178+
/// The wire parts of an extension plan, as returned by
179+
/// [`ExecutionPlanDecodeCtx::decode_extension`].
180+
#[derive(Debug)]
181+
pub struct ExtensionPlanParts<'n> {
182+
/// The plan's opaque payload, borrowed from the node it was read from.
183+
pub payload: &'n [u8],
184+
/// The plan's children, already decoded.
185+
pub children: Vec<Arc<dyn ExecutionPlan>>,
186+
}
184187

185188
/// Context handed to [`ExecutionPlan::try_to_proto`].
186189
///
@@ -253,9 +256,9 @@ impl<'a> ExecutionPlanEncodeCtx<'a> {
253256
/// single `PhysicalExtensionNode` variant and carry their own bytes.
254257
///
255258
/// Taking the name from `T` rather than as a string is what keeps the
256-
/// written name and the registered name from drifting: a mismatch would
257-
/// compile, then silently fall back to the `PhysicalExtensionCodec` on
258-
/// whichever machine does the decoding.
259+
/// written name and the registered name from drifting: a string mismatch
260+
/// would compile, then silently fall back to the `PhysicalExtensionCodec`
261+
/// on whichever machine does the decoding.
259262
///
260263
/// The name is the wire discriminator. Register `T` on the decoding session
261264
/// (see [`registry`]) and decode selects the plan's own `try_from_proto` by
@@ -272,31 +275,13 @@ impl<'a> ExecutionPlanEncodeCtx<'a> {
272275
where
273276
T: ExtensionExecutionPlan,
274277
I: IntoIterator<Item = &'b Arc<dyn ExecutionPlan>>,
275-
{
276-
self.encode_extension_named(T::PLAN_NAME, payload, children)
277-
}
278-
279-
/// [`Self::encode_extension`] with the name supplied explicitly.
280-
///
281-
/// For plans registered through
282-
/// [`ExecutionPlanRegistry::register_decoder`](registry::ExecutionPlanRegistry::register_decoder),
283-
/// whose name is not a compile-time constant of a single type. Prefer
284-
/// [`Self::encode_extension`], which cannot disagree with the registration.
285-
pub fn encode_extension_named<'b, I>(
286-
&self,
287-
plan_name: &str,
288-
payload: Vec<u8>,
289-
children: I,
290-
) -> Result<PhysicalPlanNode>
291-
where
292-
I: IntoIterator<Item = &'b Arc<dyn ExecutionPlan>>,
293278
{
294279
Ok(PhysicalPlanNode {
295280
physical_plan_type: Some(PhysicalPlanType::Extension(
296281
PhysicalExtensionNode {
297282
node: payload,
298283
inputs: self.encode_children(children)?,
299-
plan_name: Some(plan_name.to_string()),
284+
plan_name: Some(T::PLAN_NAME.to_string()),
300285
},
301286
)),
302287
})
@@ -454,10 +439,10 @@ impl<'a> ExecutionPlanDecodeCtx<'a> {
454439
"PhysicalPlanNode is not an extension node ({plan_name})"
455440
);
456441
};
457-
Ok((
458-
extension.node.as_slice(),
459-
self.decode_children(&extension.inputs)?,
460-
))
442+
Ok(ExtensionPlanParts {
443+
payload: extension.node.as_slice(),
444+
children: self.decode_children(&extension.inputs)?,
445+
})
461446
}
462447

463448
/// An expression-level decode context backed by this plan context, bound to
@@ -566,7 +551,7 @@ mod tests {
566551
let decoder = StubPlanDecoder::ok();
567552
let ctx = ExecutionPlanDecodeCtx::new(&decoder);
568553

569-
let (payload, children) = ctx
554+
let ExtensionPlanParts { payload, children } = ctx
570555
.decode_extension(&node, RegisteredExec::PLAN_NAME)
571556
.expect("decode must succeed");
572557

‎datafusion/physical-plan/src/proto/registry.rs‎

Lines changed: 48 additions & 70 deletions
Original file line numberDiff line numberDiff line change
@@ -55,14 +55,21 @@ use datafusion_proto_models::protobuf::PhysicalPlanNode;
5555
use crate::ExecutionPlan;
5656
use crate::proto::ExecutionPlanDecodeCtx;
5757

58-
/// The decode half of an extension plan's serialization, as a plain function
59-
/// pointer.
58+
/// How the registry stores a decoder internally: a function pointer to the
59+
/// monomorphized [`ExtensionExecutionPlan::try_from_proto`].
6060
///
61-
/// This is exactly the signature every migrated plan's `try_from_proto`
62-
/// associated function already has. `try_from_proto` is an inherent function
63-
/// rather than a trait method precisely so it can return
64-
/// `Arc<dyn ExecutionPlan>` and be stored like this.
65-
pub type ExecutionPlanDecoder =
61+
/// Deliberately private. [`ExtensionExecutionPlan`] is the public contract, and
62+
/// [`ExecutionPlanRegistry::decode`] is the public way to invoke one, so the
63+
/// storage can become something else (a `dyn` decoder object, to admit
64+
/// stateful or closure decoders) without a breaking change.
65+
///
66+
/// Note that returning `Arc<dyn ExecutionPlan>` is *not* what keeps
67+
/// `try_from_proto` off `dyn` dispatch — `ExecutionPlan::with_new_children`
68+
/// returns exactly that from an object-safe trait. It is that `try_from_proto`
69+
/// is a constructor: a receiver-less associated function has no `self` to
70+
/// dispatch on, so it cannot be called through a trait object and a fn pointer
71+
/// is what a registry can hold.
72+
type ExecutionPlanDecoder =
6673
fn(&PhysicalPlanNode, &ExecutionPlanDecodeCtx<'_>) -> Result<Arc<dyn ExecutionPlan>>;
6774

6875
/// An extension [`ExecutionPlan`] that serializes itself, without a
@@ -107,7 +114,7 @@ pub type ExecutionPlanDecoder =
107114
/// node: &PhysicalPlanNode,
108115
/// ctx: &ExecutionPlanDecodeCtx<'_>,
109116
/// ) -> Result<Arc<dyn ExecutionPlan>> {
110-
/// let (_payload, _children) = ctx.decode_extension(node, Self::PLAN_NAME)?;
117+
/// let parts = ctx.decode_extension(node, Self::PLAN_NAME)?;
111118
/// // ... rebuild `MyExec` from the payload and children ...
112119
/// # unimplemented!()
113120
/// }
@@ -143,9 +150,7 @@ pub trait ExtensionExecutionPlan: ExecutionPlan + Sized {
143150
#[derive(Debug, Clone, Copy)]
144151
struct RegisteredPlan {
145152
decoder: ExecutionPlanDecoder,
146-
/// `None` for decoders registered by name via
147-
/// [`ExecutionPlanRegistry::register_decoder`], which carry no Rust type.
148-
type_id: Option<TypeId>,
153+
type_id: TypeId,
149154
type_name: &'static str,
150155
}
151156

@@ -175,35 +180,12 @@ impl ExecutionPlanRegistry {
175180
T::PLAN_NAME,
176181
RegisteredPlan {
177182
decoder: T::try_from_proto,
178-
type_id: Some(TypeId::of::<T>()),
183+
type_id: TypeId::of::<T>(),
179184
type_name: type_name::<T>(),
180185
},
181186
)
182187
}
183188

184-
/// Register a decoder under an explicit `name`.
185-
///
186-
/// Escape hatch for plans whose name is not a compile-time constant (for
187-
/// example a generic plan registered once per instantiation, or a legacy
188-
/// alias kept alongside the current name). Prefer [`Self::register`].
189-
///
190-
/// Unlike [`Self::register`] this cannot tell a repeat registration from a
191-
/// collision, so *any* duplicate name is an error.
192-
pub fn register_decoder(
193-
&mut self,
194-
name: impl Into<String>,
195-
decoder: ExecutionPlanDecoder,
196-
) -> Result<()> {
197-
self.insert(
198-
name,
199-
RegisteredPlan {
200-
decoder,
201-
type_id: None,
202-
type_name: "<decoder registered by name>",
203-
},
204-
)
205-
}
206-
207189
fn insert(&mut self, name: impl Into<String>, plan: RegisteredPlan) -> Result<()> {
208190
let name = name.into();
209191
if name.is_empty() {
@@ -219,12 +201,7 @@ impl ExecutionPlanRegistry {
219201
}
220202
// Re-registering the same type is a no-op: sessions are often
221203
// configured by more than one layer of an application.
222-
Entry::Occupied(entry)
223-
if entry.get().type_id.is_some()
224-
&& entry.get().type_id == plan.type_id =>
225-
{
226-
Ok(())
227-
}
204+
Entry::Occupied(entry) if entry.get().type_id == plan.type_id => Ok(()),
228205
Entry::Occupied(entry) => config_err!(
229206
"Extension ExecutionPlan name '{}' is already registered by {}, cannot register {}. \
230207
Namespace the name with the owning crate to avoid the collision.",
@@ -235,9 +212,26 @@ impl ExecutionPlanRegistry {
235212
}
236213
}
237214

238-
/// The decoder registered under `name`, if any.
239-
pub fn decoder(&self, name: &str) -> Option<ExecutionPlanDecoder> {
240-
self.decoders.get(name).map(|plan| plan.decoder)
215+
/// Decode `node` with the decoder registered under `name`.
216+
///
217+
/// `None` means "no decoder claims this name" — the caller falls back to
218+
/// the `PhysicalExtensionCodec` chain. `Some(Err(..))` means the decoder
219+
/// that *does* own the name failed, which is fatal: falling back there
220+
/// would let another codec mis-decode the payload, the very thing the name
221+
/// exists to prevent.
222+
pub fn decode(
223+
&self,
224+
name: &str,
225+
node: &PhysicalPlanNode,
226+
ctx: &ExecutionPlanDecodeCtx<'_>,
227+
) -> Option<Result<Arc<dyn ExecutionPlan>>> {
228+
let plan = self.decoders.get(name)?;
229+
Some((plan.decoder)(node, ctx))
230+
}
231+
232+
/// Whether a decoder is registered under `name`.
233+
pub fn contains(&self, name: &str) -> bool {
234+
self.decoders.contains_key(name)
241235
}
242236

243237
/// Every registered name, in arbitrary order.
@@ -324,7 +318,7 @@ impl ExecutionPlanRegistryExt for SessionConfig {
324318
mod tests {
325319
use super::*;
326320
use crate::proto_test_util::registry_test_plan::{
327-
OtherRegisteredExec, RegisteredExec, RegisteredExecClone,
321+
OtherRegisteredExec, RegisteredExec, RegisteredExecClone, UnnamedExec,
328322
};
329323

330324
#[test]
@@ -334,7 +328,7 @@ mod tests {
334328
registry.register::<RegisteredExec>()?;
335329

336330
assert_eq!(registry.len(), 1);
337-
assert!(registry.decoder(RegisteredExec::PLAN_NAME).is_some());
331+
assert!(registry.contains(RegisteredExec::PLAN_NAME));
338332
Ok(())
339333
}
340334

@@ -369,36 +363,21 @@ mod tests {
369363
names,
370364
vec![OtherRegisteredExec::PLAN_NAME, RegisteredExec::PLAN_NAME]
371365
);
372-
assert!(registry.decoder("not.registered").is_none());
373-
Ok(())
374-
}
375-
376-
#[test]
377-
fn register_decoder_rejects_any_duplicate_name() -> Result<()> {
378-
let mut registry = ExecutionPlanRegistry::new();
379-
registry.register_decoder("alias.MyExec", RegisteredExec::try_from_proto)?;
380-
assert_eq!(registry.len(), 1);
381-
382-
let err = registry
383-
.register_decoder("alias.MyExec", RegisteredExec::try_from_proto)
384-
.expect_err("duplicate name must fail even for the same decoder");
385-
assert!(
386-
err.to_string().contains("already registered"),
387-
"unexpected error: {err}"
388-
);
366+
assert!(!registry.contains("not.registered"));
389367
Ok(())
390368
}
391369

392370
#[test]
393-
fn register_decoder_rejects_an_empty_name() {
371+
fn register_rejects_an_empty_plan_name() {
394372
let mut registry = ExecutionPlanRegistry::new();
395373
let err = registry
396-
.register_decoder("", RegisteredExec::try_from_proto)
397-
.expect_err("empty name must fail");
374+
.register::<UnnamedExec>()
375+
.expect_err("an empty PLAN_NAME must fail");
398376
assert!(
399377
err.to_string().contains("empty name"),
400378
"unexpected error: {err}"
401379
);
380+
assert!(registry.is_empty());
402381
}
403382

404383
#[test]
@@ -413,8 +392,8 @@ mod tests {
413392
.execution_plan_registry()
414393
.expect("registry must be attached to the session");
415394
assert_eq!(registry.len(), 2);
416-
assert!(registry.decoder(RegisteredExec::PLAN_NAME).is_some());
417-
assert!(registry.decoder(OtherRegisteredExec::PLAN_NAME).is_some());
395+
assert!(registry.contains(RegisteredExec::PLAN_NAME));
396+
assert!(registry.contains(OtherRegisteredExec::PLAN_NAME));
418397
Ok(())
419398
}
420399

@@ -427,8 +406,7 @@ mod tests {
427406
cloned
428407
.execution_plan_registry()
429408
.expect("registry must survive a clone")
430-
.decoder(RegisteredExec::PLAN_NAME)
431-
.is_some()
409+
.contains(RegisteredExec::PLAN_NAME)
432410
);
433411
Ok(())
434412
}

‎datafusion/physical-plan/src/proto_test_util.rs‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -330,6 +330,7 @@ pub(crate) mod registry_test_plan {
330330
use crate::execution_plan::{Boundedness, EmissionType};
331331
use crate::proto::{
332332
ExecutionPlanDecodeCtx, ExecutionPlanEncodeCtx, ExtensionExecutionPlan,
333+
ExtensionPlanParts,
333334
};
334335
use crate::{
335336
DisplayAs, DisplayFormatType, ExecutionPlan, PlanProperties,
@@ -441,7 +442,7 @@ pub(crate) mod registry_test_plan {
441442
node: &PhysicalPlanNode,
442443
ctx: &ExecutionPlanDecodeCtx<'_>,
443444
) -> Result<Arc<dyn ExecutionPlan>> {
444-
let (payload, children) =
445+
let ExtensionPlanParts { payload, children } =
445446
ctx.decode_extension(node, Self::PLAN_NAME)?;
446447
let Ok(payload) = std::str::from_utf8(payload) else {
447448
return internal_err!("{} payload is not UTF-8", $plan_name);
@@ -457,6 +458,9 @@ pub(crate) mod registry_test_plan {
457458
// independent crates could hit.
458459
extension_plan!(RegisteredExecClone, "datafusion-test.RegisteredExec");
459460
extension_plan!(OtherRegisteredExec, "datafusion-test.OtherRegisteredExec");
461+
// A plan whose author forgot to fill in `PLAN_NAME`: registration must
462+
// reject it rather than let an unnamed node reach the wire.
463+
extension_plan!(UnnamedExec, "");
460464
}
461465

462466
/// Decoder that must never run: asserts that the reject paths of a

‎datafusion/proto/src/physical_plan/mod.rs‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1296,17 +1296,23 @@ pub trait PhysicalPlanNodeExt: Sized {
12961296
// takes the codec chain exactly as before.
12971297
let registry = ctx.task_ctx().session_config().execution_plan_registry();
12981298
if let Some(plan_name) = extension.plan_name.as_deref()
1299-
&& let Some(decoder) = registry
1300-
.as_ref()
1301-
.and_then(|registry| registry.decoder(plan_name))
1299+
&& let Some(registry) = registry.as_ref()
13021300
{
13031301
let plan_decoder = ConverterPlanDecoder {
13041302
ctx,
13051303
proto_converter,
13061304
};
13071305
// The decoder receives the whole node, like every built-in
13081306
// `try_from_proto`, and decodes its own children through the ctx.
1309-
return decoder(self.node(), &ExecutionPlanDecodeCtx::new(&plan_decoder));
1307+
// `None` here means no decoder claims the name; a decode *failure*
1308+
// is returned as-is rather than falling through to the codec.
1309+
if let Some(decoded) = registry.decode(
1310+
plan_name,
1311+
self.node(),
1312+
&ExecutionPlanDecodeCtx::new(&plan_decoder),
1313+
) {
1314+
return decoded;
1315+
}
13101316
}
13111317

13121318
let inputs: Vec<Arc<dyn ExecutionPlan>> = extension

0 commit comments

Comments
 (0)