Skip to content

Commit b16bd3d

Browse files
adriangbclaude
andauthored
Add ExecutionPlan try_to_proto / try_from_proto hooks + ProjectionExec reference (apache#23495)
## Which issue does this PR close? - Part of apache#23494. ## Rationale for this change This is the foundation for porting `ExecutionPlan` serialization off the central `downcast_ref` chain in `datafusion/proto/src/physical_plan/mod.rs`, mirroring what apache#21929 did for `PhysicalExpr` (tracked in apache#22418). Everything else in apache#23494 depends on this PR; it can't be parallelized because it defines the ctx API and wiring that every per-plan follow-up uses. ## What changes are included in this PR? In `datafusion-physical-plan` (feature-gated `#[cfg(feature = "proto")]`): - `ExecutionPlan::try_to_proto(&self, ctx: &ExecutionPlanEncodeCtx) -> Result<Option<PhysicalPlanNode>>` — a new trait method defaulting to `Ok(None)` (so un-migrated plans are unchanged). - `ExecutionPlanEncodeCtx` / `ExecutionPlanDecodeCtx` plus the internal `ExecutionPlanEncode` / `ExecutionPlanDecode` dispatch traits (defined here, implemented in `datafusion-proto`) — the same dependency-inversion the `PhysicalExpr` ctx uses. The ctx exposes `encode_child`/`decode_child` (+ `_children`), `encode_expr`/`decode_expr`, and typed **bytes-only** `encode_udaf`/`decode_udaf` (+ udf/udwf) for function-carrying plans. No proto type is ever named by physical-plan beyond the pure prost types in `datafusion-proto-models`. - An `expect_plan_variant!` macro mirroring `expect_expr_variant!`. In `datafusion-proto`: - `ConverterPlanEncoder` / `ConverterPlanDecoder` implement the dispatch traits over the existing `PhysicalExtensionCodec` + converter (the function serde reuses today's `fun_definition` byte semantics exactly). The encode dispatch resolves `downcast_delegate()` first (so wrapper plans serialize as their delegate, exactly like the `downcast_ref` chain sees them), then calls the hook, then falls back to the existing downcast chain. `ProjectionExec` is migrated as the reference implementation: its old encode arm is deleted and its decode arm reduced to `ProjectionExec::try_from_proto`, so a green roundtrip proves the hook is the only path. ## Are these changes tested? Yes — the existing `proto_integration` roundtrip suite (including TPC-H) passes unchanged, which exercises the `ProjectionExec` migration end-to-end, plus a new test asserting that a wrapper plan delegating its downcast identity to `ProjectionExec` still serializes as a projection through the hook (it fails without the `downcast_delegate()` resolution). The bytes-only function serde and the required-field decode helpers — which have no in-tree plan caller until the function-carrying plans migrate — are pinned by dedicated unit tests (payload semantics, decode lookup order, error paths), addressing the patch-coverage gap. `cargo clippy --all-targets -- -D warnings`, `cargo fmt`, and `RUSTDOCFLAGS="-D warnings" cargo doc` are clean. ## Are there any user-facing changes? New public API: the `try_to_proto` trait method (with a default, so existing `ExecutionPlan` impls are unaffected) and the ctx types. No wire-format change. No API is removed: per the [API health policy](https://datafusion.apache.org/contributor-guide/api-health.html), the two per-plan `PhysicalPlanNodeExt` methods this migration replaces (`try_into_projection_physical_plan` / `try_from_projection_exec`) are kept with their original bodies under `#[deprecated(since = "55.0.0")]`. (They were also never released: `PhysicalPlanNodeExt` was introduced by apache#21929 after 54.0.0 shipped.) The per-plan follow-ups in apache#23494 will deprecate their corresponding methods the same way, with removal after the policy window. --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
1 parent d416011 commit b16bd3d

6 files changed

Lines changed: 1008 additions & 11 deletions

File tree

‎datafusion/physical-plan/src/execution_plan.rs‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -823,6 +823,27 @@ pub trait ExecutionPlan: Any + Debug + DisplayAs + Send + Sync {
823823
) -> Option<Arc<dyn ExecutionPlan>> {
824824
None
825825
}
826+
827+
/// Serialize this plan to its protobuf representation, if it knows how.
828+
///
829+
/// This is the `ExecutionPlan` analog of
830+
/// [`PhysicalExpr::try_to_proto`].
831+
///
832+
/// * `Ok(None)` (the default) — "I don't serialize myself"; the caller
833+
/// (`datafusion-proto`) falls back to the central downcast chain. Every
834+
/// un-migrated plan keeps its existing behavior.
835+
/// * `Ok(Some(node))` — fully serialized; the caller must not fall back.
836+
/// * `Err(_)` — a real failure (e.g. a child failed to serialize).
837+
///
838+
/// Only *self-contained* plans should override this — see [`crate::proto`]
839+
/// for the session-dependency boundary.
840+
#[cfg(feature = "proto")]
841+
fn try_to_proto(
842+
&self,
843+
_ctx: &crate::proto::ExecutionPlanEncodeCtx<'_>,
844+
) -> Result<Option<datafusion_proto_models::protobuf::PhysicalPlanNode>> {
845+
Ok(None)
846+
}
826847
}
827848

828849
impl dyn ExecutionPlan {

‎datafusion/physical-plan/src/lib.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,8 @@ pub mod metrics;
8888
pub mod operator_statistics;
8989
pub mod placeholder_row;
9090
pub mod projection;
91+
#[cfg(feature = "proto")]
92+
pub mod proto;
9193
pub mod recursive_query;
9294
pub mod repartition;
9395
pub mod scalar_subquery;

‎datafusion/physical-plan/src/projection.rs‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -500,6 +500,71 @@ impl ExecutionPlan for ProjectionExec {
500500
.ok()
501501
})
502502
}
503+
504+
#[cfg(feature = "proto")]
505+
fn try_to_proto(
506+
&self,
507+
ctx: &crate::proto::ExecutionPlanEncodeCtx<'_>,
508+
) -> Result<Option<datafusion_proto_models::protobuf::PhysicalPlanNode>> {
509+
use datafusion_proto_models::protobuf;
510+
let input = ctx.encode_child(self.input())?;
511+
let expr = ctx.encode_expressions(self.expr().iter().map(|p| &p.expr))?;
512+
let expr_name = self.expr().iter().map(|p| p.alias.clone()).collect();
513+
Ok(Some(protobuf::PhysicalPlanNode {
514+
physical_plan_type: Some(
515+
protobuf::physical_plan_node::PhysicalPlanType::Projection(Box::new(
516+
protobuf::ProjectionExecNode {
517+
input: Some(Box::new(input)),
518+
expr,
519+
expr_name,
520+
},
521+
)),
522+
),
523+
}))
524+
}
525+
}
526+
527+
#[cfg(feature = "proto")]
528+
impl ProjectionExec {
529+
/// Reconstruct a [`ProjectionExec`] from its protobuf representation.
530+
///
531+
/// The exact inverse of [`ExecutionPlan::try_to_proto`]: it takes the whole
532+
/// [`PhysicalPlanNode`] so every plan's `try_from_proto` shares one
533+
/// signature. Child plans and expressions are decoded recursively via the
534+
/// [`ExecutionPlanDecodeCtx`].
535+
///
536+
/// [`PhysicalPlanNode`]: datafusion_proto_models::protobuf::PhysicalPlanNode
537+
/// [`ExecutionPlan::try_to_proto`]: crate::ExecutionPlan::try_to_proto
538+
/// [`ExecutionPlanDecodeCtx`]: crate::proto::ExecutionPlanDecodeCtx
539+
pub fn try_from_proto(
540+
node: &datafusion_proto_models::protobuf::PhysicalPlanNode,
541+
ctx: &crate::proto::ExecutionPlanDecodeCtx<'_>,
542+
) -> Result<Arc<dyn ExecutionPlan>> {
543+
use datafusion_proto_models::protobuf;
544+
let projection = crate::expect_plan_variant!(
545+
node,
546+
protobuf::physical_plan_node::PhysicalPlanType::Projection,
547+
"ProjectionExec",
548+
);
549+
let input = ctx.decode_required_child(
550+
projection.input.as_deref(),
551+
"ProjectionExec",
552+
"input",
553+
)?;
554+
let input_schema = input.schema();
555+
let exprs = projection
556+
.expr
557+
.iter()
558+
.zip(projection.expr_name.iter())
559+
.map(|(expr, name)| {
560+
Ok(ProjectionExpr {
561+
expr: ctx.decode_expr(expr, input_schema.as_ref())?,
562+
alias: name.to_string(),
563+
})
564+
})
565+
.collect::<Result<Vec<ProjectionExpr>>>()?;
566+
Ok(Arc::new(ProjectionExec::try_new(exprs, input)?))
567+
}
503568
}
504569

505570
impl ProjectionStream {

0 commit comments

Comments
 (0)