Skip to content

Commit 281a955

Browse files
adriangbclaude
andcommitted
prototype: add typed bytes-only function serde to the ExecutionPlan proto ctx (apache#22419)
Adds encode_udf/udaf/udwf + decode_udf/udaf/udwf to ExecutionPlanEncodeCtx/ DecodeCtx (and the internal dispatch traits). These take datafusion-expr types + Vec<u8> and never name a proto type; the datafusion-proto adapter backs them over PhysicalExtensionCodec + the registry, owning the payload/registry lookup-order policy in one place. Wire-identical to the existing fun_definition semantics ((!buf.is_empty()).then_some(buf)). This lets function-carrying plans (Aggregate, window, ScalarSubquery) ride the polymorphic hook instead of staying as typed arms — possible because physical-plan sits above datafusion-expr (unlike the expr-side ctx). apache#23421's ScalarUdfCodec stays closed; ScalarFunctionExpr stays special-cased on the expr side where the crate cycle forces it. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01SfBPC2yYCwHUn4V8pQCVj3
1 parent 1388fb1 commit 281a955

2 files changed

Lines changed: 150 additions & 13 deletions

File tree

‎datafusion/physical-plan/src/proto.rs‎

Lines changed: 90 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -38,26 +38,31 @@
3838
//! `datafusion-physical-plan` depends on the pure prost types in
3939
//! `datafusion-proto-models` (feature `proto`), never on `datafusion-proto`.
4040
//!
41-
//! # Scope: self-contained plans only
41+
//! # Function-carrying plans
4242
//!
43-
//! These hooks are for plans that are *self-contained*: everything they need to
44-
//! round-trip is reachable from their child plans, their child expressions and
45-
//! their input schema. Plans that reference session-scoped machinery only
46-
//! `datafusion-proto` owns — the [`PhysicalExtensionCodec`] (for UDAF/UDWF and
47-
//! extension nodes) — stay as typed dispatch arms inside `datafusion-proto`,
48-
//! exactly as `ScalarFunctionExpr` does on the expression side. The decode
49-
//! context intentionally exposes the [`TaskContext`] (for function-registry and
50-
//! session-config lookups reachable from `datafusion-execution`) but *not* the
51-
//! extension codec.
43+
//! Plans that reference UD(A/W)Fs (`AggregateExec`, the window execs, …) also
44+
//! ride the hook: the context exposes typed, *bytes-only* function serde —
45+
//! [`encode_udaf`](ExecutionPlanEncodeCtx::encode_udaf) /
46+
//! [`decode_udaf`](ExecutionPlanDecodeCtx::decode_udaf) and the udf/udwf
47+
//! siblings. These take/return `datafusion-expr` types plus `Vec<u8>` and never
48+
//! name a proto type, so the `PhysicalExtensionCodec` (which only
49+
//! `datafusion-proto` can name) stays fully encapsulated behind the adapter that
50+
//! backs these traits. The lookup-order policy (payload → codec; else registry →
51+
//! codec fallback) lives once, in that adapter, rather than in every plan.
52+
//!
53+
//! This is possible because `datafusion-physical-plan` sits *above*
54+
//! `datafusion-expr` in the crate graph; the expression-side ctx (in
55+
//! `physical-expr-common`, *below* `datafusion-expr`) cannot do this, which is
56+
//! why `ScalarFunctionExpr` remains special-cased there.
5257
//!
53-
//! [`PhysicalExtensionCodec`]: (owned by `datafusion-proto`)
5458
//! [`ExecutionPlan`]: crate::ExecutionPlan
5559
5660
use std::sync::Arc;
5761

5862
use arrow::datatypes::Schema;
5963
use datafusion_common::{Result, internal_datafusion_err};
6064
use datafusion_execution::TaskContext;
65+
use datafusion_expr::{AggregateUDF, ScalarUDF, WindowUDF};
6166
use datafusion_physical_expr::PhysicalExpr;
6267
use datafusion_proto_models::protobuf::{PhysicalExprNode, PhysicalPlanNode};
6368

@@ -74,6 +79,18 @@ pub trait ExecutionPlanEncode {
7479

7580
/// Serialize a physical expression owned by the plan.
7681
fn encode_expr(&self, expr: &Arc<dyn PhysicalExpr>) -> Result<PhysicalExprNode>;
82+
83+
/// Serialize a scalar UDF to an opaque payload. `None` means "decodable by
84+
/// name alone" (built-ins). Bytes-only: no proto types cross this boundary.
85+
fn encode_udf(&self, udf: &ScalarUDF) -> Result<Option<Vec<u8>>>;
86+
87+
/// Serialize an aggregate UDF to an opaque payload. `None` means "decodable
88+
/// by name alone".
89+
fn encode_udaf(&self, udaf: &AggregateUDF) -> Result<Option<Vec<u8>>>;
90+
91+
/// Serialize a window UDF to an opaque payload. `None` means "decodable by
92+
/// name alone".
93+
fn encode_udwf(&self, udwf: &WindowUDF) -> Result<Option<Vec<u8>>>;
7794
}
7895

7996
/// Internal dispatch trait backing [`ExecutionPlanDecodeCtx`].
@@ -95,6 +112,21 @@ pub trait ExecutionPlanDecode {
95112
/// The session task context, used by plans that need the function registry
96113
/// or session configuration. Never exposes the proto extension codec.
97114
fn task_ctx(&self) -> &TaskContext;
115+
116+
/// Reconstruct a scalar UDF from its name and optional payload. Encapsulates
117+
/// the lookup-order policy (payload → codec; else registry → codec fallback)
118+
/// so no plan re-derives it. Bytes-only: no proto types cross this boundary.
119+
fn decode_udf(&self, name: &str, payload: Option<&[u8]>) -> Result<Arc<ScalarUDF>>;
120+
121+
/// Reconstruct an aggregate UDF from its name and optional payload.
122+
fn decode_udaf(
123+
&self,
124+
name: &str,
125+
payload: Option<&[u8]>,
126+
) -> Result<Arc<AggregateUDF>>;
127+
128+
/// Reconstruct a window UDF from its name and optional payload.
129+
fn decode_udwf(&self, name: &str, payload: Option<&[u8]>) -> Result<Arc<WindowUDF>>;
98130
}
99131

100132
/// Context handed to [`ExecutionPlan::try_to_proto`](crate::ExecutionPlan::try_to_proto).
@@ -140,6 +172,23 @@ impl<'a> ExecutionPlanEncodeCtx<'a> {
140172
{
141173
exprs.into_iter().map(|e| self.encode_expr(e)).collect()
142174
}
175+
176+
/// Serialize a scalar UDF to an opaque payload (`None` = built-in, decodable
177+
/// by name). No proto types cross this boundary.
178+
pub fn encode_udf(&self, udf: &ScalarUDF) -> Result<Option<Vec<u8>>> {
179+
self.encoder.encode_udf(udf)
180+
}
181+
182+
/// Serialize an aggregate UDF to an opaque payload (`None` = decodable by
183+
/// name).
184+
pub fn encode_udaf(&self, udaf: &AggregateUDF) -> Result<Option<Vec<u8>>> {
185+
self.encoder.encode_udaf(udaf)
186+
}
187+
188+
/// Serialize a window UDF to an opaque payload (`None` = decodable by name).
189+
pub fn encode_udwf(&self, udwf: &WindowUDF) -> Result<Option<Vec<u8>>> {
190+
self.encoder.encode_udwf(udwf)
191+
}
143192
}
144193

145194
/// Context handed to a plan's `try_from_proto` associated function.
@@ -203,11 +252,39 @@ impl<'a> ExecutionPlanDecodeCtx<'a> {
203252
}
204253

205254
/// The session task context (function registry + session config). Never
206-
/// exposes the proto extension codec — session-codec-dependent plans stay as
207-
/// typed dispatch arms in `datafusion-proto`.
255+
/// exposes the proto extension codec.
208256
pub fn task_ctx(&self) -> &TaskContext {
209257
self.decoder.task_ctx()
210258
}
259+
260+
/// Reconstruct a scalar UDF from its name and optional payload. The
261+
/// lookup-order policy is owned by `datafusion-proto`; no proto types cross
262+
/// this boundary.
263+
pub fn decode_udf(
264+
&self,
265+
name: &str,
266+
payload: Option<&[u8]>,
267+
) -> Result<Arc<ScalarUDF>> {
268+
self.decoder.decode_udf(name, payload)
269+
}
270+
271+
/// Reconstruct an aggregate UDF from its name and optional payload.
272+
pub fn decode_udaf(
273+
&self,
274+
name: &str,
275+
payload: Option<&[u8]>,
276+
) -> Result<Arc<AggregateUDF>> {
277+
self.decoder.decode_udaf(name, payload)
278+
}
279+
280+
/// Reconstruct a window UDF from its name and optional payload.
281+
pub fn decode_udwf(
282+
&self,
283+
name: &str,
284+
payload: Option<&[u8]>,
285+
) -> Result<Arc<WindowUDF>> {
286+
self.decoder.decode_udwf(name, payload)
287+
}
211288
}
212289

213290
/// Assert that a [`PhysicalPlanNode`] carries the expected `PhysicalPlanType`

‎datafusion/proto/src/physical_plan/mod.rs‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4365,6 +4365,26 @@ impl ExecutionPlanEncode for ConverterPlanEncoder<'_> {
43654365
self.proto_converter
43664366
.physical_expr_to_proto(expr, self.codec)
43674367
}
4368+
4369+
// Bytes-only function serde. `(!buf.is_empty()).then_some(buf)` preserves the
4370+
// existing `fun_definition` wire semantics (empty payload == encode-by-name).
4371+
fn encode_udf(&self, udf: &ScalarUDF) -> Result<Option<Vec<u8>>> {
4372+
let mut buf = vec![];
4373+
self.codec.try_encode_udf(udf, &mut buf)?;
4374+
Ok((!buf.is_empty()).then_some(buf))
4375+
}
4376+
4377+
fn encode_udaf(&self, udaf: &AggregateUDF) -> Result<Option<Vec<u8>>> {
4378+
let mut buf = vec![];
4379+
self.codec.try_encode_udaf(udaf, &mut buf)?;
4380+
Ok((!buf.is_empty()).then_some(buf))
4381+
}
4382+
4383+
fn encode_udwf(&self, udwf: &WindowUDF) -> Result<Option<Vec<u8>>> {
4384+
let mut buf = vec![];
4385+
self.codec.try_encode_udwf(udwf, &mut buf)?;
4386+
Ok((!buf.is_empty()).then_some(buf))
4387+
}
43684388
}
43694389

43704390
/// Adapter backing [`ExecutionPlanDecodeCtx`] for plans migrated to the
@@ -4396,4 +4416,44 @@ impl ExecutionPlanDecode for ConverterPlanDecoder<'_, '_> {
43964416
fn task_ctx(&self) -> &TaskContext {
43974417
self.ctx.task_ctx()
43984418
}
4419+
4420+
// Lookup-order policy, owned here so no plan re-derives it: an explicit
4421+
// payload is decoded by the codec; otherwise resolve by name from the
4422+
// registry, falling back to the codec with an empty buffer.
4423+
fn decode_udf(&self, name: &str, payload: Option<&[u8]>) -> Result<Arc<ScalarUDF>> {
4424+
match payload {
4425+
Some(buf) => self.ctx.codec().try_decode_udf(name, buf),
4426+
None => self
4427+
.ctx
4428+
.task_ctx()
4429+
.udf(name)
4430+
.or_else(|_| self.ctx.codec().try_decode_udf(name, &[])),
4431+
}
4432+
}
4433+
4434+
fn decode_udaf(
4435+
&self,
4436+
name: &str,
4437+
payload: Option<&[u8]>,
4438+
) -> Result<Arc<AggregateUDF>> {
4439+
match payload {
4440+
Some(buf) => self.ctx.codec().try_decode_udaf(name, buf),
4441+
None => self
4442+
.ctx
4443+
.task_ctx()
4444+
.udaf(name)
4445+
.or_else(|_| self.ctx.codec().try_decode_udaf(name, &[])),
4446+
}
4447+
}
4448+
4449+
fn decode_udwf(&self, name: &str, payload: Option<&[u8]>) -> Result<Arc<WindowUDF>> {
4450+
match payload {
4451+
Some(buf) => self.ctx.codec().try_decode_udwf(name, buf),
4452+
None => self
4453+
.ctx
4454+
.task_ctx()
4455+
.udwf(name)
4456+
.or_else(|_| self.ctx.codec().try_decode_udwf(name, &[])),
4457+
}
4458+
}
43994459
}

0 commit comments

Comments
 (0)