Skip to content

Commit a5ae38c

Browse files
adriangbclaude
authored andcommitted
feat(proto): thread expr encode/decode context into try_encode_expr / try_decode_expr (apache#23733)
## Which issue does this PR close? Closes apache#22920. Alternative to apache#22922 — same goal, but plumbs the per-expr encode/decode **context** (from the apache#22418 hook machinery) instead of the raw `PhysicalProtoConverterExtension`. ## Rationale for this change apache#21807 introduced the `DynamicFilterPhysicalExpr` dedup pipeline so identical references on the wire reconstruct to one shared `Arc<Inner>` via `expr_id` cache keys, and apache#22011 hooked it through the `SortExec` / `AggregateExec` / `HashJoinExec` plan codecs. The remaining gap is the **expression-level** extension path. In `serialize_physical_expr_with_converter` / `parse_physical_expr_with_converter`, the codec's `try_encode_expr` / `try_decode_expr` is reached only as a fallback after the built-in `try_to_proto` path and the `ScalarFunctionExpr` downcast — i.e. only for downstream-defined **custom `PhysicalExpr`** types. When such a codec serializes nested `PhysicalExprNode` fields *inside its own blob*, the only helper available today is the free `serialize_physical_expr` / `parse_physical_expr`, hardwired to `DefaultPhysicalProtoConverter`. So those nested exprs get `expr_id: None` on the wire and reconstruct as **distinct** `Inner` allocations — heap-max updates from a `SortExec` never reach the wrapped reference. ## What changes are included in this PR? The built-in expressions migrated under apache#22418 already receive a `PhysicalExprEncodeCtx` / `PhysicalExprDecodeCtx` in their `try_to_proto` / `try_from_proto` hooks. Those context objects bundle the dedup-aware converter **plus** the active schema and task context, and hide `PhysicalProtoConverterExtension` / `PhysicalExtensionCodec` from the expression author entirely. This PR hands the **same context** to the expr-level codec methods: ```rust fn try_decode_expr( &self, buf: &[u8], inputs: &[Arc<dyn PhysicalExpr>], ctx: &PhysicalExprDecodeCtx<'_>, // new ) -> Result<Arc<dyn PhysicalExpr>>; fn try_encode_expr( &self, node: &Arc<dyn PhysicalExpr>, buf: &mut Vec<u8>, ctx: &PhysicalExprEncodeCtx<'_>, // new ) -> Result<()>; ``` A codec that embeds nested `PhysicalExprNode`s now decodes them with `ctx.decode(..)` and encodes them with `ctx.encode_child(..)`, which: - route through any active `DeduplicatingProtoConverter` / `DeduplicatingDeserializer`, so shared inner expressions cache-hit on `expr_id`; **and** - carry the real schema and task context, so nested UDF / column references resolve against the actual registry. ### Why the context, not the raw converter (cf. apache#22922) Threading the bare `PhysicalProtoConverterExtension` still leaves the codec without the schema/registry, forcing it to fabricate a `SessionContext::new()` and hard-code a schema to call `proto_to_physical_expr` — an empty-registry footgun for any nested expr that references a UDF or column. Passing the existing `Physical{Encode,Decode}Ctx` avoids that, keeps the extension escape-hatch consistent with the per-expr proto hooks, and adds no third converter parameter to the codec API (the concern raised on apache#22922). ## Are these changes tested? Yes — `extension_codec_expr_participates_in_deduplication` builds a `BinaryExpr` whose left operand is a bare `DynamicFilterPhysicalExpr` and whose right operand is a custom `WrapperExpr` whose codec embeds the same dynamic filter inside its serialized blob (via `ctx.encode_child`). After a `DeduplicatingProtoConverter` roundtrip, an `update()` on the bare-side decoded filter is observed via `current()` on the wrapped-side filter, proving both refs back the same `Inner`. The codec needs no fabricated `SessionContext` — it decodes the nested expr through `ctx.decode`. ## Are there any user-facing changes? Yes — a **breaking change** for downstream codecs that override `try_encode_expr` / `try_decode_expr`: they must add the new `ctx` parameter (name it `_ctx` if the custom expr carries no nested `PhysicalExprNode`s). Codecs that only override the plan-level `try_encode` / `try_decode` are unaffected. Wire format is unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
1 parent 83c4411 commit a5ae38c

4 files changed

Lines changed: 220 additions & 3 deletions

File tree

‎datafusion/proto/src/physical_plan/from_proto.rs‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -383,8 +383,11 @@ pub fn parse_physical_expr_with_converter(
383383
.iter()
384384
.map(|e| proto_converter.proto_to_physical_expr(e, input_schema, ctx))
385385
.collect::<Result<_>>()?;
386-
ctx.codec()
387-
.try_decode_expr(extension.expr.as_slice(), &inputs)? as _
386+
ctx.codec().try_decode_expr(
387+
extension.expr.as_slice(),
388+
&inputs,
389+
&decode_ctx,
390+
)? as _
388391
}
389392
ExprType::Lambda(_) => LambdaExpr::try_from_proto(proto, &decode_ctx)?,
390393
ExprType::LambdaVariable(_) => {

‎datafusion/proto/src/physical_plan/mod.rs‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,8 @@ use datafusion_physical_expr::aggregate::{AggregateExprBuilder, AggregateFunctio
6262
use datafusion_physical_expr::async_scalar_function::AsyncFuncExpr;
6363
use datafusion_physical_expr::expressions::DynamicFilterPhysicalExpr;
6464
use datafusion_physical_expr::{LexOrdering, LexRequirement, PhysicalExprRef};
65+
use datafusion_physical_expr_common::physical_expr::proto_decode::PhysicalExprDecodeCtx;
66+
use datafusion_physical_expr_common::physical_expr::proto_encode::PhysicalExprEncodeCtx;
6567
use datafusion_physical_plan::aggregates::{
6668
AggregateExec, AggregateMode, LimitOptions, PhysicalGroupBy,
6769
};
@@ -4431,18 +4433,44 @@ pub trait PhysicalExtensionCodec: Debug + Send + Sync + Any {
44314433
Ok(())
44324434
}
44334435

4436+
/// Decode a custom extension expression from `buf`.
4437+
///
4438+
/// `inputs` holds the already-decoded children carried in the
4439+
/// `PhysicalExtensionExprNode.inputs` field. If the codec instead embeds
4440+
/// nested `PhysicalExprNode`s *inside* `buf`, decode them through
4441+
/// `ctx.decode(..)` (equivalently [`PhysicalExprDecodeCtx::decode`]) rather
4442+
/// than the free [`parse_physical_expr`] function: `ctx` carries the active
4443+
/// schema and task context (so UDF/column references resolve against the
4444+
/// real registry) and routes through any active `DeduplicatingDeserializer`,
4445+
/// so a shared inner expression (e.g. a `DynamicFilterPhysicalExpr`
4446+
/// referenced both from a `SortExec.filter` and from inside this blob)
4447+
/// cache-hits on its `expr_id` and re-shares one `Arc<dyn PhysicalExpr>`.
4448+
///
4449+
/// [`parse_physical_expr`]: crate::physical_plan::from_proto::parse_physical_expr
44344450
fn try_decode_expr(
44354451
&self,
44364452
_buf: &[u8],
44374453
_inputs: &[Arc<dyn PhysicalExpr>],
4454+
_ctx: &PhysicalExprDecodeCtx<'_>,
44384455
) -> Result<Arc<dyn PhysicalExpr>> {
44394456
not_impl_err!("PhysicalExtensionCodec is not provided")
44404457
}
44414458

4459+
/// Encode a custom extension expression into `buf`.
4460+
///
4461+
/// If the codec embeds nested `PhysicalExprNode`s inside `buf`, encode them
4462+
/// through `ctx.encode_child(..)` (equivalently
4463+
/// [`PhysicalExprEncodeCtx::encode_child`]) rather than the free
4464+
/// [`serialize_physical_expr`] function, so an active
4465+
/// `DeduplicatingProtoConverter` stamps matching `expr_id`s for shared
4466+
/// inner expressions. See [`Self::try_decode_expr`].
4467+
///
4468+
/// [`serialize_physical_expr`]: crate::physical_plan::to_proto::serialize_physical_expr
44424469
fn try_encode_expr(
44434470
&self,
44444471
_node: &Arc<dyn PhysicalExpr>,
44454472
_buf: &mut Vec<u8>,
4473+
_ctx: &PhysicalExprEncodeCtx<'_>,
44464474
) -> Result<()> {
44474475
not_impl_err!("PhysicalExtensionCodec is not provided")
44484476
}

‎datafusion/proto/src/physical_plan/to_proto.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -332,7 +332,7 @@ pub fn serialize_physical_expr_with_converter(
332332
})
333333
} else {
334334
let mut buf: Vec<u8> = vec![];
335-
match codec.try_encode_expr(value, &mut buf) {
335+
match codec.try_encode_expr(value, &mut buf, &ctx) {
336336
Ok(_) => {
337337
let inputs: Vec<protobuf::PhysicalExprNode> = value
338338
.children()

‎datafusion/proto/tests/cases/roundtrip_physical_plan.rs‎

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,8 @@ use datafusion_functions_aggregate::min_max::max_udaf;
120120
use datafusion_functions_aggregate::nth_value::nth_value_udaf;
121121
use datafusion_functions_aggregate::string_agg::string_agg_udaf;
122122
use datafusion_physical_expr::scalar_subquery::ScalarSubqueryExpr;
123+
use datafusion_physical_expr_common::physical_expr::proto_decode::PhysicalExprDecodeCtx;
124+
use datafusion_physical_expr_common::physical_expr::proto_encode::PhysicalExprEncodeCtx;
123125
use datafusion_proto::bytes::{
124126
physical_plan_from_bytes_with_proto_converter,
125127
physical_plan_to_bytes_with_proto_converter,
@@ -1369,6 +1371,7 @@ fn roundtrip_parquet_exec_with_custom_predicate_expr() -> Result<()> {
13691371
&self,
13701372
buf: &[u8],
13711373
inputs: &[Arc<dyn PhysicalExpr>],
1374+
_ctx: &PhysicalExprDecodeCtx<'_>,
13721375
) -> Result<Arc<dyn PhysicalExpr>> {
13731376
if buf == "CustomPredicateExpr".as_bytes() {
13741377
Ok(Arc::new(CustomPredicateExpr {
@@ -1383,6 +1386,7 @@ fn roundtrip_parquet_exec_with_custom_predicate_expr() -> Result<()> {
13831386
&self,
13841387
node: &Arc<dyn PhysicalExpr>,
13851388
buf: &mut Vec<u8>,
1389+
_ctx: &PhysicalExprEncodeCtx<'_>,
13861390
) -> Result<()> {
13871391
if node.downcast_ref::<CustomPredicateExpr>().is_some() {
13881392
buf.extend_from_slice("CustomPredicateExpr".as_bytes());
@@ -4542,3 +4546,185 @@ fn roundtrip_parquet_exec_range_output_partitioning() -> Result<()> {
45424546

45434547
Ok(())
45444548
}
4549+
4550+
/// A custom `PhysicalExpr` whose extension codec embeds a nested
4551+
/// `PhysicalExprNode` *inside its own blob* (rather than the standard
4552+
/// `PhysicalExtensionExprNode.inputs` field). This is the case that only
4553+
/// works if the expr-level codec methods receive the encode/decode context.
4554+
#[derive(Debug)]
4555+
struct WrapperExpr {
4556+
inner: Arc<dyn PhysicalExpr>,
4557+
}
4558+
4559+
impl Display for WrapperExpr {
4560+
fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
4561+
write!(f, "WrapperExpr({})", self.inner)
4562+
}
4563+
}
4564+
4565+
impl PartialEq for WrapperExpr {
4566+
fn eq(&self, other: &Self) -> bool {
4567+
self.inner.eq(&other.inner)
4568+
}
4569+
}
4570+
impl Eq for WrapperExpr {}
4571+
4572+
impl std::hash::Hash for WrapperExpr {
4573+
fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
4574+
self.inner.hash(state);
4575+
}
4576+
}
4577+
4578+
impl PhysicalExpr for WrapperExpr {
4579+
fn data_type(&self, input_schema: &Schema) -> Result<DataType> {
4580+
self.inner.data_type(input_schema)
4581+
}
4582+
fn nullable(&self, input_schema: &Schema) -> Result<bool> {
4583+
self.inner.nullable(input_schema)
4584+
}
4585+
fn evaluate(&self, _batch: &RecordBatch) -> Result<ColumnarValue> {
4586+
internal_err!("WrapperExpr is not executable in this test")
4587+
}
4588+
fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
4589+
vec![&self.inner]
4590+
}
4591+
fn with_new_children(
4592+
self: Arc<Self>,
4593+
children: Vec<Arc<dyn PhysicalExpr>>,
4594+
) -> Result<Arc<dyn PhysicalExpr>> {
4595+
Ok(Arc::new(WrapperExpr {
4596+
inner: Arc::clone(&children[0]),
4597+
}))
4598+
}
4599+
fn fmt_sql(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
4600+
std::fmt::Display::fmt(self, f)
4601+
}
4602+
}
4603+
4604+
/// Wire layout for [`WrapperExpr`]: a single nested `PhysicalExprNode`.
4605+
#[derive(Clone, PartialEq, prost::Message)]
4606+
struct WrapperExprProto {
4607+
#[prost(message, optional, boxed, tag = "1")]
4608+
inner: Option<Box<datafusion_proto::protobuf::PhysicalExprNode>>,
4609+
}
4610+
4611+
#[derive(Debug)]
4612+
struct WrapperCodec;
4613+
4614+
impl PhysicalExtensionCodec for WrapperCodec {
4615+
fn try_decode(
4616+
&self,
4617+
_buf: &[u8],
4618+
_inputs: &[Arc<dyn ExecutionPlan>],
4619+
_ctx: &TaskContext,
4620+
_proto_converter: &dyn PhysicalProtoConverterExtension,
4621+
) -> Result<Arc<dyn ExecutionPlan>> {
4622+
internal_err!("not used")
4623+
}
4624+
fn try_encode(
4625+
&self,
4626+
_node: Arc<dyn ExecutionPlan>,
4627+
_buf: &mut Vec<u8>,
4628+
_proto_converter: &dyn PhysicalProtoConverterExtension,
4629+
) -> Result<()> {
4630+
internal_err!("not used")
4631+
}
4632+
fn try_decode_expr(
4633+
&self,
4634+
buf: &[u8],
4635+
_inputs: &[Arc<dyn PhysicalExpr>],
4636+
ctx: &PhysicalExprDecodeCtx<'_>,
4637+
) -> Result<Arc<dyn PhysicalExpr>> {
4638+
let proto = WrapperExprProto::decode(buf)
4639+
.map_err(|e| internal_datafusion_err!("decode WrapperExprProto: {e}"))?;
4640+
let inner_proto = proto
4641+
.inner
4642+
.ok_or_else(|| internal_datafusion_err!("missing inner"))?;
4643+
// Decode the nested expr through the context so it resolves against
4644+
// the real schema/registry AND participates in dedup — no fabricated
4645+
// `SessionContext` or hard-coded schema required.
4646+
let inner = ctx.decode(&inner_proto)?;
4647+
Ok(Arc::new(WrapperExpr { inner }))
4648+
}
4649+
fn try_encode_expr(
4650+
&self,
4651+
node: &Arc<dyn PhysicalExpr>,
4652+
buf: &mut Vec<u8>,
4653+
ctx: &PhysicalExprEncodeCtx<'_>,
4654+
) -> Result<()> {
4655+
let wrapper = node
4656+
.downcast_ref::<WrapperExpr>()
4657+
.ok_or_else(|| internal_datafusion_err!("not WrapperExpr"))?;
4658+
// Encode the nested expr through the context so an active
4659+
// `DeduplicatingProtoConverter` stamps a matching `expr_id`.
4660+
let inner_proto = ctx.encode_child(&wrapper.inner)?;
4661+
let proto = WrapperExprProto {
4662+
inner: Some(Box::new(inner_proto)),
4663+
};
4664+
proto
4665+
.encode(buf)
4666+
.map_err(|e| internal_datafusion_err!("encode WrapperExprProto: {e}"))?;
4667+
Ok(())
4668+
}
4669+
}
4670+
4671+
/// A `DynamicFilterPhysicalExpr` referenced both as a bare expression and
4672+
/// nested inside a custom expression's codec blob must reconstruct to a
4673+
/// single shared `Inner` after roundtrip.
4674+
///
4675+
/// This exercises the expr-level codec hooks receiving the encode/decode
4676+
/// context: `try_encode_expr` routes its nested `PhysicalExprNode` through
4677+
/// `ctx.encode_child` and `try_decode_expr` through `ctx.decode`, so the
4678+
/// nested filter picks up the same `DeduplicatingProtoConverter` /
4679+
/// `DeduplicatingDeserializer` cache as the bare reference. Without the
4680+
/// context the nested expr would serialize with `expr_id: None` and decode
4681+
/// into a distinct `Inner`, breaking heap-max propagation across the
4682+
/// extension boundary in distributed execution.
4683+
#[test]
4684+
fn extension_codec_expr_participates_in_deduplication() -> Result<()> {
4685+
use prost::Message;
4686+
4687+
// A single composite expression holding TWO references to the same
4688+
// dynamic filter: bare on the left of an AND, wrapped on the right.
4689+
let dyn_filter = make_dynamic_filter();
4690+
let wrapper: Arc<dyn PhysicalExpr> = Arc::new(WrapperExpr {
4691+
inner: Arc::clone(&dyn_filter),
4692+
});
4693+
let composite: Arc<dyn PhysicalExpr> = Arc::new(BinaryExpr::new(
4694+
Arc::clone(&dyn_filter),
4695+
Operator::And,
4696+
Arc::clone(&wrapper),
4697+
));
4698+
4699+
let schema = Arc::new(Schema::new(vec![Field::new("a", DataType::Int64, false)]));
4700+
let codec = WrapperCodec;
4701+
let converter = DeduplicatingProtoConverter {};
4702+
4703+
// Encode, then round-trip through prost bytes to mimic the wire.
4704+
let proto = converter.physical_expr_to_proto(&composite, &codec)?;
4705+
let bytes = proto.encode_to_vec();
4706+
let decoded_proto =
4707+
datafusion_proto::protobuf::PhysicalExprNode::decode(bytes.as_slice()).unwrap();
4708+
4709+
let ctx = SessionContext::new();
4710+
let task_ctx = ctx.task_ctx();
4711+
let decode_ctx = PhysicalPlanDecodeContext::new(task_ctx.as_ref(), &codec);
4712+
let decoded =
4713+
converter.proto_to_physical_expr(&decoded_proto, &schema, &decode_ctx)?;
4714+
4715+
let binary = decoded
4716+
.downcast_ref::<BinaryExpr>()
4717+
.expect("must decode back to BinaryExpr");
4718+
let decoded_left = Arc::clone(binary.left());
4719+
let decoded_right = Arc::clone(binary.right());
4720+
let decoded_wrapper = decoded_right
4721+
.downcast_ref::<WrapperExpr>()
4722+
.expect("right side must decode back to WrapperExpr");
4723+
4724+
// The load-bearing check: an `update()` on the bare-side filter must be
4725+
// observable from the wrapped-side filter, proving both refs back the
4726+
// same `Inner`.
4727+
assert_dynamic_filter_update_is_visible(&decoded_left, &decoded_wrapper.inner)?;
4728+
4729+
Ok(())
4730+
}

0 commit comments

Comments
 (0)