Skip to content

Commit 7a58afe

Browse files
committed
feat(proto): plumb PhysicalProtoConverterExtension into try_encode_expr / try_decode_expr
Closes apache#22920. apache#21807 introduced the `DynamicFilterPhysicalExpr` dedup pipeline so that identical references on the wire reconstruct to one shared `Arc<Inner>` on the decode side via `expression_id` cache keys. The pipeline works end-to-end for plans built from upstream types, and apache#22011 hooked it through Sort/Aggregate/HashJoin plan codecs. However, the pipeline stops at the `PhysicalExtensionCodec` boundary. Downstream users with custom file sources or plan extension types encode/decode their nested `PhysicalExprNode` fields via `try_encode_expr` / `try_decode_expr`, which did not receive the active `PhysicalProtoConverterExtension`. The only entry points available inside such a codec were the free `serialize_physical_expr` / `parse_physical_expr` helpers, hardwired to `DefaultPhysicalProtoConverter`. So any expression inside an extension codec's serialized blob got `expr_id = None` on the wire, and the deduplicating deserializer's cache never hit for refs that crossed an extension boundary -- even when an outer `DeduplicatingProtoConverter` was in effect on the rest of the plan. Two references to the same `DynamicFilterPhysicalExpr` (one inside an extension expr, one outside) reconstructed as distinct `Inner` allocations, so heap-max updates from a TopK SortExec failed to propagate to a FileScan predicate carried by a custom file source. This matches the existing plan-level pattern (`try_encode` and `try_decode` already take a `proto_converter` parameter, added when the plan codec was wired into the converter pipeline). This change extends the expr-level methods the same way: fn try_decode_expr( &self, buf: &[u8], inputs: &[Arc<dyn PhysicalExpr>], proto_converter: &dyn PhysicalProtoConverterExtension, // new ) -> Result<Arc<dyn PhysicalExpr>>; fn try_encode_expr( &self, node: &Arc<dyn PhysicalExpr>, buf: &mut Vec<u8>, proto_converter: &dyn PhysicalProtoConverterExtension, // new ) -> Result<()>; The two internal call sites in `serialize_physical_expr_with_converter` and `parse_physical_expr_with_converter` are updated to thread the active converter through. Downstream codecs can now route nested expression encode/decode through `proto_converter.physical_expr_to_proto` / `proto_converter.proto_to_physical_expr`, picking up dedup automatically. This is a breaking change for downstream codecs that override `try_encode_expr` / `try_decode_expr` -- they will need to add the new parameter (and may pass `_proto_converter` if they don't carry nested expressions). Codecs that only override `try_encode` / `try_decode` (plan-level) are unaffected. Wire format is unchanged (`PhysicalExprNode.expr_id` is unchanged); the fix is purely on the Rust trait API. Test: a new `extension_codec_expr_participates_in_deduplication` proto-roundtrip test constructs a `BinaryExpr` whose left operand is a bare `DynamicFilterPhysicalExpr` and right operand is a custom `WrapperExpr` whose extension codec embeds the same dynamic filter inside its serialized blob. After a `DeduplicatingProtoConverter` roundtrip, an `update()` applied to the bare-side decoded filter is observed by `current()` on the wrapped-side decoded filter, proving both refs back the same `Inner`. Without the converter-aware extension codec, this test fails because the two refs end up with distinct `Inner` allocations.
1 parent b8998c7 commit 7a58afe

4 files changed

Lines changed: 262 additions & 3 deletions

File tree

‎datafusion/proto/src/physical_plan/from_proto.rs‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -412,8 +412,11 @@ pub fn parse_physical_expr_with_converter(
412412
.iter()
413413
.map(|e| proto_converter.proto_to_physical_expr(e, input_schema, ctx))
414414
.collect::<Result<_>>()?;
415-
ctx.codec()
416-
.try_decode_expr(extension.expr.as_slice(), &inputs)? as _
415+
ctx.codec().try_decode_expr(
416+
extension.expr.as_slice(),
417+
&inputs,
418+
proto_converter,
419+
)? as _
417420
}
418421
};
419422

‎datafusion/proto/src/physical_plan/mod.rs‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3940,18 +3940,36 @@ pub trait PhysicalExtensionCodec: Debug + Send + Sync + Any {
39403940
Ok(())
39413941
}
39423942

3943+
/// Decode a custom extension expression from `buf`.
3944+
///
3945+
/// Implementations whose proto carries nested `PhysicalExprNode` fields
3946+
/// should route those through `proto_converter.proto_to_physical_expr`
3947+
/// (rather than the free `parse_physical_expr` function) so that an
3948+
/// active `DeduplicatingDeserializer` can cache-hit on matching
3949+
/// `expr_id`s and re-share `Arc<dyn PhysicalExpr>` (e.g. a
3950+
/// `DynamicFilterPhysicalExpr` referenced both from a SortExec.filter
3951+
/// and from a custom file-source's predicate field).
39433952
fn try_decode_expr(
39443953
&self,
39453954
_buf: &[u8],
39463955
_inputs: &[Arc<dyn PhysicalExpr>],
3956+
_proto_converter: &dyn PhysicalProtoConverterExtension,
39473957
) -> Result<Arc<dyn PhysicalExpr>> {
39483958
not_impl_err!("PhysicalExtensionCodec is not provided")
39493959
}
39503960

3961+
/// Encode a custom extension expression into `buf`.
3962+
///
3963+
/// Implementations whose proto carries nested `PhysicalExprNode` fields
3964+
/// should route those through `proto_converter.physical_expr_to_proto`
3965+
/// (rather than the free `serialize_physical_expr` function) so that
3966+
/// an active `DeduplicatingProtoConverter` can stamp matching `expr_id`s
3967+
/// for shared inner expressions. See [`Self::try_decode_expr`].
39513968
fn try_encode_expr(
39523969
&self,
39533970
_node: &Arc<dyn PhysicalExpr>,
39543971
_buf: &mut Vec<u8>,
3972+
_proto_converter: &dyn PhysicalProtoConverterExtension,
39553973
) -> Result<()> {
39563974
not_impl_err!("PhysicalExtensionCodec is not provided")
39573975
}

‎datafusion/proto/src/physical_plan/to_proto.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -365,7 +365,7 @@ pub fn serialize_physical_expr_with_converter(
365365
})
366366
} else {
367367
let mut buf: Vec<u8> = vec![];
368-
match codec.try_encode_expr(value, &mut buf) {
368+
match codec.try_encode_expr(value, &mut buf, proto_converter) {
369369
Ok(_) => {
370370
let inputs: Vec<protobuf::PhysicalExprNode> = value
371371
.children()

‎datafusion/proto/tests/cases/roundtrip_physical_plan.rs‎

Lines changed: 238 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1161,6 +1161,7 @@ fn roundtrip_parquet_exec_with_custom_predicate_expr() -> Result<()> {
11611161
&self,
11621162
buf: &[u8],
11631163
inputs: &[Arc<dyn PhysicalExpr>],
1164+
_proto_converter: &dyn PhysicalProtoConverterExtension,
11641165
) -> Result<Arc<dyn PhysicalExpr>> {
11651166
if buf == "CustomPredicateExpr".as_bytes() {
11661167
Ok(Arc::new(CustomPredicateExpr {
@@ -1175,6 +1176,7 @@ fn roundtrip_parquet_exec_with_custom_predicate_expr() -> Result<()> {
11751176
&self,
11761177
node: &Arc<dyn PhysicalExpr>,
11771178
buf: &mut Vec<u8>,
1179+
_proto_converter: &dyn PhysicalProtoConverterExtension,
11781180
) -> Result<()> {
11791181
if node.downcast_ref::<CustomPredicateExpr>().is_some() {
11801182
buf.extend_from_slice("CustomPredicateExpr".as_bytes());
@@ -4127,3 +4129,239 @@ fn roundtrip_parquet_exec_partitioned_by_file_group() -> Result<()> {
41274129

41284130
Ok(())
41294131
}
4132+
4133+
/// Demonstrates that `try_encode_expr` / `try_decode_expr` extension codecs
4134+
/// can now participate in the dedup pipeline by routing their nested
4135+
/// `PhysicalExprNode` fields through the active
4136+
/// `PhysicalProtoConverterExtension`.
4137+
///
4138+
/// Without the `proto_converter` parameter on the expr-level codec methods,
4139+
/// any nested `PhysicalExprNode` inside a custom expression's serialized
4140+
/// blob would skip dedup (`expr_id` would be `None` on the wire). Two
4141+
/// references to the same `DynamicFilterPhysicalExpr` (one inside an
4142+
/// extension expr, one outside) would reconstruct as two distinct Arcs on
4143+
/// the decode side, breaking heap-max propagation across plan-node
4144+
/// boundaries in distributed execution.
4145+
///
4146+
/// With the converter parameter plumbed through, the extension codec
4147+
/// delegates nested expr serialization to the active deduplicating
4148+
/// converter, and both refs cache-hit on the same `expr_id` on decode.
4149+
#[test]
4150+
fn extension_codec_expr_participates_in_deduplication() -> Result<()> {
4151+
use datafusion_common::DataFusionError;
4152+
use datafusion_physical_expr::expressions::DynamicFilterPhysicalExpr;
4153+
use prost::Message;
4154+
use std::fmt::{self, Display};
4155+
4156+
// A minimal custom expression whose only logical content is a nested
4157+
// `Arc<dyn PhysicalExpr>`. Its codec routes that nested expr through
4158+
// the active `PhysicalProtoConverterExtension`.
4159+
#[derive(Debug)]
4160+
struct WrapperExpr {
4161+
inner: Arc<dyn PhysicalExpr>,
4162+
}
4163+
impl Display for WrapperExpr {
4164+
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
4165+
write!(f, "WrapperExpr({})", self.inner)
4166+
}
4167+
}
4168+
impl std::hash::Hash for WrapperExpr {
4169+
fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
4170+
self.inner.dyn_hash(state);
4171+
}
4172+
}
4173+
impl PartialEq for WrapperExpr {
4174+
fn eq(&self, other: &Self) -> bool {
4175+
self.inner.eq(&other.inner)
4176+
}
4177+
}
4178+
impl Eq for WrapperExpr {}
4179+
impl PhysicalExpr for WrapperExpr {
4180+
fn data_type(&self, input_schema: &Schema) -> Result<DataType> {
4181+
self.inner.data_type(input_schema)
4182+
}
4183+
fn nullable(&self, input_schema: &Schema) -> Result<bool> {
4184+
self.inner.nullable(input_schema)
4185+
}
4186+
fn evaluate(
4187+
&self,
4188+
_batch: &arrow::record_batch::RecordBatch,
4189+
) -> Result<datafusion::physical_plan::ColumnarValue> {
4190+
internal_err!("WrapperExpr is not executable in this test")
4191+
}
4192+
fn children(&self) -> Vec<&Arc<dyn PhysicalExpr>> {
4193+
vec![&self.inner]
4194+
}
4195+
fn with_new_children(
4196+
self: Arc<Self>,
4197+
children: Vec<Arc<dyn PhysicalExpr>>,
4198+
) -> Result<Arc<dyn PhysicalExpr>> {
4199+
Ok(Arc::new(WrapperExpr {
4200+
inner: children[0].clone(),
4201+
}))
4202+
}
4203+
fn fmt_sql(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
4204+
write!(f, "{self}")
4205+
}
4206+
}
4207+
4208+
// Wire layout: a single nested PhysicalExprNode field.
4209+
#[derive(Clone, PartialEq, prost::Message)]
4210+
struct WrapperExprProto {
4211+
#[prost(message, optional, boxed, tag = "1")]
4212+
inner: Option<Box<datafusion_proto::protobuf::PhysicalExprNode>>,
4213+
}
4214+
4215+
#[derive(Debug)]
4216+
struct WrapperCodec;
4217+
impl PhysicalExtensionCodec for WrapperCodec {
4218+
fn try_decode(
4219+
&self,
4220+
_buf: &[u8],
4221+
_inputs: &[Arc<dyn ExecutionPlan>],
4222+
_ctx: &TaskContext,
4223+
_proto_converter: &dyn PhysicalProtoConverterExtension,
4224+
) -> Result<Arc<dyn ExecutionPlan>> {
4225+
internal_err!("not used")
4226+
}
4227+
fn try_encode(
4228+
&self,
4229+
_node: Arc<dyn ExecutionPlan>,
4230+
_buf: &mut Vec<u8>,
4231+
_proto_converter: &dyn PhysicalProtoConverterExtension,
4232+
) -> Result<()> {
4233+
internal_err!("not used")
4234+
}
4235+
fn try_decode_expr(
4236+
&self,
4237+
buf: &[u8],
4238+
_inputs: &[Arc<dyn PhysicalExpr>],
4239+
proto_converter: &dyn PhysicalProtoConverterExtension,
4240+
) -> Result<Arc<dyn PhysicalExpr>> {
4241+
let proto = WrapperExprProto::decode(buf).map_err(|e| {
4242+
DataFusionError::Internal(format!("decode WrapperExprProto: {e}"))
4243+
})?;
4244+
let inner_proto = proto
4245+
.inner
4246+
.ok_or_else(|| DataFusionError::Internal("missing inner".into()))?;
4247+
let schema = Schema::new(vec![Field::new("a", DataType::Int64, false)]);
4248+
let ctx = SessionContext::new();
4249+
let task_ctx = ctx.task_ctx();
4250+
let decode_ctx = PhysicalPlanDecodeContext::new(task_ctx.as_ref(), self);
4251+
let inner = proto_converter.proto_to_physical_expr(
4252+
&inner_proto,
4253+
&schema,
4254+
&decode_ctx,
4255+
)?;
4256+
Ok(Arc::new(WrapperExpr { inner }))
4257+
}
4258+
fn try_encode_expr(
4259+
&self,
4260+
node: &Arc<dyn PhysicalExpr>,
4261+
buf: &mut Vec<u8>,
4262+
proto_converter: &dyn PhysicalProtoConverterExtension,
4263+
) -> Result<()> {
4264+
let wrapper = node
4265+
.downcast_ref::<WrapperExpr>()
4266+
.ok_or_else(|| DataFusionError::Internal("not WrapperExpr".into()))?;
4267+
let inner_proto =
4268+
proto_converter.physical_expr_to_proto(&wrapper.inner, self)?;
4269+
let proto = WrapperExprProto {
4270+
inner: Some(Box::new(inner_proto)),
4271+
};
4272+
proto.encode(buf).map_err(|e| {
4273+
DataFusionError::Internal(format!("encode WrapperExprProto: {e}"))
4274+
})?;
4275+
Ok(())
4276+
}
4277+
}
4278+
4279+
// Build a single composite expression that holds TWO references to the
4280+
// same `Arc<DynamicFilterPhysicalExpr>`: a bare one on the left of a
4281+
// BinaryExpr, and one wrapped inside our extension expr on the right.
4282+
// Roundtripping the whole composite in a single decode call exercises
4283+
// a shared dedup cache.
4284+
let dyn_filter = make_dynamic_filter();
4285+
let wrapper: Arc<dyn PhysicalExpr> = Arc::new(WrapperExpr {
4286+
inner: Arc::clone(&dyn_filter),
4287+
});
4288+
let composite: Arc<dyn PhysicalExpr> = Arc::new(BinaryExpr::new(
4289+
Arc::clone(&dyn_filter),
4290+
Operator::And,
4291+
Arc::clone(&wrapper),
4292+
));
4293+
4294+
let codec = WrapperCodec;
4295+
let converter = DeduplicatingProtoConverter {};
4296+
4297+
let proto = converter.physical_expr_to_proto(&composite, &codec)?;
4298+
// Round-trip through prost bytes to mimic the wire.
4299+
let bytes = proto.encode_to_vec();
4300+
let decoded_proto =
4301+
datafusion_proto::protobuf::PhysicalExprNode::decode(bytes.as_slice()).unwrap();
4302+
4303+
let schema = Schema::new(vec![Field::new("a", DataType::Int64, false)]);
4304+
let ctx = SessionContext::new();
4305+
let task_ctx = ctx.task_ctx();
4306+
let decode_ctx = PhysicalPlanDecodeContext::new(task_ctx.as_ref(), &codec);
4307+
let decoded =
4308+
converter.proto_to_physical_expr(&decoded_proto, &schema, &decode_ctx)?;
4309+
4310+
let binary = decoded
4311+
.downcast_ref::<BinaryExpr>()
4312+
.expect("must decode back to BinaryExpr");
4313+
let decoded_left = Arc::clone(binary.left());
4314+
let decoded_right = Arc::clone(binary.right());
4315+
let decoded_wrapper = decoded_right
4316+
.downcast_ref::<WrapperExpr>()
4317+
.expect("right side must decode back to WrapperExpr");
4318+
4319+
// Semantic referential-integrity check: both decoded sides must
4320+
// observe the same logical filter. The bare side is a
4321+
// `DynamicFilterPhysicalExpr`; the wrapped side's inner is too. An
4322+
// `update()` on one must be visible from the other via
4323+
// `current()`, proving they share the same `Inner` (the entire
4324+
// point of #21807's dedup pipeline).
4325+
//
4326+
// Without the `proto_converter` parameter on `try_encode_expr` /
4327+
// `try_decode_expr`, the wrapper codec would route its nested
4328+
// expression through `DefaultPhysicalProtoConverter` and skip dedup
4329+
// entirely, so `decoded_left` and `decoded_wrapper.inner` would
4330+
// back distinct `Inner` allocations and an update would not
4331+
// propagate across the extension boundary.
4332+
let left_dyn = decoded_left
4333+
.downcast_ref::<DynamicFilterPhysicalExpr>()
4334+
.expect("decoded bare expr must still be DynamicFilterPhysicalExpr");
4335+
let right_dyn = decoded_wrapper
4336+
.inner
4337+
.downcast_ref::<DynamicFilterPhysicalExpr>()
4338+
.expect("decoded wrapped inner must still be DynamicFilterPhysicalExpr");
4339+
4340+
// Sanity: matching expression_id is necessary for dedup to fire.
4341+
assert_eq!(
4342+
left_dyn.expression_id(),
4343+
right_dyn.expression_id(),
4344+
"shared `Inner` must report the same expression_id"
4345+
);
4346+
4347+
// The load-bearing check: prove `Inner` really is shared by updating
4348+
// one side and reading from the other.
4349+
let original_generation = left_dyn.snapshot_generation();
4350+
let new_expr: Arc<dyn PhysicalExpr> = lit(false);
4351+
left_dyn.update(Arc::clone(&new_expr))?;
4352+
assert_eq!(
4353+
right_dyn.snapshot_generation(),
4354+
original_generation + 1,
4355+
"update on the bare ref must be visible from the wrapped ref \
4356+
(same Inner)"
4357+
);
4358+
let observed = right_dyn.current()?;
4359+
assert_eq!(
4360+
format!("{observed}"),
4361+
format!("{new_expr}"),
4362+
"wrapped ref's current() must reflect the update applied to the \
4363+
bare ref"
4364+
);
4365+
4366+
Ok(())
4367+
}

0 commit comments

Comments
 (0)