Skip to content

Commit 641f176

Browse files
authored
[X-2935] proto: dedupe DynamicFilterPhysicalExpr across extension codec boundary (#60)
* [X-2935] proto: serialize and dedupe DynamicFilterPhysicalExpr (minimal port of apache#21807) Adds expression_id to DynamicFilterPhysicalExpr Inner (atomic counter, preserved across update() and with_new_children) and a corresponding PhysicalDynamicFilterNode proto message. Serializer special-cases DynamicFilterPhysicalExpr to skip snapshot() and emit the wrapper full. Deserializer reconstructs the wrapper. Combined with the existing DeduplicatingSerializer + DeduplicatingDeserializer pair on branch-53, the wire-level Arc identity dedup now works for dynamic filters too: SortExec.filter and the FileScan predicate it was pushed down to reconstruct to the same Arc<Inner> on the data server, so heap-max updates propagate end-to-end without the atlas-side share_dynamic_filters walker. Minimal port: skips upstream's ExpressionPlacement enum, snapshot() refactor, and DedupingSerializer removal. Wire layout (field 23 oneof variant + PhysicalDynamicFilterNode fields) mirrors upstream so a future DF upgrade can drop this patch mechanically. DeduplicatingSerializer and DeduplicatingDeserializer are now pub so atlas's coordinator-side and data-server-side codecs can opt in. * [X-2935] dedupe DynamicFilter by expression_id, not Arc::as_ptr Mirrors apache#21807 step 3-4: switch DeduplicatingSerializer from hashing Arc::as_ptr to hashing the expression's expression_id when available. Falls back to Arc::as_ptr for expressions without a stable id. This fixes the case FilterPushdown produces in practice: pushing a DynamicFilterPhysicalExpr from a SortExec down to a FileScan creates a new outer Arc (via with_new_children to remap columns) that still shares the same Arc<RwLock<Inner>>. Without this change, the two outer Arcs hash to different addresses and DeduplicatingDeserializer rebuilds them as two distinct Arcs on decode -- exactly the X-2935 walker's bug. Adds a new test dynamic_filter_dedup_distinct_outer_arcs_same_inner that constructs the with_new_children case and asserts dedup works. 150/150 proto integration tests pass. * [X-2935] proto: serialize SortExec's dynamic filter end-to-end Adds the final piece needed to drop the atlas-side share_dynamic_filters walker: SortExecNode now carries its internal DynamicFilterPhysicalExpr on the wire, and try_into_sort_physical_plan re-installs it via SortExec::with_dynamic_filter_expr instead of letting with_fetch(N) mint a brand-new one via create_filter() on decode. Combined with the prior two commits in this series (apache#21807 minimal port + dedup-by-expression_id), the decode side now reconstructs the SAME Arc<DynamicFilterPhysicalExpr> shared between the SortExec.filter and any pushed-down FileScan predicate that participated in the encode -- because DeduplicatingDeserializer's cache, keyed on the wire expr_id that DeduplicatingSerializer derives from expression_id, matches across plan nodes. Adds: - SortExec::dynamic_filter_expr() / with_dynamic_filter_expr() accessors - dynamic_filter field on SortExecNode proto (field 5) - to_proto: try_from_sort_exec serializes the dyn filter via proto_converter - from_proto: try_into_sort_physical_plan deserializes + reinstalls via with_dynamic_filter_expr; legacy fallback wraps a fresh Arc when the encoder snapshotted the wrapper away Test sort_exec_proto_roundtrip_preserves_dyn_filter_arc asserts end-to-end: serialize a SortExec with a known DynamicFilterPhysicalExpr, roundtrip via DeduplicatingSerializer + DeduplicatingDeserializer, the decoded SortExec's filter must carry the same expression_id. Ported from apache#22011 (minimal subset, adapted to branch-53's PhysicalProtoConverterExtension signatures). 151/151 proto integration tests pass. * [X-2935] PhysicalExtensionCodec: thread proto_converter through methods Modifies the PhysicalExtensionCodec trait so that try_encode, try_decode, try_encode_expr, and try_decode_expr receive the active PhysicalProtoConverterExtension. Codecs that embed nested PhysicalExprNode fields inside their custom proto (e.g. atlas's ReverseParquetNode.predicate) must now route those through proto_converter.proto_to_physical_expr / proto_converter.physical_expr_to_proto so the dedup cache extends across the extension boundary. Without this plumbing, an outer DeduplicatingSerializer can't reach into extension nodes -- the dedup cache stops at the extension boundary and Arc<Inner> sharing breaks for any expression carried inside an extension codec, even though the wire format already supports it via the expr_id field. With this change, atlas can finally drop the share_dynamic_filters walker once its codecs override the new methods. This is the breaking-change pattern matching the apache/datafusion upstream design (issue apache#22920) instead of additive _with_converter variants -- branch-53 is internal-only so callers can be updated in lockstep.
1 parent ddb06d5 commit 641f176

15 files changed

Lines changed: 1076 additions & 42 deletions

File tree

‎datafusion-examples/examples/custom_data_source/adapter_serialization.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -274,6 +274,7 @@ impl PhysicalExtensionCodec for AdapterPreservingCodec {
274274
buf: &[u8],
275275
inputs: &[Arc<dyn ExecutionPlan>],
276276
_ctx: &TaskContext,
277+
_proto_converter: &dyn PhysicalProtoConverterExtension,
277278
) -> Result<Arc<dyn ExecutionPlan>> {
278279
// Try to parse as our extension payload
279280
if let Ok(payload) = serde_json::from_slice::<ExtensionPayload>(buf)
@@ -302,6 +303,7 @@ impl PhysicalExtensionCodec for AdapterPreservingCodec {
302303
&self,
303304
_node: Arc<dyn ExecutionPlan>,
304305
_buf: &mut Vec<u8>,
306+
_proto_converter: &dyn PhysicalProtoConverterExtension,
305307
) -> Result<()> {
306308
// We don't need this for the example - we use serialize_physical_plan instead
307309
not_impl_err!(

‎datafusion-examples/examples/proto/composed_extension_codec.rs‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@ use datafusion::physical_plan::{DisplayAs, ExecutionPlan};
4343
use datafusion::prelude::SessionContext;
4444
use datafusion_proto::physical_plan::{
4545
AsExecutionPlan, ComposedPhysicalExtensionCodec, PhysicalExtensionCodec,
46+
PhysicalProtoConverterExtension,
4647
};
4748
use datafusion_proto::protobuf;
4849

@@ -140,6 +141,7 @@ impl PhysicalExtensionCodec for ParentPhysicalExtensionCodec {
140141
buf: &[u8],
141142
inputs: &[Arc<dyn ExecutionPlan>],
142143
_ctx: &TaskContext,
144+
_proto_converter: &dyn PhysicalProtoConverterExtension,
143145
) -> Result<Arc<dyn ExecutionPlan>> {
144146
if buf == "ParentExec".as_bytes() {
145147
Ok(Arc::new(ParentExec {
@@ -150,7 +152,12 @@ impl PhysicalExtensionCodec for ParentPhysicalExtensionCodec {
150152
}
151153
}
152154

153-
fn try_encode(&self, node: Arc<dyn ExecutionPlan>, buf: &mut Vec<u8>) -> Result<()> {
155+
fn try_encode(
156+
&self,
157+
node: Arc<dyn ExecutionPlan>,
158+
buf: &mut Vec<u8>,
159+
_proto_converter: &dyn PhysicalProtoConverterExtension,
160+
) -> Result<()> {
154161
if node.as_any().downcast_ref::<ParentExec>().is_some() {
155162
buf.extend_from_slice("ParentExec".as_bytes());
156163
Ok(())
@@ -216,6 +223,7 @@ impl PhysicalExtensionCodec for ChildPhysicalExtensionCodec {
216223
buf: &[u8],
217224
_inputs: &[Arc<dyn ExecutionPlan>],
218225
_ctx: &TaskContext,
226+
_proto_converter: &dyn PhysicalProtoConverterExtension,
219227
) -> Result<Arc<dyn ExecutionPlan>> {
220228
if buf == "ChildExec".as_bytes() {
221229
Ok(Arc::new(ChildExec {}))
@@ -224,7 +232,12 @@ impl PhysicalExtensionCodec for ChildPhysicalExtensionCodec {
224232
}
225233
}
226234

227-
fn try_encode(&self, node: Arc<dyn ExecutionPlan>, buf: &mut Vec<u8>) -> Result<()> {
235+
fn try_encode(
236+
&self,
237+
node: Arc<dyn ExecutionPlan>,
238+
buf: &mut Vec<u8>,
239+
_proto_converter: &dyn PhysicalProtoConverterExtension,
240+
) -> Result<()> {
228241
if node.as_any().downcast_ref::<ChildExec>().is_some() {
229242
buf.extend_from_slice("ChildExec".as_bytes());
230243
Ok(())

‎datafusion-examples/examples/proto/expression_deduplication.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -187,6 +187,7 @@ impl PhysicalExtensionCodec for CachingCodec {
187187
_buf: &[u8],
188188
_inputs: &[Arc<dyn ExecutionPlan>],
189189
_ctx: &TaskContext,
190+
_proto_converter: &dyn PhysicalProtoConverterExtension,
190191
) -> Result<Arc<dyn ExecutionPlan>> {
191192
datafusion::common::not_impl_err!("No custom extension nodes")
192193
}
@@ -196,6 +197,7 @@ impl PhysicalExtensionCodec for CachingCodec {
196197
&self,
197198
_node: Arc<dyn ExecutionPlan>,
198199
_buf: &mut Vec<u8>,
200+
_proto_converter: &dyn PhysicalProtoConverterExtension,
199201
) -> Result<()> {
200202
datafusion::common::not_impl_err!("No custom extension nodes")
201203
}

‎datafusion/ffi/src/proto/physical_extension_codec.rs‎

Lines changed: 34 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -141,8 +141,16 @@ unsafe extern "C" fn try_decode_fn_wrapper(
141141
.collect::<Result<Vec<_>>>();
142142
let inputs = rresult_return!(inputs);
143143

144-
let plan =
145-
rresult_return!(codec.try_decode(buf.as_ref(), &inputs, task_ctx.as_ref()));
144+
// FFI boundary doesn't propagate the active PhysicalProtoConverterExtension,
145+
// so dedup of nested expressions stops here (matches pre-#21807 behavior).
146+
let proto_converter =
147+
datafusion_proto::physical_plan::DefaultPhysicalProtoConverter {};
148+
let plan = rresult_return!(codec.try_decode(
149+
buf.as_ref(),
150+
&inputs,
151+
task_ctx.as_ref(),
152+
&proto_converter,
153+
));
146154

147155
RResult::ROk(FFI_ExecutionPlan::new(plan, None))
148156
}
@@ -156,7 +164,11 @@ unsafe extern "C" fn try_encode_fn_wrapper(
156164
let plan: Arc<dyn ExecutionPlan> = rresult_return!((&node).try_into());
157165

158166
let mut bytes = Vec::new();
159-
rresult_return!(codec.try_encode(plan, &mut bytes));
167+
// FFI boundary doesn't propagate the active PhysicalProtoConverterExtension,
168+
// so dedup of nested expressions stops here (matches pre-#21807 behavior).
169+
let proto_converter =
170+
datafusion_proto::physical_plan::DefaultPhysicalProtoConverter {};
171+
rresult_return!(codec.try_encode(plan, &mut bytes, &proto_converter));
160172

161173
RResult::ROk(bytes.into())
162174
}
@@ -327,6 +339,7 @@ impl PhysicalExtensionCodec for ForeignPhysicalExtensionCodec {
327339
buf: &[u8],
328340
inputs: &[Arc<dyn ExecutionPlan>],
329341
_ctx: &TaskContext,
342+
_proto_converter: &dyn datafusion_proto::physical_plan::PhysicalProtoConverterExtension,
330343
) -> Result<Arc<dyn ExecutionPlan>> {
331344
let inputs = inputs
332345
.iter()
@@ -340,7 +353,12 @@ impl PhysicalExtensionCodec for ForeignPhysicalExtensionCodec {
340353
Ok(plan)
341354
}
342355

343-
fn try_encode(&self, node: Arc<dyn ExecutionPlan>, buf: &mut Vec<u8>) -> Result<()> {
356+
fn try_encode(
357+
&self,
358+
node: Arc<dyn ExecutionPlan>,
359+
buf: &mut Vec<u8>,
360+
_proto_converter: &dyn datafusion_proto::physical_plan::PhysicalProtoConverterExtension,
361+
) -> Result<()> {
344362
let plan = FFI_ExecutionPlan::new(node, None);
345363
let bytes = df_result!(unsafe { (self.0.try_encode)(&self.0, plan) })?;
346364

@@ -441,6 +459,7 @@ pub(crate) mod tests {
441459
buf: &[u8],
442460
_inputs: &[Arc<dyn ExecutionPlan>],
443461
_ctx: &TaskContext,
462+
_proto_converter: &dyn datafusion_proto::physical_plan::PhysicalProtoConverterExtension,
444463
) -> Result<Arc<dyn ExecutionPlan>> {
445464
if buf[0] != Self::MAGIC_NUMBER {
446465
return exec_err!(
@@ -459,6 +478,7 @@ pub(crate) mod tests {
459478
&self,
460479
node: Arc<dyn ExecutionPlan>,
461480
buf: &mut Vec<u8>,
481+
_proto_converter: &dyn datafusion_proto::physical_plan::PhysicalProtoConverterExtension,
462482
) -> Result<()> {
463483
buf.push(Self::MAGIC_NUMBER);
464484

@@ -579,10 +599,16 @@ pub(crate) mod tests {
579599
let exec = create_test_exec();
580600
let input_execs = [create_test_exec()];
581601
let mut bytes = Vec::new();
582-
foreign_codec.try_encode(Arc::clone(&exec), &mut bytes)?;
583-
584-
let returned_exec =
585-
foreign_codec.try_decode(&bytes, &input_execs, ctx.task_ctx().as_ref())?;
602+
let proto_converter =
603+
datafusion_proto::physical_plan::DefaultPhysicalProtoConverter {};
604+
foreign_codec.try_encode(Arc::clone(&exec), &mut bytes, &proto_converter)?;
605+
606+
let returned_exec = foreign_codec.try_decode(
607+
&bytes,
608+
&input_execs,
609+
ctx.task_ctx().as_ref(),
610+
&proto_converter,
611+
)?;
586612

587613
assert!(returned_exec.as_any().is::<EmptyExec>());
588614

‎datafusion/physical-expr-common/src/physical_expr.rs‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -441,6 +441,24 @@ pub trait PhysicalExpr: Any + Send + Sync + Display + Debug + DynEq + DynHash {
441441
fn placement(&self) -> ExpressionPlacement {
442442
ExpressionPlacement::KeepInPlace
443443
}
444+
445+
/// Stable, globally-unique identifier for this [`PhysicalExpr`], if any.
446+
///
447+
/// Connected expressions (e.g. `DynamicFilterPhysicalExpr`s where two
448+
/// outer Arc instances share the same mutable inner state via
449+
/// [`PhysicalExpr::with_new_children`]) must report the same id. Proto
450+
/// (de)serialization uses this to dedupe across the wire so the
451+
/// reconstructed plan keeps `Arc<Inner>` shared.
452+
///
453+
/// Default is `None`: no identity worth preserving across a
454+
/// serialization boundary.
455+
///
456+
/// Ported from upstream apache/datafusion#21807 to make
457+
/// `DynamicFilterPhysicalExpr` dedup robust to pushdown rewriting on
458+
/// branch-53.
459+
fn expression_id(&self) -> Option<u64> {
460+
None
461+
}
444462
}
445463

446464
#[deprecated(

‎datafusion/physical-expr/src/expressions/dynamic_filters.rs‎

Lines changed: 126 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
// under the License.
1717

1818
use parking_lot::RwLock;
19+
use std::sync::atomic::{AtomicU64, Ordering};
1920
use std::{any::Any, fmt::Display, hash::Hash, sync::Arc};
2021
use tokio::sync::watch;
2122

@@ -76,21 +77,50 @@ pub struct DynamicFilterPhysicalExpr {
7677
nullable: Arc<RwLock<Option<bool>>>,
7778
}
7879

79-
#[derive(Debug)]
80-
struct Inner {
81-
/// A counter that gets incremented every time the expression is updated so that we can track changes cheaply.
82-
/// This is used for [`PhysicalExpr::snapshot_generation`] to have a cheap check for changes.
83-
generation: u64,
84-
expr: Arc<dyn PhysicalExpr>,
85-
/// Flag for quick synchronous check if filter is complete.
86-
/// This is redundant with the watch channel state, but allows us to return immediately
87-
/// from `wait_complete()` without subscribing if already complete.
88-
is_complete: bool,
80+
/// Atomic internal state of a [`DynamicFilterPhysicalExpr`].
81+
///
82+
/// `expression_id` identifies the actual filter expression. Derived
83+
/// `DynamicFilterPhysicalExpr`s (e.g. via [`PhysicalExpr::with_new_children`])
84+
/// are the same logical filter and must report the same id. Proto
85+
/// (de)serialization uses this id to dedupe references on the wire so the
86+
/// reconstructed plan keeps `Arc<Inner>` shared between e.g. SortExec.filter
87+
/// and the FileScan predicate it was pushed down to.
88+
///
89+
/// **Warning:** exposed publicly only so that `datafusion-proto` can read
90+
/// and rebuild this state. Not a stable API.
91+
///
92+
/// Ported from upstream apache/datafusion#21807 (minimal subset) so
93+
/// branch-53 can drop the atlas-side post-deserialize walker.
94+
#[derive(Clone)]
95+
pub struct Inner {
96+
/// Stable id, preserved across `update()` and `with_new_children`.
97+
pub expression_id: u64,
98+
/// Incremented on every `update()` so that [`PhysicalExpr::snapshot_generation`]
99+
/// is a cheap "did anything change?" probe.
100+
pub generation: u64,
101+
pub expr: Arc<dyn PhysicalExpr>,
102+
/// Synchronous check for completion; redundant with the watch channel
103+
/// but lets `wait_complete()` return without subscribing.
104+
pub is_complete: bool,
105+
}
106+
107+
// Exclude `expression_id` from Debug so existing roundtrip Debug comparisons
108+
// in tests still pass even when only some plan-node decoders carry the
109+
// dynamic filter through proto.
110+
impl std::fmt::Debug for Inner {
111+
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
112+
f.debug_struct("Inner")
113+
.field("generation", &self.generation)
114+
.field("expr", &self.expr)
115+
.field("is_complete", &self.is_complete)
116+
.finish()
117+
}
89118
}
90119

91120
impl Inner {
92121
fn new(expr: Arc<dyn PhysicalExpr>) -> Self {
93122
Self {
123+
expression_id: EXPR_ID_SOURCE.next(),
94124
// Start with generation 1 which gives us a different result for [`PhysicalExpr::generation`] than the default 0.
95125
// This is not currently used anywhere but it seems useful to have this simple distinction.
96126
generation: 1,
@@ -105,6 +135,26 @@ impl Inner {
105135
}
106136
}
107137

138+
/// An atomic counter used to generate monotonic u64 ids for
139+
/// `DynamicFilterPhysicalExpr` instances.
140+
struct ExpressionIdAtomicCounter {
141+
inner: AtomicU64,
142+
}
143+
144+
impl ExpressionIdAtomicCounter {
145+
const fn new() -> Self {
146+
Self {
147+
inner: AtomicU64::new(0),
148+
}
149+
}
150+
151+
fn next(&self) -> u64 {
152+
self.inner.fetch_add(1, Ordering::Relaxed)
153+
}
154+
}
155+
156+
static EXPR_ID_SOURCE: ExpressionIdAtomicCounter = ExpressionIdAtomicCounter::new();
157+
108158
impl Hash for DynamicFilterPhysicalExpr {
109159
fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
110160
// Use pointer identity of the inner Arc for stable hashing.
@@ -243,6 +293,10 @@ impl DynamicFilterPhysicalExpr {
243293
let mut current = self.inner.write();
244294
let new_generation = current.generation + 1;
245295
*current = Inner {
296+
// Preserve the expression id across updates so all references
297+
// (e.g. SortExec.filter + a pushed-down FileScan predicate)
298+
// continue to identify the same logical filter.
299+
expression_id: current.expression_id,
246300
generation: new_generation,
247301
expr: new_expr,
248302
is_complete: current.is_complete,
@@ -346,6 +400,64 @@ impl DynamicFilterPhysicalExpr {
346400

347401
write!(f, " ]")
348402
}
403+
404+
/// Return the filter's original children (before any remapping).
405+
///
406+
/// **Warning:** intended only for `datafusion-proto` (de)serialization.
407+
/// Not a stable API.
408+
pub fn original_children(&self) -> &[Arc<dyn PhysicalExpr>] {
409+
&self.children
410+
}
411+
412+
/// Return the filter's remapped children, if any have been set via
413+
/// [`PhysicalExpr::with_new_children`].
414+
///
415+
/// **Warning:** intended only for `datafusion-proto` (de)serialization.
416+
/// Not a stable API.
417+
pub fn remapped_children(&self) -> Option<&[Arc<dyn PhysicalExpr>]> {
418+
self.remapped_children.as_deref()
419+
}
420+
421+
/// Clone the atomically-captured `Inner` state.
422+
///
423+
/// **Warning:** intended only for `datafusion-proto` (de)serialization.
424+
/// Not a stable API.
425+
pub fn inner(&self) -> Inner {
426+
self.inner.read().clone()
427+
}
428+
429+
/// Rebuild a `DynamicFilterPhysicalExpr` from its stored parts. Used by
430+
/// proto deserialization to reconstruct the filter; when paired with the
431+
/// existing `DeduplicatingDeserializer` it restores shared `Arc<Inner>`
432+
/// identity across all references in the decoded plan.
433+
///
434+
/// **Warning:** intended only for `datafusion-proto` (de)serialization.
435+
/// Not a stable API.
436+
pub fn from_parts(
437+
children: Vec<Arc<dyn PhysicalExpr>>,
438+
remapped_children: Option<Vec<Arc<dyn PhysicalExpr>>>,
439+
inner: Inner,
440+
) -> Self {
441+
let state = if inner.is_complete {
442+
FilterState::Complete {
443+
generation: inner.generation,
444+
}
445+
} else {
446+
FilterState::InProgress {
447+
generation: inner.generation,
448+
}
449+
};
450+
let (state_watch, _) = watch::channel(state);
451+
452+
Self {
453+
children,
454+
remapped_children,
455+
inner: Arc::new(RwLock::new(inner)),
456+
state_watch,
457+
data_type: Arc::new(RwLock::new(None)),
458+
nullable: Arc::new(RwLock::new(None)),
459+
}
460+
}
349461
}
350462

351463
impl PhysicalExpr for DynamicFilterPhysicalExpr {
@@ -448,6 +560,10 @@ impl PhysicalExpr for DynamicFilterPhysicalExpr {
448560
// Return the current generation of the expression.
449561
self.inner.read().generation
450562
}
563+
564+
fn expression_id(&self) -> Option<u64> {
565+
Some(self.inner.read().expression_id)
566+
}
451567
}
452568

453569
#[cfg(test)]

‎datafusion/physical-expr/src/expressions/mod.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ pub use cast::{CastExpr, cast};
4545
pub use cast_column::CastColumnExpr;
4646
pub use column::{Column, col, with_new_schema};
4747
pub use datafusion_expr::utils::format_state_name;
48-
pub use dynamic_filters::DynamicFilterPhysicalExpr;
48+
pub use dynamic_filters::{DynamicFilterPhysicalExpr, Inner as DynamicFilterInner};
4949
pub use in_list::{InListExpr, in_list};
5050
pub use is_not_null::{IsNotNullExpr, is_not_null};
5151
pub use is_null::{IsNullExpr, is_null};

0 commit comments

Comments
 (0)