Skip to content

Commit e4b0494

Browse files
zhuqi-lucasMassivePizza
authored andcommitted
[X-2935] proto: dedupe DynamicFilterPhysicalExpr across extension codec boundary (#60)
* [X-2935] proto: serialize and dedupe DynamicFilterPhysicalExpr (minimal port of apache#21807) Adds expression_id to DynamicFilterPhysicalExpr Inner (atomic counter, preserved across update() and with_new_children) and a corresponding PhysicalDynamicFilterNode proto message. Serializer special-cases DynamicFilterPhysicalExpr to skip snapshot() and emit the wrapper full. Deserializer reconstructs the wrapper. Combined with the existing DeduplicatingSerializer + DeduplicatingDeserializer pair on branch-53, the wire-level Arc identity dedup now works for dynamic filters too: SortExec.filter and the FileScan predicate it was pushed down to reconstruct to the same Arc<Inner> on the data server, so heap-max updates propagate end-to-end without the atlas-side share_dynamic_filters walker. Minimal port: skips upstream's ExpressionPlacement enum, snapshot() refactor, and DedupingSerializer removal. Wire layout (field 23 oneof variant + PhysicalDynamicFilterNode fields) mirrors upstream so a future DF upgrade can drop this patch mechanically. DeduplicatingSerializer and DeduplicatingDeserializer are now pub so atlas's coordinator-side and data-server-side codecs can opt in. * [X-2935] dedupe DynamicFilter by expression_id, not Arc::as_ptr Mirrors apache#21807 step 3-4: switch DeduplicatingSerializer from hashing Arc::as_ptr to hashing the expression's expression_id when available. Falls back to Arc::as_ptr for expressions without a stable id. This fixes the case FilterPushdown produces in practice: pushing a DynamicFilterPhysicalExpr from a SortExec down to a FileScan creates a new outer Arc (via with_new_children to remap columns) that still shares the same Arc<RwLock<Inner>>. Without this change, the two outer Arcs hash to different addresses and DeduplicatingDeserializer rebuilds them as two distinct Arcs on decode -- exactly the X-2935 walker's bug. Adds a new test dynamic_filter_dedup_distinct_outer_arcs_same_inner that constructs the with_new_children case and asserts dedup works. 150/150 proto integration tests pass. * [X-2935] proto: serialize SortExec's dynamic filter end-to-end Adds the final piece needed to drop the atlas-side share_dynamic_filters walker: SortExecNode now carries its internal DynamicFilterPhysicalExpr on the wire, and try_into_sort_physical_plan re-installs it via SortExec::with_dynamic_filter_expr instead of letting with_fetch(N) mint a brand-new one via create_filter() on decode. Combined with the prior two commits in this series (apache#21807 minimal port + dedup-by-expression_id), the decode side now reconstructs the SAME Arc<DynamicFilterPhysicalExpr> shared between the SortExec.filter and any pushed-down FileScan predicate that participated in the encode -- because DeduplicatingDeserializer's cache, keyed on the wire expr_id that DeduplicatingSerializer derives from expression_id, matches across plan nodes. Adds: - SortExec::dynamic_filter_expr() / with_dynamic_filter_expr() accessors - dynamic_filter field on SortExecNode proto (field 5) - to_proto: try_from_sort_exec serializes the dyn filter via proto_converter - from_proto: try_into_sort_physical_plan deserializes + reinstalls via with_dynamic_filter_expr; legacy fallback wraps a fresh Arc when the encoder snapshotted the wrapper away Test sort_exec_proto_roundtrip_preserves_dyn_filter_arc asserts end-to-end: serialize a SortExec with a known DynamicFilterPhysicalExpr, roundtrip via DeduplicatingSerializer + DeduplicatingDeserializer, the decoded SortExec's filter must carry the same expression_id. Ported from apache#22011 (minimal subset, adapted to branch-53's PhysicalProtoConverterExtension signatures). 151/151 proto integration tests pass. * [X-2935] PhysicalExtensionCodec: thread proto_converter through methods Modifies the PhysicalExtensionCodec trait so that try_encode, try_decode, try_encode_expr, and try_decode_expr receive the active PhysicalProtoConverterExtension. Codecs that embed nested PhysicalExprNode fields inside their custom proto (e.g. atlas's ReverseParquetNode.predicate) must now route those through proto_converter.proto_to_physical_expr / proto_converter.physical_expr_to_proto so the dedup cache extends across the extension boundary. Without this plumbing, an outer DeduplicatingSerializer can't reach into extension nodes -- the dedup cache stops at the extension boundary and Arc<Inner> sharing breaks for any expression carried inside an extension codec, even though the wire format already supports it via the expr_id field. With this change, atlas can finally drop the share_dynamic_filters walker once its codecs override the new methods. This is the breaking-change pattern matching the apache/datafusion upstream design (issue apache#22920) instead of additive _with_converter variants -- branch-53 is internal-only so callers can be updated in lockstep.
1 parent de34840 commit e4b0494

12 files changed

Lines changed: 767 additions & 822 deletions

File tree

‎datafusion-examples/examples/proto/composed_extension_codec.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,7 @@ impl PhysicalExtensionCodec for ParentPhysicalExtensionCodec {
153153
buf: &mut Vec<u8>,
154154
_proto_converter: &dyn PhysicalProtoConverterExtension,
155155
) -> Result<()> {
156-
if node.is::<ParentExec>() {
156+
if node.as_any().downcast_ref::<ParentExec>().is_some() {
157157
buf.extend_from_slice("ParentExec".as_bytes());
158158
Ok(())
159159
} else {
@@ -229,7 +229,7 @@ impl PhysicalExtensionCodec for ChildPhysicalExtensionCodec {
229229
buf: &mut Vec<u8>,
230230
_proto_converter: &dyn PhysicalProtoConverterExtension,
231231
) -> Result<()> {
232-
if node.is::<ChildExec>() {
232+
if node.as_any().downcast_ref::<ChildExec>().is_some() {
233233
buf.extend_from_slice("ChildExec".as_bytes());
234234
Ok(())
235235
} else {

‎datafusion/ffi/src/proto/physical_extension_codec.rs‎

Lines changed: 19 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -148,11 +148,15 @@ unsafe extern "C" fn try_decode_fn_wrapper(
148148
.collect::<Result<Vec<_>>>();
149149
let inputs = sresult_return!(inputs);
150150

151-
let plan = sresult_return!(codec.try_decode(
151+
// FFI boundary doesn't propagate the active PhysicalProtoConverterExtension,
152+
// so dedup of nested expressions stops here (matches pre-#21807 behavior).
153+
let proto_converter =
154+
datafusion_proto::physical_plan::DefaultPhysicalProtoConverter {};
155+
let plan = rresult_return!(codec.try_decode(
152156
buf.as_ref(),
153157
&inputs,
154158
task_ctx.as_ref(),
155-
&DefaultPhysicalProtoConverter {},
159+
&proto_converter,
156160
));
157161

158162
FFI_Result::Ok(FFI_ExecutionPlan::new(plan, runtime))
@@ -167,11 +171,11 @@ unsafe extern "C" fn try_encode_fn_wrapper(
167171
let plan: Arc<dyn ExecutionPlan> = sresult_return!((&node).try_into());
168172

169173
let mut bytes = Vec::new();
170-
sresult_return!(codec.try_encode(
171-
plan,
172-
&mut bytes,
173-
&DefaultPhysicalProtoConverter {}
174-
));
174+
// FFI boundary doesn't propagate the active PhysicalProtoConverterExtension,
175+
// so dedup of nested expressions stops here (matches pre-#21807 behavior).
176+
let proto_converter =
177+
datafusion_proto::physical_plan::DefaultPhysicalProtoConverter {};
178+
rresult_return!(codec.try_encode(plan, &mut bytes, &proto_converter));
175179

176180
FFI_Result::Ok(bytes.into_iter().collect())
177181
}
@@ -346,7 +350,7 @@ impl PhysicalExtensionCodec for ForeignPhysicalExtensionCodec {
346350
buf: &[u8],
347351
inputs: &[Arc<dyn ExecutionPlan>],
348352
_ctx: &TaskContext,
349-
_proto_converter: &dyn PhysicalProtoConverterExtension,
353+
_proto_converter: &dyn datafusion_proto::physical_plan::PhysicalProtoConverterExtension,
350354
) -> Result<Arc<dyn ExecutionPlan>> {
351355
let inputs = inputs
352356
.iter()
@@ -364,7 +368,7 @@ impl PhysicalExtensionCodec for ForeignPhysicalExtensionCodec {
364368
&self,
365369
node: Arc<dyn ExecutionPlan>,
366370
buf: &mut Vec<u8>,
367-
_proto_converter: &dyn PhysicalProtoConverterExtension,
371+
_proto_converter: &dyn datafusion_proto::physical_plan::PhysicalProtoConverterExtension,
368372
) -> Result<()> {
369373
let plan = FFI_ExecutionPlan::new(node, None);
370374
let bytes = df_result!(unsafe { (self.0.try_encode)(&self.0, plan) })?;
@@ -469,7 +473,7 @@ pub(crate) mod tests {
469473
buf: &[u8],
470474
_inputs: &[Arc<dyn ExecutionPlan>],
471475
_ctx: &TaskContext,
472-
_proto_converter: &dyn PhysicalProtoConverterExtension,
476+
_proto_converter: &dyn datafusion_proto::physical_plan::PhysicalProtoConverterExtension,
473477
) -> Result<Arc<dyn ExecutionPlan>> {
474478
if buf[0] != Self::MAGIC_NUMBER {
475479
return exec_err!(
@@ -488,7 +492,7 @@ pub(crate) mod tests {
488492
&self,
489493
node: Arc<dyn ExecutionPlan>,
490494
buf: &mut Vec<u8>,
491-
_proto_converter: &dyn PhysicalProtoConverterExtension,
495+
_proto_converter: &dyn datafusion_proto::physical_plan::PhysicalProtoConverterExtension,
492496
) -> Result<()> {
493497
buf.push(Self::MAGIC_NUMBER);
494498

@@ -609,17 +613,15 @@ pub(crate) mod tests {
609613
let exec = create_test_exec();
610614
let input_execs = [create_test_exec()];
611615
let mut bytes = Vec::new();
612-
foreign_codec.try_encode(
613-
Arc::clone(&exec),
614-
&mut bytes,
615-
&DefaultPhysicalProtoConverter {},
616-
)?;
616+
let proto_converter =
617+
datafusion_proto::physical_plan::DefaultPhysicalProtoConverter {};
618+
foreign_codec.try_encode(Arc::clone(&exec), &mut bytes, &proto_converter)?;
617619

618620
let returned_exec = foreign_codec.try_decode(
619621
&bytes,
620622
&input_execs,
621623
ctx.task_ctx().as_ref(),
622-
&DefaultPhysicalProtoConverter {},
624+
&proto_converter,
623625
)?;
624626

625627
assert!(returned_exec.is::<EmptyExec>());

‎datafusion/physical-expr-common/src/physical_expr.rs‎

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -461,19 +461,20 @@ pub trait PhysicalExpr: Any + Send + Sync + Display + Debug + DynEq + DynHash {
461461
ExpressionPlacement::KeepInPlace
462462
}
463463

464-
/// Return a stable, globally-unique identifier for this [`PhysicalExpr`], if it
465-
/// has one.
464+
/// Stable, globally-unique identifier for this [`PhysicalExpr`], if any.
466465
///
467-
/// This identifier tracks which expressions which are connected (e.g. `DynamicFilterPhysicalExpr`
468-
/// where two expressions may be different but store the same mutable inner state). Tracking
469-
/// connected expressions helps preserve referential integrity within plan nodes
470-
/// during serialization and deserialization.
466+
/// Connected expressions (e.g. `DynamicFilterPhysicalExpr`s where two
467+
/// outer Arc instances share the same mutable inner state via
468+
/// [`PhysicalExpr::with_new_children`]) must report the same id. Proto
469+
/// (de)serialization uses this to dedupe across the wire so the
470+
/// reconstructed plan keeps `Arc<Inner>` shared.
471471
///
472-
/// This id must be preserved across [`PhysicalExpr::with_new_children`] or any other
473-
/// methods which may want to preserve identity.
474-
///
475-
/// Default is `None`: the expression has no identity worth preserving across a
472+
/// Default is `None`: no identity worth preserving across a
476473
/// serialization boundary.
474+
///
475+
/// Ported from upstream apache/datafusion#21807 to make
476+
/// `DynamicFilterPhysicalExpr` dedup robust to pushdown rewriting on
477+
/// branch-53.
477478
fn expression_id(&self) -> Option<u64> {
478479
None
479480
}

‎datafusion/physical-expr/src/expressions/dynamic_filters/mod.rs‎

Lines changed: 92 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717

1818
use parking_lot::RwLock;
1919
use std::sync::atomic::{AtomicU64, Ordering};
20-
use std::{fmt::Display, hash::Hash, sync::Arc};
20+
use std::{any::Any, fmt::Display, hash::Hash, sync::Arc};
2121
use tokio::sync::watch;
2222

2323
use crate::PhysicalExpr;
@@ -115,21 +115,42 @@ impl std::fmt::Debug for DynamicFilterPhysicalExpr {
115115

116116
/// Atomic internal state of a [`DynamicFilterPhysicalExpr`].
117117
///
118-
/// `expression_id` lives here because it identifies the actual filter expression `expr`.
119-
/// Derived `DynamicFilterPhysicalExpr`s (e.g. via [`PhysicalExpr::with_new_children`]) are
120-
/// the same logical filter and must report the same `expression_id`.
121-
#[derive(Clone, Debug)]
122-
struct Inner {
123-
/// A unique identifier for the expression.
124-
expression_id: u64,
125-
/// A counter that gets incremented every time the expression is updated so that we can track changes cheaply.
126-
/// This is used for [`PhysicalExpr::snapshot_generation`] to have a cheap check for changes.
127-
generation: u64,
128-
expr: Arc<dyn PhysicalExpr>,
129-
/// Flag for quick synchronous check if filter is complete.
130-
/// This is redundant with the watch channel state, but allows us to return immediately
131-
/// from `wait_complete()` without subscribing if already complete.
132-
is_complete: bool,
118+
/// `expression_id` identifies the actual filter expression. Derived
119+
/// `DynamicFilterPhysicalExpr`s (e.g. via [`PhysicalExpr::with_new_children`])
120+
/// are the same logical filter and must report the same id. Proto
121+
/// (de)serialization uses this id to dedupe references on the wire so the
122+
/// reconstructed plan keeps `Arc<Inner>` shared between e.g. SortExec.filter
123+
/// and the FileScan predicate it was pushed down to.
124+
///
125+
/// **Warning:** exposed publicly only so that `datafusion-proto` can read
126+
/// and rebuild this state. Not a stable API.
127+
///
128+
/// Ported from upstream apache/datafusion#21807 (minimal subset) so
129+
/// branch-53 can drop the atlas-side post-deserialize walker.
130+
#[derive(Clone)]
131+
pub struct Inner {
132+
/// Stable id, preserved across `update()` and `with_new_children`.
133+
pub expression_id: u64,
134+
/// Incremented on every `update()` so that [`PhysicalExpr::snapshot_generation`]
135+
/// is a cheap "did anything change?" probe.
136+
pub generation: u64,
137+
pub expr: Arc<dyn PhysicalExpr>,
138+
/// Synchronous check for completion; redundant with the watch channel
139+
/// but lets `wait_complete()` return without subscribing.
140+
pub is_complete: bool,
141+
}
142+
143+
// Exclude `expression_id` from Debug so existing roundtrip Debug comparisons
144+
// in tests still pass even when only some plan-node decoders carry the
145+
// dynamic filter through proto.
146+
impl std::fmt::Debug for Inner {
147+
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
148+
f.debug_struct("Inner")
149+
.field("generation", &self.generation)
150+
.field("expr", &self.expr)
151+
.field("is_complete", &self.is_complete)
152+
.finish()
153+
}
133154
}
134155

135156
impl Inner {
@@ -150,6 +171,26 @@ impl Inner {
150171
}
151172
}
152173

174+
/// An atomic counter used to generate monotonic u64 ids for
175+
/// `DynamicFilterPhysicalExpr` instances.
176+
struct ExpressionIdAtomicCounter {
177+
inner: AtomicU64,
178+
}
179+
180+
impl ExpressionIdAtomicCounter {
181+
const fn new() -> Self {
182+
Self {
183+
inner: AtomicU64::new(0),
184+
}
185+
}
186+
187+
fn next(&self) -> u64 {
188+
self.inner.fetch_add(1, Ordering::Relaxed)
189+
}
190+
}
191+
192+
static EXPR_ID_SOURCE: ExpressionIdAtomicCounter = ExpressionIdAtomicCounter::new();
193+
153194
impl Hash for DynamicFilterPhysicalExpr {
154195
fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
155196
// Use pointer identity of the inner Arc for stable hashing.
@@ -328,7 +369,9 @@ impl DynamicFilterPhysicalExpr {
328369
let mut current = self.inner.write();
329370
let new_generation = current.generation + 1;
330371
*current = Inner {
331-
// Preserve the expression id across updates.
372+
// Preserve the expression id across updates so all references
373+
// (e.g. SortExec.filter + a pushed-down FileScan predicate)
374+
// continue to identify the same logical filter.
332375
expression_id: current.expression_id,
333376
generation: new_generation,
334377
expr: new_expr,
@@ -459,9 +502,39 @@ impl DynamicFilterPhysicalExpr {
459502
write!(f, " ]")
460503
}
461504

505+
/// Return the filter's original children (before any remapping).
506+
///
507+
/// **Warning:** intended only for `datafusion-proto` (de)serialization.
508+
/// Not a stable API.
509+
pub fn original_children(&self) -> &[Arc<dyn PhysicalExpr>] {
510+
&self.children
511+
}
512+
513+
/// Return the filter's remapped children, if any have been set via
514+
/// [`PhysicalExpr::with_new_children`].
515+
///
516+
/// **Warning:** intended only for `datafusion-proto` (de)serialization.
517+
/// Not a stable API.
518+
pub fn remapped_children(&self) -> Option<&[Arc<dyn PhysicalExpr>]> {
519+
self.remapped_children.as_deref()
520+
}
521+
522+
/// Clone the atomically-captured `Inner` state.
523+
///
524+
/// **Warning:** intended only for `datafusion-proto` (de)serialization.
525+
/// Not a stable API.
526+
pub fn inner(&self) -> Inner {
527+
self.inner.read().clone()
528+
}
529+
462530
/// Rebuild a `DynamicFilterPhysicalExpr` from its stored parts. Used by
463-
/// proto deserialization.
464-
fn from_parts(
531+
/// proto deserialization to reconstruct the filter; when paired with the
532+
/// existing `DeduplicatingDeserializer` it restores shared `Arc<Inner>`
533+
/// identity across all references in the decoded plan.
534+
///
535+
/// **Warning:** intended only for `datafusion-proto` (de)serialization.
536+
/// Not a stable API.
537+
pub fn from_parts(
465538
children: Vec<Arc<dyn PhysicalExpr>>,
466539
remapped_children: Option<Vec<Arc<dyn PhysicalExpr>>>,
467540
inner: Inner,

‎datafusion/physical-plan/src/sorts/sort.rs‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1104,17 +1104,20 @@ impl SortExec {
11041104
}
11051105

11061106
/// Returns the dynamic filter expression for this sort (TopK), if set.
1107+
///
1108+
/// Used by `datafusion-proto` to roundtrip the SortExec's internal
1109+
/// `DynamicFilterPhysicalExpr` so the decode side can re-share its
1110+
/// `Arc<Inner>` with the pushed-down FileScan predicate
1111+
/// (apache/datafusion#22011 minimal port for branch-53).
11071112
pub fn dynamic_filter_expr(&self) -> Option<Arc<DynamicFilterPhysicalExpr>> {
11081113
self.filter.as_ref().map(|f| f.read().expr())
11091114
}
11101115

1111-
/// Replace the dynamic filter expression for this sort.
1112-
///
1113-
///
1114-
/// Resets any internal state which may depend on the previous dynamic filter.
1116+
/// Install a `DynamicFilterPhysicalExpr` as this sort's TopK dyn filter,
1117+
/// validating that its children reference columns in the input schema.
11151118
///
1116-
/// Validates that the filter's children reference valid columns in
1117-
/// the sort's input schema.
1119+
/// Used by proto deserialization to restore Arc identity across the wire;
1120+
/// see [`SortExec::dynamic_filter_expr`].
11181121
pub fn with_dynamic_filter_expr(
11191122
mut self,
11201123
filter: Arc<DynamicFilterPhysicalExpr>,

‎datafusion/proto-models/proto/datafusion.proto‎

Lines changed: 22 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1024,8 +1024,14 @@ message PhysicalExprNode {
10241024

10251025
PhysicalHashExprNode hash_expr = 21;
10261026

1027-
PhysicalScalarSubqueryExprNode scalar_subquery = 22;
1028-
1027+
// Ported from apache/datafusion#21807 (minimal subset). Allows a
1028+
// DynamicFilterPhysicalExpr to roundtrip through proto with its wrapper
1029+
// intact, so the receiver (combined with DeduplicatingDeserializer)
1030+
// can rebuild a shared `Arc<Inner>` across e.g. SortExec.filter and the
1031+
// FileScan predicate pushed down from it.
1032+
//
1033+
// Field 22 is intentionally skipped to match upstream's field number
1034+
// for a future scalar_subquery variant atlas doesn't need yet.
10291035
PhysicalDynamicFilterNode dynamic_filter = 23;
10301036

10311037
PhysicalHigherOrderUdfNode higher_order_udf = 24;
@@ -1034,12 +1040,18 @@ message PhysicalExprNode {
10341040
}
10351041
}
10361042

1043+
// Ported from apache/datafusion#21807. See `dynamic_filter = 23` above.
10371044
message PhysicalDynamicFilterNode {
10381045
repeated PhysicalExprNode children = 1;
10391046
repeated PhysicalExprNode remapped_children = 2;
10401047
uint64 generation = 3;
10411048
PhysicalExprNode inner_expr = 4;
10421049
bool is_complete = 5;
1050+
// Stable identity of this dynamic filter. References with the same id
1051+
// MUST deserialize to the same `Arc<DynamicFilterPhysicalExpr>` so heap-max
1052+
// updates propagate from SortExec to the FileScan predicate it was
1053+
// pushed down to.
1054+
uint64 expression_id = 6;
10431055
}
10441056

10451057
message PhysicalScalarUdfNode {
@@ -1458,7 +1470,14 @@ message SortExecNode {
14581470
// Maximum number of highest/lowest rows to fetch; negative means no limit
14591471
int64 fetch = 3;
14601472
bool preserve_partitioning = 4;
1461-
// Optional dynamic filter expression for TopK pushdown.
1473+
1474+
// Ported from apache/datafusion#22011 (minimal subset). Carries the
1475+
// SortExec's internal `DynamicFilterPhysicalExpr` so the decode side
1476+
// can install it via `with_dynamic_filter_expr` instead of letting
1477+
// `with_fetch(...).create_filter()` mint a brand-new one. Combined with
1478+
// the #21807 PhysicalDynamicFilterNode + dedup-by-expression_id, the
1479+
// resulting decoded plan re-shares the `Arc<Inner>` with the pushed-down
1480+
// FileScan predicate (X-2935 walker becomes redundant).
14621481
PhysicalExprNode dynamic_filter = 5;
14631482
}
14641483

0 commit comments

Comments
 (0)