You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit e4b0494
Browse filesBrowse the repository at this point in the historyBrowse files
[X-2935] proto: dedupe DynamicFilterPhysicalExpr across extension codec boundary (#60)
* [X-2935] proto: serialize and dedupe DynamicFilterPhysicalExpr (minimal port of apache#21807)
Adds expression_id to DynamicFilterPhysicalExpr Inner (atomic counter,
preserved across update() and with_new_children) and a corresponding
PhysicalDynamicFilterNode proto message. Serializer special-cases
DynamicFilterPhysicalExpr to skip snapshot() and emit the wrapper full.
Deserializer reconstructs the wrapper.
Combined with the existing DeduplicatingSerializer + DeduplicatingDeserializer
pair on branch-53, the wire-level Arc identity dedup now works for dynamic
filters too: SortExec.filter and the FileScan predicate it was pushed down
to reconstruct to the same Arc<Inner> on the data server, so heap-max
updates propagate end-to-end without the atlas-side share_dynamic_filters
walker.
Minimal port: skips upstream's ExpressionPlacement enum, snapshot()
refactor, and DedupingSerializer removal. Wire layout (field 23 oneof
variant + PhysicalDynamicFilterNode fields) mirrors upstream so a future
DF upgrade can drop this patch mechanically.
DeduplicatingSerializer and DeduplicatingDeserializer are now pub so
atlas's coordinator-side and data-server-side codecs can opt in.
* [X-2935] dedupe DynamicFilter by expression_id, not Arc::as_ptr
Mirrors apache#21807 step 3-4: switch DeduplicatingSerializer
from hashing Arc::as_ptr to hashing the expression's expression_id when
available. Falls back to Arc::as_ptr for expressions without a stable id.
This fixes the case FilterPushdown produces in practice: pushing a
DynamicFilterPhysicalExpr from a SortExec down to a FileScan creates a
new outer Arc (via with_new_children to remap columns) that still shares
the same Arc<RwLock<Inner>>. Without this change, the two outer Arcs hash
to different addresses and DeduplicatingDeserializer rebuilds them as
two distinct Arcs on decode -- exactly the X-2935 walker's bug.
Adds a new test dynamic_filter_dedup_distinct_outer_arcs_same_inner that
constructs the with_new_children case and asserts dedup works.
150/150 proto integration tests pass.
* [X-2935] proto: serialize SortExec's dynamic filter end-to-end
Adds the final piece needed to drop the atlas-side share_dynamic_filters
walker: SortExecNode now carries its internal DynamicFilterPhysicalExpr
on the wire, and try_into_sort_physical_plan re-installs it via
SortExec::with_dynamic_filter_expr instead of letting with_fetch(N)
mint a brand-new one via create_filter() on decode.
Combined with the prior two commits in this series (apache#21807 minimal port
+ dedup-by-expression_id), the decode side now reconstructs the SAME
Arc<DynamicFilterPhysicalExpr> shared between the SortExec.filter and
any pushed-down FileScan predicate that participated in the encode --
because DeduplicatingDeserializer's cache, keyed on the wire expr_id
that DeduplicatingSerializer derives from expression_id, matches across
plan nodes.
Adds:
- SortExec::dynamic_filter_expr() / with_dynamic_filter_expr() accessors
- dynamic_filter field on SortExecNode proto (field 5)
- to_proto: try_from_sort_exec serializes the dyn filter via proto_converter
- from_proto: try_into_sort_physical_plan deserializes + reinstalls via
with_dynamic_filter_expr; legacy fallback wraps a fresh Arc when the
encoder snapshotted the wrapper away
Test sort_exec_proto_roundtrip_preserves_dyn_filter_arc asserts
end-to-end: serialize a SortExec with a known DynamicFilterPhysicalExpr,
roundtrip via DeduplicatingSerializer + DeduplicatingDeserializer, the
decoded SortExec's filter must carry the same expression_id.
Ported from apache#22011 (minimal subset, adapted to
branch-53's PhysicalProtoConverterExtension signatures).
151/151 proto integration tests pass.
* [X-2935] PhysicalExtensionCodec: thread proto_converter through methods
Modifies the PhysicalExtensionCodec trait so that try_encode, try_decode,
try_encode_expr, and try_decode_expr receive the active
PhysicalProtoConverterExtension. Codecs that embed nested PhysicalExprNode
fields inside their custom proto (e.g. atlas's ReverseParquetNode.predicate)
must now route those through proto_converter.proto_to_physical_expr /
proto_converter.physical_expr_to_proto so the dedup cache extends across
the extension boundary.
Without this plumbing, an outer DeduplicatingSerializer can't reach
into extension nodes -- the dedup cache stops at the extension boundary
and Arc<Inner> sharing breaks for any expression carried inside an
extension codec, even though the wire format already supports it via
the expr_id field. With this change, atlas can finally drop the
share_dynamic_filters walker once its codecs override the new methods.
This is the breaking-change pattern matching the apache/datafusion
upstream design (issue apache#22920) instead of additive _with_converter
variants -- branch-53 is internal-only so callers can be updated in
lockstep.
0 commit comments