Skip to content

Commit 206813f

Browse files
adriangbclaude
andauthored
refactor(proto): destructure plan and proto structs in join serde hooks (apache#24164)
## Which issue does this PR close? <!-- No dedicated issue; this is follow-up cleanup for the proto hook migration EPIC. --> - Part of apache#23494. ## Rationale for this change EPIC apache#23494 moved every built-in `ExecutionPlan` off the central `downcast_ref` chain in `datafusion-proto` onto per-plan hooks: a `try_to_proto` override inside `impl ExecutionPlan for FooExec`, and an inherent `FooExec::try_from_proto`. Those hooks read plan state through **getters** (`self.filter()`, `self.join_type()`, …). That makes adding a field to a plan struct invisible to serialization — nothing breaks at compile time, the new field is just silently not serialized, and the bug only shows up as state quietly disappearing on a round-trip. This is not hypothetical. `HashJoinExec.fetch` was dropped on round-trip for exactly this reason (fixed in apache#24165) — the field was added to the struct and the encoder, still calling getters, never learned about it. Destructuring removes the failure mode. If `try_to_proto` starts with an exhaustive `let Self { … } = self;` with **no `..`**, adding a field to the plan struct is a compile error in the encoder. If `try_from_proto` destructures the prost-generated node struct the same way, adding a field to the `.proto` is a compile error in every decoder. The prost structs are plain, all-`pub`, and not `#[non_exhaustive]`, so this compiles today. ## What changes are included in this PR? Applies the pattern to the join plans, one commit per plan: - `CrossJoinExec` - `NestedLoopJoinExec` - `SortMergeJoinExec` - `SymmetricHashJoinExec` - `HashExpr` / `HashTableLookupExpr` (`joins/hash_join/partitioned_hash_eval.rs`) - `HashJoinExec` Fields that genuinely are not serialized bind to `_` and carry a short comment saying why — derived at construction, runtime state, or recomputed by `new`/`try_new`/the builder on decode. For example: ```rust let Self { left, right, on, filter, join_type, sort_options, null_equality, // derived from the children's schemas by `try_new` on decode schema: _, // runtime metrics, not part of the plan metrics: _, // recomputed from `on` and `sort_options` by `try_new` on decode left_sort_exprs: _, // recomputed from `on` and `sort_options` by `try_new` on decode right_sort_exprs: _, // recomputed by `try_new` on decode cache: _, } = self; ``` `HashTableLookupExpr::try_to_proto` deliberately serializes none of its state — it holds a runtime `Arc<Map>` and is replaced with `lit(true)`. It gets a destructure with every field bound to `_`, so that adding a field there forces a decision instead of passing unnoticed. For `HashJoinExec`, the serialized bindings are `left`, `right`, `on`, `filter`, `join_type`, `mode`, `projection`, `null_equality`, `null_aware`, `dynamic_filter` and `fetch`. The six ignored fields are `join_schema`, `left_fut`, `random_state`, `metrics`, `column_indices` and `cache` — `random_state` is the fixed `HASH_JOIN_SEED` constant that the builder sets identically on decode, and the rest are runtime state or recomputed by the builder. **No additional unserialized fields were found.** Every field on the join plans that is not written to the proto is genuinely derived, recomputed on decode, or runtime-only. **This is a pure refactor. The wire format is byte-for-byte unchanged and no behavior changes.** Every getter that was replaced returns exactly the field it was replaced with. The only incidental changes are in `SymmetricHashJoinExec::try_from_proto` and `HashJoinExec::try_from_proto`, where four `internal_datafusion_err!` calls switched to inlined format args now that the field is bound locally; the message text they produce is identical. ## Are these changes tested? Covered by the existing round-trip tests — that is the point of the change: if the destructures had drifted from the encoders, the plans would fail to round-trip. Run locally: - `cargo test -p datafusion-proto --test proto_integration` — 215 passed, including all five hash-join round-trip tests (`roundtrip_hash_join`, `roundtrip_hash_join_fetch`, `roundtrip_hash_join_projection_states`, `test_hash_join_with_dynamic_filter_roundtrip`, `roundtrip_sym_hash_join`) - `cargo test -p datafusion-physical-plan` — 1641 passed - `cargo clippy --all-targets --workspace --features avro,integration-tests,extended_tests -- -D warnings` - `cargo fmt --all` ## Are there any user-facing changes? No. --------- Co-authored-by: Claude Opus 5 <[email protected]>
1 parent 5091b42 commit 206813f

6 files changed

Lines changed: 326 additions & 180 deletions

File tree

‎datafusion/physical-plan/src/joins/cross_join.rs‎

Lines changed: 24 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -462,8 +462,23 @@ impl ExecutionPlan for CrossJoinExec {
462462
) -> Result<Option<datafusion_proto_models::protobuf::PhysicalPlanNode>> {
463463
use datafusion_proto_models::protobuf;
464464

465-
let left = ctx.encode_child(self.left())?;
466-
let right = ctx.encode_child(self.right())?;
465+
// Destructure exhaustively (no `..`) so that a newly added field is a
466+
// compile error here instead of being silently left out of the proto.
467+
let Self {
468+
left,
469+
right,
470+
// derived from the children's schemas by `new` on decode
471+
schema: _,
472+
// runtime build-side state, not part of the plan
473+
left_fut: _,
474+
// runtime metrics, not part of the plan
475+
metrics: _,
476+
// recomputed by `new` on decode
477+
cache: _,
478+
} = self;
479+
480+
let left = ctx.encode_child(left)?;
481+
let right = ctx.encode_child(right)?;
467482

468483
Ok(Some(protobuf::PhysicalPlanNode {
469484
physical_plan_type: Some(
@@ -492,16 +507,13 @@ impl CrossJoinExec {
492507
"CrossJoinExec",
493508
);
494509

495-
let left = ctx.decode_required_child(
496-
crossjoin.left.as_deref(),
497-
"CrossJoinExec",
498-
"left",
499-
)?;
500-
let right = ctx.decode_required_child(
501-
crossjoin.right.as_deref(),
502-
"CrossJoinExec",
503-
"right",
504-
)?;
510+
// Destructure exhaustively (no `..`) so that a newly added proto field
511+
// is a compile error here instead of being silently ignored.
512+
let protobuf::CrossJoinExecNode { left, right } = &**crossjoin;
513+
514+
let left = ctx.decode_required_child(left.as_deref(), "CrossJoinExec", "left")?;
515+
let right =
516+
ctx.decode_required_child(right.as_deref(), "CrossJoinExec", "right")?;
505517

506518
Ok(Arc::new(CrossJoinExec::new(left, right)))
507519
}

‎datafusion/physical-plan/src/joins/hash_join/exec.rs‎

Lines changed: 81 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -1859,11 +1859,39 @@ impl ExecutionPlan for HashJoinExec {
18591859
) -> Result<Option<datafusion_proto_models::protobuf::PhysicalPlanNode>> {
18601860
use datafusion_proto_models::protobuf;
18611861

1862-
let left = ctx.encode_child(self.left())?;
1863-
let right = ctx.encode_child(self.right())?;
1862+
// Destructure exhaustively (no `..`) so that a newly added field is a
1863+
// compile error here instead of being silently left out of the proto.
1864+
let Self {
1865+
left,
1866+
right,
1867+
on,
1868+
filter,
1869+
join_type,
1870+
mode,
1871+
projection,
1872+
null_equality,
1873+
null_aware,
1874+
dynamic_filter,
1875+
fetch,
1876+
// derived from the children's schemas by the builder on decode
1877+
join_schema: _,
1878+
// runtime build-side state, not part of the plan
1879+
left_fut: _,
1880+
// the fixed `HASH_JOIN_SEED` constant, set identically by the
1881+
// builder on decode
1882+
random_state: _,
1883+
// runtime metrics, not part of the plan
1884+
metrics: _,
1885+
// recomputed by the builder on decode
1886+
column_indices: _,
1887+
// recomputed by the builder on decode
1888+
cache: _,
1889+
} = self;
1890+
1891+
let left = ctx.encode_child(left)?;
1892+
let right = ctx.encode_child(right)?;
18641893

1865-
let on = self
1866-
.on()
1894+
let on = on
18671895
.iter()
18681896
.map(|(l, r)| -> Result<protobuf::JoinOn> {
18691897
Ok(protobuf::JoinOn {
@@ -1873,27 +1901,28 @@ impl ExecutionPlan for HashJoinExec {
18731901
})
18741902
.collect::<Result<Vec<_>>>()?;
18751903

1876-
let join_type = crate::joins::proto::join_type_to_proto(*self.join_type());
1877-
let null_equality =
1878-
crate::joins::proto::null_equality_to_proto(self.null_equality());
1904+
let join_type = crate::joins::proto::join_type_to_proto(*join_type);
1905+
let null_equality = crate::joins::proto::null_equality_to_proto(*null_equality);
18791906
// `PartitionMode` is specific to `HashJoinExec`, so its conversion stays
18801907
// inline (by-name on purpose: the enums are numbered differently).
1881-
let partition_mode = match self.partition_mode() {
1908+
let partition_mode = match mode {
18821909
PartitionMode::CollectLeft => protobuf::PartitionMode::CollectLeft,
18831910
PartitionMode::Partitioned => protobuf::PartitionMode::Partitioned,
18841911
PartitionMode::Auto => protobuf::PartitionMode::Auto,
18851912
};
18861913

1887-
let filter = self
1888-
.filter()
1914+
let filter = filter
1915+
.as_ref()
18891916
.map(|f| crate::joins::proto::join_filter_to_proto(f, ctx))
18901917
.transpose()?;
18911918

1892-
let dynamic_filter = self
1893-
.dynamic_expressions_produced()
1894-
.into_iter()
1895-
.next()
1896-
.map(|expr| ctx.encode_expr(&expr))
1919+
let dynamic_filter = dynamic_filter
1920+
.as_ref()
1921+
.map(|df| {
1922+
let df_expr: Arc<dyn PhysicalExpr> =
1923+
Arc::clone(&df.filter) as Arc<dyn PhysicalExpr>;
1924+
ctx.encode_expr(&df_expr)
1925+
})
18971926
.transpose()?;
18981927

18991928
Ok(Some(protobuf::PhysicalPlanNode {
@@ -1914,14 +1943,14 @@ impl ExecutionPlan for HashJoinExec {
19141943
// single-element sentinel `[u32::MAX]` (never a valid column
19151944
// index); every other state is sent as-is. See
19161945
// `try_from_proto` for the matching decoder.
1917-
projection: match self.projection.as_ref() {
1946+
projection: match projection.as_ref() {
19181947
None => Vec::new(),
19191948
Some(v) if v.is_empty() => vec![u32::MAX],
19201949
Some(v) => v.iter().map(|x| *x as u32).collect(),
19211950
},
1922-
null_aware: self.null_aware,
1951+
null_aware: *null_aware,
19231952
dynamic_filter,
1924-
fetch: self.fetch.map(|f| f as u64),
1953+
fetch: fetch.map(|f| f as u64),
19251954
},
19261955
)),
19271956
),
@@ -1946,18 +1975,29 @@ impl HashJoinExec {
19461975
"HashJoinExec",
19471976
);
19481977

1949-
let left =
1950-
ctx.decode_required_child(hashjoin.left.as_deref(), "HashJoinExec", "left")?;
1951-
let right = ctx.decode_required_child(
1952-
hashjoin.right.as_deref(),
1953-
"HashJoinExec",
1954-
"right",
1955-
)?;
1978+
// Destructure exhaustively (no `..`) so that a newly added proto field
1979+
// is a compile error here instead of being silently ignored.
1980+
let protobuf::HashJoinExecNode {
1981+
left,
1982+
right,
1983+
on,
1984+
join_type,
1985+
partition_mode,
1986+
null_equality,
1987+
filter,
1988+
projection,
1989+
null_aware,
1990+
dynamic_filter,
1991+
fetch,
1992+
} = &**hashjoin;
1993+
1994+
let left = ctx.decode_required_child(left.as_deref(), "HashJoinExec", "left")?;
1995+
let right =
1996+
ctx.decode_required_child(right.as_deref(), "HashJoinExec", "right")?;
19561997
let left_schema = left.schema();
19571998
let right_schema = right.schema();
19581999

1959-
let on: Vec<(PhysicalExprRef, PhysicalExprRef)> = hashjoin
1960-
.on
2000+
let on: Vec<(PhysicalExprRef, PhysicalExprRef)> = on
19612001
.iter()
19622002
.map(|col| {
19632003
let l = ctx.decode_required_expr(
@@ -1976,38 +2016,32 @@ impl HashJoinExec {
19762016
})
19772017
.collect::<Result<_>>()?;
19782018

1979-
let join_type = crate::joins::proto::join_type_from_proto(
1980-
hashjoin.join_type,
1981-
"HashJoinExec",
1982-
)?;
2019+
let join_type =
2020+
crate::joins::proto::join_type_from_proto(*join_type, "HashJoinExec")?;
19832021
let null_equality = crate::joins::proto::null_equality_from_proto(
1984-
hashjoin.null_equality,
2022+
*null_equality,
19852023
"HashJoinExec",
19862024
)?;
19872025
// `PartitionMode` is specific to `HashJoinExec`, so its conversion stays
19882026
// inline (by-name on purpose: the enums are numbered differently).
1989-
let partition_mode = match protobuf::PartitionMode::try_from(
1990-
hashjoin.partition_mode,
1991-
)
1992-
.map_err(|_| {
1993-
internal_datafusion_err!(
1994-
"HashJoinExec: unknown PartitionMode {}",
1995-
hashjoin.partition_mode
1996-
)
1997-
})? {
2027+
let partition_mode = match protobuf::PartitionMode::try_from(*partition_mode)
2028+
.map_err(|_| {
2029+
internal_datafusion_err!(
2030+
"HashJoinExec: unknown PartitionMode {partition_mode}"
2031+
)
2032+
})? {
19982033
protobuf::PartitionMode::CollectLeft => PartitionMode::CollectLeft,
19992034
protobuf::PartitionMode::Partitioned => PartitionMode::Partitioned,
20002035
protobuf::PartitionMode::Auto => PartitionMode::Auto,
20012036
};
20022037

2003-
let filter = hashjoin
2004-
.filter
2038+
let filter = filter
20052039
.as_ref()
20062040
.map(|f| crate::joins::proto::join_filter_from_proto(f, ctx, "HashJoinExec"))
20072041
.transpose()?;
20082042

20092043
// Preserve the empty-projection sentinel written by `try_to_proto`.
2010-
let projection = match hashjoin.projection.as_slice() {
2044+
let projection = match projection.as_slice() {
20112045
[] => None,
20122046
[u32::MAX] => Some(Vec::new()),
20132047
indices => Some(indices.iter().map(|i| *i as usize).collect()),
@@ -2023,8 +2057,7 @@ impl HashJoinExec {
20232057
// wrong limit but the worst one, silently turning the query into an
20242058
// empty result. Report the out-of-range value instead. Please do not
20252059
// "simplify" this back to `as usize`.
2026-
let fetch = hashjoin
2027-
.fetch
2060+
let fetch = fetch
20282061
.map(|f| {
20292062
usize::try_from(f).map_err(|_| {
20302063
plan_datafusion_err!(
@@ -2039,11 +2072,11 @@ impl HashJoinExec {
20392072
.with_projection(projection)
20402073
.with_partition_mode(partition_mode)
20412074
.with_null_equality(null_equality)
2042-
.with_null_aware(hashjoin.null_aware)
2075+
.with_null_aware(*null_aware)
20432076
.with_fetch(fetch)
20442077
.build()?;
20452078

2046-
if let Some(dynamic_filter_proto) = &hashjoin.dynamic_filter {
2079+
if let Some(dynamic_filter_proto) = dynamic_filter {
20472080
// The dynamic filter is a `DynamicFilterPhysicalExpr` over the probe
20482081
// (right) side; decode against the right schema then downcast.
20492082
let dynamic_filter_expr =

‎datafusion/physical-plan/src/joins/hash_join/partitioned_hash_eval.rs‎

Lines changed: 37 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -208,14 +208,25 @@ impl PhysicalExpr for HashExpr {
208208
ctx: &datafusion_physical_expr_common::physical_expr::proto_encode::PhysicalExprEncodeCtx<'_>,
209209
) -> Result<Option<datafusion_proto_models::protobuf::PhysicalExprNode>> {
210210
use datafusion_proto_models::protobuf;
211-
let on_columns = ctx.encode_children_expressions(&self.on_columns)?;
211+
212+
// Destructure exhaustively (no `..`) so that a newly added field is a
213+
// compile error here instead of being silently left out of the proto.
214+
let Self {
215+
on_columns,
216+
random_state,
217+
description,
218+
} = self;
219+
220+
let on_columns = ctx.encode_children_expressions(on_columns)?;
212221
Ok(Some(protobuf::PhysicalExprNode {
213222
expr_id: None,
214223
expr_type: Some(protobuf::physical_expr_node::ExprType::HashExpr(
215224
protobuf::PhysicalHashExprNode {
216225
on_columns,
217-
seed0: self.seed(),
218-
description: self.description.clone(),
226+
// only the seed is serialized; `RandomState` is rebuilt
227+
// from it by `SeededRandomState::with_seed` on decode
228+
seed0: random_state.seed(),
229+
description: description.clone(),
219230
},
220231
)),
221232
}))
@@ -243,11 +254,19 @@ impl HashExpr {
243254
Some(protobuf::physical_expr_node::ExprType::HashExpr(h)) => h,
244255
_ => return internal_err!("PhysicalExprNode is not a HashExpr"),
245256
};
246-
let on_columns = ctx.decode_children_expressions(&hash_expr.on_columns)?;
257+
// Destructure exhaustively (no `..`) so that a newly added proto field
258+
// is a compile error here instead of being silently ignored.
259+
let protobuf::PhysicalHashExprNode {
260+
on_columns,
261+
seed0,
262+
description,
263+
} = hash_expr;
264+
265+
let on_columns = ctx.decode_children_expressions(on_columns)?;
247266
Ok(Arc::new(HashExpr::new(
248267
on_columns,
249-
SeededRandomState::with_seed(hash_expr.seed0),
250-
hash_expr.description.clone(),
268+
SeededRandomState::with_seed(*seed0),
269+
description.clone(),
251270
)))
252271
}
253272
}
@@ -394,6 +413,18 @@ impl PhysicalExpr for HashTableLookupExpr {
394413
use datafusion_proto_models::protobuf;
395414
use datafusion_proto_models::protobuf::physical_expr_node::ExprType;
396415

416+
// Destructure exhaustively (no `..`) so that a newly added field is a
417+
// compile error here, forcing a decision about whether the lit(true)
418+
// replacement below is still the right thing to emit.
419+
let Self {
420+
// deliberately not serialized: the whole expression is replaced
421+
// with lit(true), see the comment below
422+
on_columns: _,
423+
random_state: _,
424+
map: _,
425+
description: _,
426+
} = self;
427+
397428
// HashTableLookupExpr holds a runtime Arc<Map> (the build-side hash
398429
// table) that cannot be serialized, so it is replaced with lit(true).
399430
//

0 commit comments

Comments
 (0)