Skip to content

Commit a4e071d

Browse files
adriangbclaude
andauthored
refactor(proto): destructure plan and proto structs in core plan serde hooks (apache#24167)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Follow-up cleanup for EPIC apache#23494; does not close an issue on its own. ## Rationale for this change EPIC apache#23494 moved every built-in `ExecutionPlan` off the central `downcast_ref` chain in `datafusion-proto` onto per-plan hooks: a `try_to_proto` override on the `ExecutionPlan` trait and an inherent `FooExec::try_from_proto`, both living in the plan's own module. Those hooks currently read plan state through **getters** (`self.expr()`, `self.fetch()`, ...). That means adding a field to a plan struct is invisible to serialization: nothing breaks, nothing warns, the field is just silently dropped on round-trip. This is not hypothetical — it has already cost us a real bug: `HashJoinExec.fetch` is not serialized and is lost on round-trip (being fixed separately). Starting each hook with an *exhaustive* destructure turns that class of mistake into a compile error: * **Encode side** — destructure `self` naming every field, no `..`. Add a field to `SortExec` and `SortExec::try_to_proto` stops compiling until you either serialize it or bind it to `_` with a comment saying why it does not need to be (derived at construction, runtime state, recomputed on decode). * **Decode side** — destructure the prost node struct the same way. The generated structs are plain, all-`pub` and not `#[non_exhaustive]`, so adding a field to the `.proto` becomes a compile error in every decoder rather than a silent omission. ## What changes are included in this PR? A mechanical, behavior-preserving refactor of the serde hooks for the core (non-join, non-aggregate, non-window) plans in `datafusion/physical-plan/src/`: `SortExec`, `SortPreservingMergeExec`, `GlobalLimitExec`, `LocalLimitExec`, `FilterExec`, `ProjectionExec`, `RepartitionExec`, `UnionExec`, `InterleaveExec`, `CoalesceBatchesExec`, `CoalescePartitionsExec`, `CooperativeExec`, `BufferExec`, `EmptyExec`, `PlaceholderRowExec`, `ExplainExec`, `ScalarSubqueryExec`. **The wire format is unchanged — byte for byte — and no behavior changes.** The same values are written to and read from the same proto fields; only the way they are reached in Rust changes. Split into four commits, one per group of plans, each of which builds green. ### Fields the refactor documented as not serialized Writing the destructures surfaced and pinned down, in comments at the point of use, which fields do not survive a round-trip today. All of these are pre-existing; **none of them are changed here**, because fixing any of them means changing the wire format, which is a separate decision: * `SortPreservingMergeExec::enable_round_robin_repartition` — no field on `SortPreservingMergeExecNode`; decoding always restores the `true` default from `SortPreservingMergeExec::new`. * `GlobalLimitExec::required_ordering` and `LocalLimitExec::required_ordering` — no field on the limit nodes; they are set by the `enforce_sorting` optimizer rule, so a decoded plan starts with `None`. Everything else that binds to `_` is genuinely not plan shape: runtime metrics (`metrics` / `metrics_set`), execution-time state (`RepartitionExec::state`, `ScalarSubqueryExec::subquery_future` and `results`), values derived at construction from other serialized fields (`SortExec::common_sort_prefix`), the positional `ScalarSubqueryLink::index`, and the `cache` / `properties` plan properties that are recomputed on decode. Two small notes for reviewers: * `RepartitionExec` keeps its output partitioning *inside* `cache` (`partitioning()` is `&self.cache.partitioning`), so `cache` is bound rather than `_`-ed there and is read for the serialized `partitioning` field. * `EmptyExec` / `PlaceholderRowExec` now read the encoded partition count from the `partitions` field instead of going through `properties().output_partitioning().partition_count()`. The two are kept in sync by `with_partitions`, the only writer besides `new`, so the encoded value is identical. ## Are these changes tested? Covered by the existing round-trip tests, which are the real proof that the wire format did not move. Locally on this branch: * `cargo test -p datafusion-proto --test proto_integration` — 214 passed, 0 failed * `cargo test -p datafusion-physical-plan` — 1640 passed, 0 failed * `cargo clippy -p datafusion-physical-plan --all-targets --all-features -- -D warnings` — clean * `cargo fmt --all` No new tests are added here: the change adds no new behavior to test, and the compile-error property it introduces is enforced by rustc rather than by a test. **On the comprehensiveness of that existing coverage** (thanks @andygrove for pushing on this — the original wording oversold it): * `SortPreservingMergeExec`, updated by this PR, has **no** round-trip test at all — `SortPreservingMerge` does not appear anywhere in `roundtrip_physical_plan.rs`. The other 16 plans touched here each have at least one dedicated `roundtrip_*` test. * Separately, `roundtrip_test` asserts on `format!("{plan:?}")`, so it cannot catch a dropped field that a plan's `Debug` impl doesn't print. That is precisely how the `HashJoinExec::fetch` bug (apache#24165) went unnoticed. Both are now tracked in apache#24171, with the test work split out into its own PR so this mechanical refactor and the coverage improvements can be reviewed independently. Sequencing is up to reviewers — happy to land the tests first if that's preferred. ## Are there any user-facing changes? No. No public API changes, no wire format changes, no behavior changes. --------- Co-authored-by: Claude Opus 5 <[email protected]>
1 parent e2bdc2c commit a4e071d

15 files changed

Lines changed: 425 additions & 156 deletions

File tree

‎datafusion/physical-plan/src/buffer.rs‎

Lines changed: 18 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -332,13 +332,24 @@ impl ExecutionPlan for BufferExec {
332332
ctx: &crate::proto::ExecutionPlanEncodeCtx<'_>,
333333
) -> Result<Option<datafusion_proto_models::protobuf::PhysicalPlanNode>> {
334334
use datafusion_proto_models::protobuf;
335-
let input = ctx.encode_child(self.input())?;
335+
// Destructure exhaustively (no `..`) so that adding a field to
336+
// `BufferExec` is a compile error here until it is either serialized or
337+
// explicitly documented as not needing to be.
338+
let Self {
339+
input,
340+
// Derived from the input's properties at construction time.
341+
properties: _,
342+
capacity,
343+
// Runtime metrics, not part of the plan shape.
344+
metrics: _,
345+
} = self;
346+
let input = ctx.encode_child(input)?;
336347
Ok(Some(protobuf::PhysicalPlanNode {
337348
physical_plan_type: Some(
338349
protobuf::physical_plan_node::PhysicalPlanType::Buffer(Box::new(
339350
protobuf::BufferExecNode {
340351
input: Some(Box::new(input)),
341-
capacity: self.capacity() as u64,
352+
capacity: *capacity as u64,
342353
},
343354
)),
344355
),
@@ -363,9 +374,11 @@ impl BufferExec {
363374
protobuf::physical_plan_node::PhysicalPlanType::Buffer,
364375
"BufferExec",
365376
);
366-
let input =
367-
ctx.decode_required_child(buffer.input.as_deref(), "BufferExec", "input")?;
368-
Ok(Arc::new(BufferExec::new(input, buffer.capacity as usize)))
377+
// Destructure exhaustively so that a new field on `BufferExecNode` is a
378+
// compile error here rather than a silently dropped field.
379+
let protobuf::BufferExecNode { input, capacity } = &**buffer;
380+
let input = ctx.decode_required_child(input.as_deref(), "BufferExec", "input")?;
381+
Ok(Arc::new(BufferExec::new(input, *capacity as usize)))
369382
}
370383
}
371384

‎datafusion/physical-plan/src/coalesce_batches.rs‎

Lines changed: 27 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -323,14 +323,26 @@ impl ExecutionPlan for CoalesceBatchesExec {
323323
ctx: &crate::proto::ExecutionPlanEncodeCtx<'_>,
324324
) -> Result<Option<datafusion_proto_models::protobuf::PhysicalPlanNode>> {
325325
use datafusion_proto_models::protobuf;
326-
let input = ctx.encode_child(self.input())?;
326+
// Destructure exhaustively (no `..`) so that adding a field to
327+
// `CoalesceBatchesExec` is a compile error here until it is either
328+
// serialized or explicitly documented as not needing to be.
329+
let Self {
330+
input,
331+
target_batch_size,
332+
fetch,
333+
// Runtime metrics, not part of the plan shape.
334+
metrics: _,
335+
// Derived plan properties, recomputed on decode.
336+
cache: _,
337+
} = self;
338+
let input = ctx.encode_child(input)?;
327339
Ok(Some(protobuf::PhysicalPlanNode {
328340
physical_plan_type: Some(
329341
protobuf::physical_plan_node::PhysicalPlanType::CoalesceBatches(
330342
Box::new(protobuf::CoalesceBatchesExecNode {
331343
input: Some(Box::new(input)),
332-
target_batch_size: self.target_batch_size() as u32,
333-
fetch: self.fetch().map(|n| n as u32),
344+
target_batch_size: *target_batch_size as u32,
345+
fetch: fetch.map(|n| n as u32),
334346
}),
335347
),
336348
),
@@ -361,14 +373,19 @@ impl CoalesceBatchesExec {
361373
protobuf::physical_plan_node::PhysicalPlanType::CoalesceBatches,
362374
"CoalesceBatchesExec",
363375
);
364-
let input = ctx.decode_required_child(
365-
coalesce_batches.input.as_deref(),
366-
"CoalesceBatchesExec",
367-
"input",
368-
)?;
376+
// Destructure exhaustively so that a new field on
377+
// `CoalesceBatchesExecNode` is a compile error here rather than a
378+
// silently dropped field.
379+
let protobuf::CoalesceBatchesExecNode {
380+
input,
381+
target_batch_size,
382+
fetch,
383+
} = &**coalesce_batches;
384+
let input =
385+
ctx.decode_required_child(input.as_deref(), "CoalesceBatchesExec", "input")?;
369386
Ok(Arc::new(
370-
CoalesceBatchesExec::new(input, coalesce_batches.target_batch_size as usize)
371-
.with_fetch(coalesce_batches.fetch.map(|f| f as usize)),
387+
CoalesceBatchesExec::new(input, *target_batch_size as usize)
388+
.with_fetch(fetch.map(|f| f as usize)),
372389
))
373390
}
374391
}

‎datafusion/physical-plan/src/coalesce_partitions.rs‎

Lines changed: 19 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -384,13 +384,24 @@ impl ExecutionPlan for CoalescePartitionsExec {
384384
ctx: &crate::proto::ExecutionPlanEncodeCtx<'_>,
385385
) -> Result<Option<datafusion_proto_models::protobuf::PhysicalPlanNode>> {
386386
use datafusion_proto_models::protobuf;
387-
let input = ctx.encode_child(self.input())?;
387+
// Destructure exhaustively (no `..`) so that adding a field to
388+
// `CoalescePartitionsExec` is a compile error here until it is either
389+
// serialized or explicitly documented as not needing to be.
390+
let Self {
391+
input,
392+
// Runtime metrics, not part of the plan shape.
393+
metrics: _,
394+
// Derived plan properties, recomputed on decode.
395+
cache: _,
396+
fetch,
397+
} = self;
398+
let input = ctx.encode_child(input)?;
388399
Ok(Some(protobuf::PhysicalPlanNode {
389400
physical_plan_type: Some(
390401
protobuf::physical_plan_node::PhysicalPlanType::Merge(Box::new(
391402
protobuf::CoalescePartitionsExecNode {
392403
input: Some(Box::new(input)),
393-
fetch: self.fetch().map(|f| f as u32),
404+
fetch: fetch.map(|f| f as u32),
394405
},
395406
)),
396407
),
@@ -417,14 +428,17 @@ impl CoalescePartitionsExec {
417428
protobuf::physical_plan_node::PhysicalPlanType::Merge,
418429
"CoalescePartitionsExec",
419430
);
431+
// Destructure exhaustively so that a new field on
432+
// `CoalescePartitionsExecNode` is a compile error here rather than a
433+
// silently dropped field.
434+
let protobuf::CoalescePartitionsExecNode { input, fetch } = &**merge;
420435
let input = ctx.decode_required_child(
421-
merge.input.as_deref(),
436+
input.as_deref(),
422437
"CoalescePartitionsExec",
423438
"input",
424439
)?;
425440
Ok(Arc::new(
426-
CoalescePartitionsExec::new(input)
427-
.with_fetch(merge.fetch.map(|f| f as usize)),
441+
CoalescePartitionsExec::new(input).with_fetch(fetch.map(|f| f as usize)),
428442
))
429443
}
430444
}

‎datafusion/physical-plan/src/coop.rs‎

Lines changed: 15 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -402,7 +402,15 @@ impl ExecutionPlan for CooperativeExec {
402402
ctx: &crate::proto::ExecutionPlanEncodeCtx<'_>,
403403
) -> Result<Option<datafusion_proto_models::protobuf::PhysicalPlanNode>> {
404404
use datafusion_proto_models::protobuf;
405-
let input = ctx.encode_child(self.input())?;
405+
// Destructure exhaustively (no `..`) so that adding a field to
406+
// `CooperativeExec` is a compile error here until it is either
407+
// serialized or explicitly documented as not needing to be.
408+
let Self {
409+
input,
410+
// Derived from the input's properties at construction time.
411+
properties: _,
412+
} = self;
413+
let input = ctx.encode_child(input)?;
406414
Ok(Some(protobuf::PhysicalPlanNode {
407415
physical_plan_type: Some(
408416
protobuf::physical_plan_node::PhysicalPlanType::Cooperative(Box::new(
@@ -432,11 +440,12 @@ impl CooperativeExec {
432440
protobuf::physical_plan_node::PhysicalPlanType::Cooperative,
433441
"CooperativeExec",
434442
);
435-
let input = ctx.decode_required_child(
436-
cooperative.input.as_deref(),
437-
"CooperativeExec",
438-
"input",
439-
)?;
443+
// Destructure exhaustively so that a new field on
444+
// `CooperativeExecNode` is a compile error here rather than a silently
445+
// dropped field.
446+
let protobuf::CooperativeExecNode { input } = &**cooperative;
447+
let input =
448+
ctx.decode_required_child(input.as_deref(), "CooperativeExec", "input")?;
440449
Ok(Arc::new(CooperativeExec::new(input)))
441450
}
442451
}

‎datafusion/physical-plan/src/empty.rs‎

Lines changed: 16 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -214,16 +214,22 @@ impl ExecutionPlan for EmptyExec {
214214
_ctx: &crate::proto::ExecutionPlanEncodeCtx<'_>,
215215
) -> Result<Option<datafusion_proto_models::protobuf::PhysicalPlanNode>> {
216216
use datafusion_proto_models::protobuf;
217-
let schema = self.schema().as_ref().try_into()?;
217+
// Destructure exhaustively (no `..`) so that adding a field to
218+
// `EmptyExec` is a compile error here until it is either serialized or
219+
// explicitly documented as not needing to be.
220+
let Self {
221+
schema,
222+
partitions,
223+
// Derived from `schema` and `partitions`, recomputed on decode.
224+
cache: _,
225+
} = self;
226+
let schema = schema.as_ref().try_into()?;
218227
Ok(Some(protobuf::PhysicalPlanNode {
219228
physical_plan_type: Some(
220229
protobuf::physical_plan_node::PhysicalPlanType::Empty(
221230
protobuf::EmptyExecNode {
222231
schema: Some(schema),
223-
partitions: self
224-
.properties()
225-
.output_partitioning()
226-
.partition_count() as u32,
232+
partitions: *partitions as u32,
227233
},
228234
),
229235
),
@@ -244,15 +250,18 @@ impl EmptyExec {
244250
protobuf::physical_plan_node::PhysicalPlanType::Empty,
245251
"EmptyExec",
246252
);
247-
let schema = empty.schema.as_ref().ok_or_else(|| {
253+
// Destructure exhaustively so that a new field on `EmptyExecNode` is a
254+
// compile error here rather than a silently dropped field.
255+
let protobuf::EmptyExecNode { schema, partitions } = empty;
256+
let schema = schema.as_ref().ok_or_else(|| {
248257
datafusion_common::internal_datafusion_err!(
249258
"EmptyExec is missing required field 'schema'"
250259
)
251260
})?;
252261
let schema = Arc::new(arrow::datatypes::Schema::try_from(schema)?);
253262
// A zero (absent) partition count comes from a plan encoded before the
254263
// field existed, which always meant a single partition.
255-
let partitions = empty.partitions.max(1) as usize;
264+
let partitions = (*partitions).max(1) as usize;
256265
Ok(Arc::new(EmptyExec::new(schema).with_partitions(partitions)))
257266
}
258267
}

‎datafusion/physical-plan/src/explain.rs‎

Lines changed: 23 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -215,17 +215,26 @@ impl ExecutionPlan for ExplainExec {
215215
) -> Result<Option<datafusion_proto_models::protobuf::PhysicalPlanNode>> {
216216
use datafusion_proto_models::protobuf;
217217

218+
// Destructure exhaustively (no `..`) so that adding a field to
219+
// `ExplainExec` is a compile error here until it is either serialized
220+
// or explicitly documented as not needing to be.
221+
let Self {
222+
schema,
223+
stringified_plans,
224+
verbose,
225+
// Derived from `schema`, recomputed on decode.
226+
cache: _,
227+
} = self;
218228
Ok(Some(protobuf::PhysicalPlanNode {
219229
physical_plan_type: Some(
220230
protobuf::physical_plan_node::PhysicalPlanType::Explain(
221231
protobuf::ExplainExecNode {
222-
schema: Some(self.schema().as_ref().try_into()?),
223-
stringified_plans: self
224-
.stringified_plans()
232+
schema: Some(schema.as_ref().try_into()?),
233+
stringified_plans: stringified_plans
225234
.iter()
226235
.map(stringified_plan_to_proto)
227236
.collect(),
228-
verbose: self.verbose(),
237+
verbose: *verbose,
229238
},
230239
),
231240
),
@@ -247,19 +256,25 @@ impl ExplainExec {
247256
protobuf::physical_plan_node::PhysicalPlanType::Explain,
248257
"ExplainExec",
249258
);
250-
let schema = explain.schema.as_ref().ok_or_else(|| {
259+
// Destructure exhaustively so that a new field on `ExplainExecNode` is
260+
// a compile error here rather than a silently dropped field.
261+
let protobuf::ExplainExecNode {
262+
schema,
263+
stringified_plans,
264+
verbose,
265+
} = explain;
266+
let schema = schema.as_ref().ok_or_else(|| {
251267
datafusion_common::internal_datafusion_err!(
252268
"ExplainExec is missing required field 'schema'"
253269
)
254270
})?;
255271
Ok(Arc::new(ExplainExec::new(
256272
Arc::new(arrow::datatypes::Schema::try_from(schema)?),
257-
explain
258-
.stringified_plans
273+
stringified_plans
259274
.iter()
260275
.map(stringified_plan_from_proto)
261276
.collect(),
262-
explain.verbose,
277+
*verbose,
263278
)))
264279
}
265280
}

0 commit comments

Comments
 (0)