Skip to content

Commit f3f381c

Browse files
refactor(proto): migrate single-child plans (apache#23710)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes apache#23500. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Physical plan serialization currently relies on centralized type dispatch in `datafusion-proto`. This separates serialization from the plans that own the relevant state and requires every built-in plan to be handled specially. Moving serialization into each plan keeps that logic with its owner and allows the central dispatch chain to be removed incrementally. `CoalesceBatchesExec` is deprecated but remains a supported public type. Migrating its existing serialization keeps the dispatch model consistent without extending its lifetime or changing its behavior. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - Add per-plan protobuf encoding and decoding for: - `CoalesceBatchesExec` - `CoalescePartitionsExec` - `CooperativeExec` - `BufferExec` - Route decoding through each plan's `try_from_proto` implementation. - Remove the corresponding centralized encoding branches. - Retain the old public conversion helpers as deprecated compatibility APIs. - Preserve the existing protobuf wire representation. - Add roundtrip coverage for `CooperativeExec` and `BufferExec`. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> There are no runtime behavior or protobuf wire-format changes. The old `PhysicalPlanNodeExt` conversion helpers remain available but are deprecated in favor of the per-plan serialization hooks. --------- Signed-off-by: Jiawei Zhao <[email protected]>
1 parent 02a902c commit f3f381c

6 files changed

Lines changed: 336 additions & 110 deletions

File tree

‎datafusion/physical-plan/src/buffer.rs‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -298,6 +298,48 @@ impl ExecutionPlan for BufferExec {
298298
Ok(Arc::new(Self::new(new_input, self.capacity)) as Arc<dyn ExecutionPlan>)
299299
})
300300
}
301+
302+
#[cfg(feature = "proto")]
303+
fn try_to_proto(
304+
&self,
305+
ctx: &crate::proto::ExecutionPlanEncodeCtx<'_>,
306+
) -> Result<Option<datafusion_proto_models::protobuf::PhysicalPlanNode>> {
307+
use datafusion_proto_models::protobuf;
308+
let input = ctx.encode_child(self.input())?;
309+
Ok(Some(protobuf::PhysicalPlanNode {
310+
physical_plan_type: Some(
311+
protobuf::physical_plan_node::PhysicalPlanType::Buffer(Box::new(
312+
protobuf::BufferExecNode {
313+
input: Some(Box::new(input)),
314+
capacity: self.capacity() as u64,
315+
},
316+
)),
317+
),
318+
}))
319+
}
320+
}
321+
322+
#[cfg(feature = "proto")]
323+
impl BufferExec {
324+
/// Reconstruct a [`BufferExec`] from its protobuf representation.
325+
///
326+
/// The exact inverse of [`ExecutionPlan::try_to_proto`].
327+
///
328+
/// [`ExecutionPlan::try_to_proto`]: crate::ExecutionPlan::try_to_proto
329+
pub fn try_from_proto(
330+
node: &datafusion_proto_models::protobuf::PhysicalPlanNode,
331+
ctx: &crate::proto::ExecutionPlanDecodeCtx<'_>,
332+
) -> Result<Arc<dyn ExecutionPlan>> {
333+
use datafusion_proto_models::protobuf;
334+
let buffer = crate::expect_plan_variant!(
335+
node,
336+
protobuf::physical_plan_node::PhysicalPlanType::Buffer,
337+
"BufferExec",
338+
);
339+
let input =
340+
ctx.decode_required_child(buffer.input.as_deref(), "BufferExec", "input")?;
341+
Ok(Arc::new(BufferExec::new(input, buffer.capacity as usize)))
342+
}
301343
}
302344

303345
/// Represents anything that occupies a capacity in a [MemoryBufferedStream].

‎datafusion/physical-plan/src/coalesce_batches.rs‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -290,6 +290,61 @@ impl ExecutionPlan for CoalesceBatchesExec {
290290
) as Arc<dyn ExecutionPlan>)
291291
})
292292
}
293+
294+
#[cfg(feature = "proto")]
295+
fn try_to_proto(
296+
&self,
297+
ctx: &crate::proto::ExecutionPlanEncodeCtx<'_>,
298+
) -> Result<Option<datafusion_proto_models::protobuf::PhysicalPlanNode>> {
299+
use datafusion_proto_models::protobuf;
300+
let input = ctx.encode_child(self.input())?;
301+
Ok(Some(protobuf::PhysicalPlanNode {
302+
physical_plan_type: Some(
303+
protobuf::physical_plan_node::PhysicalPlanType::CoalesceBatches(
304+
Box::new(protobuf::CoalesceBatchesExecNode {
305+
input: Some(Box::new(input)),
306+
target_batch_size: self.target_batch_size() as u32,
307+
fetch: self.fetch().map(|n| n as u32),
308+
}),
309+
),
310+
),
311+
}))
312+
}
313+
}
314+
315+
#[cfg(feature = "proto")]
316+
#[expect(deprecated)]
317+
impl CoalesceBatchesExec {
318+
/// Reconstruct a [`CoalesceBatchesExec`] from its protobuf representation.
319+
///
320+
/// The exact inverse of [`ExecutionPlan::try_to_proto`]: it takes the whole
321+
/// [`PhysicalPlanNode`] so every plan's `try_from_proto` shares one
322+
/// signature. The child plan is decoded recursively via the
323+
/// [`ExecutionPlanDecodeCtx`].
324+
///
325+
/// [`PhysicalPlanNode`]: datafusion_proto_models::protobuf::PhysicalPlanNode
326+
/// [`ExecutionPlan::try_to_proto`]: crate::ExecutionPlan::try_to_proto
327+
/// [`ExecutionPlanDecodeCtx`]: crate::proto::ExecutionPlanDecodeCtx
328+
pub fn try_from_proto(
329+
node: &datafusion_proto_models::protobuf::PhysicalPlanNode,
330+
ctx: &crate::proto::ExecutionPlanDecodeCtx<'_>,
331+
) -> Result<Arc<dyn ExecutionPlan>> {
332+
use datafusion_proto_models::protobuf;
333+
let coalesce_batches = crate::expect_plan_variant!(
334+
node,
335+
protobuf::physical_plan_node::PhysicalPlanType::CoalesceBatches,
336+
"CoalesceBatchesExec",
337+
);
338+
let input = ctx.decode_required_child(
339+
coalesce_batches.input.as_deref(),
340+
"CoalesceBatchesExec",
341+
"input",
342+
)?;
343+
Ok(Arc::new(
344+
CoalesceBatchesExec::new(input, coalesce_batches.target_batch_size as usize)
345+
.with_fetch(coalesce_batches.fetch.map(|f| f as usize)),
346+
))
347+
}
293348
}
294349

295350
/// Stream for [`CoalesceBatchesExec`]. See [`CoalesceBatchesExec`] for more details.

‎datafusion/physical-plan/src/coalesce_partitions.rs‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -346,6 +346,56 @@ impl ExecutionPlan for CoalescePartitionsExec {
346346
}
347347
})
348348
}
349+
350+
#[cfg(feature = "proto")]
351+
fn try_to_proto(
352+
&self,
353+
ctx: &crate::proto::ExecutionPlanEncodeCtx<'_>,
354+
) -> Result<Option<datafusion_proto_models::protobuf::PhysicalPlanNode>> {
355+
use datafusion_proto_models::protobuf;
356+
let input = ctx.encode_child(self.input())?;
357+
Ok(Some(protobuf::PhysicalPlanNode {
358+
physical_plan_type: Some(
359+
protobuf::physical_plan_node::PhysicalPlanType::Merge(Box::new(
360+
protobuf::CoalescePartitionsExecNode {
361+
input: Some(Box::new(input)),
362+
fetch: self.fetch().map(|f| f as u32),
363+
},
364+
)),
365+
),
366+
}))
367+
}
368+
}
369+
370+
#[cfg(feature = "proto")]
371+
impl CoalescePartitionsExec {
372+
/// Reconstruct a [`CoalescePartitionsExec`] from its protobuf representation.
373+
///
374+
/// The exact inverse of [`ExecutionPlan::try_to_proto`]. Note the protobuf
375+
/// variant is named `Merge` (node [`CoalescePartitionsExecNode`]).
376+
///
377+
/// [`CoalescePartitionsExecNode`]: datafusion_proto_models::protobuf::CoalescePartitionsExecNode
378+
/// [`ExecutionPlan::try_to_proto`]: crate::ExecutionPlan::try_to_proto
379+
pub fn try_from_proto(
380+
node: &datafusion_proto_models::protobuf::PhysicalPlanNode,
381+
ctx: &crate::proto::ExecutionPlanDecodeCtx<'_>,
382+
) -> Result<Arc<dyn ExecutionPlan>> {
383+
use datafusion_proto_models::protobuf;
384+
let merge = crate::expect_plan_variant!(
385+
node,
386+
protobuf::physical_plan_node::PhysicalPlanType::Merge,
387+
"CoalescePartitionsExec",
388+
);
389+
let input = ctx.decode_required_child(
390+
merge.input.as_deref(),
391+
"CoalescePartitionsExec",
392+
"input",
393+
)?;
394+
Ok(Arc::new(
395+
CoalescePartitionsExec::new(input)
396+
.with_fetch(merge.fetch.map(|f| f as usize)),
397+
))
398+
}
349399
}
350400

351401
#[cfg(test)]

‎datafusion/physical-plan/src/coop.rs‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -369,6 +369,50 @@ impl ExecutionPlan for CooperativeExec {
369369
}
370370
}
371371
}
372+
373+
#[cfg(feature = "proto")]
374+
fn try_to_proto(
375+
&self,
376+
ctx: &crate::proto::ExecutionPlanEncodeCtx<'_>,
377+
) -> Result<Option<datafusion_proto_models::protobuf::PhysicalPlanNode>> {
378+
use datafusion_proto_models::protobuf;
379+
let input = ctx.encode_child(self.input())?;
380+
Ok(Some(protobuf::PhysicalPlanNode {
381+
physical_plan_type: Some(
382+
protobuf::physical_plan_node::PhysicalPlanType::Cooperative(Box::new(
383+
protobuf::CooperativeExecNode {
384+
input: Some(Box::new(input)),
385+
},
386+
)),
387+
),
388+
}))
389+
}
390+
}
391+
392+
#[cfg(feature = "proto")]
393+
impl CooperativeExec {
394+
/// Reconstruct a [`CooperativeExec`] from its protobuf representation.
395+
///
396+
/// The exact inverse of [`ExecutionPlan::try_to_proto`].
397+
///
398+
/// [`ExecutionPlan::try_to_proto`]: crate::ExecutionPlan::try_to_proto
399+
pub fn try_from_proto(
400+
node: &datafusion_proto_models::protobuf::PhysicalPlanNode,
401+
ctx: &crate::proto::ExecutionPlanDecodeCtx<'_>,
402+
) -> Result<Arc<dyn ExecutionPlan>> {
403+
use datafusion_proto_models::protobuf;
404+
let cooperative = crate::expect_plan_variant!(
405+
node,
406+
protobuf::physical_plan_node::PhysicalPlanType::Cooperative,
407+
"CooperativeExec",
408+
);
409+
let input = ctx.decode_required_child(
410+
cooperative.input.as_deref(),
411+
"CooperativeExec",
412+
"input",
413+
)?;
414+
Ok(Arc::new(CooperativeExec::new(input)))
415+
}
372416
}
373417

374418
/// Creates a [`CooperativeStream`] wrapper around the given [`RecordBatchStream`].

0 commit comments

Comments
 (0)