Skip to content

Commit a589b4b

Browse files
adriangbclaude
andauthored
refactor(proto): put Partitioning / sort-expression serde on the types (apache#24003)
## Which issue does this PR close? - Part of apache#23494 (the `try_to_proto` / `try_from_proto` migration). Precursor for apache#23497 / apache#23683 / apache#23498 and for the remaining per-plan migrations that need to encode an output partitioning or an ordering. ## Rationale for this change `Partitioning`'s protobuf conversion currently exists in three places: - inline in `RepartitionExec::try_to_proto`, - inline in `RepartitionExec::try_from_proto`, - in `datafusion-proto`'s `serialize_partitioning` / `parse_protobuf_partitioning`. Every plan or data source migrated to the hooks that has an output partitioning has so far copied it again — apache#23683 is about to add a fourth copy for `FileScanConfig`. The flat `PhysicalSortExprNode` encoding is the same story, one level down and more widespread: `AggregateExec`'s ordering requirement, `SymmetricHashJoinExec`'s left and right sort expressions, the window expressions, and range partitioning each hand-roll the same `map` / `collect` over `PhysicalSortExprNode { expr, asc, nulls_first }`, on both the encode and the decode side. apache#23683 (`output_ordering`) and apache#23752 / apache#23781 (the sinks' required ordering) are about to add two more. There is no reason for this logic to live in the callers: it converts a `Partitioning` (and the `PhysicalSortExpr`s and `ScalarValue`s inside it), and needs nothing from the plan level beyond the ability to encode a child expression. ## What changes are included in this PR? Put the single copy next to the types that own it, taking the expression-level context (`datafusion-physical-expr` and `-common` already carry the `proto` feature): - `PhysicalSortExpr::try_to_proto` / `try_from_proto` (`physical-expr-common`) - `sort_exprs_try_to_proto` / `sort_exprs_try_from_proto` (`physical-expr-common`), the sequence form every caller actually needs - `Partitioning::try_to_proto` / `try_from_proto` (`physical-expr`) So plan hooks can reach them, `ExecutionPlanEncodeCtx` / `ExecutionPlanDecodeCtx` now back the expression-level contexts (`PhysicalExprEncode` / `PhysicalExprDecode`) and hand one out via `expr_ctx()`. That bridge is useful beyond partitioning: from here on any plan hook can pass its ctx straight to an expression-level conversion, which is the shape the rest of the migration wants. The sequence encoder is generic over `Borrow<PhysicalSortExpr>`, so one function serves a `LexOrdering`, a `&[PhysicalSortExpr]`, and a `LexRequirement` mapped through `PhysicalSortExpr::from`. The decoder returns the expressions rather than a `LexOrdering`, because callers disagree on what an empty list means: "no ordering declared" for a scan, an error for an operator that requires one. Routed through the new methods: - `RepartitionExec` and `datafusion-proto`'s central serializer, which also retires `serialize_range_partitioning`, `serialize_range_split_point`, `parse_protobuf_range_partitioning` and `parse_protobuf_range_split_point`, - `AggregateExec`'s ordering requirement, `SymmetricHashJoinExec`'s left/right sort expressions, and the window expressions — encode and decode each. Net: −380 / +458 lines with the new tests included; production code shrinks. The next operator that needs partitioning or ordering serde writes one line instead of sixty. ## Are these changes tested? Yes. - New unit tests for the sequence helpers (option and order fidelity, owned `LexRequirement` input, encode-error propagation, missing inner expression), using the existing `proto_test_util` stubs. - The existing round-trip suites cover the rest, and now exercise the shared path: all four partitioning variants (`datafusion-proto`'s `roundtrip_physical_plan` tests exercise the central serializer, `RepartitionExec`'s own hook tests exercise the plan path), plus aggregate `ORDER BY`, window `ORDER BY` and symmetric-hash-join sort expressions for the ordering helpers. `datafusion-proto`: 209 passed / 0 failed. - Lib suites for the three changed crates: 1613 + 1651 + 80 passed / 0 failed. - Full workspace run: 10344 passed (`cargo test --profile ci --workspace --lib --tests --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption`). The only failures are 8 backtrace-symbolization tests in `datafusion-common` and 4 `datafusion-cli` tests that hard-code a repo-relative `parquet-testing/` path; both are artifacts of running from a linked worktree on macOS, in crates this PR does not touch, and the `datafusion-cli` four pass once that path resolves. - `cargo fmt` and `ci/scripts/rust_clippy.sh` clean. ## Are there any user-facing changes? The protobuf wire format is unchanged. Additive API: - `Partitioning::try_to_proto` / `try_from_proto`, `PhysicalSortExpr::try_to_proto` / `try_from_proto`, `sort_exprs_try_to_proto` / `sort_exprs_try_from_proto` (feature `proto`). - `ExecutionPlanEncodeCtx::expr_ctx()` / `ExecutionPlanDecodeCtx::expr_ctx(schema)`. Two behavior differences, both in error paths: - Out-of-range partition counts now return an error instead of wrapping (`as usize`) or panicking (`try_into().unwrap()` in `parse_protobuf_hash_partitioning`). - A missing sort-expression child now reports which field is missing (`PhysicalSortExpr is missing required field 'expr'`) instead of `Unexpected empty physical expression`, and the same message now replaces the three bespoke ones in `AggregateExec`, `SymmetricHashJoinExec` and the window expressions. Four private helpers in `datafusion-proto` are removed (`serialize_range_partitioning`, `serialize_range_split_point`, `parse_protobuf_range_partitioning`, `parse_protobuf_range_split_point`); the public `serialize_partitioning` / `parse_protobuf_partitioning` keep their signatures and behavior. --------- Co-authored-by: Claude Opus 5 <[email protected]>
1 parent 39d5064 commit a589b4b

10 files changed

Lines changed: 696 additions & 395 deletions

File tree

‎datafusion/physical-expr-common/src/sort_expr.rs‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -183,6 +183,102 @@ impl PhysicalSortExpr {
183183
}
184184
}
185185

186+
/// Protobuf conversions for [`PhysicalSortExpr`].
187+
///
188+
/// This is the flat [`PhysicalSortExprNode`] representation used wherever the
189+
/// wire format stores an ordering (scan output orderings, range partitioning,
190+
/// window frames, …). It is *not* the `PhysicalExprNode::Sort` wrapping that
191+
/// `SortExec` uses for its own `expr` field.
192+
///
193+
/// [`PhysicalSortExprNode`]: datafusion_proto_models::protobuf::PhysicalSortExprNode
194+
#[cfg(feature = "proto")]
195+
impl PhysicalSortExpr {
196+
/// Serialize this sort expression, encoding its child expression through
197+
/// `ctx`.
198+
pub fn try_to_proto(
199+
&self,
200+
ctx: &crate::physical_expr::proto_encode::PhysicalExprEncodeCtx<'_>,
201+
) -> Result<datafusion_proto_models::protobuf::PhysicalSortExprNode> {
202+
Ok(datafusion_proto_models::protobuf::PhysicalSortExprNode {
203+
expr: Some(Box::new(ctx.encode_child(&self.expr)?)),
204+
asc: !self.options.descending,
205+
nulls_first: self.options.nulls_first,
206+
})
207+
}
208+
209+
/// Reconstruct a [`PhysicalSortExpr`] from its protobuf representation.
210+
pub fn try_from_proto(
211+
node: &datafusion_proto_models::protobuf::PhysicalSortExprNode,
212+
ctx: &crate::physical_expr::proto_decode::PhysicalExprDecodeCtx<'_>,
213+
) -> Result<Self> {
214+
let expr = ctx.decode_required_expression(
215+
node.expr.as_deref(),
216+
"PhysicalSortExpr",
217+
"expr",
218+
)?;
219+
Ok(PhysicalSortExpr {
220+
expr,
221+
options: SortOptions {
222+
descending: !node.asc,
223+
nulls_first: node.nulls_first,
224+
},
225+
})
226+
}
227+
}
228+
229+
/// Serialize a sequence of sort expressions into the flat
230+
/// [`PhysicalSortExprNode`] list the wire format uses for an ordering.
231+
///
232+
/// Accepts anything that yields [`PhysicalSortExpr`]s by value or by reference,
233+
/// so a [`LexOrdering`], a `&[PhysicalSortExpr]`, or a [`LexRequirement`]
234+
/// mapped through [`PhysicalSortExpr::from`] all work:
235+
///
236+
/// ```ignore
237+
/// let nodes = sort_exprs_try_to_proto(ordering.iter(), ctx)?;
238+
/// let nodes = sort_exprs_try_to_proto(
239+
/// requirement.iter().map(|req| PhysicalSortExpr::from(req.clone())),
240+
/// ctx,
241+
/// )?;
242+
/// ```
243+
///
244+
/// The `PhysicalSortExprNodeCollection` message some plans use is just this
245+
/// list in a wrapper, so those callers wrap the result themselves rather than
246+
/// this function guessing which shape they mean.
247+
///
248+
/// [`PhysicalSortExprNode`]: datafusion_proto_models::protobuf::PhysicalSortExprNode
249+
#[cfg(feature = "proto")]
250+
pub fn sort_exprs_try_to_proto<E: std::borrow::Borrow<PhysicalSortExpr>>(
251+
exprs: impl IntoIterator<Item = E>,
252+
ctx: &crate::physical_expr::proto_encode::PhysicalExprEncodeCtx<'_>,
253+
) -> Result<Vec<datafusion_proto_models::protobuf::PhysicalSortExprNode>> {
254+
exprs
255+
.into_iter()
256+
.map(|expr| expr.borrow().try_to_proto(ctx))
257+
.collect()
258+
}
259+
260+
/// Reconstruct a sequence of sort expressions from the flat
261+
/// [`PhysicalSortExprNode`] list, the counterpart of
262+
/// [`sort_exprs_try_to_proto`].
263+
///
264+
/// Returns the expressions rather than a [`LexOrdering`] or a
265+
/// [`LexRequirement`], because callers differ in what an empty list means:
266+
/// `LexOrdering::new` / `LexRequirement::new` return `None` for it, which is
267+
/// "no ordering declared" for a scan and an error for an operator that requires
268+
/// one.
269+
///
270+
/// [`PhysicalSortExprNode`]: datafusion_proto_models::protobuf::PhysicalSortExprNode
271+
#[cfg(feature = "proto")]
272+
pub fn sort_exprs_try_from_proto(
273+
nodes: &[datafusion_proto_models::protobuf::PhysicalSortExprNode],
274+
ctx: &crate::physical_expr::proto_decode::PhysicalExprDecodeCtx<'_>,
275+
) -> Result<Vec<PhysicalSortExpr>> {
276+
nodes
277+
.iter()
278+
.map(|node| PhysicalSortExpr::try_from_proto(node, ctx))
279+
.collect()
280+
}
281+
186282
impl PartialEq for PhysicalSortExpr {
187283
fn eq(&self, other: &Self) -> bool {
188284
self.options == other.options && self.expr.eq(&other.expr)

0 commit comments

Comments
 (0)