Repository navigation
Commit a589b4b
refactor(proto): put Partitioning / sort-expression serde on the types (apache#24003)
## Which issue does this PR close?
- Part of apache#23494 (the `try_to_proto` / `try_from_proto` migration).
Precursor for apache#23497 / apache#23683 / apache#23498 and for the remaining per-plan
migrations that need to encode an output partitioning or an ordering.
## Rationale for this change
`Partitioning`'s protobuf conversion currently exists in three places:
- inline in `RepartitionExec::try_to_proto`,
- inline in `RepartitionExec::try_from_proto`,
- in `datafusion-proto`'s `serialize_partitioning` /
`parse_protobuf_partitioning`.
Every plan or data source migrated to the hooks that has an output
partitioning has so far copied it again — apache#23683 is about to add a
fourth
copy for `FileScanConfig`.
The flat `PhysicalSortExprNode` encoding is the same story, one level
down and
more widespread: `AggregateExec`'s ordering requirement,
`SymmetricHashJoinExec`'s
left and right sort expressions, the window expressions, and range
partitioning
each hand-roll the same `map` / `collect` over
`PhysicalSortExprNode { expr, asc, nulls_first }`, on both the encode
and the
decode side. apache#23683 (`output_ordering`) and apache#23752 / apache#23781 (the sinks'
required
ordering) are about to add two more.
There is no reason for this logic to live in the callers: it converts a
`Partitioning` (and the `PhysicalSortExpr`s and `ScalarValue`s inside
it), and
needs nothing from the plan level beyond the ability to encode a child
expression.
## What changes are included in this PR?
Put the single copy next to the types that own it, taking the
expression-level
context (`datafusion-physical-expr` and `-common` already carry the
`proto`
feature):
- `PhysicalSortExpr::try_to_proto` / `try_from_proto`
(`physical-expr-common`)
- `sort_exprs_try_to_proto` / `sort_exprs_try_from_proto`
(`physical-expr-common`),
the sequence form every caller actually needs
- `Partitioning::try_to_proto` / `try_from_proto` (`physical-expr`)
So plan hooks can reach them, `ExecutionPlanEncodeCtx` /
`ExecutionPlanDecodeCtx`
now back the expression-level contexts (`PhysicalExprEncode` /
`PhysicalExprDecode`) and hand one out via `expr_ctx()`. That bridge is
useful
beyond partitioning: from here on any plan hook can pass its ctx
straight to an
expression-level conversion, which is the shape the rest of the
migration wants.
The sequence encoder is generic over `Borrow<PhysicalSortExpr>`, so one
function
serves a `LexOrdering`, a `&[PhysicalSortExpr]`, and a `LexRequirement`
mapped
through `PhysicalSortExpr::from`. The decoder returns the expressions
rather
than a `LexOrdering`, because callers disagree on what an empty list
means: "no
ordering declared" for a scan, an error for an operator that requires
one.
Routed through the new methods:
- `RepartitionExec` and `datafusion-proto`'s central serializer, which
also
retires `serialize_range_partitioning`, `serialize_range_split_point`,
`parse_protobuf_range_partitioning` and
`parse_protobuf_range_split_point`,
- `AggregateExec`'s ordering requirement, `SymmetricHashJoinExec`'s
left/right
sort expressions, and the window expressions — encode and decode each.
Net: −380 / +458 lines with the new tests included; production code
shrinks. The
next operator that needs partitioning or ordering serde writes one line
instead
of sixty.
## Are these changes tested?
Yes.
- New unit tests for the sequence helpers (option and order fidelity,
owned
`LexRequirement` input, encode-error propagation, missing inner
expression),
using the existing `proto_test_util` stubs.
- The existing round-trip suites cover the rest, and now exercise the
shared
path: all four partitioning variants (`datafusion-proto`'s
`roundtrip_physical_plan` tests exercise the central serializer,
`RepartitionExec`'s own hook tests exercise the plan path), plus
aggregate
`ORDER BY`, window `ORDER BY` and symmetric-hash-join sort expressions
for the
ordering helpers. `datafusion-proto`: 209 passed / 0 failed.
- Lib suites for the three changed crates: 1613 + 1651 + 80 passed / 0
failed.
- Full workspace run: 10344 passed
(`cargo test --profile ci --workspace --lib --tests --features
avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption`).
The only failures are 8 backtrace-symbolization tests in
`datafusion-common`
and 4 `datafusion-cli` tests that hard-code a repo-relative
`parquet-testing/`
path; both are artifacts of running from a linked worktree on macOS, in
crates
this PR does not touch, and the `datafusion-cli` four pass once that
path
resolves.
- `cargo fmt` and `ci/scripts/rust_clippy.sh` clean.
## Are there any user-facing changes?
The protobuf wire format is unchanged.
Additive API:
- `Partitioning::try_to_proto` / `try_from_proto`,
`PhysicalSortExpr::try_to_proto` /
`try_from_proto`, `sort_exprs_try_to_proto` /
`sort_exprs_try_from_proto`
(feature `proto`).
- `ExecutionPlanEncodeCtx::expr_ctx()` /
`ExecutionPlanDecodeCtx::expr_ctx(schema)`.
Two behavior differences, both in error paths:
- Out-of-range partition counts now return an error instead of wrapping
(`as usize`) or panicking (`try_into().unwrap()` in
`parse_protobuf_hash_partitioning`).
- A missing sort-expression child now reports which field is missing
(`PhysicalSortExpr is missing required field 'expr'`) instead of
`Unexpected empty physical expression`, and the same message now
replaces the
three bespoke ones in `AggregateExec`, `SymmetricHashJoinExec` and the
window
expressions.
Four private helpers in `datafusion-proto` are removed
(`serialize_range_partitioning`, `serialize_range_split_point`,
`parse_protobuf_range_partitioning`,
`parse_protobuf_range_split_point`); the
public `serialize_partitioning` / `parse_protobuf_partitioning` keep
their
signatures and behavior.
---------
Co-authored-by: Claude Opus 5 <[email protected]>1 parent 39d5064 commit a589b4b
10 files changed
Lines changed: 696 additions & 395 deletions
File tree
- datafusion
- physical-expr-common/src
- physical-expr/src
- physical-plan/src
- aggregates
- joins
- repartition
- windows
- proto
- src/physical_plan
- tests/cases
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
183 | 183 | | |
184 | 184 | | |
185 | 185 | | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
186 | 282 | | |
187 | 283 | | |
188 | 284 | | |
| |||
0 commit comments