Repository navigation
Commit a4e071d
refactor(proto): destructure plan and proto structs in core plan serde hooks (apache#24167)
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes apache#123` indicates that this PR will close issue apache#123.
-->
- Follow-up cleanup for EPIC apache#23494; does not close an issue on its own.
## Rationale for this change
EPIC apache#23494 moved every built-in `ExecutionPlan` off the central
`downcast_ref`
chain in `datafusion-proto` onto per-plan hooks: a `try_to_proto`
override on
the `ExecutionPlan` trait and an inherent `FooExec::try_from_proto`,
both living
in the plan's own module.
Those hooks currently read plan state through **getters**
(`self.expr()`,
`self.fetch()`, ...). That means adding a field to a plan struct is
invisible to
serialization: nothing breaks, nothing warns, the field is just silently
dropped on round-trip. This is not hypothetical — it has already cost us
a real
bug: `HashJoinExec.fetch` is not serialized and is lost on round-trip
(being
fixed separately).
Starting each hook with an *exhaustive* destructure turns that class of
mistake
into a compile error:
* **Encode side** — destructure `self` naming every field, no `..`. Add
a field
to `SortExec` and `SortExec::try_to_proto` stops compiling until you
either
serialize it or bind it to `_` with a comment saying why it does not
need to
be (derived at construction, runtime state, recomputed on decode).
* **Decode side** — destructure the prost node struct the same way. The
generated structs are plain, all-`pub` and not `#[non_exhaustive]`, so
adding
a field to the `.proto` becomes a compile error in every decoder rather
than a
silent omission.
## What changes are included in this PR?
A mechanical, behavior-preserving refactor of the serde hooks for the
core
(non-join, non-aggregate, non-window) plans in
`datafusion/physical-plan/src/`:
`SortExec`, `SortPreservingMergeExec`, `GlobalLimitExec`,
`LocalLimitExec`,
`FilterExec`, `ProjectionExec`, `RepartitionExec`, `UnionExec`,
`InterleaveExec`, `CoalesceBatchesExec`, `CoalescePartitionsExec`,
`CooperativeExec`, `BufferExec`, `EmptyExec`, `PlaceholderRowExec`,
`ExplainExec`, `ScalarSubqueryExec`.
**The wire format is unchanged — byte for byte — and no behavior
changes.** The
same values are written to and read from the same proto fields; only the
way
they are reached in Rust changes.
Split into four commits, one per group of plans, each of which builds
green.
### Fields the refactor documented as not serialized
Writing the destructures surfaced and pinned down, in comments at the
point of
use, which fields do not survive a round-trip today. All of these are
pre-existing; **none of them are changed here**, because fixing any of
them
means changing the wire format, which is a separate decision:
* `SortPreservingMergeExec::enable_round_robin_repartition` — no field
on
`SortPreservingMergeExecNode`; decoding always restores the `true`
default
from `SortPreservingMergeExec::new`.
* `GlobalLimitExec::required_ordering` and
`LocalLimitExec::required_ordering` —
no field on the limit nodes; they are set by the `enforce_sorting`
optimizer
rule, so a decoded plan starts with `None`.
Everything else that binds to `_` is genuinely not plan shape: runtime
metrics
(`metrics` / `metrics_set`), execution-time state
(`RepartitionExec::state`,
`ScalarSubqueryExec::subquery_future` and `results`), values derived at
construction from other serialized fields
(`SortExec::common_sort_prefix`), the
positional `ScalarSubqueryLink::index`, and the `cache` / `properties`
plan
properties that are recomputed on decode.
Two small notes for reviewers:
* `RepartitionExec` keeps its output partitioning *inside* `cache`
(`partitioning()` is `&self.cache.partitioning`), so `cache` is bound
rather
than `_`-ed there and is read for the serialized `partitioning` field.
* `EmptyExec` / `PlaceholderRowExec` now read the encoded partition
count from
the `partitions` field instead of going through
`properties().output_partitioning().partition_count()`. The two are kept
in
sync by `with_partitions`, the only writer besides `new`, so the encoded
value
is identical.
## Are these changes tested?
Covered by the existing round-trip tests, which are the real proof that
the wire
format did not move. Locally on this branch:
* `cargo test -p datafusion-proto --test proto_integration` — 214
passed, 0 failed
* `cargo test -p datafusion-physical-plan` — 1640 passed, 0 failed
* `cargo clippy -p datafusion-physical-plan --all-targets --all-features
-- -D warnings` — clean
* `cargo fmt --all`
No new tests are added here: the change adds no new behavior to test,
and the
compile-error property it introduces is enforced by rustc rather than by
a test.
**On the comprehensiveness of that existing coverage** (thanks
@andygrove for
pushing on this — the original wording oversold it):
* `SortPreservingMergeExec`, updated by this PR, has **no** round-trip
test at
all — `SortPreservingMerge` does not appear anywhere in
`roundtrip_physical_plan.rs`. The other 16 plans touched here each have
at
least one dedicated `roundtrip_*` test.
* Separately, `roundtrip_test` asserts on `format!("{plan:?}")`, so it
cannot
catch a dropped field that a plan's `Debug` impl doesn't print. That is
precisely how the `HashJoinExec::fetch` bug (apache#24165) went unnoticed.
Both are now tracked in apache#24171, with the test work split out into its
own PR so
this mechanical refactor and the coverage improvements can be reviewed
independently. Sequencing is up to reviewers — happy to land the tests
first if
that's preferred.
## Are there any user-facing changes?
No. No public API changes, no wire format changes, no behavior changes.
---------
Co-authored-by: Claude Opus 5 <[email protected]>1 parent e2bdc2c commit a4e071d
15 files changed
Lines changed: 425 additions & 156 deletions
File tree
- datafusion/physical-plan/src
- repartition
- sorts
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
332 | 332 | | |
333 | 333 | | |
334 | 334 | | |
335 | | - | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
336 | 347 | | |
337 | 348 | | |
338 | 349 | | |
339 | 350 | | |
340 | 351 | | |
341 | | - | |
| 352 | + | |
342 | 353 | | |
343 | 354 | | |
344 | 355 | | |
| |||
363 | 374 | | |
364 | 375 | | |
365 | 376 | | |
366 | | - | |
367 | | - | |
368 | | - | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
369 | 382 | | |
370 | 383 | | |
371 | 384 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
323 | 323 | | |
324 | 324 | | |
325 | 325 | | |
326 | | - | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
327 | 339 | | |
328 | 340 | | |
329 | 341 | | |
330 | 342 | | |
331 | 343 | | |
332 | | - | |
333 | | - | |
| 344 | + | |
| 345 | + | |
334 | 346 | | |
335 | 347 | | |
336 | 348 | | |
| |||
361 | 373 | | |
362 | 374 | | |
363 | 375 | | |
364 | | - | |
365 | | - | |
366 | | - | |
367 | | - | |
368 | | - | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
369 | 386 | | |
370 | | - | |
371 | | - | |
| 387 | + | |
| 388 | + | |
372 | 389 | | |
373 | 390 | | |
374 | 391 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
384 | 384 | | |
385 | 385 | | |
386 | 386 | | |
387 | | - | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
388 | 399 | | |
389 | 400 | | |
390 | 401 | | |
391 | 402 | | |
392 | 403 | | |
393 | | - | |
| 404 | + | |
394 | 405 | | |
395 | 406 | | |
396 | 407 | | |
| |||
417 | 428 | | |
418 | 429 | | |
419 | 430 | | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
420 | 435 | | |
421 | | - | |
| 436 | + | |
422 | 437 | | |
423 | 438 | | |
424 | 439 | | |
425 | 440 | | |
426 | | - | |
427 | | - | |
| 441 | + | |
428 | 442 | | |
429 | 443 | | |
430 | 444 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
402 | 402 | | |
403 | 403 | | |
404 | 404 | | |
405 | | - | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
406 | 414 | | |
407 | 415 | | |
408 | 416 | | |
| |||
432 | 440 | | |
433 | 441 | | |
434 | 442 | | |
435 | | - | |
436 | | - | |
437 | | - | |
438 | | - | |
439 | | - | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
440 | 449 | | |
441 | 450 | | |
442 | 451 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
214 | 214 | | |
215 | 215 | | |
216 | 216 | | |
217 | | - | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
218 | 227 | | |
219 | 228 | | |
220 | 229 | | |
221 | 230 | | |
222 | 231 | | |
223 | | - | |
224 | | - | |
225 | | - | |
226 | | - | |
| 232 | + | |
227 | 233 | | |
228 | 234 | | |
229 | 235 | | |
| |||
244 | 250 | | |
245 | 251 | | |
246 | 252 | | |
247 | | - | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
248 | 257 | | |
249 | 258 | | |
250 | 259 | | |
251 | 260 | | |
252 | 261 | | |
253 | 262 | | |
254 | 263 | | |
255 | | - | |
| 264 | + | |
256 | 265 | | |
257 | 266 | | |
258 | 267 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
215 | 215 | | |
216 | 216 | | |
217 | 217 | | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
218 | 228 | | |
219 | 229 | | |
220 | 230 | | |
221 | 231 | | |
222 | | - | |
223 | | - | |
224 | | - | |
| 232 | + | |
| 233 | + | |
225 | 234 | | |
226 | 235 | | |
227 | 236 | | |
228 | | - | |
| 237 | + | |
229 | 238 | | |
230 | 239 | | |
231 | 240 | | |
| |||
247 | 256 | | |
248 | 257 | | |
249 | 258 | | |
250 | | - | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
251 | 267 | | |
252 | 268 | | |
253 | 269 | | |
254 | 270 | | |
255 | 271 | | |
256 | 272 | | |
257 | | - | |
258 | | - | |
| 273 | + | |
259 | 274 | | |
260 | 275 | | |
261 | 276 | | |
262 | | - | |
| 277 | + | |
263 | 278 | | |
264 | 279 | | |
265 | 280 | | |
| |||
0 commit comments