You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 49bb0f1
Browse filesBrowse the repository at this point in the historyBrowse files
fix: preserve EmptyExec and PlaceholderRowExec partition count across proto round-trip (apache#23643)
## Which issue does this PR close?
- Closesapache#23642.
## Rationale for this change
`EmptyExecNode` and `PlaceholderRowExecNode` encoded only a schema, so
the partition count set by `EmptyExec::with_partitions(n)` was silently
lost across a physical-plan round-trip: a plan reporting `n` partitions
before serialization reported `1` after.
This is silent data loss rather than an error — the decoded plan is
well-formed but describes a different plan than the one encoded. It
bites any setup that plans in one process and executes in another. In
Ballista, an optimizer rule collapses provably-empty sub-plans into an
`EmptyExec` inheriting the replaced node's partition count; the
scheduler sizes the stage's task count from that count, ships the plan
to an executor, and every task above partition 0 fails with:
```
Internal("Assertion failed: partition < self.partitions: EmptyExec invalid partition 1 (expected less than 1)")
```
## What changes are included in this PR?
- Add a `uint32 partitions` field to `EmptyExecNode` and
`PlaceholderRowExecNode` in `datafusion.proto`, and regenerate the
prost/pbjson code via `regen.sh`.
- Write `partitions` on encode and apply it via `with_partitions(...)`
on decode.
Both `EmptyExec` and `PlaceholderRowExec` mirror their private
`partitions` field into `PlanProperties` as `UnknownPartitioning(n)`, so
the encoder reads the count via
`properties().output_partitioning().partition_count()` on the existing
`ExecutionPlan` trait. No new public API is added to
`datafusion-physical-plan`.
The wire format stays compatible in both directions. Plans encoded
before this field existed carry no value for it, which prost surfaces as
`0`; decode maps that to the previous default of `1`. Plans encoded
after this change add a field that older readers skip.
Note for apache#23501, which migrates these nodes to the `try_to_proto` /
`try_from_proto` pattern: that issue specifies "schema only" and a
byte-for-byte identical wire format, which would reintroduce this bug.
The `partitions` field should be folded into that rewrite.
## Are these changes tested?
Yes, three new tests in `roundtrip_physical_plan.rs`: partition-count
round-trips for `EmptyExec` and `PlaceholderRowExec`, plus one decoding
`partitions: 0` nodes directly to pin the backward-compatibility
mapping. The round-trip tests were confirmed to fail against the unfixed
decoder with exactly the reported symptom (4 partitions decoding to 1).
## Are there any user-facing changes?
No API changes to `datafusion-physical-plan`. Encoded plans gain a new
protobuf field, which is backward and forward compatible as described
above. The generated `EmptyExecNode` and `PlaceholderRowExecNode`
structs gain a `partitions` field, which breaks downstream code
constructing them with an exhaustive struct literal; this is documented
in the 55.0.0 upgrade guide.
0 commit comments