Skip to content

Commit 49bb0f1

Browse files
andygrovekosiew
authored andcommitted
fix: preserve EmptyExec and PlaceholderRowExec partition count across proto round-trip (apache#23643)
## Which issue does this PR close? - Closes apache#23642. ## Rationale for this change `EmptyExecNode` and `PlaceholderRowExecNode` encoded only a schema, so the partition count set by `EmptyExec::with_partitions(n)` was silently lost across a physical-plan round-trip: a plan reporting `n` partitions before serialization reported `1` after. This is silent data loss rather than an error — the decoded plan is well-formed but describes a different plan than the one encoded. It bites any setup that plans in one process and executes in another. In Ballista, an optimizer rule collapses provably-empty sub-plans into an `EmptyExec` inheriting the replaced node's partition count; the scheduler sizes the stage's task count from that count, ships the plan to an executor, and every task above partition 0 fails with: ``` Internal("Assertion failed: partition < self.partitions: EmptyExec invalid partition 1 (expected less than 1)") ``` ## What changes are included in this PR? - Add a `uint32 partitions` field to `EmptyExecNode` and `PlaceholderRowExecNode` in `datafusion.proto`, and regenerate the prost/pbjson code via `regen.sh`. - Write `partitions` on encode and apply it via `with_partitions(...)` on decode. Both `EmptyExec` and `PlaceholderRowExec` mirror their private `partitions` field into `PlanProperties` as `UnknownPartitioning(n)`, so the encoder reads the count via `properties().output_partitioning().partition_count()` on the existing `ExecutionPlan` trait. No new public API is added to `datafusion-physical-plan`. The wire format stays compatible in both directions. Plans encoded before this field existed carry no value for it, which prost surfaces as `0`; decode maps that to the previous default of `1`. Plans encoded after this change add a field that older readers skip. Note for apache#23501, which migrates these nodes to the `try_to_proto` / `try_from_proto` pattern: that issue specifies "schema only" and a byte-for-byte identical wire format, which would reintroduce this bug. The `partitions` field should be folded into that rewrite. ## Are these changes tested? Yes, three new tests in `roundtrip_physical_plan.rs`: partition-count round-trips for `EmptyExec` and `PlaceholderRowExec`, plus one decoding `partitions: 0` nodes directly to pin the backward-compatibility mapping. The round-trip tests were confirmed to fail against the unfixed decoder with exactly the reported symptom (4 partitions decoding to 1). ## Are there any user-facing changes? No API changes to `datafusion-physical-plan`. Encoded plans gain a new protobuf field, which is backward and forward compatible as described above. The generated `EmptyExecNode` and `PlaceholderRowExecNode` structs gain a `partitions` field, which breaks downstream code constructing them with an exhaustive struct literal; this is documented in the 55.0.0 upgrade guide.
1 parent 1836a28 commit 49bb0f1

6 files changed

Lines changed: 154 additions & 5 deletions

File tree

‎datafusion/proto-models/proto/datafusion.proto‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1369,10 +1369,16 @@ message JoinOn {
13691369

13701370
message EmptyExecNode {
13711371
datafusion_common.Schema schema = 1;
1372+
// Number of output partitions. Absent (0) means a single partition, so that
1373+
// plans encoded before this field existed decode to the previous default.
1374+
uint32 partitions = 2;
13721375
}
13731376

13741377
message PlaceholderRowExecNode {
13751378
datafusion_common.Schema schema = 1;
1379+
// Number of output partitions. Absent (0) means a single partition, so that
1380+
// plans encoded before this field existed decode to the previous default.
1381+
uint32 partitions = 2;
13761382
}
13771383

13781384
message ProjectionExecNode {

‎datafusion/proto-models/src/generated/pbjson.rs‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5857,10 +5857,16 @@ impl serde::Serialize for EmptyExecNode {
58575857
if self.schema.is_some() {
58585858
len += 1;
58595859
}
5860+
if self.partitions != 0 {
5861+
len += 1;
5862+
}
58605863
let mut struct_ser = serializer.serialize_struct("datafusion.EmptyExecNode", len)?;
58615864
if let Some(v) = self.schema.as_ref() {
58625865
struct_ser.serialize_field("schema", v)?;
58635866
}
5867+
if self.partitions != 0 {
5868+
struct_ser.serialize_field("partitions", &self.partitions)?;
5869+
}
58645870
struct_ser.end()
58655871
}
58665872
}
@@ -5872,11 +5878,13 @@ impl<'de> serde::Deserialize<'de> for EmptyExecNode {
58725878
{
58735879
const FIELDS: &[&str] = &[
58745880
"schema",
5881+
"partitions",
58755882
];
58765883

58775884
#[allow(clippy::enum_variant_names)]
58785885
enum GeneratedField {
58795886
Schema,
5887+
Partitions,
58805888
}
58815889
impl<'de> serde::Deserialize<'de> for GeneratedField {
58825890
fn deserialize<D>(deserializer: D) -> std::result::Result<GeneratedField, D::Error>
@@ -5899,6 +5907,7 @@ impl<'de> serde::Deserialize<'de> for EmptyExecNode {
58995907
{
59005908
match value {
59015909
"schema" => Ok(GeneratedField::Schema),
5910+
"partitions" => Ok(GeneratedField::Partitions),
59025911
_ => Err(serde::de::Error::unknown_field(value, FIELDS)),
59035912
}
59045913
}
@@ -5919,6 +5928,7 @@ impl<'de> serde::Deserialize<'de> for EmptyExecNode {
59195928
V: serde::de::MapAccess<'de>,
59205929
{
59215930
let mut schema__ = None;
5931+
let mut partitions__ = None;
59225932
while let Some(k) = map_.next_key()? {
59235933
match k {
59245934
GeneratedField::Schema => {
@@ -5927,10 +5937,19 @@ impl<'de> serde::Deserialize<'de> for EmptyExecNode {
59275937
}
59285938
schema__ = map_.next_value()?;
59295939
}
5940+
GeneratedField::Partitions => {
5941+
if partitions__.is_some() {
5942+
return Err(serde::de::Error::duplicate_field("partitions"));
5943+
}
5944+
partitions__ =
5945+
Some(map_.next_value::<::pbjson::private::NumberDeserialize<_>>()?.0)
5946+
;
5947+
}
59305948
}
59315949
}
59325950
Ok(EmptyExecNode {
59335951
schema: schema__,
5952+
partitions: partitions__.unwrap_or_default(),
59345953
})
59355954
}
59365955
}
@@ -22125,10 +22144,16 @@ impl serde::Serialize for PlaceholderRowExecNode {
2212522144
if self.schema.is_some() {
2212622145
len += 1;
2212722146
}
22147+
if self.partitions != 0 {
22148+
len += 1;
22149+
}
2212822150
let mut struct_ser = serializer.serialize_struct("datafusion.PlaceholderRowExecNode", len)?;
2212922151
if let Some(v) = self.schema.as_ref() {
2213022152
struct_ser.serialize_field("schema", v)?;
2213122153
}
22154+
if self.partitions != 0 {
22155+
struct_ser.serialize_field("partitions", &self.partitions)?;
22156+
}
2213222157
struct_ser.end()
2213322158
}
2213422159
}
@@ -22140,11 +22165,13 @@ impl<'de> serde::Deserialize<'de> for PlaceholderRowExecNode {
2214022165
{
2214122166
const FIELDS: &[&str] = &[
2214222167
"schema",
22168+
"partitions",
2214322169
];
2214422170

2214522171
#[allow(clippy::enum_variant_names)]
2214622172
enum GeneratedField {
2214722173
Schema,
22174+
Partitions,
2214822175
}
2214922176
impl<'de> serde::Deserialize<'de> for GeneratedField {
2215022177
fn deserialize<D>(deserializer: D) -> std::result::Result<GeneratedField, D::Error>
@@ -22167,6 +22194,7 @@ impl<'de> serde::Deserialize<'de> for PlaceholderRowExecNode {
2216722194
{
2216822195
match value {
2216922196
"schema" => Ok(GeneratedField::Schema),
22197+
"partitions" => Ok(GeneratedField::Partitions),
2217022198
_ => Err(serde::de::Error::unknown_field(value, FIELDS)),
2217122199
}
2217222200
}
@@ -22187,6 +22215,7 @@ impl<'de> serde::Deserialize<'de> for PlaceholderRowExecNode {
2218722215
V: serde::de::MapAccess<'de>,
2218822216
{
2218922217
let mut schema__ = None;
22218+
let mut partitions__ = None;
2219022219
while let Some(k) = map_.next_key()? {
2219122220
match k {
2219222221
GeneratedField::Schema => {
@@ -22195,10 +22224,19 @@ impl<'de> serde::Deserialize<'de> for PlaceholderRowExecNode {
2219522224
}
2219622225
schema__ = map_.next_value()?;
2219722226
}
22227+
GeneratedField::Partitions => {
22228+
if partitions__.is_some() {
22229+
return Err(serde::de::Error::duplicate_field("partitions"));
22230+
}
22231+
partitions__ =
22232+
Some(map_.next_value::<::pbjson::private::NumberDeserialize<_>>()?.0)
22233+
;
22234+
}
2219822235
}
2219922236
}
2220022237
Ok(PlaceholderRowExecNode {
2220122238
schema: schema__,
22239+
partitions: partitions__.unwrap_or_default(),
2220222240
})
2220322241
}
2220422242
}

‎datafusion/proto-models/src/generated/prost.rs‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2068,11 +2068,19 @@ pub struct JoinOn {
20682068
pub struct EmptyExecNode {
20692069
#[prost(message, optional, tag = "1")]
20702070
pub schema: ::core::option::Option<super::datafusion_common::Schema>,
2071+
/// Number of output partitions. Absent (0) means a single partition, so that
2072+
/// plans encoded before this field existed decode to the previous default.
2073+
#[prost(uint32, tag = "2")]
2074+
pub partitions: u32,
20712075
}
20722076
#[derive(Clone, PartialEq, ::prost::Message)]
20732077
pub struct PlaceholderRowExecNode {
20742078
#[prost(message, optional, tag = "1")]
20752079
pub schema: ::core::option::Option<super::datafusion_common::Schema>,
2080+
/// Number of output partitions. Absent (0) means a single partition, so that
2081+
/// plans encoded before this field existed decode to the previous default.
2082+
#[prost(uint32, tag = "2")]
2083+
pub partitions: u32,
20762084
}
20772085
#[derive(Clone, PartialEq, ::prost::Message)]
20782086
pub struct ProjectionExecNode {

‎datafusion/proto/src/physical_plan/mod.rs‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1710,7 +1710,10 @@ pub trait PhysicalPlanNodeExt: Sized {
17101710
_proto_converter: &dyn PhysicalProtoConverterExtension,
17111711
) -> Result<Arc<dyn ExecutionPlan>> {
17121712
let schema = Arc::new(convert_required!(empty.schema)?);
1713-
Ok(Arc::new(EmptyExec::new(schema)))
1713+
// A zero (absent) partition count comes from a plan encoded before the
1714+
// field existed, which always meant a single partition.
1715+
let partitions = empty.partitions.max(1) as usize;
1716+
Ok(Arc::new(EmptyExec::new(schema).with_partitions(partitions)))
17141717
}
17151718

17161719
fn try_into_placeholder_row_physical_plan(
@@ -1719,7 +1722,12 @@ pub trait PhysicalPlanNodeExt: Sized {
17191722
_ctx: &PhysicalPlanDecodeContext<'_>,
17201723
) -> Result<Arc<dyn ExecutionPlan>> {
17211724
let schema = Arc::new(convert_required!(placeholder.schema)?);
1722-
Ok(Arc::new(PlaceholderRowExec::new(schema)))
1725+
// A zero (absent) partition count comes from a plan encoded before the
1726+
// field existed, which always meant a single partition.
1727+
let partitions = placeholder.partitions.max(1) as usize;
1728+
Ok(Arc::new(
1729+
PlaceholderRowExec::new(schema).with_partitions(partitions),
1730+
))
17231731
}
17241732

17251733
fn try_into_sort_physical_plan(
@@ -3015,6 +3023,8 @@ pub trait PhysicalPlanNodeExt: Sized {
30153023
Ok(protobuf::PhysicalPlanNode {
30163024
physical_plan_type: Some(PhysicalPlanType::Empty(protobuf::EmptyExecNode {
30173025
schema: Some(schema),
3026+
partitions: empty.properties().output_partitioning().partition_count()
3027+
as u32,
30183028
})),
30193029
})
30203030
}
@@ -3028,6 +3038,8 @@ pub trait PhysicalPlanNodeExt: Sized {
30283038
physical_plan_type: Some(PhysicalPlanType::PlaceholderRow(
30293039
protobuf::PlaceholderRowExecNode {
30303040
schema: Some(schema),
3041+
partitions: empty.properties().output_partitioning().partition_count()
3042+
as u32,
30313043
},
30323044
)),
30333045
})

‎datafusion/proto/tests/cases/roundtrip_physical_plan.rs‎

Lines changed: 55 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -88,9 +88,9 @@ use datafusion::physical_plan::windows::{
8888
create_udwf_window_expr,
8989
};
9090
use datafusion::physical_plan::{
91-
DisplayAs, DisplayFormatType, ExecutionPlan, InputOrderMode, Partitioning,
92-
PhysicalExpr, PlanProperties, RangePartitioning, SendableRecordBatchStream,
93-
SplitPoint, Statistics, displayable,
91+
DisplayAs, DisplayFormatType, ExecutionPlan, ExecutionPlanProperties, InputOrderMode,
92+
Partitioning, PhysicalExpr, PlanProperties, RangePartitioning,
93+
SendableRecordBatchStream, SplitPoint, Statistics, displayable,
9494
};
9595
use datafusion::prelude::{ParquetReadOptions, SessionContext};
9696
use datafusion::scalar::ScalarValue;
@@ -235,6 +235,58 @@ fn roundtrip_empty() -> Result<()> {
235235
roundtrip_test(Arc::new(EmptyExec::new(Arc::new(Schema::empty()))))
236236
}
237237

238+
#[test]
239+
fn roundtrip_empty_with_partitions() -> Result<()> {
240+
let ctx = SessionContext::new();
241+
let codec = DefaultPhysicalExtensionCodec {};
242+
let proto_converter = DefaultPhysicalProtoConverter {};
243+
let plan = Arc::new(EmptyExec::new(Arc::new(Schema::empty())).with_partitions(4));
244+
let plan = roundtrip_test_and_return(plan, &ctx, &codec, &proto_converter)?;
245+
assert_eq!(plan.output_partitioning().partition_count(), 4);
246+
Ok(())
247+
}
248+
249+
#[test]
250+
fn roundtrip_placeholder_row_with_partitions() -> Result<()> {
251+
let ctx = SessionContext::new();
252+
let codec = DefaultPhysicalExtensionCodec {};
253+
let proto_converter = DefaultPhysicalProtoConverter {};
254+
let plan =
255+
Arc::new(PlaceholderRowExec::new(Arc::new(Schema::empty())).with_partitions(4));
256+
let plan = roundtrip_test_and_return(plan, &ctx, &codec, &proto_converter)?;
257+
assert_eq!(plan.output_partitioning().partition_count(), 4);
258+
Ok(())
259+
}
260+
261+
/// Plans encoded before `partitions` was added carry no value for it, which
262+
/// decodes as zero and must be treated as the previous default of one.
263+
#[test]
264+
fn decode_empty_and_placeholder_row_without_partitions() -> Result<()> {
265+
let ctx = SessionContext::new();
266+
let codec = DefaultPhysicalExtensionCodec {};
267+
let schema: protobuf::Schema = (&Schema::empty()).try_into()?;
268+
269+
for physical_plan_type in [
270+
protobuf::physical_plan_node::PhysicalPlanType::Empty(protobuf::EmptyExecNode {
271+
schema: Some(schema.clone()),
272+
partitions: 0,
273+
}),
274+
protobuf::physical_plan_node::PhysicalPlanType::PlaceholderRow(
275+
protobuf::PlaceholderRowExecNode {
276+
schema: Some(schema.clone()),
277+
partitions: 0,
278+
},
279+
),
280+
] {
281+
let node = protobuf::PhysicalPlanNode {
282+
physical_plan_type: Some(physical_plan_type),
283+
};
284+
let plan = node.try_into_physical_plan(ctx.task_ctx().as_ref(), &codec)?;
285+
assert_eq!(plan.output_partitioning().partition_count(), 1);
286+
}
287+
Ok(())
288+
}
289+
238290
#[derive(Debug)]
239291
struct DowncastDelegatingExec {
240292
inner: Arc<dyn ExecutionPlan>,

‎docs/source/library-user-guide/upgrading/55.0.0.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -598,3 +598,36 @@ metadata and schema fingerprint match.
598598
- Pass the current schema fingerprint to `CachedFileMetadata::is_valid_for`.
599599

600600
See [PR #23201](https://github.com/apache/datafusion/pull/23201) for details.
601+
602+
### `EmptyExecNode` and `PlaceholderRowExecNode` gained a `partitions` field
603+
604+
The generated protobuf structs `EmptyExecNode` and `PlaceholderRowExecNode`
605+
encoded only a schema, so the partition count set by `EmptyExec::with_partitions`
606+
was silently dropped when a physical plan was serialized and deserialized: a plan
607+
that reported `n` partitions before encoding reported `1` after. Both messages now
608+
carry a `partitions` field that round-trips the count.
609+
610+
**Who is affected:**
611+
612+
- Users constructing `EmptyExecNode` or `PlaceholderRowExecNode` with an
613+
exhaustive struct literal.
614+
615+
**Migration guide:**
616+
617+
Set the new field, or fill it from `Default`:
618+
619+
```rust,ignore
620+
// Before
621+
EmptyExecNode { schema: Some(schema) }
622+
623+
// After
624+
EmptyExecNode { schema: Some(schema), partitions: 4 }
625+
// or
626+
EmptyExecNode { schema: Some(schema), ..Default::default() }
627+
```
628+
629+
The wire format stays compatible in both directions. Plans encoded before this
630+
field existed decode as a single partition, the previous default, and plans
631+
encoded after it add a field that older readers ignore.
632+
633+
See [PR #23643](https://github.com/apache/datafusion/pull/23643) for details.

0 commit comments

Comments
 (0)