Skip to content

Commit 4af8735

Browse files
committed
fix: correct stale comments and vacuous-batch gaps from review
An Opus-model review of 131796d caught three loose ends: - The SchemaConformingStream/conform_stream_schema doc comment claimed InterleaveExec::try_new re-validates child field data types the same way UnionExec::try_new does via calculate_union. It doesn't -- InterleaveExec::compute_properties builds fresh EquivalenceProperties without that check. Corrected the comment to state the real invariant (InterleaveExec inputs come from an already-validated UnionExec via the optimizer) and note that a genuine mismatch would now surface as an explicit error rather than a silently corrupt batch. - test_interleave_conforms_batch_schema and union_all_widening_cast_also_fixes_nullable iterated collected batches without asserting the batch list was non-empty, so both could pass vacuously if no rows came through. - union_nullable_spill.rs's regression test doc comment asserted UnionExec returns child streams "without schema coercion" -- no longer true after this fix. Updated it to explain the test now guards the SpillManager-level fix (#21292) specifically, since UnionExec itself no longer produces mismatched-nullability batches.
1 parent 131796d commit 4af8735

3 files changed

Lines changed: 33 additions & 22 deletions

File tree

‎datafusion/core/tests/memory_limit/union_nullable_spill.rs‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -103,10 +103,15 @@ fn build_task_ctx(pool_size: usize) -> Arc<datafusion_execution::TaskContext> {
103103
/// have mismatched nullability (one child's `val` is non-nullable, the other's
104104
/// is nullable with NULLs). A tiny FairSpillPool forces all batches to spill.
105105
///
106-
/// UnionExec returns child streams without schema coercion, so batches from
107-
/// different children carry different per-field nullability into the shared
108-
/// SpillPool. The IPC writer must use the SpillManager's canonical (nullable)
109-
/// schema — not the first batch's schema — so readback batches are valid.
106+
/// `UnionExec` now re-stamps every child batch with its own declared (nullable)
107+
/// schema before they reach `RepartitionExec` (see
108+
/// <https://github.com/apache/datafusion/issues/15394>), so this no longer
109+
/// exercises mismatched-nullability batches arriving at the SpillManager via
110+
/// `UnionExec` specifically. It's kept as a regression test for the
111+
/// SpillManager fix itself: the IPC writer must use the SpillManager's
112+
/// canonical schema -- not the first batch's schema -- so readback batches
113+
/// stay valid for any caller that does hand it batches with differing
114+
/// nullability. See <https://github.com/apache/datafusion/issues/21292>.
110115
///
111116
/// Otherwise, sort_batch will panic with
112117
/// `Column 'val' is declared as non-nullable but contains null values`

‎datafusion/core/tests/sql/union_nullable.rs‎

Lines changed: 7 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ async fn union_all_mismatched_nullable(
8080
/// alone is not enough here -- it reflects the raw, pre-`Analyzer` plan (see
8181
/// `SessionState::create_logical_plan`), which for a `UNION` still has the
8282
/// first leg's un-coerced type.
83-
async fn analyzed_schema(df: &DataFrame) -> Result<Schema> {
83+
fn analyzed_schema(df: &DataFrame) -> Result<Schema> {
8484
Ok(df
8585
.clone()
8686
.into_optimized_plan()?
@@ -93,7 +93,7 @@ async fn analyzed_schema(df: &DataFrame) -> Result<Schema> {
9393
/// query's analyzed output schema field-for-field -- including
9494
/// nullability -- no matter which leg it came from.
9595
async fn assert_every_batch_matches_declared_schema(df: DataFrame) -> Result<()> {
96-
let declared_schema = analyzed_schema(&df).await?;
96+
let declared_schema = analyzed_schema(&df)?;
9797

9898
let batches = df.collect().await?;
9999
assert!(!batches.is_empty());
@@ -115,8 +115,7 @@ async fn assert_every_batch_matches_declared_schema(df: DataFrame) -> Result<()>
115115
async fn union_all_same_type_left_not_null_right_nullable() -> Result<()> {
116116
let df = union_all_mismatched_nullable(false, true).await?;
117117
assert!(
118-
analyzed_schema(&df)
119-
.await?
118+
analyzed_schema(&df)?
120119
.field_with_name("status")?
121120
.is_nullable()
122121
);
@@ -127,8 +126,7 @@ async fn union_all_same_type_left_not_null_right_nullable() -> Result<()> {
127126
async fn union_all_same_type_left_nullable_right_not_null() -> Result<()> {
128127
let df = union_all_mismatched_nullable(true, false).await?;
129128
assert!(
130-
analyzed_schema(&df)
131-
.await?
129+
analyzed_schema(&df)?
132130
.field_with_name("status")?
133131
.is_nullable()
134132
);
@@ -138,7 +136,7 @@ async fn union_all_same_type_left_nullable_right_not_null() -> Result<()> {
138136
#[tokio::test]
139137
async fn union_all_same_type_both_not_null_stays_not_null() -> Result<()> {
140138
let df = union_all_mismatched_nullable(false, false).await?;
141-
let declared_schema = analyzed_schema(&df).await?;
139+
let declared_schema = analyzed_schema(&df)?;
142140
assert!(
143141
!declared_schema.field_with_name("status")?.is_nullable(),
144142
"status should remain NOT NULL when neither leg is nullable"
@@ -190,14 +188,15 @@ async fn union_all_widening_cast_also_fixes_nullable() -> Result<()> {
190188
)
191189
.await?;
192190

193-
let declared_schema = analyzed_schema(&df).await?;
191+
let declared_schema = analyzed_schema(&df)?;
194192
assert_eq!(
195193
declared_schema.field_with_name("val")?.data_type(),
196194
&DataType::Int64
197195
);
198196
assert!(declared_schema.field_with_name("val")?.is_nullable());
199197

200198
let batches = df.collect().await?;
199+
assert!(!batches.is_empty());
201200
for batch in &batches {
202201
assert_eq!(batch.schema().as_ref(), &declared_schema);
203202
}

‎datafusion/physical-plan/src/union.rs‎

Lines changed: 17 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -67,15 +67,20 @@ use tokio::macros::support::thread_rng_n;
6767
/// Wraps a child stream so that every batch it yields is re-stamped with
6868
/// `schema` instead of the child's own schema.
6969
///
70-
/// This is used by [`UnionExec`] when a child's output schema disagrees
71-
/// with the union's declared output schema -- in practice this only
72-
/// happens for nullability (the union's schema is nullable wherever *any*
73-
/// input's field is, but casts are only inserted between inputs when the
74-
/// *type* differs, not when only nullability does). Re-stamping is safe:
75-
/// the union's schema can only be *more* permissive than any single
76-
/// input's (nullability is combined with logical OR, never narrowed), and
77-
/// the underlying data type is unaffected -- only the `Field::nullable`
78-
/// flag (and equivalent metadata) changes.
70+
/// This is used by both [`UnionExec`] and [`InterleaveExec`] when a child's
71+
/// output schema disagrees with the operator's declared output schema --
72+
/// in practice this only happens for nullability (the declared schema is
73+
/// nullable wherever *any* input's field is, but casts are only inserted
74+
/// between inputs when the *type* differs, not when only nullability
75+
/// does). For [`UnionExec`], [`UnionExec::try_new`] guarantees this: it
76+
/// calls `calculate_union`, which rejects any input whose field data types
77+
/// don't match the computed union schema. [`InterleaveExec::try_new`] does
78+
/// not repeat that check -- its inputs are only ever produced by the
79+
/// optimizer rewriting an already-validated `UnionExec`, whose children's
80+
/// types are therefore already known to agree -- but if this wrapper ever
81+
/// did see a genuine data type mismatch (e.g. from a hand-built
82+
/// `InterleaveExec`), `RecordBatch::try_new_with_options` below reports it
83+
/// as an error rather than silently yielding a corrupt batch.
7984
struct SchemaConformingStream {
8085
schema: SchemaRef,
8186
inner: SendableRecordBatchStream,
@@ -1094,7 +1099,9 @@ mod tests {
10941099
let interleave_schema = interleave.schema();
10951100
assert!(interleave_schema.field(0).is_nullable());
10961101

1097-
for batch in collect(interleave, task_ctx).await? {
1102+
let batches = collect(interleave, task_ctx).await?;
1103+
assert!(!batches.is_empty());
1104+
for batch in &batches {
10981105
assert_eq!(batch.schema(), interleave_schema);
10991106
}
11001107

0 commit comments

Comments
 (0)