Skip to content

Commit 4c6098d

Browse files
dariocurrkosiew
authored andcommitted
fix: UnionExec now conforms each batch to the union's declared schema (apache#23861)
## Which issue does this PR close? - Closes apache#23862. - Related to apache#15394. ## Rationale for this change `UNION ALL` between an input whose column is `NOT NULL` and an input where the same column is nullable produces a valid, correctly-typed logical plan — the analyzer already OR's nullability across legs in `coerce_union_schema` (`datafusion/optimizer/src/analyzer/type_coercion.rs`), so the union's *declared* schema correctly reports the field as nullable. The bug is at execution time: `UnionExec::execute()` hands out each child's `RecordBatch`es completely unchanged. A leg whose column was already `NOT NULL` (and therefore needed no `CAST` from the analyzer) keeps emitting batches with a `NOT NULL` field, contradicting the union's own declared (nullable) schema. DataFusion's own execution tolerates this silently, but any consumer that checks schema equality across batches from the same stream — most notably `pyarrow.Table.from_batches` via the Arrow C Stream FFI used by the `datafusion` Python bindings — rejects the stream with `ArrowInvalid: Schema at index N was different`, even though every individual `SELECT` runs fine on its own. ### Minimal reproducible example (Python) ```python import pyarrow as pa from datafusion import SessionContext ctx = SessionContext() ctx.register_record_batch( "table_a", pa.record_batch( {"id": [1, 2], "status": ["ok", "ok"]}, schema=pa.schema([("id", pa.int64()), ("status", pa.string())]), # NOT NULL ), ) ctx.register_record_batch( "table_b", pa.record_batch( {"id": [3, 4], "status": ["done", None]}, schema=pa.schema([("id", pa.int64()), ("status", pa.string(), True)]), # nullable ), ) df = ctx.sql("SELECT id, status FROM table_a UNION ALL SELECT id, status FROM table_b") print(df.schema()) # status: string, nullable -- correct df.to_pandas() # raises pyarrow.lib.ArrowInvalid: Schema at index 1 was different ``` The same root cause is why `apache#16627` had to make the sqllogictest `convert_batches` helper tolerant of this exact mismatch instead of failing, and why `apache#15603` (stale, closed for inactivity) attempted a similar fix at the physical-execution layer but didn't land. ## What changes are included in this PR? - `datafusion/physical-plan/src/union.rs`: `UnionExec::execute()` now compares each child stream's schema against `UnionExec`'s own declared schema, and if they disagree, wraps the child stream in a small new `SchemaConformingStream` that re-stamps every batch with the union's schema before yielding it. This is always safe: the union's schema can only be *more* permissive than any single input's (nullability is combined with logical OR, never narrowed — see the existing `coerce_union_schema` docs), and only the `Field::nullable` metadata changes; the underlying array data and data type are untouched. - `datafusion/core/tests/sql/union_nullable.rs` (new): regression tests covering same-type nullable/non-nullable mismatches in both leg orders, the "both legs NOT NULL" case (schema should stay `NOT NULL`), and a case where one leg also needs a real `CAST` (`Int32` -> `Int64`) in addition to the nullability fix. `InterleaveExec` (used for sorted unions) may have an analogous issue, but I kept this PR scoped to plain `UnionExec`, which is what's reported in apache#23862 / apache#15394 and reproduces the Python-binding failure above. ## Are these changes tested? Yes — added `datafusion/core/tests/sql/union_nullable.rs` with 4 new tests. I verified each one fails with a clear schema-mismatch assertion on `main` (i.e. before this fix) and passes with it applied. Also ran the full `datafusion-physical-plan` and `datafusion-optimizer` unit suites, `union.slt`/`union_by_name.slt` sqllogictests, and `cargo fmt`/`clippy` (`--no-deps`, since an unrelated pre-existing dead-code lint in `datafusion-physical-expr` fails `-D warnings` on `main` even without this change). ## Are there any user-facing changes? `UNION ALL` results now consistently report the analyzer's declared nullability on every batch, regardless of which leg produced it. No public API changes.
1 parent c850f19 commit 4c6098d

4 files changed

Lines changed: 342 additions & 6 deletions

File tree

‎datafusion/core/tests/memory_limit/union_nullable_spill.rs‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -103,10 +103,15 @@ fn build_task_ctx(pool_size: usize) -> Arc<datafusion_execution::TaskContext> {
103103
/// have mismatched nullability (one child's `val` is non-nullable, the other's
104104
/// is nullable with NULLs). A tiny FairSpillPool forces all batches to spill.
105105
///
106-
/// UnionExec returns child streams without schema coercion, so batches from
107-
/// different children carry different per-field nullability into the shared
108-
/// SpillPool. The IPC writer must use the SpillManager's canonical (nullable)
109-
/// schema — not the first batch's schema — so readback batches are valid.
106+
/// `UnionExec` now re-stamps every child batch with its own declared (nullable)
107+
/// schema before they reach `RepartitionExec` (see
108+
/// <https://github.com/apache/datafusion/issues/15394>), so this no longer
109+
/// exercises mismatched-nullability batches arriving at the SpillManager via
110+
/// `UnionExec` specifically. It's kept as a regression test for the
111+
/// SpillManager fix itself: the IPC writer must use the SpillManager's
112+
/// canonical schema -- not the first batch's schema -- so readback batches
113+
/// stay valid for any caller that does hand it batches with differing
114+
/// nullability. See <https://github.com/apache/datafusion/issues/21292>.
110115
///
111116
/// Otherwise, sort_batch will panic with
112117
/// `Column 'val' is declared as non-nullable but contains null values`

‎datafusion/core/tests/sql/mod.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,7 @@ mod runtime_config;
7171
pub mod select;
7272
mod sql_api;
7373
mod union_comparison;
74+
mod union_nullable;
7475
mod unparser;
7576

7677
async fn register_aggregate_csv_by_sql(ctx: &SessionContext) {
Lines changed: 204 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,204 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
//! Regression tests asserting that every batch yielded by a `UNION ALL`
19+
//! reports the union's own declared schema, even when the same column is
20+
//! `NOT NULL` on one leg and nullable on another. See
21+
//! <https://github.com/apache/datafusion/issues/15394>.
22+
23+
use std::sync::Arc;
24+
25+
use arrow::array::{Int64Array, RecordBatch, StringArray};
26+
use arrow::datatypes::{DataType, Field, Schema};
27+
use datafusion::prelude::*;
28+
use datafusion_common::Result;
29+
30+
/// Builds two single-partition tables that agree on `id`/`status` types but
31+
/// disagree on whether `status` is nullable, then runs `UNION ALL` over them.
32+
async fn union_all_mismatched_nullable(
33+
left_nullable: bool,
34+
right_nullable: bool,
35+
) -> Result<DataFrame> {
36+
let ctx = SessionContext::new();
37+
38+
let schema_a = Arc::new(Schema::new(vec![
39+
Field::new("id", DataType::Int64, false),
40+
Field::new("status", DataType::Utf8, left_nullable),
41+
]));
42+
let batch_a = RecordBatch::try_new(
43+
Arc::clone(&schema_a),
44+
vec![
45+
Arc::new(Int64Array::from(vec![1, 2])),
46+
Arc::new(StringArray::from(vec!["ok", "ok"])),
47+
],
48+
)?;
49+
ctx.register_batch("table_a", batch_a)?;
50+
51+
let schema_b = Arc::new(Schema::new(vec![
52+
Field::new("id", DataType::Int64, false),
53+
Field::new("status", DataType::Utf8, right_nullable),
54+
]));
55+
let status_values: Vec<Option<&str>> = if right_nullable {
56+
vec![Some("done"), None]
57+
} else {
58+
vec![Some("done"), Some("also-done")]
59+
};
60+
let batch_b = RecordBatch::try_new(
61+
Arc::clone(&schema_b),
62+
vec![
63+
Arc::new(Int64Array::from(vec![3, 4])),
64+
Arc::new(StringArray::from(status_values)),
65+
],
66+
)?;
67+
ctx.register_batch("table_b", batch_b)?;
68+
69+
ctx.sql(
70+
"SELECT id, status FROM table_a \
71+
UNION ALL \
72+
SELECT id, status FROM table_b",
73+
)
74+
.await
75+
}
76+
77+
/// The schema DataFusion actually commits to for a query: the logical plan
78+
/// after the `Analyzer` (which includes the `UNION` nullability/type
79+
/// coercion this test targets) and `Optimizer` have run. `DataFrame::schema`
80+
/// alone is not enough here -- it reflects the raw, pre-`Analyzer` plan (see
81+
/// `SessionState::create_logical_plan`), which for a `UNION` still has the
82+
/// first leg's un-coerced type.
83+
fn analyzed_schema(df: &DataFrame) -> Result<Schema> {
84+
Ok(df
85+
.clone()
86+
.into_optimized_plan()?
87+
.schema()
88+
.as_arrow()
89+
.clone())
90+
}
91+
92+
/// Every `RecordBatch` actually produced by a `UNION ALL` must match the
93+
/// query's analyzed output schema field-for-field -- including
94+
/// nullability -- no matter which leg it came from.
95+
async fn assert_every_batch_matches_declared_schema(df: DataFrame) -> Result<()> {
96+
let declared_schema = analyzed_schema(&df)?;
97+
98+
let batches = df.collect().await?;
99+
assert!(!batches.is_empty());
100+
for batch in &batches {
101+
assert_eq!(
102+
batch.schema().as_ref(),
103+
&declared_schema,
104+
"a UNION ALL leg produced a RecordBatch whose schema disagrees \
105+
with the union's declared output schema (commonly a dropped \
106+
nullable flag) -- this is what downstream consumers that check \
107+
schema equality across batches (e.g. pyarrow) reject with \
108+
`ArrowInvalid: Schema at index N was different`"
109+
);
110+
}
111+
Ok(())
112+
}
113+
114+
#[tokio::test]
115+
async fn union_all_same_type_left_not_null_right_nullable() -> Result<()> {
116+
let df = union_all_mismatched_nullable(false, true).await?;
117+
assert!(
118+
analyzed_schema(&df)?
119+
.field_with_name("status")?
120+
.is_nullable()
121+
);
122+
assert_every_batch_matches_declared_schema(df).await
123+
}
124+
125+
#[tokio::test]
126+
async fn union_all_same_type_left_nullable_right_not_null() -> Result<()> {
127+
let df = union_all_mismatched_nullable(true, false).await?;
128+
assert!(
129+
analyzed_schema(&df)?
130+
.field_with_name("status")?
131+
.is_nullable()
132+
);
133+
assert_every_batch_matches_declared_schema(df).await
134+
}
135+
136+
#[tokio::test]
137+
async fn union_all_same_type_both_not_null_stays_not_null() -> Result<()> {
138+
let df = union_all_mismatched_nullable(false, false).await?;
139+
let declared_schema = analyzed_schema(&df)?;
140+
assert!(
141+
!declared_schema.field_with_name("status")?.is_nullable(),
142+
"status should remain NOT NULL when neither leg is nullable"
143+
);
144+
assert_every_batch_matches_declared_schema(df).await
145+
}
146+
147+
/// Same bug, but the coercion also has to widen the *type* (Int32 -> Int64)
148+
/// on one leg. The leg that already matched the target type still needed
149+
/// its nullability reconciled at execution time, independent of whichever
150+
/// legs needed a `CAST`.
151+
#[tokio::test]
152+
async fn union_all_widening_cast_also_fixes_nullable() -> Result<()> {
153+
use arrow::array::Int32Array;
154+
155+
let ctx = SessionContext::new();
156+
157+
let schema_a = Arc::new(Schema::new(vec![
158+
Field::new("id", DataType::Int64, false),
159+
Field::new("val", DataType::Int32, false),
160+
]));
161+
let batch_a = RecordBatch::try_new(
162+
Arc::clone(&schema_a),
163+
vec![
164+
Arc::new(Int64Array::from(vec![1, 2])),
165+
Arc::new(Int32Array::from(vec![10, 20])),
166+
],
167+
)?;
168+
ctx.register_batch("table_a", batch_a)?;
169+
170+
let schema_b = Arc::new(Schema::new(vec![
171+
Field::new("id", DataType::Int64, false),
172+
Field::new("val", DataType::Int64, true),
173+
]));
174+
let batch_b = RecordBatch::try_new(
175+
Arc::clone(&schema_b),
176+
vec![
177+
Arc::new(Int64Array::from(vec![3, 4])),
178+
Arc::new(Int64Array::from(vec![Some(30), None])),
179+
],
180+
)?;
181+
ctx.register_batch("table_b", batch_b)?;
182+
183+
let df = ctx
184+
.sql(
185+
"SELECT id, val FROM table_a \
186+
UNION ALL \
187+
SELECT id, val FROM table_b",
188+
)
189+
.await?;
190+
191+
let declared_schema = analyzed_schema(&df)?;
192+
assert_eq!(
193+
declared_schema.field_with_name("val")?.data_type(),
194+
&DataType::Int64
195+
);
196+
assert!(declared_schema.field_with_name("val")?.is_nullable());
197+
198+
let batches = df.collect().await?;
199+
assert!(!batches.is_empty());
200+
for batch in &batches {
201+
assert_eq!(batch.schema().as_ref(), &declared_schema);
202+
}
203+
Ok(())
204+
}

0 commit comments

Comments
 (0)