Repository navigation
Commit 4c6098d
fix: UnionExec now conforms each batch to the union's declared schema (apache#23861)
## Which issue does this PR close?
- Closes apache#23862.
- Related to apache#15394.
## Rationale for this change
`UNION ALL` between an input whose column is `NOT NULL` and an input
where the same column is nullable produces a valid, correctly-typed
logical plan — the analyzer already OR's nullability across legs in
`coerce_union_schema`
(`datafusion/optimizer/src/analyzer/type_coercion.rs`), so the union's
*declared* schema correctly reports the field as nullable.
The bug is at execution time: `UnionExec::execute()` hands out each
child's `RecordBatch`es completely unchanged. A leg whose column was
already `NOT NULL` (and therefore needed no `CAST` from the analyzer)
keeps emitting batches with a `NOT NULL` field, contradicting the
union's own declared (nullable) schema. DataFusion's own execution
tolerates this silently, but any consumer that checks schema equality
across batches from the same stream — most notably
`pyarrow.Table.from_batches` via the Arrow C Stream FFI used by the
`datafusion` Python bindings — rejects the stream with `ArrowInvalid:
Schema at index N was different`, even though every individual `SELECT`
runs fine on its own.
### Minimal reproducible example (Python)
```python
import pyarrow as pa
from datafusion import SessionContext
ctx = SessionContext()
ctx.register_record_batch(
"table_a",
pa.record_batch(
{"id": [1, 2], "status": ["ok", "ok"]},
schema=pa.schema([("id", pa.int64()), ("status", pa.string())]), # NOT NULL
),
)
ctx.register_record_batch(
"table_b",
pa.record_batch(
{"id": [3, 4], "status": ["done", None]},
schema=pa.schema([("id", pa.int64()), ("status", pa.string(), True)]), # nullable
),
)
df = ctx.sql("SELECT id, status FROM table_a UNION ALL SELECT id, status FROM table_b")
print(df.schema()) # status: string, nullable -- correct
df.to_pandas() # raises pyarrow.lib.ArrowInvalid: Schema at index 1 was different
```
The same root cause is why `apache#16627` had to make the sqllogictest
`convert_batches` helper tolerant of this exact mismatch instead of
failing, and why `apache#15603` (stale, closed for inactivity) attempted a
similar fix at the physical-execution layer but didn't land.
## What changes are included in this PR?
- `datafusion/physical-plan/src/union.rs`: `UnionExec::execute()` now
compares each child stream's schema against `UnionExec`'s own declared
schema, and if they disagree, wraps the child stream in a small new
`SchemaConformingStream` that re-stamps every batch with the union's
schema before yielding it. This is always safe: the union's schema can
only be *more* permissive than any single input's (nullability is
combined with logical OR, never narrowed — see the existing
`coerce_union_schema` docs), and only the `Field::nullable` metadata
changes; the underlying array data and data type are untouched.
- `datafusion/core/tests/sql/union_nullable.rs` (new): regression tests
covering same-type nullable/non-nullable mismatches in both leg orders,
the "both legs NOT NULL" case (schema should stay `NOT NULL`), and a
case where one leg also needs a real `CAST` (`Int32` -> `Int64`) in
addition to the nullability fix.
`InterleaveExec` (used for sorted unions) may have an analogous issue,
but I kept this PR scoped to plain `UnionExec`, which is what's reported
in apache#23862 / apache#15394 and reproduces the Python-binding failure above.
## Are these changes tested?
Yes — added `datafusion/core/tests/sql/union_nullable.rs` with 4 new
tests. I verified each one fails with a clear schema-mismatch assertion
on `main` (i.e. before this fix) and passes with it applied. Also ran
the full `datafusion-physical-plan` and `datafusion-optimizer` unit
suites, `union.slt`/`union_by_name.slt` sqllogictests, and `cargo
fmt`/`clippy` (`--no-deps`, since an unrelated pre-existing dead-code
lint in `datafusion-physical-expr` fails `-D warnings` on `main` even
without this change).
## Are there any user-facing changes?
`UNION ALL` results now consistently report the analyzer's declared
nullability on every batch, regardless of which leg produced it. No
public API changes.1 parent c850f19 commit 4c6098d
4 files changed
Lines changed: 342 additions & 6 deletions
File tree
- datafusion
- core/tests
- memory_limit
- sql
- physical-plan/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
103 | 103 | | |
104 | 104 | | |
105 | 105 | | |
106 | | - | |
107 | | - | |
108 | | - | |
109 | | - | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
110 | 115 | | |
111 | 116 | | |
112 | 117 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
71 | 71 | | |
72 | 72 | | |
73 | 73 | | |
| 74 | + | |
74 | 75 | | |
75 | 76 | | |
76 | 77 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
0 commit comments