Repository navigation
REGRESSION: UNION: DataFrame::schema() reports the first input's type, but execution returns the coerced type #27023
Description
Describe the bug
For a UNION ALL whose inputs have different column types, DataFrame::schema() (the plan as built by the SQL planner) reports the first input's type, but the batches the query produces have the type chosen later by the TypeCoercion analyzer. A caller that checks results against the declared schema sees a type change between planning and execution.
#14380 describes the root cause (the Union node takes the left input's schema and is only coerced later). This issue is about a user-visible effect, made more common by a 54.0 change: union coercion now uses type_union_coercion, which picks the finer timestamp unit. 53.1 kept the left unit in the case below, so declared and executed types used to agree.
To Reproduce
use datafusion::prelude::*;
#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
let ctx = SessionContext::new();
for sql in [
// Timestamp(us, UTC) UNION ALL CAST(NULL AS TIMESTAMP), i.e. Timestamp(ns, None)
"SELECT t FROM (SELECT arrow_cast(arrow_cast(1, 'Timestamp(Microsecond, None)'), \
'Timestamp(Microsecond, Some(\"UTC\"))') AS t UNION ALL SELECT CAST(NULL AS TIMESTAMP) AS t) u",
// Utf8 UNION ALL LargeUtf8
"SELECT s FROM (SELECT arrow_cast('a', 'Utf8') AS s \
UNION ALL SELECT arrow_cast('b', 'LargeUtf8') AS s) u",
] {
let df = ctx.sql(sql).await?;
let declared = df.schema().field(0).data_type().clone();
let executed = df.collect().await?[0].schema().field(0).data_type().clone();
println!("declared={declared:?} executed={executed:?}");
}
Ok(())
}Output on 55.2.0:
declared=Timestamp(Microsecond, Some("UTC")) executed=Timestamp(Nanosecond, Some("UTC"))
declared=Utf8 executed=LargeUtf8
Output on 53.1.0:
declared=Timestamp(Microsecond, Some("UTC")) executed=Timestamp(Microsecond, Some("UTC"))
declared=Utf8 executed=LargeUtf8
(Utf8 UNION ALL Utf8View behaves the same way: declared Utf8, executed Utf8View, on both versions.)
Expected behavior
DataFrame::schema() should describe the batches the query returns: either the Union is built with the coerced schema (#14380), or the declared schema reflects the analyzer's coercion.
Additional context
Found while upgrading an embedded engine from 53.1 to 55.2. Its export path checks every executed batch against the declared schema, and this mismatch refused queries that had passed on 53.1. The workaround is to cast every union branch to one explicit type.
Activity
alamb commentedon Nov 3, 2026
Thanks @mikewolfd -- I added this to the list of things to fix for 56:
56.0.0(Oct/Nov 2026) #24461[-]UNION: DataFrame::schema() reports the first input's type, but execution returns the coerced type[/-][+]REGRESSION: UNION: DataFrame::schema() reports the first input's type, but execution returns the coerced type[/+]alamb commentedon Nov 3, 2026
I reproduced this and confirmed the timestamp case is a regression, so I have marked this issue as such.
I ran the program from the description (plus a
Utf8/Utf8Viewcase) against datafusion 53.1.0, 54.1.0, 55.2.0 and currentmain(181a87a):Summary:
timeunit_coercionto pick the finer unit. Before that changeTimestamp(us, UTC)UNION ALLTimestamp(ns, None)coerced toTimestamp(us, UTC), which happened to match the first input and so the declared schema. Now it coerces toTimestamp(ns, UTC)and the declared schema no longer matches.Utf8/LargeUtf8andUtf8/Utf8Viewcases: not a regression. They mismatch on every version tested, back to 53.1.0. That is the long-standing root cause tracked in Create UNION plan node with correct schema #14380: theUnionnode is built with the first input's schema and only coerced later by theTypeCoercionanalyzer.So the underlying bug is old, but #22759 widened the set of queries where it is visible, and in particular broke the timestamp query above which passed on 53.1 and 54.1.
fix: infer UNION output types before analysis (apache#27139)
[branch-56] fix: infer UNION output types before analysis (backport #…