Each Union branch should produce fields with nullable: true if any of the branches produces fields with nullable: true on the same position

Describe the bug

It was found in #15242 that the nullability of Literal is simply based on its scalar value and does not take its nullability in the input schema into account.

To Reproduce

This issue cannot be seen in the CLI, as it doesn't do any strict type checking of record batches when displaying results. However, it can be observed in SLTs where normalize::convert_batches ensures that each record batch's schema is the same as that of the first batch.

Take the following query for example:

select x, y, z from t3 union all by name select z, y, x, 'd' as zz from t3;

UNION is a case where record batch streams from each input will be merged into an output stream.

zz should be considered as nullable in the output schema because it is not found in the lhs. However, the current behavior is that record batches streamed from the rhs do not have zz as null because of the current implementation of PhysicalExpr::nullable for Literal.

Expected behavior

The nullability of Literal should be determined by its surrounding context and value, not only its value.

Additional context

No response

I've been trying to find where to resolve this issue but my understanding of the core of DF is currently too limited to uncover a solution. I've created a test though that exhibits the issue

#[tokio::test]
async fn union_by_name_literal_is_null_and_not_null() -> Result<()> {
    let str_array_1 = StringArray::from(vec![Some("a1")]);
    let str_array_2 = StringArray::from(vec![Some("a2")]);
    let str_array_3 = StringArray::from(vec![Some("b1")]);

    let batch_1 = RecordBatch::try_from_iter(vec![
        ("a", Arc::new(str_array_1) as ArrayRef),
    ])?;
    let batch_2 = RecordBatch::try_from_iter(vec![
        ("a", Arc::new(str_array_2) as ArrayRef),
        ("b", Arc::new(str_array_3) as ArrayRef),
    ])?;

    let ctx = SessionContext::new();
    ctx.register_batch("union_batch_1", batch_1)?;
    ctx.register_batch("union_batch_2", batch_2)?;

    let df1 = ctx.table("union_batch_1").await?;
    let df2 = ctx.table("union_batch_2").await?;

    let batches = df2.union_by_name(df1)?.collect().await?;
    let schema = batches[0].schema();

    for batch in batches {
        // Verify schema is the same for all batches
        if !schema.contains(&batch.schema()) {
            return Err(DataFusionError::Internal(
                format!(
                    "Schema mismatch. Previously had\n{:#?}\n\nGot:\n{:#?}",
                    &schema,
                    batch.schema()
                ),
            ));
        }
    }

    Ok(())
}

Above is based upon #15489

Nullability of Literal is not determined by surrounding context

I wouldn't say that nullability of a literal is to be determined.
Rather, if anything is tbd, it's UNION branches need to be 'coerced' to have same nullability.

Let me update the title.

fix: UnionExec now conforms each batch to the union's declared schema

fix: UnionExec now conforms each batch to the union's declared schema (…

physical-plan: coerce UNION/INTERLEAVE schema mismatches at plan time

physical-plan: coerce UNION/INTERLEAVE schema mismatches at plan time (…

fix: UnionExec now conforms each batch to the union's declared schema (…

physical-plan: coerce UNION/INTERLEAVE schema mismatches at plan time (…

fix(exec): give project() output exactly the requested schema metadata

Navigation Menu

Repository navigation

Each Union branch should produce fields with `nullable: true` if any of the branches produces fields with `nullable: true` on the same position #15394

Describe the bug

To Reproduce

Expected behavior

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Each Union branch should produce fields with nullable: true if any of the branches produces fields with nullable: true on the same position #15394

Description

Describe the bug

To Reproduce

Expected behavior

Additional context

Activity

Omega359 commented on Mar 29, 2025

findepi commented on Jun 30, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Each Union branch should produce fields with `nullable: true` if any of the branches produces fields with `nullable: true` on the same position #15394