Skip to content

Internal error: CAST over an expression simplified to a metadata-carrying column, under UNION ALL + aggregate (physical vs logical field metadata) #26178

Open

Description

@TRO-Wolf

Describe the bug

A CAST over an expression that the optimizer simplifies to a bare column carrying field metadata raises an internal error when it sits in the branches of a UNION ALL under an aggregate:

Internal error: Physical input schema should be the same as the one converted from logical input schema. Differences:
	- field metadata at index 0 [x]: (physical) {"PARQUET:field_id": "1"} vs (logical) {}.

Reproduced on DataFusion 54.1.0 (through datafusion-python 54.1.0, Linux x86_64). Not checked against main.

To Reproduce

import datafusion, pyarrow as pa

schema = pa.schema([
    pa.field("b", pa.int64(), nullable=False, metadata={"PARQUET:field_id": "1"}),
    pa.field("n", pa.int64(), metadata={"PARQUET:field_id": "2"}),
])
t = pa.table({"b": [1, 2, 3], "n": [1, None, 3]}, schema=schema)

ctx = datafusion.SessionContext()
ctx.register_record_batches("t", [t.to_batches()])

ctx.sql("""
    SELECT max(x) AS m, count(x) AS c FROM
      (SELECT CAST(coalesce(b, 7) AS BIGINT) AS x FROM t
       UNION ALL
       SELECT CAST(coalesce(b, 7) AS BIGINT) AS x FROM t) q
""").collect()

b is NOT NULL, so coalesce(b, 7) simplifies to b.

Controls on the same build, all of which answer:

Query Result
the same query without the UNION ALL m = 3, c = 3
UNION ALL over CAST(b AS BIGINT) (no coalesce) m = 3, c = 6
the failing query against the same data registered without field metadata m = 3, c = 6

Expected behavior

m = 3, c = 6.

Additional context

Reading the code, the logical side (cast_output_field) and the physical side (CastExpr's resolved target field) both propagate the child's field for a default-target cast, so they agree when both are derived from the same plan state. Here they are not: the UNION's schema was computed before the coalesce was simplified away (child = a function result, no metadata), while the physical plan is built from the simplified branches (child = the column, with its metadata).

Two possible directions, offered as a suggestion only:

  1. Recompute the union schema after expression simplification. This fixes the UNION shape.
  2. Treat a cast's output as a new value that carries none of its child's field metadata, on both the logical and the physical side. This would fix every boundary whose schema is fixed before simplification, and it also removes a visible oddity: today SELECT CAST(col AS <type>) returns a field that still carries the source column's PARQUET:field_id.

Possibly related: #19049 (closed; UNION ALL plus optimize_projections with Parquet field metadata), #25978 and #25369 (logical/physical mismatch from a rewrite under an aggregate).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions