You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Internal error: CAST over an expression simplified to a metadata-carrying column, under UNION ALL + aggregate (physical vs logical field metadata) #26178
A CAST over an expression that the optimizer simplifies to a bare column carrying field metadata raises an internal error when it sits in the branches of a UNION ALL under an aggregate:
Internal error: Physical input schema should be the same as the one converted from logical input schema. Differences:
- field metadata at index 0 [x]: (physical) {"PARQUET:field_id": "1"} vs (logical) {}.
Reproduced on DataFusion 54.1.0 (through datafusion-python 54.1.0, Linux x86_64). Not checked against main.
To Reproduce
importdatafusion, pyarrowaspaschema=pa.schema([
pa.field("b", pa.int64(), nullable=False, metadata={"PARQUET:field_id": "1"}),
pa.field("n", pa.int64(), metadata={"PARQUET:field_id": "2"}),
])
t=pa.table({"b": [1, 2, 3], "n": [1, None, 3]}, schema=schema)
ctx=datafusion.SessionContext()
ctx.register_record_batches("t", [t.to_batches()])
ctx.sql(""" SELECT max(x) AS m, count(x) AS c FROM (SELECT CAST(coalesce(b, 7) AS BIGINT) AS x FROM t UNION ALL SELECT CAST(coalesce(b, 7) AS BIGINT) AS x FROM t) q""").collect()
b is NOT NULL, so coalesce(b, 7) simplifies to b.
Controls on the same build, all of which answer:
Query
Result
the same query without the UNION ALL
m = 3, c = 3
UNION ALL over CAST(b AS BIGINT) (no coalesce)
m = 3, c = 6
the failing query against the same data registered without field metadata
m = 3, c = 6
Expected behavior
m = 3, c = 6.
Additional context
Reading the code, the logical side (cast_output_field) and the physical side (CastExpr's resolved target field) both propagate the child's field for a default-target cast, so they agree when both are derived from the same plan state. Here they are not: the UNION's schema was computed before the coalesce was simplified away (child = a function result, no metadata), while the physical plan is built from the simplified branches (child = the column, with its metadata).
Two possible directions, offered as a suggestion only:
Recompute the union schema after expression simplification. This fixes the UNION shape.
Treat a cast's output as a new value that carries none of its child's field metadata, on both the logical and the physical side. This would fix every boundary whose schema is fixed before simplification, and it also removes a visible oddity: today SELECT CAST(col AS <type>) returns a field that still carries the source column's PARQUET:field_id.
Possibly related: #19049 (closed; UNION ALL plus optimize_projections with Parquet field metadata), #25978 and #25369 (logical/physical mismatch from a rewrite under an aggregate).
Activity
docs(cast-view-agg-nullability-1): record the upstream issue apache/d…