You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Inconsistent and incorrect struct field coercion #14396
When coercing structs with different types DataFusion is inconsistent in its behavior. Sometimes it errors and in other times it is inconsistent
To Reproduce
Consider two structs with two fields
createtabletasvalues
(
{ 'foo': 'baz' },
{ 'xxx': arrow_cast('blarg', 'Utf8View') } -- column4 has List of Struct w/ Utf8View and a different field name
);
They can be coerced to the same field using UNION,
> select column1 from t UNION ALL select column2 from t;
+-------------+
| column1 |
+-------------+
| {c0: baz} |
| {c0: blarg} |
+-------------+
2 row(s) fetched.
Elapsed 0.018 seconds.
Likewise with CASE they are coerced to the same value
>select CASE WHEN 1=2 THEN column1 ELSE column2 END from t ;
+-----------------------------------------------------------------+
| CASE WHEN Int64(1) = Int64(2) THEN t.column1 ELSE t.column2 END |
+-----------------------------------------------------------------+
| {c0: blarg} |
+-----------------------------------------------------------------+1 row(s) fetched.
Elapsed 0.010 seconds.
So starting with reproduction from what was reported in the issue.
With the following DDL.
create table t as values
(
{ 'foo': 'baz' },
{ 'xxx': arrow_cast('blarg', 'Utf8View') } -- column4 has List of Struct w/ Utf8View and a different field name
);
The first query now correctly fails:
> select column1 from t UNION ALL select column2 from t;
This feature is not implemented: Unsupported CAST from Struct("xxx": Utf8View) to Struct("foo": Utf8View)
But the second query still works:
select CASE WHEN 1=2 THEN column1 ELSE column2 END from t ;
+-----------------------------------------------------------------+
| CASE WHEN Int64(1) = Int64(2) THEN t.column1 ELSE t.column2 END |
+-----------------------------------------------------------------+
| {xxx: blarg} |
+-----------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.001 seconds.
There is some separate related prior art here #19955 - where positional fallback coercion was removed for casting and I think thats the issue here as well - so im thinking to remove that fallback.
Activity
Update tests per apache#14396
Do not rename struct fields when coercing types in `CASE` (#14384)
Lordworms commentedon Feb 2, 2025
take
Do not rename struct fields when coercing types in `CASE` (apache#14384)
alamb commentedon Feb 3, 2025
@Omega359 notes an additional test that would be good:
fix: Support struct casting with field reordering and count changes
fix: Add name-based struct coercion for CASE expressions and struct t…
fix: Support struct casting with field reordering and count changes
fix: Add name-based struct coercion for CASE expressions and struct t…
matthewmturner commentedon Nov 5, 2026
I will look into this
matthewmturner commentedon Nov 8, 2026
So starting with reproduction from what was reported in the issue.
With the following DDL.
The first query now correctly fails:
But the second query still works:
matthewmturner commentedon Nov 8, 2026
The additional expected cases from the description also work
matthewmturner commentedon Nov 8, 2026
There is some separate related prior art here #19955 - where positional fallback coercion was removed for casting and I think thats the issue here as well - so im thinking to remove that fallback.
Improve struct type coercion (apache#27251)