Skip to content

Inconsistent and incorrect struct field coercion #14396

Closed

Description

@alamb

Describe the bug

When coercing structs with different types DataFusion is inconsistent in its behavior. Sometimes it errors and in other times it is inconsistent

To Reproduce

Consider two structs with two fields

create table t as values
(
 { 'foo': 'baz' }, 
 { 'xxx': arrow_cast('blarg', 'Utf8View') } -- column4 has List of Struct w/ Utf8View and a different field name
);

They can be coerced to the same field using UNION,

> select column1 from t UNION ALL select column2 from t;
+-------------+
| column1     |
+-------------+
| {c0: baz}   |
| {c0: blarg} |
+-------------+
2 row(s) fetched.
Elapsed 0.018 seconds.

Likewise with CASE they are coerced to the same value

> select CASE WHEN 1=2 THEN column1 ELSE column2 END from t ;
+-----------------------------------------------------------------+
| CASE WHEN Int64(1) = Int64(2) THEN t.column1 ELSE t.column2 END |
+-----------------------------------------------------------------+
| {c0: blarg}                                                     |
+-----------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.010 seconds.

Expected behavior

I believe the field names will be different after

However I expect all of the above queries to fail as the field names are incorrect

I expect the following cases to work:

  1. Field names are the same but in different order
  2. Field names are the same but the value types need to be coerced themselves (see case on Type Coercion fails for List with inner type struct which has large/view types #14154)

Additional context

@jayzhan211 suggests: https://github.com/apache/datafusion/pull/14384/files#r1937492704

Yes, I think type union resolution is the correct on for CASE

Activity

  1. added
    bugSomething isn't working
    on Feb 1, 2025
  2. Lordworms commented on Feb 2, 2025

    @Lordworms
    Contributor

    take

  3. alamb commented on Feb 3, 2025

    @alamb
    ContributorAuthor
  4. matthewmturner commented on Nov 5, 2026

    @matthewmturner
    Contributor

    I will look into this

  5. matthewmturner commented on Nov 8, 2026

    @matthewmturner
    Contributor

    So starting with reproduction from what was reported in the issue.

    With the following DDL.

    create table t as values
    (
     { 'foo': 'baz' }, 
     { 'xxx': arrow_cast('blarg', 'Utf8View') } -- column4 has List of Struct w/ Utf8View and a different field name
    );
    

    The first query now correctly fails:

    > select column1 from t UNION ALL select column2 from t;
    This feature is not implemented: Unsupported CAST from Struct("xxx": Utf8View) to Struct("foo": Utf8View)
    

    But the second query still works:

    select CASE WHEN 1=2 THEN column1 ELSE column2 END from t ;
    +-----------------------------------------------------------------+
    | CASE WHEN Int64(1) = Int64(2) THEN t.column1 ELSE t.column2 END |
    +-----------------------------------------------------------------+
    | {xxx: blarg}                                                    |
    +-----------------------------------------------------------------+
    1 row(s) fetched.
    Elapsed 0.001 seconds.
    
  6. matthewmturner commented on Nov 8, 2026

    @matthewmturner
    Contributor

    The additional expected cases from the description also work

    > create table u as values
    (
     { 'foo': 'baz', 'bar': 'bux' },
     { 'bar': 'b', 'foo': 'bt' }
    );
    0 row(s) fetched.
    Elapsed 0.002 seconds.
    
    > SELECT column1 FROM u UNION ALL SELECT column2 FROM u;
    +----------------------+
    | column1              |
    +----------------------+
    | {foo: baz, bar: bux} |
    | {foo: bt, bar: b}    |
    +----------------------+
    2 row(s) fetched.
    Elapsed 0.034 seconds.
    
    > create table v as values
    (
     { 'foo': 'baz', 'bar': 'bux' },
     { 'bar': 'b', 'foo': arrow_cast('bt', 'Utf8View') }
    );
    0 row(s) fetched.
    Elapsed 0.002 seconds.
    
    > DESCRIBE SELECT column1 FROM v UNION ALL SELECT column2 FROM v;
    +-------------+--------------------------------------+-------------+
    | column_name | data_type                            | is_nullable |
    +-------------+--------------------------------------+-------------+
    | column1     | Struct("foo": Utf8View, "bar": Utf8) | NO          |
    +-------------+--------------------------------------+-------------+
    1 row(s) fetched.
    Elapsed 0.001 seconds.
    
  7. matthewmturner commented on Nov 8, 2026

    @matthewmturner
    Contributor

    There is some separate related prior art here #19955 - where positional fallback coercion was removed for casting and I think thats the issue here as well - so im thinking to remove that fallback.

  8. added a commit that references this issue on Nov 10, 2026
    c8265b6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions