Skip to content

Commit c8265b6

Browse files
Improve struct type coercion (#27251)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #14396 ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. Please explain the problem you are trying to solve in terms of the user-visible behavior, rather than the implementation. For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the implementation. "COUNT(DISTINCT) returns wrong results when the column contains nulls" is the user-visible problem. --> Type coercion for structs could result in incorrect results. Namely, because of a fallback to coercing by position structs with different fields could end up returning results when an error is the expected result. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here, but it is sometimes worth providing a summary of the individual changes in this PR. --> We remove the position fallback which follows a previously established pattern [here](#19955) for struct casting. We also ensure struct field names match before moving to struct coercion. Commit [c4813bf](c4813bf) -> Baseline sqllogictests that show the expected failure and do not pass without fix Commit [c3afa0d](c3afa0d) -> Remove position based fallback and require struct field names to match ## What is the testing strategy for this PR? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code Briefly describe how this PR is tested, and point to the specific tests you added. For example: 'This new feature is covered by the `sqllogictest` cases added in `foo.slt`'. If this PR does not add tests, explain why. For example, if the change is already covered by existing tests, please mention it. You should also check the `codecov` bot reply on this PR to confirm the changed code is exercised. --> A initial commit with tests demonstrating the expected behavior were added and then fixes applied on top of that. The tests were focused on coercion of struct types when using `UNION`, `CASE`, and comparison between struct types. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please add the `api change` label. --> Errors that previously incorrectly succeeded will now fail and this class of error is handled completely during type coercion now so errors will be reflected as such.
1 parent 14b3edd commit c8265b6

4 files changed

Lines changed: 220 additions & 56 deletions

File tree

‎datafusion/expr-common/src/type_coercion/binary.rs‎

Lines changed: 91 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -969,7 +969,8 @@ pub fn try_type_union_resolution_with_struct(
969969
/// [`comparison_coercion`], which prefers numeric types so that ordering and
970970
/// equality follow numeric semantics.
971971
pub fn type_union_coercion(lhs_type: &DataType, rhs_type: &DataType) -> Option<DataType> {
972-
if lhs_type.equals_datatype(rhs_type) {
972+
if lhs_type.equals_datatype(rhs_type) && struct_field_names_match(lhs_type, rhs_type)
973+
{
973974
return Some(lhs_type.clone());
974975
}
975976
binary_numeric_coercion(lhs_type, rhs_type)
@@ -1018,7 +1019,8 @@ fn comparison_coercion_inner<F>(
10181019
where
10191020
F: Fn(&DataType, &DataType) -> Option<DataType> + Copy,
10201021
{
1021-
if lhs_type.equals_datatype(rhs_type) {
1022+
if lhs_type.equals_datatype(rhs_type) && struct_field_names_match(lhs_type, rhs_type)
1023+
{
10221024
// same type => equality is possible
10231025
return Some(lhs_type.clone());
10241026
}
@@ -1433,29 +1435,73 @@ where
14331435
return None;
14341436
}
14351437

1436-
// If the two structs have exactly the same set of field names (possibly in
1437-
// different order), prefer name-based coercion. Otherwise fall back to
1438-
// positional coercion which preserves backward compatibility.
1439-
//
1440-
// Name-based coercion is used in:
1441-
// 1. Array construction: [s1, s2] where s1 and s2 have reordered fields
1442-
// 2. UNION operations: different field orders unified by name
1443-
// 3. VALUES clauses: heterogeneous struct rows unified by field name
1444-
// 4. JOIN conditions: structs with matching field names
1445-
// 5. Window functions: partitions/orders by struct fields
1446-
// 6. Aggregate functions: collecting structs with reordered fields
1447-
//
1448-
// See docs/source/user-guide/sql/struct_coercion.md for detailed examples.
1438+
// Structs coerce only when they have the same set of field names
1439+
// (any order). Field names are part of the struct type, and the
1440+
// cast layer matches fields by name, so no other common type is
1441+
// executable. See docs/source/user-guide/sql/struct_coercion.md.
14491442
if fields_have_same_names(lhs_fields, rhs_fields) {
14501443
return coerce_struct_by_name(lhs_fields, rhs_fields, coerce_fn);
14511444
}
14521445

1453-
coerce_struct_by_position(lhs_fields, rhs_fields, coerce_fn)
1446+
None
14541447
}
14551448
_ => None,
14561449
}
14571450
}
14581451

1452+
/// Return true if the field names of every nested struct agree.
1453+
///
1454+
/// Guards the `equals_datatype` fast paths: `equals_datatype` ignores struct
1455+
/// field names, but struct casts match fields by name, so types that differ
1456+
/// only in struct field names must not count as equal. Container labels
1457+
/// (list items, map entries and their key/value fields, union variants) stay
1458+
/// ignored because casts do not depend on them.
1459+
///
1460+
/// Call this only after `equals_datatype` returned true, so the two type
1461+
/// trees have the same shape.
1462+
fn struct_field_names_match(lhs_type: &DataType, rhs_type: &DataType) -> bool {
1463+
use arrow::datatypes::DataType::*;
1464+
match (lhs_type, rhs_type) {
1465+
(Struct(lhs), Struct(rhs)) => {
1466+
lhs.len() == rhs.len()
1467+
&& lhs.iter().zip(rhs.iter()).all(|(l, r)| {
1468+
l.name() == r.name()
1469+
&& struct_field_names_match(l.data_type(), r.data_type())
1470+
})
1471+
}
1472+
(List(l), List(r))
1473+
| (LargeList(l), LargeList(r))
1474+
| (ListView(l), ListView(r))
1475+
| (LargeListView(l), LargeListView(r))
1476+
| (FixedSizeList(l, _), FixedSizeList(r, _)) => {
1477+
struct_field_names_match(l.data_type(), r.data_type())
1478+
}
1479+
// Map entry key/value fields are positional; check only their types.
1480+
(Map(l, _), Map(r, _)) => match (l.data_type(), r.data_type()) {
1481+
(Struct(le), Struct(re)) => {
1482+
le.len() == re.len()
1483+
&& le.iter().zip(re.iter()).all(|(lf, rf)| {
1484+
struct_field_names_match(lf.data_type(), rf.data_type())
1485+
})
1486+
}
1487+
_ => true,
1488+
},
1489+
(Dictionary(_, l), Dictionary(_, r)) => struct_field_names_match(l, r),
1490+
(RunEndEncoded(_, l), RunEndEncoded(_, r)) => {
1491+
struct_field_names_match(l.data_type(), r.data_type())
1492+
}
1493+
// Union variants are identified by type id, not by listing order.
1494+
(Union(l, _), Union(r, _)) => l.iter().all(|(lid, lf)| {
1495+
r.iter()
1496+
.find(|(rid, _)| *rid == lid)
1497+
.is_some_and(|(_, rf)| {
1498+
struct_field_names_match(lf.data_type(), rf.data_type())
1499+
})
1500+
}),
1501+
_ => true,
1502+
}
1503+
}
1504+
14591505
/// Return true if every left-field name exists in the right fields (and lengths are equal).
14601506
///
14611507
/// # Assumptions
@@ -1524,36 +1570,6 @@ where
15241570
Some(Struct(coerced.into()))
15251571
}
15261572

1527-
/// Coerce two structs positionally (left-to-right) using `coerce_fn`.
1528-
/// Preserves field names from the left struct and uses combined nullability.
1529-
fn coerce_struct_by_position<F>(
1530-
lhs_fields: &Fields,
1531-
rhs_fields: &Fields,
1532-
coerce_fn: F,
1533-
) -> Option<DataType>
1534-
where
1535-
F: Fn(&DataType, &DataType) -> Option<DataType> + Copy,
1536-
{
1537-
use arrow::datatypes::DataType::*;
1538-
1539-
// First coerce individual types; fail early if any pair cannot be coerced.
1540-
let coerced_types: Vec<DataType> = lhs_fields
1541-
.iter()
1542-
.zip(rhs_fields.iter())
1543-
.map(|(l, r)| coerce_fn(l.data_type(), r.data_type()))
1544-
.collect::<Option<Vec<DataType>>>()?;
1545-
1546-
// Build final fields preserving left-side names and combined nullability.
1547-
let orig_pairs = lhs_fields.iter().zip(rhs_fields.iter());
1548-
let fields: Vec<FieldRef> = coerced_types
1549-
.into_iter()
1550-
.zip(orig_pairs)
1551-
.map(|(datatype, (lhs, rhs))| coerce_fields(datatype, lhs, rhs))
1552-
.collect();
1553-
1554-
Some(Struct(fields.into()))
1555-
}
1556-
15571573
/// returns the result of coercing two fields to a common type
15581574
fn coerce_fields(common_type: DataType, lhs: &FieldRef, rhs: &FieldRef) -> FieldRef {
15591575
let is_nullable = lhs.is_nullable() || rhs.is_nullable();
@@ -1574,14 +1590,33 @@ where
15741590
use arrow::datatypes::DataType::*;
15751591
match (lhs_type, rhs_type) {
15761592
(Map(lhs_field, lhs_ordered), Map(rhs_field, rhs_ordered)) => {
1577-
struct_coercion(lhs_field.data_type(), rhs_field.data_type(), coerce_fn).map(
1578-
|key_value_type| {
1579-
Map(
1580-
Arc::new((**lhs_field).clone().with_data_type(key_value_type)),
1581-
*lhs_ordered && *rhs_ordered,
1582-
)
1583-
},
1584-
)
1593+
// A map entry struct has exactly two positional fields (key,
1594+
// value) with arbitrary names; coerce their types positionally
1595+
// and keep the left side's names.
1596+
let (Struct(lhs_entry), Struct(rhs_entry)) =
1597+
(lhs_field.data_type(), rhs_field.data_type())
1598+
else {
1599+
return None;
1600+
};
1601+
if lhs_entry.len() != 2 || rhs_entry.len() != 2 {
1602+
return None;
1603+
}
1604+
let entry_fields: Vec<FieldRef> = lhs_entry
1605+
.iter()
1606+
.zip(rhs_entry.iter())
1607+
.map(|(lhs, rhs)| {
1608+
coerce_fn(lhs.data_type(), rhs.data_type())
1609+
.map(|common_type| coerce_fields(common_type, lhs, rhs))
1610+
})
1611+
.collect::<Option<Vec<_>>>()?;
1612+
Some(Map(
1613+
Arc::new(
1614+
(**lhs_field)
1615+
.clone()
1616+
.with_data_type(Struct(entry_fields.into())),
1617+
),
1618+
*lhs_ordered && *rhs_ordered,
1619+
))
15851620
}
15861621
_ => None,
15871622
}
@@ -1858,7 +1893,8 @@ fn string_concat_coercion(lhs_type: &DataType, rhs_type: &DataType) -> Option<Si
18581893
}
18591894

18601895
fn array_coercion(lhs_type: &DataType, rhs_type: &DataType) -> Option<DataType> {
1861-
if lhs_type.equals_datatype(rhs_type) {
1896+
if lhs_type.equals_datatype(rhs_type) && struct_field_names_match(lhs_type, rhs_type)
1897+
{
18621898
Some(lhs_type.to_owned())
18631899
} else {
18641900
None

‎datafusion/expr-common/src/type_coercion/binary/tests/comparison.rs‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1232,3 +1232,42 @@ fn test_comparison_coercion_with_session_timezone() {
12321232
Some(DataType::Utf8)
12331233
);
12341234
}
1235+
1236+
#[test]
1237+
fn union_variants_match_by_type_id_not_position() {
1238+
use arrow::datatypes::{UnionFields, UnionMode};
1239+
1240+
// Exercises `struct_field_names_match`, the guard on the
1241+
// `equals_datatype` fast paths. Union variants are identified by type id,
1242+
// not by listing order.
1243+
fn struct_type(name: &str) -> DataType {
1244+
DataType::Struct(Fields::from(vec![Field::new(name, DataType::Utf8, true)]))
1245+
}
1246+
fn union_type(variants: &[(i8, &str, DataType)]) -> DataType {
1247+
DataType::Union(
1248+
UnionFields::try_new(
1249+
variants.iter().map(|(id, _, _)| *id),
1250+
variants.iter().map(|(_, name, data_type)| {
1251+
Field::new(*name, data_type.clone(), true)
1252+
}),
1253+
)
1254+
.unwrap(),
1255+
UnionMode::Dense,
1256+
)
1257+
}
1258+
1259+
// Same child types throughout, so `equals_datatype` holds for every pair
1260+
// below and the guard is what decides.
1261+
let foo = struct_type("foo");
1262+
let bar = struct_type("bar");
1263+
1264+
let ordered = union_type(&[(0, "v0", foo.clone()), (1, "v1", bar.clone())]);
1265+
let reordered = union_type(&[(1, "v1", bar.clone()), (0, "v0", foo.clone())]);
1266+
// Same id -> variant mapping, different listing order: a match.
1267+
assert!(struct_field_names_match(&ordered, &reordered));
1268+
1269+
// Positions line up by name, but the id-matched variants have swapped
1270+
// struct names: not a match.
1271+
let swapped_ids = union_type(&[(1, "v0", foo.clone()), (0, "v1", bar)]);
1272+
assert!(!struct_field_names_match(&ordered, &swapped_ids));
1273+
}

‎datafusion/sqllogictest/test_files/struct.slt‎

Lines changed: 75 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1481,12 +1481,86 @@ drop table target_struct_insert;
14811481

14821482
# Test 10: CASE expression with different struct field orders
14831483
query II
1484-
select
1484+
select
14851485
(case when true then {a: 1, b: 2} else {b: 3, a: 4} end)['a'] as a_val,
14861486
(case when true then {a: 1, b: 2} else {b: 3, a: 4} end)['b'] as b_val;
14871487
----
14881488
1 2
14891489

1490+
############################
1491+
# Mismatched Field Names
1492+
############################
1493+
1494+
# Structs with different field names have no common type, so implicit
1495+
# coercion must fail at plan time.
1496+
# See https://github.com/apache/datafusion/issues/14396
1497+
1498+
statement ok
1499+
create table mismatched_struct_names as values
1500+
(
1501+
{ 'foo': 'baz' },
1502+
{ 'xxx': arrow_cast('blarg', 'Utf8View') }
1503+
);
1504+
1505+
# UNION of structs with different field names fails at plan time
1506+
statement error Error during planning: Incompatible inputs for Union: Previous inputs were of type Struct\("foo": Utf8\), but got incompatible type Struct\("xxx": Utf8View\) on column 'column2'
1507+
select column1 from mismatched_struct_names union all select column2 from mismatched_struct_names;
1508+
1509+
# CASE with mismatched struct branches fails at plan time, even though the
1510+
# mismatched branch is dead code
1511+
statement error Error during planning: Failed to coerce then \(Struct\("foo": Utf8\)\) and else \(Struct\("xxx": Utf8View\)\) to common types in CASE WHEN expression
1512+
select case when 1=2 then column1 else column2 end from mismatched_struct_names;
1513+
1514+
# Comparison of structs with different field names fails at plan time
1515+
statement error Error during planning: Cannot infer common argument type for comparison operation Struct\("foo": Utf8\) = Struct\("xxx": Utf8View\)
1516+
select column1 = column2 from mismatched_struct_names;
1517+
1518+
# Comparison with an anonymous struct literal (fields c0, c1) also fails
1519+
statement error Error during planning: Cannot infer common argument type for comparison operation Struct\("foo": Utf8\) = Struct\("c0": Utf8\)
1520+
select column1 = struct('baz') from mismatched_struct_names;
1521+
1522+
statement ok
1523+
drop table mismatched_struct_names;
1524+
1525+
# Structs with identical field types but different field names must also
1526+
# fail. These take the "types are already equal" fast path, which ignores
1527+
# struct field names unless guarded.
1528+
1529+
statement error Error during planning: Failed to coerce then \(Struct\("foo": Utf8\)\) and else \(Struct\("xxx": Utf8\)\) to common types in CASE WHEN expression
1530+
select case when 1=2 then { 'foo': 'baz' } else { 'xxx': 'blarg' } end;
1531+
1532+
statement error Error during planning: Cannot infer common argument type for comparison operation Struct\("foo": Utf8\) = Struct\("xxx": Utf8\)
1533+
select { 'foo': 'baz' } = { 'xxx': 'blarg' };
1534+
1535+
# Array containment over structs with different field names also fails.
1536+
# The error differs from the ones above because the analyzer rewrites `@>`
1537+
# to the `array_has_all` function before binary operator coercion runs, so
1538+
# the function signature coercion reports the failure.
1539+
statement error Error during planning: Failed to coerce arguments to satisfy a call to 'array_has_all' function
1540+
select [{ 'foo': 1 }] @> [{ 'xxx': 1 }];
1541+
1542+
# Same field names in a different order, with a field type that needs
1543+
# coercion (Utf8 vs Utf8View), still unify by name.
1544+
1545+
statement ok
1546+
create table reordered_struct_names as values
1547+
({ 'foo': 'baz', 'bar': 'bux' }),
1548+
({ 'bar': 'b', 'foo': arrow_cast('bt', 'Utf8View') });
1549+
1550+
query ?T
1551+
select column1, arrow_typeof(column1) from reordered_struct_names order by column1['foo'];
1552+
----
1553+
{bar: bux, foo: baz} Struct("bar": Utf8, "foo": Utf8View)
1554+
{bar: b, foo: bt} Struct("bar": Utf8, "foo": Utf8View)
1555+
1556+
query ?
1557+
select case when 1=2 then { 'foo': 'baz', 'bar': 'bux' } else { 'bar': 'b', 'foo': arrow_cast('bt', 'Utf8View') } end;
1558+
----
1559+
{bar: b, foo: bt}
1560+
1561+
statement ok
1562+
drop table reordered_struct_names;
1563+
14901564
############################
14911565
# JOIN Coercion Tests
14921566
############################

‎docs/source/user-guide/sql/struct_coercion.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -175,6 +175,21 @@ SELECT [
175175

176176
### Limitations
177177

178+
**Implicit coercion requires identical field name sets.** Two structs coerce to a common type only when they have the same field names (in any order). Structs with different field names have no common type, and the query fails at plan time:
179+
180+
```sql
181+
-- Different field names: foo vs xxx
182+
SELECT {foo: 'baz'} UNION ALL SELECT {xxx: 'blarg'};
183+
-- Error: Incompatible inputs for Union
184+
185+
SELECT CASE WHEN a = 1 THEN {foo: 'baz'} ELSE {xxx: 'blarg'} END FROM t;
186+
-- Error: Failed to coerce then (...) and else (...) to common types
187+
-- in CASE WHEN expression
188+
189+
SELECT {foo: 'baz'} = {xxx: 'blarg'};
190+
-- Error: Cannot infer common argument type for comparison operation
191+
```
192+
178193
**Field count must match exactly.** If structs have different numbers of fields and their field names don't completely overlap, the query will fail:
179194

180195
```sql

0 commit comments

Comments
 (0)