Support Union data type coercion #18825

Is your feature request related to a problem or challenge?

Union arrays currently cannot be used in comparison operations with scalar values or other types. When attempting to filter or compare a union column against a constant value (e.g., where union_column = 67), datafusion fails during type coercion

This prevents common filtering and selection ops on union-typed data. At work, we interact with a lot of JSON and accessing data with -> returns Union types. Currently the following query will err:

select * from records
where json_val->'id' = 123

Describe the solution you'd like

• Write a union_coercion function that handles union-to-scalar and union-to-union type coercion, following the pattern established by other composite types like struct_coercion and map_coercion
• Extend comparison_coercion to chain to union_coercion as part of its coercion attempts

On the arrow-side, comparison kernel support for unions are being developed:

• Add comparison support for Union arrays in the cmp kernel arrow-rs#8837
• Extend comparison support for Union arrays against an opaque array arrow-rs#8881