Repository navigation
Closed
Labels
enhancementNew feature or requestNew feature or request
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
This ticket is designed to capture the work needed to properly support Arrow Struct types in DataFusion
https://arrow.apache.org/datafusion/user-guide/sql/sql_status.html says that nested types are not supported; The are not fully supported, but there are parts of the support already present such as a way to serialize them via ArrowWriter and using field["nested_field"] syntax
Describe the solution you'd like
Research, and describe / implement what is else remains for proper support.
Array (ListArray) support:
- [Epic] General ticket for the concept of the practical implementation of
ARRAY#6980 - Support
FixedSizeListin array methods #6560 - Implement
unnestfunction #6555 - Support array sort for nested array #9252
Map (MapArray) support:
Struct (StructArray) support:
- Create a struct with the specified field names and values #5861
- Support DuckDB style stuct syntax #9820
- Create
Structtable with explicit type and name #10207 - Support unnest for struct data type #10264
- Support recursive unnest #10660
- Converge StructFunc and NamedStructFunc to one func #11204
- The struct value should not have duplicate and null name #11438
- Implement nested identifier access ( "Nested identifiers not yet supported" ) #11445
Union (UnionArray) support
Other
Known issues so far:
- [Question] Are composite types supported ? #2179 from @Cheappie
- How to build a field of type Struct with DataFusion SQL? #2043 from @lquerel
- Feature request for support for struct and arry data types #3617 from @kesavkolla
- Nested coalesce #6074
- Add note about structured field access to supported SQL docs #1222
- Introduce ProjectionMask To Allow Nested Projection Pushdown #2581
- make_array returns FixedSizeList which has proven to be difficult to work with. #6446
- Add array expressions #6075
- New functions for working with arrays #6119
- Support empty array #6561
- Supports NULL in arrays #6556
- New array method
array_contains#6557 - Write
Signaturemethod for list datatypes. #6559 - New concatenation operator for working with arrays #6603
- Implement
AnyOpandAllOpoperators #6602 - Support nested structs in Struct expression #6598
- Constructing and Destructing Objects (JSON) #6631
- Feature request for support for struct and arry data types #3617
- Trouble getting fancy with ARRAY_AGG (DISTINCT ARRAY_AGG) #6743
- Multi-field structs from builtin scalar fn incompatible with UDAF #7012
- Support parquet statistics for struct columns #8334
- Add the ability to create tables with deeply nested schemas in SQL #11746
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Activity
nl5887 commentedon Jun 9, 2022
This https://github.com/apache/arrow-datafusion/blob/master/datafusion/core/src/physical_plan/file_format/mod.rs#L238 is one reason of errors related to column projection. It compares the complete enum, failing on different field order.
Arrow has a method to compare data types (https://github.com/apache/arrow-rs/blob/master/arrow/src/datatypes/datatype.rs#L674). I think this method should me made public, and used in above.
Currently datafusion uses match_field_names (default true), https://github.com/apache/arrow-rs/blob/master/arrow/src/record_batch.rs#L153 causing the error.
alamb commentedon Jun 10, 2022
Thanks for the investigation @nl5887 -- that sounds definitely plausible. Feel free to file a PR with proposed changed -- we would love to review them
nl5887 commentedon Jun 26, 2022
This one is also related: #2581
tv42 commentedon Feb 22, 2023
Reminder to write docs: #1222
[-]Properly support arrow `Struct` types / Composite type in DataFusion[/-][+][EPIC] Properly support arrow `Struct` types / Composite type in DataFusion[/+][-][EPIC] Properly support arrow `Struct` types / Composite type in DataFusion[/-][+][EPIC] Improved support for nested / structured types (`Struct` , `List`, `ListArray`, and other Composite types)[/+]alexwilcoxson-rel commentedon Aug 30, 2023
Potential to add to list #7012
alamb commentedon Mar 27, 2024
We are starting to make progress on struct support --
There is a PR up to support
named_struct#9743 and work afoot to support nicer literal syntax: #9820 🚀toaiduongdh commentedon Apr 27, 2024
Hi, i think unnest support for struct can be an item in this epic right?
alamb commentedon Apr 27, 2024
That would make sense to me -- is there a ticket that describes what this means?
duongcongtoai commentedon Apr 27, 2024
i created a ticket: #10264
alamb commentedon Apr 30, 2024
Thank you. I added this to the list in the ticket description
duongcongtoai commentedon May 25, 2024
I added an issue to support recursive unnest: #10660, i think it shoul belong to this epic
alamb commentedon May 25, 2024
Added
goldmedal commentedon Jul 12, 2024
I added an issue to check the duplicate or null name for struct: #11438
Throne3d commentedon Jul 13, 2024
I think #11445 is related to this epic
alamb commentedon Jul 14, 2024
Thank you -- added
TheBuilderJR commentedon Aug 13, 2024
Right now datafusion doesn't support struct evolution very well. Imagine you have a struct named
customDatawith fieldsomeOptionEnabledin one parquet file, later down the line you add a new fieldnewAddedOptionto thecustomDatastruct in another parquet file. Currently when you try andSELECT * FROM tableyou'll get this error:Feels like we should handle this more gracefully. cc @alamb
I'm happy to make contributions if someone can point me to the right places to look.
alamb commentedon Aug 15, 2024
I agree
My suggestion is to start with filing a ticket with a self contained reproducer (either rust code or SQL) that shows what you are trying to do.
This would likely become part of the test of any code improvement we make, as well as providing some more detail for other contributors to help point to the right place in the code
alamb commentedon Feb 13, 2025
I think we will claim datafusion has pretty good structured type support now, so let's close this epic and treat additional work as new issues / items
TheBuilderJR commentedon Feb 18, 2025
@alamb I just tried this on 45.0 and it still doesn't seem supported:
Do I need to flip a flag or something?
TheBuilderJR commentedon Feb 19, 2025
Filed an issue here: