You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Substrait literal conversion should preserve nested nullability #22095
Is your feature request related to a problem or challenge?
DataFusion’s Substrait consumer currently normalizes some nested literal fields to nullable when converting Substrait literals into ScalarValues.
This is intentional today because VirtualTable / Values consumption converts the schema and literal rows separately. DataFusion later materializes the values into Arrow arrays and builds a RecordBatch using the schema. Arrow requires nested field types to match exactly, including nullability, so a mismatch like this can fail even
when the value itself is valid:
Add an expected-type-aware Substrait literal conversion path.
For VirtualTable / Values, the consumer has access to the NamedStruct schema. When converting each literal row, pass the expected top-level Field into literal conversion, and recursively use the expected nested fields for Struct, List, and Map literals.
The existing schema-less literal conversion path should remain available for ordinary expression literals where no expected field/type is available.
Activity
nathanb9 commentedon May 11, 2026
if its okay, I can take this @neilconway
neilconway commentedon May 11, 2026
@nathanb9 Absolutely! It make make sense to wait until #12727 is landed to avoid merge conflicts, but that's up to you.
fix(substrait): preserve nested nullability
huymq1710 commentedon Aug 7, 2026
Hi @neilconway , @nathanb9
Since this issue has been open without action for a while, I picked it up and opened #24158