Repository navigation
Commit f1ab86d
Add FixedSizeList support for recursive struct schema adaptation (apache#22980)
## Which issue does this PR close?
* Part of apache#20835
## Rationale for this change
`FixedSizeList` containing `Struct` values was not handled by the
existing recursive nested adaptation logic used for schema evolution. As
a result, planner-time compatibility checks, nested cast detection, and
runtime casting did not support additive struct evolution within
`FixedSizeList` containers.
This change adds `FixedSizeList` support and verifies planner/runtime
parity so that planning allows exactly the cases runtime can adapt while
continuing to reject incompatible schema changes.
## What changes are included in this PR?
* Extend `cast_column` to support recursive casting of `FixedSizeList`
values when source and target list sizes match.
* Add `FixedSizeList` handling to:
* `requires_nested_struct_cast`
* `validate_data_type_compatibility`
* Implement recursive casting of nested `Struct` values contained in
`FixedSizeList`.
* Preserve planner/runtime parity by validating child type compatibility
before runtime fallback logic is applied.
* Add handling for null-parent `FixedSizeList` entries by masking hidden
child values before retrying casts, avoiding failures caused by
semantically inaccessible child data.
* Refactor list and list-view casting helpers to use Arrow `AsArray`
accessors.
## Are these changes tested?
Yes.
The following tests were added:
* `test_cast_fixed_size_list_struct`
* `test_validate_fixed_size_list_struct_compatibility`
*
`test_validate_fixed_size_list_struct_missing_non_nullable_field_rejected`
* `test_validate_fixed_size_list_struct_size_mismatch_rejected`
* `test_cast_fixed_size_list_struct_all_null`
*
`test_fixed_size_list_struct_planner_runtime_parity_on_incompatible_type`
*
`test_cast_fixed_size_list_struct_missing_non_nullable_field_runtime_rejected`
*
`test_cast_fixed_size_list_struct_ignores_hidden_child_values_for_null_parent`
Existing coverage in `test_requires_nested_struct_cast` was also
extended to include `FixedSizeList` cases.
These tests cover:
* Additive nullable nested-field evolution
* All-null and partially null list cases
* Incompatible nested type changes
* Non-nullable field addition rejection
* Planner/runtime parity validation
## Are there any user-facing changes?
No user-facing changes. This is an internal enhancement to nested schema
adaptation and casting behavior for `FixedSizeList<Struct>` types.
## LLM-generated code disclosure
This PR includes LLM-generated code and comments. All LLM-generated
content has been manually reviewed.
---------
Co-authored-by: Andrew Lamb <[email protected]>1 parent abec311 commit f1ab86d
2 files changed
Lines changed: 496 additions & 52 deletions
0 commit comments