Skip to content

Commit f1ab86d

Browse files
kosiewalamb
andauthored
Add FixedSizeList support for recursive struct schema adaptation (apache#22980)
## Which issue does this PR close? * Part of apache#20835 ## Rationale for this change `FixedSizeList` containing `Struct` values was not handled by the existing recursive nested adaptation logic used for schema evolution. As a result, planner-time compatibility checks, nested cast detection, and runtime casting did not support additive struct evolution within `FixedSizeList` containers. This change adds `FixedSizeList` support and verifies planner/runtime parity so that planning allows exactly the cases runtime can adapt while continuing to reject incompatible schema changes. ## What changes are included in this PR? * Extend `cast_column` to support recursive casting of `FixedSizeList` values when source and target list sizes match. * Add `FixedSizeList` handling to: * `requires_nested_struct_cast` * `validate_data_type_compatibility` * Implement recursive casting of nested `Struct` values contained in `FixedSizeList`. * Preserve planner/runtime parity by validating child type compatibility before runtime fallback logic is applied. * Add handling for null-parent `FixedSizeList` entries by masking hidden child values before retrying casts, avoiding failures caused by semantically inaccessible child data. * Refactor list and list-view casting helpers to use Arrow `AsArray` accessors. ## Are these changes tested? Yes. The following tests were added: * `test_cast_fixed_size_list_struct` * `test_validate_fixed_size_list_struct_compatibility` * `test_validate_fixed_size_list_struct_missing_non_nullable_field_rejected` * `test_validate_fixed_size_list_struct_size_mismatch_rejected` * `test_cast_fixed_size_list_struct_all_null` * `test_fixed_size_list_struct_planner_runtime_parity_on_incompatible_type` * `test_cast_fixed_size_list_struct_missing_non_nullable_field_runtime_rejected` * `test_cast_fixed_size_list_struct_ignores_hidden_child_values_for_null_parent` Existing coverage in `test_requires_nested_struct_cast` was also extended to include `FixedSizeList` cases. These tests cover: * Additive nullable nested-field evolution * All-null and partially null list cases * Incompatible nested type changes * Non-nullable field addition rejection * Planner/runtime parity validation ## Are there any user-facing changes? No user-facing changes. This is an internal enhancement to nested schema adaptation and casting behavior for `FixedSizeList<Struct>` types. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed. --------- Co-authored-by: Andrew Lamb <[email protected]>
1 parent abec311 commit f1ab86d

2 files changed

Lines changed: 496 additions & 52 deletions

File tree

0 commit comments

Comments
 (0)