Repository navigation
Commit 710c748
Match Parquet map key/value by position in nested schema coercion (apache#25342)
# Match Parquet map key/value by position in nested schema coercion
## Which issue does this PR close?
- Follow-up to apache#25193 (which closed apache#25192). No separate issue.
## Rationale for this change
apache#25193 made `apply_file_schema_type_coercions` recurse into nested types
so that, for example, a nested `Utf8` column in a Parquet file is
decoded directly as `Utf8View` when the table schema asks for it,
instead of being cast afterwards.
For `Map` columns the recursion delegated the `key`/`value` children to
the by-name `Struct` path, even though the code comment said container
children match by position. Parquet readers always name map children
`key_value`/`key`/`value`, while Arrow producers commonly use
`entries`/`keys`/`values` (e.g. `MapBuilder` defaults). With that naming
mismatch the view coercion was silently skipped for maps and queries
kept the slower cast path.
## What changes are included in this PR?
- `Map` key and value children are now matched by position, regardless
of their names. Field names, nullability, metadata, the container kind,
`FixedSizeList` width and `Map` ordering continue to come from the file
schema.
- `apply_file_schema_type_coercions` is restructured around a recursive
`coerce_data_type` helper that operates on `DataType`s directly. This
removes the throwaway single-field `Schema`s built for each nested
level, the `needs_*` pre-scan flags (the result is compared to the input
instead), and an unreachable match arm in the container branch (the one
uncovered line in the apache#25193 codecov report).
- The doc comment now describes the nested matching rules (structs by
name, lists and maps by position) and which parts of the result always
come from the file.
Behaviour is otherwise unchanged; the existing nested parquet `.slt`
files (`schema_evolution_nested`, `parquet_nested_schema_pruning`,
`parquet_filter_pushdown`) pass.
## What is the testing strategy for this PR?
Unit tests in `schema_coercion.rs`:
- `nested_coercion_preserves_list_containers` now uses a table-side
child that differs from the file-side child in name, nullability,
metadata, `FixedSizeList` width and `Map` ordering, and asserts the
file's values win. Mutation testing against apache#25193 showed that taking
the width, ordering or child field from the table survived the previous
version of this test; a wrong `FixedSizeList` width is accepted silently
by parquet-rs and would decode data with the wrong list size.
- `nested_coercion_matches_map_entries_by_position`: Arrow-named table
map vs Parquet-named file map, plus a map with a different number of
entry children that must be left alone.
- `nested_coercion_map_reader_produces_string_views`: writes a file with
Parquet's map naming and no embedded Arrow schema, coerces against an
Arrow-named table schema, and checks the Parquet reader accepts the
schema and produces `StringViewArray` keys and values.
- `nested_coercion_list_inside_struct`.
The two map tests fail on `main` without the fix.
## Are there any user-facing changes?
No API changes. Map columns whose table schema uses `keys`/`values`
naming now get the same nested view/string coercion as structs and
lists, avoiding a cast during scan.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01LA4Rks1XaN8wxpszj1hXQQ
---------
Co-authored-by: Claude <[email protected]>
Co-authored-by: Copilot Autofix powered by AI <[email protected]>1 parent 42f3888 commit 710c748
1 file changed
Lines changed: 577 additions & 164 deletions
0 commit comments