You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Avro read fails with ResolveError when the file has a map/list/struct field not in the read schema #4288
0.12.0 (latest release); also reproduced on main (e3564a5)
Please describe the bug 🐞
Reading an Avro file fails with ResolveError when the file schema has a map, list, or struct field that is not in the read schema:
pyiceberg.exceptions.ResolveError: File/read schema are not aligned for map, got None
at pyiceberg/avro/resolver.py SchemaPartnerAccessor.map_key_partner
Why this matters: The spec resolves fields by ID and asks readers to be permissive: its manifest read rules say a field the reader's version doesn't define should be ignored ("Ignore the field"). Iceberg Java/Spark, DuckDB, and Snowflake skip unknown manifest fields of any type. PyIceberg skips them only when they are primitives, so a manifest with an extra nested field makes the table unreadable. For example, a data_file map field that the reader's version doesn't define fails the scan.
Cause:ReadSchemaResolver.struct gives a file-only primitive a None position, so it is read and dropped. For nested types, though, SchemaPartnerAccessor.field_partner, list_element_partner, map_key_partner, and map_value_partner raise when the partner is None.
Minimal reproduction: write a manifest entry with fastavro, using MANIFEST_ENTRY_SCHEMAS[2] plus an extra optional map<int, binary> field in data_file. Then read it with AvroFile(read_schema=MANIFEST_ENTRY_SCHEMAS[2], ...).
Activity