Skip to content

Avro read fails with ResolveError when the file has a map/list/struct field not in the read schema #4288

Closed

Description

@haizhou-zhao

Apache Iceberg version

0.12.0 (latest release); also reproduced on main (e3564a5)

Please describe the bug 🐞

Reading an Avro file fails with ResolveError when the file schema has a map, list, or struct field that is not in the read schema:

pyiceberg.exceptions.ResolveError: File/read schema are not aligned for map, got None
  at pyiceberg/avro/resolver.py  SchemaPartnerAccessor.map_key_partner

Why this matters: The spec resolves fields by ID and asks readers to be permissive: its manifest read rules say a field the reader's version doesn't define should be ignored ("Ignore the field"). Iceberg Java/Spark, DuckDB, and Snowflake skip unknown manifest fields of any type. PyIceberg skips them only when they are primitives, so a manifest with an extra nested field makes the table unreadable. For example, a data_file map field that the reader's version doesn't define fails the scan.

Cause: ReadSchemaResolver.struct gives a file-only primitive a None position, so it is read and dropped. For nested types, though, SchemaPartnerAccessor.field_partner, list_element_partner, map_key_partner, and map_value_partner raise when the partner is None.

Minimal reproduction: write a manifest entry with fastavro, using MANIFEST_ENTRY_SCHEMAS[2] plus an extra optional map<int, binary> field in data_file. Then read it with AvroFile(read_schema=MANIFEST_ENTRY_SCHEMAS[2], ...).

Willingness to contribute

  • I can contribute a fix for this bug independently

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions