You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 7fe9328
Browse filesBrowse the repository at this point in the historyBrowse files
fix: fall back when a schema repeats a Parquet field id
Under `spark.sql.parquet.fieldId.read.enabled` Spark resolves each
requested field to the one Parquet field carrying its id, and raises
FOUND_DUPLICATE_FIELD_IN_FIELD_ID_LOOKUP_MODE when more than one
answers. Comet never looked at field ids, so a requested schema that
repeats one was read positionally and returned rows where Spark raises.
The ids ride in the requested schema's `StructField.metadata`, so this
is decidable from the plan. `DataTypeSupport` gains a
`hasDuplicateFieldIds` predicate beside the existing duplicate-name one,
and `CometScanTypeChecker` declines a field list that trips it.
Both entry points are overridden, because they see different things.
`isTypeSupported` is handed each field's data type, so it sees nested
structs as the trait's recursion reaches them but can never compare two
top-level fields; `isSchemaSupported` is where the schema's own field
list is available. Each list is inspected exactly once, so nothing is
re-walked at an enclosing level.
`ArrowCachedBatchSerializer.supportsType` also accepted a struct with
duplicate child names, so caching such a relation stored it in Comet's
Arrow format, which Java Arrow cannot import back because it keys struct
children by name. One more schema check delegates it to Spark's default
cache format, alongside the interval types already excluded there, and
the two copies of that predicate in the shuffle gate now call the shared
one rather than spelling it out inline.
No Parquet decoding changes; see #5786 for that path.
Closes#5801.
0 commit comments