You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 4aa737f
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](iceberg) Carry the timestamptz marker into the ancestor read of a missing equality key
### What problem does this PR solve?
Issue Number: None
Related PR: #69845
Problem Summary:
When an Iceberg equality-delete key is a nested field that an older data
file does not store, but the file does store the key's parent struct,
`_build_missing_equality_delete_key_expr` reads the whole parent so that
the key stays NULL under a NULL parent. It added that read through
`_append_file_scan_column`, a bare top-level projection without timestamp
markers. Earlier in this PR, `annotate_file_schema` started rewriting each
marked timestamptz leaf of the Parquet file schema to TIMESTAMPTZ. The
file block is built from that schema, but the Parquet reader takes the
timestamp semantics only from the projection.
Reproduction: an older Parquet file holds a nullable
`payload {existing INT, ts INT96 timestamptz}` (Spark's encoding), the
table later added `payload.k`, and a non-empty equality delete is keyed on
`payload.k`. Scans of that file failed in `check_block_column_type`:
Parquet read `payload` as STRUCT<existing, DATETIMEV2(6)> into a
STRUCT<existing, TIMESTAMPTZ(6)> file block. Projecting `payload` did not
help: adding the root as a predicate column replaced the projection its
mapping had marked. Before this PR the file schema kept the physical
types, so the same read succeeded.
The fix moves the annotated projection the hidden-key path already built
(`top_level` plus `attach_timestamp_semantics` from the annotated file
column) into `equality_delete_root_projection()`. The missing key's
ancestor, the hidden key and the delete file's keys all use it now. The
ancestor is still added as a predicate column through
`FileScanRequestBuilder`, so a root already in the request is merged or
moved as before. A plan without markers builds the same request as before.
### Release note
None
### Check List (For Author)
- Test: Unit Test (to be run)
- New `IcebergV2ReaderTest.IcebergMissingNestedEqualityKeyAncestorKeepsInt96TimestampTzMarker`:
in Asia/Shanghai, with the key's parent holding an INT96 timestamptz
sibling, checks the surviving rows with the parent hidden and
projected, that the projected sibling reads as an instant, and a plan
without the marker
- Behavior changed: No
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 <[email protected]>
0 commit comments