You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 79477f0
Browse filesBrowse the repository at this point in the historyBrowse files
docs(spec): record the v3 answer and the write-it-ourselves option
Two questions, and the first rules out the obvious hope for the second.
Iceberg v3 would not help. §13.7's cost is pyiceberg deep-copying
TableMetadata, and TableMetadataV3 is the same pydantic model reached by
the same update path — v3 changes the format, not the library's object
handling. Recorded as §13.8 along with what v3 WOULD bring: variant for
the tabular-JSON case, nanosecond timestamps, default column values. And
what it would not: row lineage does not replace litelink_offset, since a
_row_id exists only once the row is committed and §7 filters the buffer
leg on an offset the table has not seen.
Neither 0.11.1 nor 0.12.0rc1 will write v3 at all —
NotImplementedError, apache/iceberg-python#1551 — so this is moot before
it is arguable.
That the cost is pyiceberg's rather than Iceberg's is what makes writing
the metadata directly an option. The library already writes its own
Parquet at a claimed path, fsynced before the commit, so pyiceberg is
only doing the registration: a manifest entry, a manifest, a manifest
list, a metadata.json and a pointer swap. Each is a documented format
plus a SQLite row update the library already makes atomically.
§1's "used, not reimplemented" turns on a distinction worth stating —
the format is the commitment, the library is an implementation choice —
and the real risk is drift, since metadata that is subtly wrong still
opens locally and breaks external readers later, in someone else's
engine.
Two cheaper things to rule out first: committing every Nth seal rather
than every seal, which trades commit count for buffer size and needs no
new format code, and the upstream fix, since the deep copy is not
load-bearing.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
0 commit comments