Skip to content

Commit 80e28ad

Browse files
nhobin219claude
andcommitted
docs(spec): record the v3 answer and the write-it-ourselves option
Two questions, and the first rules out the obvious hope for the second. Iceberg v3 would not help. §13.7's cost is pyiceberg deep-copying TableMetadata, and TableMetadataV3 is the same pydantic model reached by the same update path — v3 changes the format, not the library's object handling. Recorded as §13.8 along with what v3 WOULD bring: variant for the tabular-JSON case, nanosecond timestamps, default column values. And what it would not: row lineage does not replace litelink_offset, since a _row_id exists only once the row is committed and §7 filters the buffer leg on an offset the table has not seen. Neither 0.11.1 nor 0.12.0rc1 will write v3 at all — NotImplementedError, apache/iceberg-python#1551 — so this is moot before it is arguable. That the cost is pyiceberg's rather than Iceberg's is what makes writing the metadata directly an option. The library already writes its own Parquet at a claimed path, fsynced before the commit, so pyiceberg is only doing the registration: a manifest entry, a manifest, a manifest list, a metadata.json and a pointer swap. Each is a documented format plus a SQLite row update the library already makes atomically. §1's "used, not reimplemented" turns on a distinction worth stating — the format is the commitment, the library is an implementation choice — and the real risk is drift, since metadata that is subtly wrong still opens locally and breaks external readers later, in someone else's engine. Two cheaper things to rule out first: committing every Nth seal rather than every seal, which trades commit count for buffer size and needs no new format code, and the upstream fix, since the deep copy is not load-bearing. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
1 parent 1202e1b commit 80e28ad

1 file changed

Lines changed: 62 additions & 0 deletions

File tree

‎docs/SPEC.md‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1112,6 +1112,39 @@ The consequence worth planning for is that local disk holds roughly
11121112
`compact_below` over time so yesterday's output is tomorrow's input, tiered compaction, or
11131113
the honest possibility that unbounded local retention is not a supported configuration.
11141114

1115+
### Registering files without pyiceberg's commit path
1116+
1117+
The cost above is pyiceberg's, not Iceberg's, which makes a fourth option available: write
1118+
the metadata directly. **The library already writes its own Parquet** — at a path claimed in
1119+
`sealing` before the bytes exist (I2), fsynced before the commit (I1) — so pyiceberg is only
1120+
doing the registration. That registration is a manifest entry, a manifest, a manifest list,
1121+
a `metadata.json`, and a pointer swap, and every one of those is a documented file format
1122+
plus a SQLite row update the library already knows how to make atomically (I16).
1123+
1124+
The attraction is that appending an entry does not inherently require copying the whole
1125+
table metadata twenty-seven times. The objection is §1's principle that *"everything Iceberg
1126+
provides is used, not reimplemented"*, and the distinction that principle turns on is worth
1127+
stating: using the **format** is the commitment, using the **library** is an implementation
1128+
choice. Files that conform are still Iceberg. The real risk is drift — hand-written metadata
1129+
that is subtly wrong still opens locally and breaks the external readers the archive exists
1130+
for, and it breaks them later, in someone else's engine.
1131+
1132+
Two cheaper things should be ruled out first, because both are small and neither risks the
1133+
format:
1134+
1135+
- **Commit less often.** A seal must make rows durable, which it does by writing and
1136+
fsyncing Parquet; it does not have to register them in the same breath. Registering every
1137+
Nth seal cuts commit count by N, and the rows stay readable throughout because §7 serves
1138+
anything above the table's extent from the buffer — the same window step 3 already leaves
1139+
open, just wider. What it costs is a larger buffer, which §7 measures as the variable cost
1140+
of a read.
1141+
- **Wait for the upstream fix.** The deep copy is not load-bearing; it is how
1142+
`update_table_metadata` is written today.
1143+
1144+
If those are not enough, hand-writing the commit is a bounded piece of work with one hard
1145+
requirement: an external engine must read the result. That is a test before it is a design —
1146+
attach something that is not pyiceberg and assert it sees what litelink says is there.
1147+
11151148
**The archive has the same commit cost and it does not matter in the same way.** Its file
11161149
count is unbounded by design — it is the full history — so registering into it rewrites
11171150
manifests against a table that only grows. But that happens in §5, which is lazy,
@@ -1126,6 +1159,35 @@ The consequence worth planning for is that local disk holds roughly
11261159
feeling a cost that was supposed to have been moved off it. The remote table wants the same
11271160
manifest-merge properties as the local one for that reason, and §5's throughput is worth a
11281161
number rather than an assumption.
1162+
8. **Iceberg v3.** Tables are written at format-version 2 because pyiceberg will not write
1163+
anything else — `NotImplementedError: Writing V3 is not yet supported`, at 0.11.1 and at
1164+
0.12.0rc1, tracked upstream as apache/iceberg-python#1551. That is the whole of the current
1165+
answer. Nothing about this design prefers v2.
1166+
1167+
Three things in v3 would matter here, and the one that looks decisive is not.
1168+
1169+
**Variant**, for semi-structured data, is the interesting one and the furthest away —
1170+
pyiceberg has no `VariantType` at any version yet. The target workload is tabular JSON off a
1171+
websocket, where today the choice is to parse every field into a column or keep the frame as
1172+
text. Variant is the third option: store the frame, address into it, let the engine prune.
1173+
1174+
**Nanosecond timestamps.** Temporal columns are refused today because their round trip
1175+
through SQLite's storage classes is untested, and `timestamp[ns]` pyiceberg rejected
1176+
outright. The examples carry epoch nanoseconds in an `int64` as a result — honest, and it
1177+
loses the type. v3 has the types and pyiceberg already models them.
1178+
1179+
**Default column values**, which would make §9's add-a-column less lossy: an older file
1180+
could read a declared default rather than null.
1181+
1182+
**Row lineage does not replace `litelink_offset`**, though it is the obvious candidate. v3
1183+
gives each row a table-level `_row_id`, which answers §2's objection that Iceberg's sequence
1184+
numbers are per *snapshot* rather than per row. It does not answer the other half. A
1185+
`_row_id` is assigned when the row is committed to the table, and the tier boundary needs an
1186+
identifier that exists while the row is still in the buffer — §7 filters the buffer leg on
1187+
an offset the table has not seen. The library keeps owning that column under v3.
1188+
1189+
**Deletion vectors** are irrelevant rather than useful: §1 has no updates and no deletes,
1190+
and the only rows that leave do so as whole files leaving a snapshot.
11291191

11301192
---
11311193

0 commit comments

Comments
 (0)