Skip to content

Commit 1cb149f

Browse files
committed
docs: describe the footer reads native Iceberg writes make on object stores
The JVM reads each natively written file's footer back to compute its Iceberg metrics. Say that this is two reads per file, made one after another, which on S3 or GCS are GET requests, so a write of many small files pays two round trips for each.
1 parent 70bb11d commit 1cb149f

2 files changed

Lines changed: 13 additions & 1 deletion

File tree

‎docs/source/user-guide/latest/iceberg-writes.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -293,7 +293,14 @@ bounds carried over from the native writer's tracked state. iceberg-java's metad
293293
— metrics modes, the inferred-column cap
294294
(`write.metadata.metrics.max-inferred-column-defaults`), bound truncation, and list/map bounds
295295
suppression — are therefore applied by iceberg-java's own code regardless of what the native
296-
writer reports. This costs one footer-sized ranged read per written file at write time.
296+
writer reports.
297+
298+
iceberg-java's writer takes these metrics from the footer it still holds in memory, but the native
299+
path reads each footer back from storage: two small reads per written file, one for the footer's
300+
length and one for the footer itself, made one file after another before the task finishes. On S3
301+
or GCS each read is a GET request, so the task waits two request round trips per file. That is
302+
small next to uploading a large file, but a write that produces many small files, such as a fanout
303+
write over many partitions, pays it for every one of them.
297304

298305
## Failure handling
299306

‎docs/source/user-guide/latest/migration-guide.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,11 @@ new default unless it sets `spark.comet.write.iceberg.enabled=false`.
9999
`spark.comet.write.iceberg.splitOperator.enabled` is now a testing setting, and setting it to
100100
`false` does not turn the split operator off. See [Iceberg Writes](iceberg-writes.md).
101101

102+
To compute the Iceberg metrics of a natively written file, Comet reads the file's Parquet footer
103+
back from storage, where iceberg-java's writer takes them from memory. On S3 or GCS that is two GET
104+
requests per file, which adds up for a write that produces many small files. See
105+
[Native Parquet write eligibility](iceberg-writes.md#native-parquet-write-eligibility).
106+
102107
The native writer's buffers count against Comet's off-heap memory pool, where iceberg-java's buffers
103108
sit on the JVM heap. A fanout write keeps a data file open for every partition a task writes to, and
104109
each open file holds the row group it is writing, so a task that writes to many partitions can need

0 commit comments

Comments
 (0)