Skip to content

Commit e238121

Browse files
committed
fix: release an Iceberg file's reservation once its row groups leave memory
A data file's share was its written size capped at one row group, so every fanout file that had flushed stayed charged a whole row group until it closed. A local file keeps nothing it has flushed, so a pool of eight 128 KiB row groups failed a twelve-partition write whose files held almost nothing. Each file's OutputFile is now wrapped so that its writer counts the bytes that leave memory on their way to storage, and the file reports what it has written less those. A local file releases a row group as it is handed on. S3 and GCS do not: OpenDAL collects what a file flushes into parts of at least 5 MiB and holds each part until the next one is complete or the file closes, so a file there keeps what it has flushed since its last upload charged.
1 parent 9183992 commit e238121

6 files changed

Lines changed: 462 additions & 42 deletions

File tree

‎.ai/skills/review-comet-iceberg-write-pr/SKILL.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,6 +147,9 @@ Read the ownership table in the contributor guide before reviewing any change ne
147147
`PartitionFeed`s, or anything a new feed holds back) must be added to what `run_write_task` reserves. A
148148
builder that constructs `ParquetWriterBuilder` directly leaves its files out of the task's
149149
memory reservation, so a wide fanout write grows past the pool instead of failing its task.
150+
A storage scheme newly supported for writes needs its entry in
151+
`StorageWrites::for_location`: a file reports its flushed row groups until its storage has
152+
written them out, which a local file does at once and an object store only part by part.
150153

151154
## 5. Plan Shape
152155

‎docs/source/contributor-guide/iceberg-writes.md‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -264,15 +264,20 @@ Points where Comet adapts iceberg-rust to match iceberg-java:
264264

265265
iceberg-rust keeps each open file writer private inside its rolling and partitioning writers, so
266266
Comet wraps `ParquetWriterBuilder` in `MeteredParquetWriterBuilder`, whose files report what they
267-
hold in memory. After every batch `run_write_task` resizes the task's reservation to what the open
268-
files report plus the rows each `PartitionFeed` holds, for the dictionary choice or for pacing, and a resize the pool refuses fails the task. What
269-
that figure covers, and what it misses, is described under
267+
hold in memory. It hands `ParquetWriter` each file's `OutputFile` behind a `CountedOutput`, whose
268+
writer counts the bytes that leave memory on their way to storage, and a file reports what it has
269+
written less those. When they leave depends on the storage (`StorageWrites`). After every batch
270+
`run_write_task` resizes the task's reservation to what the open files report plus the rows each
271+
`PartitionFeed` holds, for the dictionary choice or for pacing, and a resize the pool refuses fails
272+
the task. What that figure covers, and what it misses, is described under
270273
[Native writers](memory_management.md#native-writers).
271274

272275
`FileIO` comes from `load_file_io` in `iceberg_common.rs`, shared with the native scan. It picks
273276
the storage backend from the data location's scheme and wires in Comet's S3 credential bridge when
274277
one is configured. For writes the bridge fails closed: if a configured provider cannot initialize,
275-
the task fails rather than writing with the default credential chain.
278+
the task fails rather than writing with the default credential chain. A storage scheme newly
279+
supported for writes also needs its entry in `StorageWrites::for_location`, which treats an unknown
280+
scheme as one that uploads in parts.
276281

277282
### Native to JVM: the task payload
278283

‎docs/source/contributor-guide/memory_management.md‎

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -391,11 +391,16 @@ the executor.
391391
([#5304](https://github.com/apache/datafusion-comet/issues/5304)).
392392
- The Iceberg writer keeps one file open per partition in a fanout write, so its reservation grows
393393
with the number of partitions a task writes. iceberg-rust's `ParquetWriter` does not expose
394-
parquet-rs's `memory_size`, only the bytes written so far plus the in-progress row group's
395-
encoded size, so each open file reports that figure capped at the row-group size
396-
(`write.parquet.row-group-size-bytes`). The reservation also covers the rows each partition holds
397-
back, first for its dictionary choice and then until they fill the 1000-row unit the rolling
398-
writer is fed in. It does not cover what
394+
parquet-rs's `memory_size`, only every byte the file has written, its in-progress row group's
395+
encoded size included. So each file's writer counts the bytes that leave memory on their way to
396+
storage, and the file reports what it has written less those. A local file writes a flushed row
397+
group out at once, so it reports its in-progress row group and the few KiB parquet-rs buffers in
398+
front of storage. S3 and GCS take a file in parts of at least 5 MiB, and OpenDAL holds each part
399+
in memory until the next one is complete or the file closes, so there a file also reports what
400+
it has flushed since its last part was uploaded. At the default row-group size
401+
(`write.parquet.row-group-size-bytes`, 128 MiB) that is the last row group it flushed. The
402+
reservation also covers the rows each partition holds back, first for its dictionary choice and
403+
then until they fill the 1000-row unit the rolling writer is fed in. It does not cover what
399404
parquet-rs holds beyond the encoded size: dictionary hash tables, unencoded dictionary indices
400405
and buffer capacity. Native writes decline Bloom filters today, and neither figure would include
401406
them.

‎docs/source/user-guide/latest/iceberg-writes.md‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -262,14 +262,15 @@ attempt id is embedded in its data file names.
262262

263263
The native writer's buffers are charged to Comet's memory pool, the off-heap budget Comet's other
264264
native operators draw on, where iceberg-java's buffers sit on the JVM heap. A fanout write keeps a
265-
data file open for every partition a task writes to, and each open file holds up to one row group
266-
(`write.parquet.row-group-size-bytes`) in memory, so a task writing to many partitions needs memory
267-
in proportion to them. When the pool cannot grant it, the task fails with a `CometNativeException`
268-
reading `Additional allocation failed for IcebergWriteExec` instead of exceeding the executor's
269-
memory, and Spark retries it like any other task failure. Such a write fits in less memory with the
270-
fanout writer disabled (`write.spark.fanout.enabled=false`): Spark then sorts each task's rows by
271-
partition, and the task keeps one file open at a time. A smaller row-group size also helps.
272-
Otherwise the write needs a larger `spark.memory.offHeap.size`.
265+
data file open for every partition a task writes to. Each open file holds the row group it is
266+
writing in memory, up to `write.parquet.row-group-size-bytes`, and on S3 or GCS also the last row
267+
group it flushed, which is uploaded once the next one is complete or the file closes. So a task
268+
writing to many partitions needs memory in proportion to them. When the pool cannot grant it, the
269+
task fails with a `CometNativeException` reading `Additional allocation failed for IcebergWriteExec`
270+
instead of exceeding the executor's memory, and Spark retries it like any other task failure. Such a
271+
write fits in less memory with the fanout writer disabled (`write.spark.fanout.enabled=false`):
272+
Spark then sorts each task's rows by partition, and the task keeps one file open at a time. A
273+
smaller row-group size also helps. Otherwise the write needs a larger `spark.memory.offHeap.size`.
273274

274275
Partial results are never committed. The commit set is exactly the commit messages returned by
275276
successful tasks — a failed task contributes none — and if the job fails, the driver-side

‎native/core/src/execution/operators/iceberg_common.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -574,7 +574,7 @@ fn env_region_present() -> bool {
574574
/// The JVM write gate (`CometIcebergNativeWrite.storageScheme`) mirrors this rule exactly, case
575575
/// included. Change both together, and keep the cases in
576576
/// `scheme_of_extracts_scheme_from_all_uri_forms` in step with its `storageScheme` test.
577-
fn scheme_of(path: &str) -> &str {
577+
pub(crate) fn scheme_of(path: &str) -> &str {
578578
match path.split_once(':') {
579579
Some((scheme, _)) if !scheme.is_empty() && !scheme.contains('/') => scheme,
580580
_ => "file",

0 commit comments

Comments
 (0)