You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 8c783aa
Browse filesBrowse the repository at this point in the historyBrowse files
feat: charge native write buffers to the task's memory pool (#6247)
* feat: charge native write buffers to the task's memory pool
Neither native writer reserved what it buffers, so a native Parquet or Iceberg
write held its row groups in memory that no budget accounted for. A fanout
Iceberg write multiplies that by the number of partitions a task writes, and
when it outgrew the executor's memory overhead the container was killed.
Both writers now register one memory consumer per task and resize it after
every batch. ParquetWriterExec reserves parquet-rs's estimate of its
in-progress row group. The Iceberg writer wraps iceberg-rust's
ParquetWriterBuilder so that every open file reports what it holds, capped at
the row-group size because iceberg-rust exposes only the bytes written plus
the in-progress row group, and adds the rows its pacers hold back. Neither
writer can spill, so a resize the pool refuses fails the task with a native
out-of-memory error that Spark can retry.
Closes#5648.
Closes#6115.
* fix: release an Iceberg file's reservation once its row groups leave memory
A data file's share was its written size capped at one row group, so every fanout file that had
flushed stayed charged a whole row group until it closed. A local file keeps nothing it has
flushed, so a pool of eight 128 KiB row groups failed a twelve-partition write whose files held
almost nothing.
Each file's OutputFile is now wrapped so that its writer counts the bytes that leave memory on
their way to storage, and the file reports what it has written less those. A local file releases
a row group as it is handed on. S3 and GCS do not: OpenDAL collects what a file flushes into parts
of at least 5 MiB and holds each part until the next one is complete or the file closes, so a file
there keeps what it has flushed since its last upload charged.
Copy file name to clipboardExpand all lines: docs/source/contributor-guide/iceberg-writes.md
+23-11Lines changed: 23 additions & 11 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -272,10 +272,22 @@ Points where Comet adapts iceberg-rust to match iceberg-java:
272
272
([#5776](https://github.com/apache/datafusion-comet/issues/5776)). Manifest order becomes the row
273
273
order of an unordered read, so any map iteration that reaches the output needs the same care.
274
274
275
+
iceberg-rust keeps each open file writer private inside its rolling and partitioning writers, so
276
+
Comet wraps `ParquetWriterBuilder` in `MeteredParquetWriterBuilder`, whose files report what they
277
+
hold in memory. It hands `ParquetWriter` each file's `OutputFile` behind a `CountedOutput`, whose
278
+
writer counts the bytes that leave memory on their way to storage, and a file reports what it has
279
+
written less those. When they leave depends on the storage (`StorageWrites`). After every batch
280
+
`run_write_task` resizes the task's reservation to what the open files report plus the rows each
281
+
`PartitionFeed` holds, for the dictionary choice or for pacing, and a resize the pool refuses fails
282
+
the task. What that figure covers, and what it misses, is described under
|`CometIcebergWriteActionSuite`| End-to-end writes through the split plan and the native writer: parity with iceberg-java, row-level DML, partition evolution, file order, cleanup on task and job failure, AQE re-planning. |
402
-
|`CometIcebergWriteDetectionSuite`| One case per eligibility rule, accepted and declined. |
403
-
|`CometIcebergSystemFunctionSuite`| Native `bucket`, `truncate`, `years`/`months`/`days`/`hours`, which keep a partitioned write's hash distribution and sort native end to end. |
404
-
|`CometIcebergRewriteActionSuite`| Iceberg's `rewrite_data_files` with the split plan and the native writer. |
405
-
|`IcebergWriteProtoTranslationSuite`| Translation of properties into `IcebergParquetWriteSettings` and the writer mode. |
406
-
| Rust tests in `iceberg_write.rs` and in `iceberg_partition_*.rs`| File rolling on the 1000-row grid, fanout order, clustered input checks, cleanup guard, manifest round trip, partition path rendering, partition values past `chrono`'s calendar. |
407
-
| Rust tests in `iceberg_dictionary.rs`| The per-column dictionary choice against answers recorded from parquet-mr, including columns either side of its cut-off. |
408
-
|`CometIcebergWriteBenchmark`| Native versus iceberg-java for unpartitioned, clustered, fanout and copy-on-write delete writes. It checks each arm's plan before timing it. |
|`CometIcebergWriteActionSuite`| End-to-end writes through the split plan and the native writer: parity with iceberg-java, row-level DML, partition evolution, file order, cleanup on task and job failure, a fanout write outgrowing the memory pool, AQE re-planning. |
414
+
|`CometIcebergWriteDetectionSuite`| One case per eligibility rule, accepted and declined. |
415
+
|`CometIcebergSystemFunctionSuite`| Native `bucket`, `truncate`, `years`/`months`/`days`/`hours`, which keep a partitioned write's hash distribution and sort native end to end. |
416
+
|`CometIcebergRewriteActionSuite`| Iceberg's `rewrite_data_files` with the split plan and the native writer. |
417
+
|`IcebergWriteProtoTranslationSuite`| Translation of properties into `IcebergParquetWriteSettings` and the writer mode. |
418
+
| Rust tests in `iceberg_write.rs` and in `iceberg_partition_*.rs`| File rolling on the 1000-row grid, fanout order, clustered input checks, cleanup guard, manifest round trip, memory reservation, partition path rendering, partition values past `chrono`'s calendar.|
419
+
| Rust tests in `iceberg_dictionary.rs`| The per-column dictionary choice against answers recorded from parquet-mr, including columns either side of its cut-off. |
420
+
|`CometIcebergWriteBenchmark`| Native versus iceberg-java for unpartitioned, clustered, fanout and copy-on-write delete writes. It checks each arm's plan before timing it. |
409
421
410
422
The Comet suites run against the Iceberg version each Spark profile pins in `spark/pom.xml`: 1.5.2
411
423
for Spark 3.4, 1.8.1 for 3.5, 1.10.0 for 4.0 and 4.2, and 1.11.0 for 4.1. Only the default profile
0 commit comments