Skip to content

Commit 8c783aa

Browse files
authored
feat: charge native write buffers to the task's memory pool (#6247)
* feat: charge native write buffers to the task's memory pool Neither native writer reserved what it buffers, so a native Parquet or Iceberg write held its row groups in memory that no budget accounted for. A fanout Iceberg write multiplies that by the number of partitions a task writes, and when it outgrew the executor's memory overhead the container was killed. Both writers now register one memory consumer per task and resize it after every batch. ParquetWriterExec reserves parquet-rs's estimate of its in-progress row group. The Iceberg writer wraps iceberg-rust's ParquetWriterBuilder so that every open file reports what it holds, capped at the row-group size because iceberg-rust exposes only the bytes written plus the in-progress row group, and adds the rows its pacers hold back. Neither writer can spill, so a resize the pool refuses fails the task with a native out-of-memory error that Spark can retry. Closes #5648. Closes #6115. * fix: release an Iceberg file's reservation once its row groups leave memory A data file's share was its written size capped at one row group, so every fanout file that had flushed stayed charged a whole row group until it closed. A local file keeps nothing it has flushed, so a pool of eight 128 KiB row groups failed a twelve-partition write whose files held almost nothing. Each file's OutputFile is now wrapped so that its writer counts the bytes that leave memory on their way to storage, and the file reports what it has written less those. A local file releases a row group as it is handed on. S3 and GCS do not: OpenDAL collects what a file flushes into parts of at least 5 MiB and holds each part until the next one is complete or the file closes, so a file there keeps what it has flushed since its last upload charged.
1 parent 45e91be commit 8c783aa

11 files changed

Lines changed: 1330 additions & 38 deletions

File tree

‎.ai/skills/review-comet-iceberg-write-pr/SKILL.md‎

Lines changed: 18 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -152,6 +152,14 @@ Read the ownership table in the contributor guide before reviewing any change ne
152152
`SparkWrite.abort` keeps working.
153153
- [ ] The user-visible exception type is still what Spark's own write path throws on each Spark
154154
version ([#6143](https://github.com/apache/datafusion-comet/issues/6143)).
155+
- [ ] **Every open file and every held-back row is reserved.** Files must be opened through
156+
`MeteredParquetWriterBuilder`, and rows the writer holds outside iceberg-rust (the
157+
`PartitionFeed`s, or anything a new feed holds back) must be added to what `run_write_task` reserves. A
158+
builder that constructs `ParquetWriterBuilder` directly leaves its files out of the task's
159+
memory reservation, so a wide fanout write grows past the pool instead of failing its task.
160+
A storage scheme newly supported for writes needs its entry in
161+
`StorageWrites::for_location`: a file reports its flushed row groups until its storage has
162+
written them out, which a local file does at once and an object store only part by part.
155163

156164
## 5. Plan Shape
157165

@@ -189,16 +197,16 @@ Read the ownership table in the contributor guide before reviewing any change ne
189197

190198
## 7. Tests
191199

192-
| Suite | Covers |
193-
| ----------------------------------- | ------------------------------------------------------------------------------------ |
194-
| `CometIcebergWriteActionSuite` | Both layers end to end, parity with iceberg-java, DML, failure cleanup, AQE re-plans |
195-
| `CometIcebergWriteDetectionSuite` | One case per eligibility rule |
196-
| `CometIcebergSystemFunctionSuite` | Native partition transforms keeping partitioned writes native |
197-
| `CometIcebergRewriteActionSuite` | `rewrite_data_files` through the split plan and the native writer |
198-
| `IcebergWriteProtoTranslationSuite` | Property to `IcebergParquetWriteSettings` translation |
199-
| Rust tests in `iceberg_write.rs` | Rolling, fanout order, clustered checks, cleanup guard, manifest round trip |
200-
| `iceberg_partition_path.rs` tests | Partition path rendering against iceberg-java |
201-
| `CometIcebergWriteBenchmark` | Native versus iceberg-java throughput |
200+
| Suite | Covers |
201+
| ----------------------------------- | ----------------------------------------------------------------------------------------------- |
202+
| `CometIcebergWriteActionSuite` | Both layers end to end, parity with iceberg-java, DML, failure cleanup, AQE re-plans |
203+
| `CometIcebergWriteDetectionSuite` | One case per eligibility rule |
204+
| `CometIcebergSystemFunctionSuite` | Native partition transforms keeping partitioned writes native |
205+
| `CometIcebergRewriteActionSuite` | `rewrite_data_files` through the split plan and the native writer |
206+
| `IcebergWriteProtoTranslationSuite` | Property to `IcebergParquetWriteSettings` translation |
207+
| Rust tests in `iceberg_write.rs` | Rolling, fanout order, clustered checks, cleanup guard, manifest round trip, memory reservation |
208+
| `iceberg_partition_path.rs` tests | Partition path rendering against iceberg-java |
209+
| `CometIcebergWriteBenchmark` | Native versus iceberg-java throughput |
202210

203211
Ask specifically:
204212

‎docs/source/contributor-guide/iceberg-writes.md‎

Lines changed: 23 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -272,10 +272,22 @@ Points where Comet adapts iceberg-rust to match iceberg-java:
272272
([#5776](https://github.com/apache/datafusion-comet/issues/5776)). Manifest order becomes the row
273273
order of an unordered read, so any map iteration that reaches the output needs the same care.
274274

275+
iceberg-rust keeps each open file writer private inside its rolling and partitioning writers, so
276+
Comet wraps `ParquetWriterBuilder` in `MeteredParquetWriterBuilder`, whose files report what they
277+
hold in memory. It hands `ParquetWriter` each file's `OutputFile` behind a `CountedOutput`, whose
278+
writer counts the bytes that leave memory on their way to storage, and a file reports what it has
279+
written less those. When they leave depends on the storage (`StorageWrites`). After every batch
280+
`run_write_task` resizes the task's reservation to what the open files report plus the rows each
281+
`PartitionFeed` holds, for the dictionary choice or for pacing, and a resize the pool refuses fails
282+
the task. What that figure covers, and what it misses, is described under
283+
[Native writers](memory_management.md#native-writers).
284+
275285
`FileIO` comes from `load_file_io` in `iceberg_common.rs`, shared with the native scan. It picks
276286
the storage backend from the data location's scheme and wires in Comet's S3 credential bridge when
277287
one is configured. For writes the bridge fails closed: if a configured provider cannot initialize,
278-
the task fails rather than writing with the default credential chain.
288+
the task fails rather than writing with the default credential chain. A storage scheme newly
289+
supported for writes also needs its entry in `StorageWrites::for_location`, which treats an unknown
290+
scheme as one that uploads in parts.
279291

280292
The JVM eligibility gate fails closed before building that `FileIO` for an `s3` or `s3a` data
281293
location. It checks effective `fs.s3a.*` Hadoop settings against the six keys translated by
@@ -396,16 +408,16 @@ the writer: run the write suites and the Iceberg Spark tests. The pin policy is
396408

397409
## Testing
398410

399-
| Suite | What it covers |
400-
| ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
401-
| `CometIcebergWriteActionSuite` | End-to-end writes through the split plan and the native writer: parity with iceberg-java, row-level DML, partition evolution, file order, cleanup on task and job failure, AQE re-planning. |
402-
| `CometIcebergWriteDetectionSuite` | One case per eligibility rule, accepted and declined. |
403-
| `CometIcebergSystemFunctionSuite` | Native `bucket`, `truncate`, `years`/`months`/`days`/`hours`, which keep a partitioned write's hash distribution and sort native end to end. |
404-
| `CometIcebergRewriteActionSuite` | Iceberg's `rewrite_data_files` with the split plan and the native writer. |
405-
| `IcebergWriteProtoTranslationSuite` | Translation of properties into `IcebergParquetWriteSettings` and the writer mode. |
406-
| Rust tests in `iceberg_write.rs` and in `iceberg_partition_*.rs` | File rolling on the 1000-row grid, fanout order, clustered input checks, cleanup guard, manifest round trip, partition path rendering, partition values past `chrono`'s calendar. |
407-
| Rust tests in `iceberg_dictionary.rs` | The per-column dictionary choice against answers recorded from parquet-mr, including columns either side of its cut-off. |
408-
| `CometIcebergWriteBenchmark` | Native versus iceberg-java for unpartitioned, clustered, fanout and copy-on-write delete writes. It checks each arm's plan before timing it. |
411+
| Suite | What it covers |
412+
| ---------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
413+
| `CometIcebergWriteActionSuite` | End-to-end writes through the split plan and the native writer: parity with iceberg-java, row-level DML, partition evolution, file order, cleanup on task and job failure, a fanout write outgrowing the memory pool, AQE re-planning. |
414+
| `CometIcebergWriteDetectionSuite` | One case per eligibility rule, accepted and declined. |
415+
| `CometIcebergSystemFunctionSuite` | Native `bucket`, `truncate`, `years`/`months`/`days`/`hours`, which keep a partitioned write's hash distribution and sort native end to end. |
416+
| `CometIcebergRewriteActionSuite` | Iceberg's `rewrite_data_files` with the split plan and the native writer. |
417+
| `IcebergWriteProtoTranslationSuite` | Translation of properties into `IcebergParquetWriteSettings` and the writer mode. |
418+
| Rust tests in `iceberg_write.rs` and in `iceberg_partition_*.rs` | File rolling on the 1000-row grid, fanout order, clustered input checks, cleanup guard, manifest round trip, memory reservation, partition path rendering, partition values past `chrono`'s calendar. |
419+
| Rust tests in `iceberg_dictionary.rs` | The per-column dictionary choice against answers recorded from parquet-mr, including columns either side of its cut-off. |
420+
| `CometIcebergWriteBenchmark` | Native versus iceberg-java for unpartitioned, clustered, fanout and copy-on-write delete writes. It checks each arm's plan before timing it. |
409421

410422
The Comet suites run against the Iceberg version each Spark profile pins in `spark/pom.xml`: 1.5.2
411423
for Spark 3.4, 1.8.1 for 3.5, 1.10.0 for 4.0 and 4.2, and 1.11.0 for 4.1. Only the default profile

‎docs/source/contributor-guide/memory_management.md‎

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -375,6 +375,40 @@ Native operators reserve through DataFusion's `MemoryConsumer` / `MemoryReservat
375375

376376
An operator that never calls `try_grow` is invisible to the pool no matter how much memory it uses.
377377

378+
### Native writers
379+
380+
Both native writers reserve what they hold between batches through a single consumer per task,
381+
`ParquetWriterExec[N]` or `IcebergWriteExec[N]`, resized after every batch. Neither can spill, so
382+
when the pool refuses a resize the task fails with a `CometNativeException` whose message starts
383+
`Additional allocation failed for` and names the consumer. That is a task failure Spark can retry.
384+
Unreserved, the same memory would count only toward the container limit, where exceeding it kills
385+
the executor.
386+
387+
- `ParquetWriterExec` writes one file per task and reserves parquet-rs's estimate of the file's
388+
in-progress row group (`ArrowWriter::memory_size`). The estimate counts encoded pages, encoder
389+
buffers, dictionaries and Bloom filters. The writer keeps parquet-rs's default row-group limit of
390+
1Mi rows and sets no byte limit, so a wide schema can hold a large row group
391+
([#5304](https://github.com/apache/datafusion-comet/issues/5304)).
392+
- The Iceberg writer keeps one file open per partition in a fanout write, so its reservation grows
393+
with the number of partitions a task writes. iceberg-rust's `ParquetWriter` does not expose
394+
parquet-rs's `memory_size`, only every byte the file has written, its in-progress row group's
395+
encoded size included. So each file's writer counts the bytes that leave memory on their way to
396+
storage, and the file reports what it has written less those. A local file writes a flushed row
397+
group out at once, so it reports its in-progress row group and the few KiB parquet-rs buffers in
398+
front of storage. S3 and GCS take a file in parts of at least 5 MiB, and OpenDAL holds each part
399+
in memory until the next one is complete or the file closes, so there a file also reports what
400+
it has flushed since its last part was uploaded. At the default row-group size
401+
(`write.parquet.row-group-size-bytes`, 128 MiB) that is the last row group it flushed. The
402+
reservation also covers the rows each partition holds back, first for its dictionary choice and
403+
then until they fill the 1000-row unit the rolling writer is fed in. It does not cover what
404+
parquet-rs holds beyond the encoded size: dictionary hash tables, unencoded dictionary indices
405+
and buffer capacity. Native writes decline Bloom filters today, and neither figure would include
406+
them.
407+
408+
The writers register one consumer per task rather than one per open file because every consumer
409+
registered with `fair_unified` lowers the share of every other consumer in the task. A consumer per
410+
partition would shrink the task's other operators' shares as a fanout write widened.
411+
378412
## Crossing the FFI boundary
379413

380414
Batches move between the JVM and native over the Arrow C Data and C Stream interfaces, which are
@@ -425,7 +459,8 @@ The pool tracks _declared reservations_. Container RSS counts _pages the process
425459
diverge for several structural reasons:
426460

427461
- **Undeclared allocations.** Arrow array builders, expression kernels producing intermediate
428-
arrays, decompression buffers, Parquet metadata structures, `object_store` request buffers, and
462+
arrays, decompression buffers, Parquet metadata structures, the part of a native writer's buffers
463+
its estimate misses (see [Native writers](#native-writers)), `object_store` request buffers, and
429464
tokio's own machinery all allocate without reserving. Only operators that were explicitly written
430465
to reserve show up in the pool.
431466
- **Rounding and padding.** Arrow buffers are padded to 64-byte boundaries and builders grow by

‎docs/source/user-guide/latest/iceberg-writes.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -299,6 +299,18 @@ failure), the error propagates as an ordinary Spark task failure and Spark's tas
299299
re-executes it — through the native writer again. Retries cannot collide: each attempt's task
300300
attempt id is embedded in its data file names.
301301

302+
The native writer's buffers are charged to Comet's memory pool, the off-heap budget Comet's other
303+
native operators draw on, where iceberg-java's buffers sit on the JVM heap. A fanout write keeps a
304+
data file open for every partition a task writes to. Each open file holds the row group it is
305+
writing in memory, up to `write.parquet.row-group-size-bytes`, and on S3 or GCS also the last row
306+
group it flushed, which is uploaded once the next one is complete or the file closes. So a task
307+
writing to many partitions needs memory in proportion to them. When the pool cannot grant it, the
308+
task fails with a `CometNativeException` reading `Additional allocation failed for IcebergWriteExec`
309+
instead of exceeding the executor's memory, and Spark retries it like any other task failure. Such a
310+
write fits in less memory with the fanout writer disabled (`write.spark.fanout.enabled=false`):
311+
Spark then sorts each task's rows by partition, and the task keeps one file open at a time. A
312+
smaller row-group size also helps. Otherwise the write needs a larger `spark.memory.offHeap.size`.
313+
302314
Partial results are never committed. The commit set is exactly the commit messages returned by
303315
successful tasks — a failed task contributes none — and if the job fails, the driver-side
304316
commit operator aborts without committing anything. A failed task attempt also deletes the

‎native/core/src/execution/memory_pools/mod.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,8 @@ mod plan_pool;
2222
mod spark_memory;
2323
mod spill_replay;
2424
mod task_shared;
25+
#[cfg(test)]
26+
pub(crate) mod testing;
2527
mod unified_pool;
2628

2729
use datafusion::execution::memory_pool::{MemoryPool, TrackConsumersPool, UnboundedMemoryPool};
Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
//! Memory pools for tests of operators that reserve memory.
19+
20+
use std::fmt;
21+
use std::sync::atomic::{AtomicUsize, Ordering};
22+
23+
use datafusion::common::Result;
24+
use datafusion::execution::memory_pool::{GreedyMemoryPool, MemoryPool, MemoryReservation};
25+
26+
/// A [`GreedyMemoryPool`] that also records the most it ever had reserved, so a test can check
27+
/// what an operator reserved while it ran, not only that it gave everything back.
28+
#[derive(Debug)]
29+
pub(crate) struct PeakMemoryPool {
30+
inner: GreedyMemoryPool,
31+
peak: AtomicUsize,
32+
}
33+
34+
impl PeakMemoryPool {
35+
pub(crate) fn new(limit: usize) -> Self {
36+
Self {
37+
inner: GreedyMemoryPool::new(limit),
38+
peak: AtomicUsize::new(0),
39+
}
40+
}
41+
42+
pub(crate) fn peak(&self) -> usize {
43+
self.peak.load(Ordering::Relaxed)
44+
}
45+
46+
fn record(&self) {
47+
self.peak
48+
.fetch_max(self.inner.reserved(), Ordering::Relaxed);
49+
}
50+
}
51+
52+
impl fmt::Display for PeakMemoryPool {
53+
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
54+
write!(f, "PeakMemoryPool({}, peak {})", self.inner, self.peak())
55+
}
56+
}
57+
58+
impl MemoryPool for PeakMemoryPool {
59+
fn name(&self) -> &str {
60+
"PeakMemoryPool"
61+
}
62+
63+
fn grow(&self, reservation: &MemoryReservation, additional: usize) {
64+
self.inner.grow(reservation, additional);
65+
self.record();
66+
}
67+
68+
fn shrink(&self, reservation: &MemoryReservation, shrink: usize) {
69+
self.inner.shrink(reservation, shrink);
70+
}
71+
72+
fn try_grow(&self, reservation: &MemoryReservation, additional: usize) -> Result<()> {
73+
self.inner.try_grow(reservation, additional)?;
74+
self.record();
75+
Ok(())
76+
}
77+
78+
fn reserved(&self) -> usize {
79+
self.inner.reserved()
80+
}
81+
}

0 commit comments

Comments
 (0)