@@ -135,11 +135,15 @@ requests per file, which adds up for a write that produces many small files. See
135135The native writer's buffers count against Comet's off-heap memory pool, where iceberg-java's buffers
136136sit on the JVM heap. A fanout write keeps a data file open for every partition a task writes to, and
137137each open file holds the row group it is writing, so a task that writes to many partitions can need
138- more memory than the pool grants it. The task then fails with
139- ` Additional allocation failed for IcebergWriteExec ` . Disabling the fanout writer
140- (` write.spark.fanout.enabled=false ` ), a smaller ` write.parquet.row-group-size-bytes ` or a larger
141- ` spark.memory.offHeap.size ` lets such a write fit, and ` spark.comet.write.iceberg.enabled=false `
142- writes it with iceberg-java as before. See [ Failure handling] ( iceberg-writes.md#failure-handling ) .
138+ more memory than the pool grants it. The task then writes out and closes the partitions holding the
139+ most memory, whose next rows open new files, so the write finishes with more, smaller files than
140+ iceberg-java's would. Disabling the fanout writer (` write.spark.fanout.enabled=false ` ) or a larger
141+ ` spark.memory.offHeap.size ` avoids the extra files. A write that keeps one file open, unpartitioned
142+ or clustered, has no partition to close, so when that file outgrows the pool the task fails with
143+ ` Additional allocation failed for IcebergWriteExec ` . A smaller ` write.parquet.row-group-size-bytes `
144+ or a larger ` spark.memory.offHeap.size ` lets such a write fit. With
145+ ` spark.comet.write.iceberg.enabled=false ` , iceberg-java writes both, as before. See
146+ [ Failure handling] ( iceberg-writes.md#failure-handling ) .
143147
144148## Upgrading to Comet 1.1.0
145149
0 commit comments