Skip to content

Commit 9e5a8be

Browse files
committed
docs: describe fanout writes closing partitions in the 1.2.0 upgrade guide
1 parent 5eba983 commit 9e5a8be

1 file changed

Lines changed: 9 additions & 5 deletions

File tree

‎docs/source/user-guide/latest/migration-guide.md‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -135,11 +135,15 @@ requests per file, which adds up for a write that produces many small files. See
135135
The native writer's buffers count against Comet's off-heap memory pool, where iceberg-java's buffers
136136
sit on the JVM heap. A fanout write keeps a data file open for every partition a task writes to, and
137137
each open file holds the row group it is writing, so a task that writes to many partitions can need
138-
more memory than the pool grants it. The task then fails with
139-
`Additional allocation failed for IcebergWriteExec`. Disabling the fanout writer
140-
(`write.spark.fanout.enabled=false`), a smaller `write.parquet.row-group-size-bytes` or a larger
141-
`spark.memory.offHeap.size` lets such a write fit, and `spark.comet.write.iceberg.enabled=false`
142-
writes it with iceberg-java as before. See [Failure handling](iceberg-writes.md#failure-handling).
138+
more memory than the pool grants it. The task then writes out and closes the partitions holding the
139+
most memory, whose next rows open new files, so the write finishes with more, smaller files than
140+
iceberg-java's would. Disabling the fanout writer (`write.spark.fanout.enabled=false`) or a larger
141+
`spark.memory.offHeap.size` avoids the extra files. A write that keeps one file open, unpartitioned
142+
or clustered, has no partition to close, so when that file outgrows the pool the task fails with
143+
`Additional allocation failed for IcebergWriteExec`. A smaller `write.parquet.row-group-size-bytes`
144+
or a larger `spark.memory.offHeap.size` lets such a write fit. With
145+
`spark.comet.write.iceberg.enabled=false`, iceberg-java writes both, as before. See
146+
[Failure handling](iceberg-writes.md#failure-handling).
143147

144148
## Upgrading to Comet 1.1.0
145149

0 commit comments

Comments
 (0)