Skip to content

Commit 1929e05

Browse files
committed
Say the split applies to Iceberg writes and that the native writer requires it
1 parent d345015 commit 1929e05

1 file changed

Lines changed: 5 additions & 5 deletions

File tree

‎content/blog/2026-10-01-datafusion-comet-1.1.0.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,7 @@ and validates against the catalog. Adaptive Query Execution (AQE) re-plans the q
6060
the operator itself sits outside AQE, so the file writing can't adapt to how its input actually ran. And because
6161
file writing is bundled with the metadata and commit steps, there is no separate piece for Comet to take over.
6262

63-
Setting `spark.comet.write.iceberg.splitOperator.enabled=true` splits eligible writes into two operators:
63+
Setting `spark.comet.write.iceberg.splitOperator.enabled=true` splits eligible Iceberg writes into two operators:
6464

6565
1. **`IcebergWrite`** writes data files on the executors and returns each task's commit message. When AQE is
6666
enabled, as it is by default, it runs inside AQE along with the query feeding it. The split also works with
@@ -77,10 +77,10 @@ exactly as if Comet weren't there.
7777

7878
### Writing Parquet natively
7979

80-
Also setting `spark.comet.iceberg.write.enabled=true` hands each task's Parquet writing to iceberg-rust. The
81-
driver passes the native writer everything it needs: schema, partition spec, data location, Parquet settings,
82-
writer mode, and object-store configuration. Each task writes its files and returns their metadata as an
83-
in-memory Iceberg manifest.
80+
The native writer requires the split. Also setting `spark.comet.iceberg.write.enabled=true` hands each task's
81+
Parquet writing to iceberg-rust. The driver passes the native writer everything it needs: schema, partition spec,
82+
data location, Parquet settings, writer mode, and object-store configuration. Each task writes its files and
83+
returns their metadata as an in-memory Iceberg manifest.
8484

8585
From there, iceberg-java takes over again. The JVM reads the manifest, recomputes each file's metrics from its
8686
Parquet footer, and builds the same commit message the JVM writer would have produced. Snapshots, manifest

0 commit comments

Comments
 (0)