Skip to content

Commit fd051bc

Browse files
committed
Address @viirya's review of the Comet 1.1.0 post
Drop the AQE framing from the split-operator section, which raised more questions than it answered, and explain the split by what matters for the native writer: file writing was bundled with metadata and commit. Say plainly that iceberg-java still writes the files with only the split enabled, and reword "not just a valid one".
1 parent 1929e05 commit fd051bc

1 file changed

Lines changed: 13 additions & 18 deletions

File tree

‎content/blog/2026-10-01-datafusion-comet-1.1.0.md‎

Lines changed: 13 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -56,24 +56,19 @@ disabled by default**, and it only engages under fairly strict conditions.
5656
### Splitting the write operator
5757

5858
Spark writes an Iceberg table with a single operator that writes the data files, writes the metadata, commits,
59-
and validates against the catalog. Adaptive Query Execution (AQE) re-plans the query feeding that operator, but
60-
the operator itself sits outside AQE, so the file writing can't adapt to how its input actually ran. And because
61-
file writing is bundled with the metadata and commit steps, there is no separate piece for Comet to take over.
59+
and validates against the catalog. Because file writing is bundled with the metadata and commit steps, there was
60+
no separate piece for Comet to replace.
6261

6362
Setting `spark.comet.write.iceberg.splitOperator.enabled=true` splits eligible Iceberg writes into two operators:
6463

65-
1. **`IcebergWrite`** writes data files on the executors and returns each task's commit message. When AQE is
66-
enabled, as it is by default, it runs inside AQE along with the query feeding it. The split also works with
67-
AQE disabled.
68-
2. **`IcebergCommit`** collects the commit messages on the driver and performs the normal Iceberg commit, once,
69-
outside AQE.
64+
1. **`IcebergWrite`** writes data files on the executors and returns each task's commit message.
65+
2. **`IcebergCommit`** collects the commit messages on the driver and performs the normal Iceberg commit, once.
7066

71-
On its own, the split doesn't change who writes the files; that's still iceberg-java. It moves file writing inside
72-
AQE and separates it from the commit, which gives the native writer a place to plug in. It covers `INSERT INTO`
73-
and DataFrame `append`, static and dynamic `INSERT OVERWRITE`, and copy-on-write `DELETE`, `UPDATE`, and `MERGE`,
74-
on every supported Spark version. Merge-on-read writes are left alone. When Comet can't split a write (an
75-
unrecognized write class, CTAS on Spark 3.4, or a write that needs Spark's commit coordinator), it plans the write
76-
exactly as if Comet weren't there.
67+
With only the split enabled, iceberg-java still writes the files; the native writer, described next, replaces
68+
that step. The split covers `INSERT INTO` and DataFrame `append`, static and dynamic `INSERT OVERWRITE`, and
69+
copy-on-write `DELETE`, `UPDATE`, and `MERGE`, on every supported Spark version. Merge-on-read writes are left
70+
alone. When Comet can't split a write (an unrecognized write class, CTAS on Spark 3.4, or a write that needs
71+
Spark's commit coordinator), it plans the write exactly as if Comet weren't there.
7772

7873
### Writing Parquet natively
7974

@@ -95,10 +90,10 @@ To see which path a write took, check the physical plan: a native write shows `C
9590

9691
### Matching iceberg-java
9792

98-
The goal is the table iceberg-java would have written, not just a valid one. Manifest metrics drive pruning for
99-
every future reader, so rather than trust the native writer's numbers, Comet recomputes them with iceberg-java's
100-
own code. Parity tests write the same rows through both writers and compare the committed counts and bounds. The
101-
cost is one small footer read per file.
93+
The native writer aims to write the same table iceberg-java would, down to the metadata. Manifest metrics drive
94+
pruning for every future reader, so rather than trust the native writer's numbers, Comet recomputes them with
95+
iceberg-java's own code. Parity tests write the same rows through both writers and compare the committed counts
96+
and bounds. The cost is one small footer read per file.
10297

10398
Eligibility is an allowlist. A write goes native only if its whole configuration matches the documented set of
10499
supported settings. Anything else, including properties added by future Iceberg versions, falls back to

0 commit comments

Comments
 (0)