@@ -56,24 +56,19 @@ disabled by default**, and it only engages under fairly strict conditions.
5656### Splitting the write operator
5757
5858Spark writes an Iceberg table with a single operator that writes the data files, writes the metadata, commits,
59- and validates against the catalog. Adaptive Query Execution (AQE) re-plans the query feeding that operator, but
60- the operator itself sits outside AQE, so the file writing can't adapt to how its input actually ran. And because
61- file writing is bundled with the metadata and commit steps, there is no separate piece for Comet to take over.
59+ and validates against the catalog. Because file writing is bundled with the metadata and commit steps, there was
60+ no separate piece for Comet to replace.
6261
6362Setting ` spark.comet.write.iceberg.splitOperator.enabled=true ` splits eligible Iceberg writes into two operators:
6463
65- 1 . ** ` IcebergWrite ` ** writes data files on the executors and returns each task's commit message. When AQE is
66- enabled, as it is by default, it runs inside AQE along with the query feeding it. The split also works with
67- AQE disabled.
68- 2 . ** ` IcebergCommit ` ** collects the commit messages on the driver and performs the normal Iceberg commit, once,
69- outside AQE.
64+ 1 . ** ` IcebergWrite ` ** writes data files on the executors and returns each task's commit message.
65+ 2 . ** ` IcebergCommit ` ** collects the commit messages on the driver and performs the normal Iceberg commit, once.
7066
71- On its own, the split doesn't change who writes the files; that's still iceberg-java. It moves file writing inside
72- AQE and separates it from the commit, which gives the native writer a place to plug in. It covers ` INSERT INTO `
73- and DataFrame ` append ` , static and dynamic ` INSERT OVERWRITE ` , and copy-on-write ` DELETE ` , ` UPDATE ` , and ` MERGE ` ,
74- on every supported Spark version. Merge-on-read writes are left alone. When Comet can't split a write (an
75- unrecognized write class, CTAS on Spark 3.4, or a write that needs Spark's commit coordinator), it plans the write
76- exactly as if Comet weren't there.
67+ With only the split enabled, iceberg-java still writes the files; the native writer, described next, replaces
68+ that step. The split covers ` INSERT INTO ` and DataFrame ` append ` , static and dynamic ` INSERT OVERWRITE ` , and
69+ copy-on-write ` DELETE ` , ` UPDATE ` , and ` MERGE ` , on every supported Spark version. Merge-on-read writes are left
70+ alone. When Comet can't split a write (an unrecognized write class, CTAS on Spark 3.4, or a write that needs
71+ Spark's commit coordinator), it plans the write exactly as if Comet weren't there.
7772
7873### Writing Parquet natively
7974
@@ -95,10 +90,10 @@ To see which path a write took, check the physical plan: a native write shows `C
9590
9691### Matching iceberg-java
9792
98- The goal is the table iceberg-java would have written, not just a valid one . Manifest metrics drive pruning for
99- every future reader, so rather than trust the native writer's numbers, Comet recomputes them with iceberg-java's
100- own code. Parity tests write the same rows through both writers and compare the committed counts and bounds. The
101- cost is one small footer read per file.
93+ The native writer aims to write the same table iceberg-java would, down to the metadata . Manifest metrics drive
94+ pruning for every future reader, so rather than trust the native writer's numbers, Comet recomputes them with
95+ iceberg-java's own code. Parity tests write the same rows through both writers and compare the committed counts
96+ and bounds. The cost is one small footer read per file.
10297
10398Eligibility is an allowlist. A write goes native only if its whole configuration matches the documented set of
10499supported settings. Anything else, including properties added by future Iceberg versions, falls back to
0 commit comments