Skip to content

Commit e555dba

Browse files
authored
feat: make native Iceberg writes opt-in for 1.2.0 (#6962)
spark.comet.write.iceberg.enabled now defaults to false again, so 1.2.0 ships the native Iceberg writer as a supported, opt-in feature instead of turning it on for every eligible write. The rest of #6664 stays: the setting is in the query execution category, it plans the split operator and the native writer by itself, merge-on-read writes keep Spark's operator, and spark.comet.exec.enabled=false keeps Spark's write operator. The Iceberg Spark test diffs and both Iceberg write benchmarks set the setting themselves, so they still run the native writer. CometIcebergWriteActionSuite now checks that an application that sets neither write flag gets Spark's own operator, and that the write flag alone writes an eligible write natively under the split plan. The merge-on-read and transition-reversion tests set the write flag themselves. The user guide, FAQ, Gluten comparison, contributor guide, roadmap and Iceberg write review skill describe the native writer as opt-in, and the 1.2.0 upgrade-guide entry covers the move from 1.1.0's testing settings to the one supported setting.
1 parent b513921 commit e555dba

13 files changed

Lines changed: 135 additions & 114 deletions

File tree

‎.ai/skills/review-comet-iceberg-write-pr/SKILL.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -49,11 +49,11 @@ not, inherits it. There is no query that fails to tell you.
4949
| Native writer | `CometIcebergNativeWrite` (gate and serde), `IcebergWriteProtoTranslation`, `CometIcebergWriteExec`, `iceberg_write.rs` |
5050
| Shared with the scan | `iceberg_common.rs` (`load_file_io`, `storage_factory_for`, `scheme_of`), `IcebergReflection`, `NativeConfig` S3 translation |
5151

52-
`spark.comet.write.iceberg.enabled` switches both write layers and is on by default, so a change to
53-
the split plan affects every Iceberg write, including writes that never reach the native writer, and
54-
a change to the native writer affects every eligible one. The testing-only
55-
`spark.comet.write.iceberg.splitOperator.enabled` plans the split operator with the native writer
56-
off. A change to a shared file affects the native Iceberg scan too.
52+
`spark.comet.write.iceberg.enabled` switches both write layers. It is off by default, but once a
53+
user turns it on, a change to the split plan affects every Iceberg write, including writes that
54+
never reach the native writer, and a change to the native writer affects every eligible one. The
55+
testing-only `spark.comet.write.iceberg.splitOperator.enabled` plans the split operator with the
56+
native writer off. A change to a shared file affects the native Iceberg scan too.
5757

5858
## 2. Direction of the Gate Change
5959

‎docs/source/about/faq.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -141,9 +141,9 @@ Common causes of a small speedup are:
141141
### Does Comet support Delta Lake, Apache Hudi, or Apache Paimon tables?
142142

143143
Not yet. Comet accelerates [Apache Iceberg](../user-guide/latest/iceberg.md) tables, with a native
144-
reader and [native writes](../user-guide/latest/iceberg-writes.md) that are both enabled by default.
145-
It does not accelerate scans of Delta Lake, Hudi, or Paimon tables, so Spark reads those. Native
146-
Delta Lake reads are in development; see the
144+
reader that is enabled by default and opt-in
145+
[native writes](../user-guide/latest/iceberg-writes.md). It does not accelerate scans of Delta Lake,
146+
Hudi, or Paimon tables, so Spark reads those. Native Delta Lake reads are in development; see the
147147
[roadmap](../contributor-guide/roadmap.md#delta-lake-support). Hudi and Paimon are not on the
148148
roadmap.
149149

‎docs/source/about/gluten_comparison.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -118,10 +118,10 @@ covers a broader set of table formats overall.
118118
Comet provides a native Iceberg scan built on iceberg-rust. It has been tested with Iceberg 1.5 and 1.8 through 1.11
119119
and supports Iceberg spec v1, v2, and v3, schema evolution, time travel and branch reads, positional and equality
120120
deletes and deletion vectors, encrypted v3 tables, REST catalogs, and S3-compatible object storage. For Iceberg
121-
writes, Comet writes the data files of eligible writes natively through iceberg-rust by default, and iceberg-java
122-
writes the rest and commits every write (see [Comet Iceberg writes]). Comet does not currently provide native
123-
integrations for Delta Lake, Hudi, or Paimon. See the [Comet Iceberg guide] for the full list of supported features
124-
and known limitations.
121+
writes, Comet can write the data files of eligible writes natively through iceberg-rust, an opt-in feature, and
122+
iceberg-java writes the rest and commits every write (see [Comet Iceberg writes]). Comet does not currently provide
123+
native integrations for Delta Lake, Hudi, or Paimon. See the [Comet Iceberg guide] for the full list of supported
124+
features and known limitations.
125125

126126
[Comet Iceberg guide]: /user-guide/latest/iceberg.md
127127
[Comet Iceberg writes]: /user-guide/latest/iceberg-writes.md

‎docs/source/contributor-guide/iceberg-spark-tests.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,8 @@ Here is an overview of the changes that the diffs make to Iceberg:
3434
- Enable Comet's Iceberg write path, the split-operator plan (`IcebergCommit -> IcebergWrite`) and the native
3535
(iceberg-rust) Parquet writer, alongside the native scan in every Comet-configured session, so that Iceberg's
3636
write, commit, and row-level-operation tests run through it. `spark.comet.write.iceberg.enabled` switches both
37-
and is on by default; the diffs set it explicitly, together with the testing-only
37+
and is off by default for users, so these suites are where it runs against Iceberg's own tests. The diffs set
38+
it explicitly, together with the testing-only
3839
`spark.comet.write.iceberg.splitOperator.enabled`. That flag is redundant for the normal write path, but
3940
it is still what plans merge-on-read (`WriteDelta`) writes as the split plan on Spark 3.5+, so Iceberg's
4041
row-level-operation tests exercise that path. See [#5259]

‎docs/source/contributor-guide/iceberg-writes.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -37,11 +37,11 @@ Two layers, the second built on the first, both switched by `spark.comet.write.i
3737
| The split-operator plan | The plan shape. Spark's single V2 write operator becomes `IcebergCommit` over `IcebergWrite`, and iceberg-java writes the data files inside it. |
3838
| The native writer | Who writes the data files. An eligible `IcebergWrite` becomes `CometIcebergWrite`, which writes Parquet with iceberg-rust. |
3939

40-
The flag defaults to `true` since Comet 1.2.0, so a change to the split plan reaches every Iceberg
41-
write and a change to the native writer reaches every eligible one. Set to `false`, it plans Spark's
42-
own write operator. The testing-only `spark.comet.write.iceberg.splitOperator.enabled` plans the
43-
split operator with the native writer off, which the suites use to compare the two writers under
44-
the same plan. The rollout and its criteria are tracked in
40+
The flag defaults to `false`, which plans Spark's own write operator. With it on, a change to the
41+
split plan reaches every Iceberg write and a change to the native writer reaches every eligible one.
42+
The testing-only `spark.comet.write.iceberg.splitOperator.enabled` plans the split operator with the
43+
native writer off, which the suites use to compare the two writers under the same plan. The
44+
criteria for turning the flag on by default are tracked in
4545
[#5644](https://github.com/apache/datafusion-comet/issues/5644) under the epic
4646
[#5649](https://github.com/apache/datafusion-comet/issues/5649).
4747

‎docs/source/contributor-guide/roadmap.md‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -137,14 +137,14 @@ enabled by default ([#1625]).
137137

138138
## Iceberg Table Writes
139139

140-
Since Comet 1.2.0, Comet writes Iceberg tables natively by default, controlled by one setting,
141-
`spark.comet.write.iceberg.enabled` ([#5644]). Comet splits Spark's Iceberg V2 write operator into separate writer
142-
and committer operators, so the query feeding the write becomes visible to AQE and to Comet's columnar rules
143-
([#4658]), and delegates each task's Parquet write to `iceberg-rust` when the write passes an eligibility check
144-
([#5361]); writes that don't pass keep Iceberg Java's writer. Merge-on-read writes keep Spark's write operator
145-
until the native writer can write them ([#6240]). The remaining work toward the original goal of [#4322], an ETL job that runs end to end in native code, is
146-
tracked in [#5649]: correctness fixes, failure handling that matches Iceberg Java, and broader coverage. See
147-
[Iceberg Writes](iceberg-writes.md) for how the write path works.
140+
Comet can write Iceberg tables natively, controlled by one setting, `spark.comet.write.iceberg.enabled`, which is
141+
off by default. Comet splits Spark's Iceberg V2 write operator into separate writer and committer operators, so the
142+
query feeding the write becomes visible to AQE and to Comet's columnar rules ([#4658]), and delegates each task's
143+
Parquet write to `iceberg-rust` when the write passes an eligibility check ([#5361]); writes that don't pass keep
144+
Iceberg Java's writer. Merge-on-read writes keep Spark's write operator until the native writer can write them
145+
([#6240]). The remaining work toward the original goal of [#4322], an ETL job that runs end to end in native code,
146+
is tracked in [#5649]: correctness fixes, failure handling that matches Iceberg Java, broader coverage, and enabling
147+
the setting by default ([#5644]). See [Iceberg Writes](iceberg-writes.md) for how the write path works.
148148

149149
[#4322]: https://github.com/apache/datafusion-comet/issues/4322
150150
[#4658]: https://github.com/apache/datafusion-comet/pull/4658

‎docs/source/user-guide/latest/datasources.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,8 @@ Arrow format, allowing the Comet pipeline to take over after that, but the proce
2929

3030
### Apache Iceberg
3131

32-
Comet accelerates Iceberg scans of Parquet files and, since Comet 1.2.0, writes the data files of
33-
eligible Iceberg writes natively. See the [Iceberg Guide] and [Iceberg Writes](iceberg-writes.md)
32+
Comet accelerates Iceberg scans of Parquet files and can write the data files of eligible Iceberg
33+
writes natively, an opt-in feature. See the [Iceberg Guide] and [Iceberg Writes](iceberg-writes.md)
3434
for more information.
3535

3636
[iceberg guide]: iceberg.md

‎docs/source/user-guide/latest/iceberg-writes.md‎

Lines changed: 19 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -19,13 +19,15 @@
1919

2020
# Iceberg Writes: Comet's Split-Operator Plan and Native Writer
2121

22-
Since Comet 1.2.0, Comet plans Iceberg writes with the split-operator plan described below, and
23-
writes the data files of each eligible write natively with iceberg-rust. A write that is not
24-
eligible falls back to iceberg-java's writer. Natively written files differ from iceberg-java's in
25-
the ways listed under [Accepted divergences](#accepted-divergences).
22+
With `spark.comet.write.iceberg.enabled=true`, Comet plans Iceberg writes with the split-operator
23+
plan described below, and writes the data files of each eligible write natively with iceberg-rust.
24+
A write that is not eligible falls back to iceberg-java's writer. Natively written files differ
25+
from iceberg-java's in the ways listed under [Accepted divergences](#accepted-divergences).
2626

27-
`spark.comet.write.iceberg.enabled` switches both. Set it to `false` to plan Spark's own write
28-
operator instead, as Comet 1.1.0 did, so that iceberg-java writes every data file.
27+
The setting is off by default, so Spark plans its own write operator and iceberg-java writes every
28+
data file. Validate the native writer against your own workloads before enabling it, including
29+
the costs described under [Native Parquet write eligibility](#native-parquet-write-eligibility)
30+
and [Failure handling](#failure-handling).
2931

3032
## Overview
3133

@@ -37,8 +39,8 @@ data-file writing cannot be re-planned in response to how its input ran. And bec
3739
writing is bundled with the metadata and commit steps, there is no separate step for Comet to
3840
replace.
3941

40-
When `spark.comet.write.iceberg.enabled=true`, the default, Comet rewrites eligible Iceberg writes
41-
into two operators:
42+
When `spark.comet.write.iceberg.enabled=true`, Comet rewrites eligible Iceberg writes into two
43+
operators:
4244

4345
1. **`IcebergWrite`** — writes the data files on the executors, exactly as iceberg-java does
4446
today, and returns each task's serialized commit message. This operator and the sub-query
@@ -86,9 +88,8 @@ already has, even if they do not fill its first page.
8688

8789
## Configuration
8890

89-
Iceberg writes need only the standard Comet and Iceberg setup (see [`iceberg.md`](iceberg.md)).
90-
`spark.comet.write.iceberg.enabled` is on by default; set it to `false` to plan Spark's own write
91-
operator instead.
91+
Standard Comet + Iceberg setup (see [`iceberg.md`](iceberg.md)) plus the write setting, which is
92+
off by default:
9293

9394
```
9495
# Standard Comet / Iceberg wiring
@@ -98,6 +99,9 @@ spark.sql.catalog.<name>=org.apache.iceberg.spark.SparkCatalog
9899
spark.sql.catalog.<name>.type=hadoop # or hive / glue / rest / ...
99100
spark.sql.catalog.<name>.warehouse=...
100101
102+
# Split-operator plan and native Parquet writer
103+
spark.comet.write.iceberg.enabled=true
104+
101105
# Lets writes whose input is a local relation (INSERT ... VALUES, a local DataFrame) use the
102106
# native writer; see "Native Parquet write eligibility" below
103107
spark.comet.exec.localTableScan.enabled=true
@@ -139,7 +143,7 @@ changes.
139143

140144
The rewrite is skipped — and the write runs through Spark's stock combined operator — when:
141145

142-
- `spark.comet.write.iceberg.enabled` is set to `false` and the testing setting
146+
- `spark.comet.write.iceberg.enabled` is `false` (the default) and the testing setting
143147
`spark.comet.write.iceberg.splitOperator.enabled`, which Comet 1.1.0 documented as a user setting,
144148
is not on;
145149
- Comet is disabled (`spark.comet.enabled=false`), or its native execution is
@@ -160,7 +164,7 @@ trade-off, only no plan change.
160164

161165
## Native Parquet write eligibility
162166

163-
When `spark.comet.write.iceberg.enabled=true`, the default
167+
When `spark.comet.write.iceberg.enabled=true`
164168
([#5361](https://github.com/apache/datafusion-comet/pull/5361)), the `IcebergWrite` operator's
165169
per-task Parquet write is delegated to [iceberg-rust](https://github.com/apache/iceberg-rust).
166170
The native writer must produce the same outcome as iceberg-java — the same Parquet features,
@@ -376,9 +380,8 @@ unchanged.
376380
## Accepted divergences
377381

378382
Some differences between parquet-mr and the pinned parquet-rs / iceberg-rust are unconditional —
379-
they apply to every native write and cannot be configured away. Leaving
380-
`spark.comet.write.iceberg.enabled` at its default of `true` accepts them, and setting it to
381-
`false` avoids them. They fall into three classes with very
383+
they apply to every native write and cannot be configured away. Enabling
384+
`spark.comet.write.iceberg.enabled` accepts them. They fall into three classes with very
382385
different blast radius: differences confined to the physical bytes of a data file (cosmetic —
383386
no reader decision is based on them), differences visible in manifest metadata (these outlive
384387
the write and feed later readers' pruning decisions, so each one is analyzed individually

‎docs/source/user-guide/latest/iceberg.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -198,9 +198,10 @@ The following scenarios will fall back to the JVM Iceberg reader:
198198
change. The check uses the table's schema history, so the fallback stays after those files are
199199
rewritten
200200

201-
Writes are not covered by this list. By default Comet plans an Iceberg write with its split-operator
202-
plan and writes eligible data files natively; see [Iceberg Writes](iceberg-writes.md) for the plan,
203-
the native writer, and when each applies.
201+
Writes are not covered by this list. By default Iceberg writes use Spark's own writer. With
202+
`spark.comet.write.iceberg.enabled=true`, Comet plans an Iceberg write with its split-operator plan
203+
and writes eligible data files natively; see [Iceberg Writes](iceberg-writes.md) for the plan, the
204+
native writer, and when each applies.
204205

205206
### Iceberg UDFs
206207

‎docs/source/user-guide/latest/migration-guide.md‎

Lines changed: 17 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -88,42 +88,23 @@ operators, such as the scan of a Data Source V2 connector.
8888

8989
### Iceberg Writes
9090

91-
`spark.comet.write.iceberg.enabled` now defaults to `true`. Comet plans an Iceberg `INSERT INTO`,
92-
`INSERT OVERWRITE`, and copy-on-write `DELETE`, `UPDATE` or `MERGE` as two operators, `IcebergWrite`
93-
under `IcebergCommit`, in place of Spark's single V2 write operator, and writes the data files of
94-
each eligible write natively with iceberg-rust. A write that is not eligible still uses
95-
iceberg-java's writer, and iceberg-java still commits every write. The table holds the same rows,
96-
but natively written files differ from iceberg-java's in the ways listed under
97-
[Accepted divergences](iceberg-writes.md#accepted-divergences), and explain output and the Spark UI
98-
show the new operators. Set `spark.comet.write.iceberg.enabled=false` to plan Spark's own operator,
99-
as in Comet 1.1.0. Comet 1.1.0 named this setting `spark.comet.iceberg.write.enabled`, in the
100-
testing category, and Comet now ignores that name, so a deployment that set it to `false` gets the
101-
new default unless it sets `spark.comet.write.iceberg.enabled=false`.
102-
`spark.comet.write.iceberg.splitOperator.enabled` is now a testing setting, and setting it to
103-
`false` does not turn the split operator off. Comet 1.1.0 documented it as the switch for the split
104-
plan, so a deployment that still sets it to `true` keeps the split plan when
105-
`spark.comet.write.iceberg.enabled=false`, with iceberg-java writing the data files, and on Spark
106-
3.5+ it also plans merge-on-read writes as the split plan. Unset that setting as well as setting
107-
`spark.comet.write.iceberg.enabled=false` to plan Spark's own operator. See
108-
[Iceberg Writes](iceberg-writes.md).
109-
110-
To compute the Iceberg metrics of a natively written file, Comet reads the file's Parquet footer
111-
back from storage, where iceberg-java's writer takes them from memory. On S3 or GCS that is two GET
112-
requests per file, which adds up for a write that produces many small files. See
113-
[Native Parquet write eligibility](iceberg-writes.md#native-parquet-write-eligibility).
114-
115-
The native writer's buffers count against Comet's off-heap memory pool, where iceberg-java's buffers
116-
sit on the JVM heap. A fanout write keeps a data file open for every partition a task writes to, and
117-
each open file holds the row group it is writing, so a task that writes to many partitions can need
118-
more memory than the pool grants it. The task then writes out and closes the partitions holding the
119-
most memory, whose next rows open new files, so the write finishes with more, smaller files than
120-
iceberg-java's would. Disabling the fanout writer (`write.spark.fanout.enabled=false`) or a larger
121-
`spark.memory.offHeap.size` avoids the extra files. A write that keeps one file open, unpartitioned
122-
or clustered, has no partition to close, so when that file outgrows the pool the task fails with
123-
`Additional allocation failed for IcebergWriteExec`. A smaller `write.parquet.row-group-size-bytes`
124-
or a larger `spark.memory.offHeap.size` lets such a write fit. With
125-
`spark.comet.write.iceberg.enabled=false`, iceberg-java writes both, as before. See
126-
[Failure handling](iceberg-writes.md#failure-handling).
91+
Comet's Iceberg write path is now switched by one setting, `spark.comet.write.iceberg.enabled`,
92+
which stays off by default. With it on, Comet plans an Iceberg `INSERT INTO`, `INSERT OVERWRITE`,
93+
and copy-on-write `DELETE`, `UPDATE` or `MERGE` as two operators, `IcebergWrite` under
94+
`IcebergCommit`, in place of Spark's single V2 write operator, and writes the data files of each
95+
eligible write natively with iceberg-rust. A write that is not eligible uses iceberg-java's writer,
96+
and iceberg-java commits every write. Before turning it on, see [Iceberg Writes](iceberg-writes.md)
97+
for how natively written files differ from iceberg-java's, the footer reads the native path makes on
98+
S3 or GCS, and how the native writer draws on Comet's off-heap memory pool.
99+
100+
Comet 1.1.0 named this setting `spark.comet.iceberg.write.enabled`, in the testing category, and it
101+
took effect only together with `spark.comet.write.iceberg.splitOperator.enabled`. Comet now ignores
102+
the old name, so a deployment that turned native writes on that way must set
103+
`spark.comet.write.iceberg.enabled=true` to keep them.
104+
`spark.comet.write.iceberg.splitOperator.enabled` is now a testing setting. Comet 1.1.0 documented
105+
it as the switch for the split plan, and a deployment that still sets it to `true` keeps the split
106+
plan, with iceberg-java writing the data files, and on Spark 3.5+ it now also plans merge-on-read
107+
writes as the split plan. Unset it to plan Spark's own operator.
127108

128109
## Upgrading to Comet 1.1.0
129110

0 commit comments

Comments
 (0)