You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit e555dba
Browse filesBrowse the repository at this point in the historyBrowse files
feat: make native Iceberg writes opt-in for 1.2.0 (#6962)
spark.comet.write.iceberg.enabled now defaults to false again, so 1.2.0
ships the native Iceberg writer as a supported, opt-in feature instead
of turning it on for every eligible write. The rest of #6664 stays: the
setting is in the query execution category, it plans the split operator
and the native writer by itself, merge-on-read writes keep Spark's
operator, and spark.comet.exec.enabled=false keeps Spark's write
operator.
The Iceberg Spark test diffs and both Iceberg write benchmarks set the
setting themselves, so they still run the native writer.
CometIcebergWriteActionSuite now checks that an application that sets
neither write flag gets Spark's own operator, and that the write flag
alone writes an eligible write natively under the split plan. The
merge-on-read and transition-reversion tests set the write flag
themselves.
The user guide, FAQ, Gluten comparison, contributor guide, roadmap and
Iceberg write review skill describe the native writer as opt-in, and
the 1.2.0 upgrade-guide entry covers the move from 1.1.0's testing
settings to the one supported setting.
Copy file name to clipboardExpand all lines: docs/source/contributor-guide/iceberg-writes.md
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -37,11 +37,11 @@ Two layers, the second built on the first, both switched by `spark.comet.write.i
37
37
| The split-operator plan | The plan shape. Spark's single V2 write operator becomes `IcebergCommit` over `IcebergWrite`, and iceberg-java writes the data files inside it. |
38
38
| The native writer | Who writes the data files. An eligible `IcebergWrite` becomes `CometIcebergWrite`, which writes Parquet with iceberg-rust. |
39
39
40
-
The flag defaults to `true` since Comet 1.2.0, so a change to the split plan reaches every Iceberg
41
-
write and a change to the native writer reaches every eligible one. Set to `false`, it plans Spark's
42
-
own write operator. The testing-only `spark.comet.write.iceberg.splitOperator.enabled` plans the
43
-
split operator with the native writer off, which the suites use to compare the two writers under
44
-
the same plan. The rollout and its criteria are tracked in
40
+
The flag defaults to `false`, which plans Spark's own write operator. With it on, a change to the
41
+
split plan reaches every Iceberg write and a change to the native writer reaches every eligible one.
42
+
The testing-only `spark.comet.write.iceberg.splitOperator.enabled` plans the split operator with the
43
+
native writer off, which the suites use to compare the two writers under the same plan. The
44
+
criteria for turning the flag on by default are tracked in
45
45
[#5644](https://github.com/apache/datafusion-comet/issues/5644) under the epic
until the native writer can write them ([#6240]). The remaining work toward the original goal of [#4322], an ETL job that runs end to end in native code, is
146
-
tracked in [#5649]: correctness fixes, failure handling that matches Iceberg Java, and broader coverage. See
147
-
[Iceberg Writes](iceberg-writes.md) for how the write path works.
140
+
Comet can write Iceberg tables natively, controlled by one setting,`spark.comet.write.iceberg.enabled`, which is
141
+
off by default. Comet splits Spark's Iceberg V2 write operator into separate writer and committer operators, so the
142
+
query feeding the write becomes visible to AQE and to Comet's columnar rules ([#4658]), and delegates each task's
143
+
Parquet write to `iceberg-rust` when the write passes an eligibility check ([#5361]); writes that don't pass keep
144
+
Iceberg Java's writer. Merge-on-read writes keep Spark's write operator until the native writer can write them
145
+
([#6240]). The remaining work toward the original goal of [#4322], an ETL job that runs end to end in native code,
146
+
is tracked in [#5649]: correctness fixes, failure handling that matches Iceberg Java, broader coverage, and enabling
147
+
the setting by default ([#5644]). See [Iceberg Writes](iceberg-writes.md) for how the write path works.
spark.sql.catalog.<name>.type=hadoop # or hive / glue / rest / ...
99
100
spark.sql.catalog.<name>.warehouse=...
100
101
102
+
# Split-operator plan and native Parquet writer
103
+
spark.comet.write.iceberg.enabled=true
104
+
101
105
# Lets writes whose input is a local relation (INSERT ... VALUES, a local DataFrame) use the
102
106
# native writer; see "Native Parquet write eligibility" below
103
107
spark.comet.exec.localTableScan.enabled=true
@@ -139,7 +143,7 @@ changes.
139
143
140
144
The rewrite is skipped — and the write runs through Spark's stock combined operator — when:
141
145
142
-
-`spark.comet.write.iceberg.enabled` is set to `false` and the testing setting
146
+
-`spark.comet.write.iceberg.enabled` is `false` (the default) and the testing setting
143
147
`spark.comet.write.iceberg.splitOperator.enabled`, which Comet 1.1.0 documented as a user setting,
144
148
is not on;
145
149
- Comet is disabled (`spark.comet.enabled=false`), or its native execution is
@@ -160,7 +164,7 @@ trade-off, only no plan change.
160
164
161
165
## Native Parquet write eligibility
162
166
163
-
When `spark.comet.write.iceberg.enabled=true`, the default
167
+
When `spark.comet.write.iceberg.enabled=true`
164
168
([#5361](https://github.com/apache/datafusion-comet/pull/5361)), the `IcebergWrite` operator's
165
169
per-task Parquet write is delegated to [iceberg-rust](https://github.com/apache/iceberg-rust).
166
170
The native writer must produce the same outcome as iceberg-java — the same Parquet features,
@@ -376,9 +380,8 @@ unchanged.
376
380
## Accepted divergences
377
381
378
382
Some differences between parquet-mr and the pinned parquet-rs / iceberg-rust are unconditional —
379
-
they apply to every native write and cannot be configured away. Leaving
380
-
`spark.comet.write.iceberg.enabled` at its default of `true` accepts them, and setting it to
381
-
`false` avoids them. They fall into three classes with very
383
+
they apply to every native write and cannot be configured away. Enabling
384
+
`spark.comet.write.iceberg.enabled` accepts them. They fall into three classes with very
382
385
different blast radius: differences confined to the physical bytes of a data file (cosmetic —
383
386
no reader decision is based on them), differences visible in manifest metadata (these outlive
384
387
the write and feed later readers' pruning decisions, so each one is analyzed individually
0 commit comments