Skip to content

Commit cb90a1b

Browse files
committed
docs: explain the 1.1.0 split flag in the Iceberg writes upgrade guide
A deployment that still sets spark.comet.write.iceberg.splitOperator.enabled=true keeps the split plan when spark.comet.write.iceberg.enabled=false. Say so in the upgrade entry and the fallback list, add a test that pins the combination, and note that the flag still plans merge-on-read writes in the Iceberg Spark tests.
1 parent 9e5a8be commit cb90a1b

4 files changed

Lines changed: 41 additions & 3 deletions

File tree

‎docs/source/contributor-guide/iceberg-spark-tests.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,9 @@ Here is an overview of the changes that the diffs make to Iceberg:
3535
(iceberg-rust) Parquet writer, alongside the native scan in every Comet-configured session, so that Iceberg's
3636
write, commit, and row-level-operation tests run through it. `spark.comet.write.iceberg.enabled` switches both
3737
and is on by default; the diffs set it explicitly, together with the testing-only
38-
`spark.comet.write.iceberg.splitOperator.enabled`, which it makes redundant. See [#5259]
38+
`spark.comet.write.iceberg.splitOperator.enabled`. That flag is redundant for the normal write path, but
39+
it is still what plans merge-on-read (`WriteDelta`) writes as the split plan on Spark 3.5+, so Iceberg's
40+
row-level-operation tests exercise that path. See [#5259]
3941
- Enable `spark.comet.exec.localTableScan.enabled` in the same sessions. `CometIcebergNativeWrite` sets
4042
`requiresNativeChildren`, so without this flag a write fed by an inline `VALUES` list keeps Spark's row-based
4143
`LocalTableScanExec`, the conversion is declined, and the write silently runs on the JVM writer. Many Iceberg

‎docs/source/user-guide/latest/iceberg-writes.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -139,7 +139,9 @@ changes.
139139

140140
The rewrite is skipped — and the write runs through Spark's stock combined operator — when:
141141

142-
- `spark.comet.write.iceberg.enabled` is set to `false`;
142+
- `spark.comet.write.iceberg.enabled` is set to `false` and the testing setting
143+
`spark.comet.write.iceberg.splitOperator.enabled`, which Comet 1.1.0 documented as a user setting,
144+
is not on;
143145
- Comet is disabled (`spark.comet.enabled=false`), or its native execution is
144146
(`spark.comet.exec.enabled=false`);
145147
- Comet is in plan-only mode (`spark.comet.explain.planOnly.enabled=true`);

‎docs/source/user-guide/latest/migration-guide.md‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,12 @@ as in Comet 1.1.0. Comet 1.1.0 named this setting `spark.comet.iceberg.write.ena
125125
testing category, and Comet now ignores that name, so a deployment that set it to `false` gets the
126126
new default unless it sets `spark.comet.write.iceberg.enabled=false`.
127127
`spark.comet.write.iceberg.splitOperator.enabled` is now a testing setting, and setting it to
128-
`false` does not turn the split operator off. See [Iceberg Writes](iceberg-writes.md).
128+
`false` does not turn the split operator off. Comet 1.1.0 documented it as the switch for the split
129+
plan, so a deployment that still sets it to `true` keeps the split plan when
130+
`spark.comet.write.iceberg.enabled=false`, with iceberg-java writing the data files, and on Spark
131+
3.5+ it also plans merge-on-read writes as the split plan. Unset that setting as well as setting
132+
`spark.comet.write.iceberg.enabled=false` to plan Spark's own operator. See
133+
[Iceberg Writes](iceberg-writes.md).
129134

130135
To compute the Iceberg metrics of a natively written file, Comet reads the file's Parquet footer
131136
back from storage, where iceberg-java's writer takes them from memory. On S3 or GCS that is two GET

‎spark/src/test/scala/org/apache/comet/CometIcebergWriteActionSuite.scala‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -148,6 +148,35 @@ class CometIcebergWriteActionSuite
148148
}
149149
}
150150

151+
// Comet 1.1.0 documented the split flag as a user setting, so a deployment can carry it into 1.2.0.
152+
// With it on, `spark.comet.write.iceberg.enabled=false` only turns the native writer off: the split
153+
// plan stays, with iceberg-java writing the files. The upgrade guide tells such deployments to
154+
// unset the split flag to get Spark's own operator back.
155+
test(
156+
"the split flag keeps the split plan with iceberg-java's writer when the write flag is off") {
157+
assume(icebergAvailable, "Iceberg not available in classpath")
158+
withIcebergCatalog { _ =>
159+
createTable("split_flag_left_on", partitionSpec = "")
160+
val snapshot = captureWrite("split_flag_left_on") {
161+
withSessionConf(
162+
CometConf.COMET_ICEBERG_WRITE_SPLIT_OPERATOR_ENABLED.key -> Some("true"),
163+
CometConf.COMET_ICEBERG_NATIVE_WRITE_ENABLED.key -> Some("false")) {
164+
spark.sql(
165+
"INSERT INTO cat.db.split_flag_left_on VALUES " +
166+
"(1, 'us-east', 10.5), (2, 'us-west', 20.3), (3, 'eu', 30.7)")
167+
}
168+
}
169+
assertExactlyOneCommit(snapshot)
170+
val nativeWrites = snapshot.plans.flatMap { plan =>
171+
collectWithSubqueries(plan) { case e: CometIcebergWriteExec => e }
172+
}
173+
assert(
174+
nativeWrites.isEmpty,
175+
"expected iceberg-java's IcebergWriteExec. Plans:\n" + snapshot.plans.mkString("\n--\n"))
176+
assertRows("split_flag_left_on", expectedIds = Seq(1, 2, 3))
177+
}
178+
}
179+
151180
// The suite pins both write flags, so this test unsets them to see what an application that
152181
// sets neither gets. A Parquet scan is a native input, so the write is eligible.
153182
// https://github.com/apache/datafusion-comet/issues/5644

0 commit comments

Comments
 (0)