Skip to content

Commit bc4be39

Browse files
authored
docs: add 1.1.0 upgrade notes for fixed, deprecated and removed settings (#6248)
The 1.1.0 section of the upgrade guide said only that no legacy key is needed and that JDK 17 is required. Several changes since 1.0.0 need no legacy key but can still change what an existing deployment does: - #6191 reads spark.comet.shuffle.native.writeBufferSize, spark.comet.maxTempDirectorySize and spark.memory.offHeap.size in the units they are written in. - #6195 disables Comet on the spark.sql.extensions path when off-heap memory is disabled, and checks the shuffle manager that is running. - #6163 deprecates spark.comet.exec.memoryPool.fraction. - #6066 removes three settings that applied only to on-heap mode. Add an entry for each, naming the setting that restores the old effect where there is one, and let the guide's introduction cover changes like these alongside the ones that need a legacy key. The fraction and spark.comet.exec.sortMergeJoinWithJoinFilter.enabled said they "will be removed in a future release", but the versioning policy removes a covered key only in a major release. Say that instead.
1 parent f7952de commit bc4be39

3 files changed

Lines changed: 82 additions & 13 deletions

File tree

‎docs/source/user-guide/latest/migration-guide.md‎

Lines changed: 71 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,11 @@ A **behavior change** is one where the same query, run over the same data, with
2727
set configuration, produces a different result or a different error than it did in the previous
2828
release. Comet's [versioning policy](../../about/versioning_policy.md) permits these in a minor
2929
release only when a `spark.comet.legacy.*` configuration key restores the previous behavior, so
30-
every entry below names such a key.
30+
every behavior change below names such a key.
31+
32+
A release's section can also list changes that need no legacy key but can still change what an
33+
existing deployment does, such as a fix that makes Comet apply a setting as documented, or a setting
34+
that was deprecated or removed. Each of these entries names the settings involved.
3135

3236
Two kinds of change are deliberately absent from this guide:
3337

@@ -55,11 +59,76 @@ key is removed.
5559

5660
## Upgrading to Comet 1.1.0
5761

58-
Comet `1.1.0` makes no behavior changes that need a `spark.comet.legacy.*` key.
62+
Comet `1.1.0` makes no behavior changes that need a `spark.comet.legacy.*` key. The changes below
63+
need none either, but check whether any of them applies to your deployment.
5964

6065
Comet `1.1.0` requires JDK 17 or later. JDK 11 is no longer supported. See
6166
[Installing Comet](installation.md) for the supported Java, Scala, and Spark versions.
6267

68+
### Settings That Now Take Effect as Documented
69+
70+
Comet `1.0.0` misread three size settings. Comet `1.1.0` reads each of them as documented, so a job
71+
that sets one of them can behave differently after the upgrade. In each case, a setting that
72+
already exists restores the old effect.
73+
74+
- `spark.comet.shuffle.native.writeBufferSize` was read in MiB but used as a number of bytes, so
75+
the native shuffle writer ran with a 1-byte write buffer by default, and a value of `64m` gave it
76+
64 bytes. The setting is now read in bytes with a default of 1 MiB, and a value with a unit means
77+
what it says. Each shuffle task holds a few of these buffers in native memory that no memory pool
78+
tracks, so check any value you set with a unit against `spark.executor.memoryOverhead`. A bare
79+
number keeps its old meaning, so writing the number without its unit restores the old buffer
80+
size. Values of 2 GiB or more are now rejected.
81+
- `spark.comet.maxTempDirectorySize` was ignored when it was written with a unit, such as `10g`, and
82+
the 100 GB default applied instead. It is now enforced, so a query that spills more than that
83+
amount now fails. Remove the setting to keep the old limit. See
84+
[Limiting Spill Disk Usage](tuning/memory.md#limiting-spill-disk-usage).
85+
- `spark.memory.offHeap.size` was read as MiB when it was written as a bare number of bytes. The
86+
`fair_unified` memory pool's per-operator shares were therefore about a million times too large,
87+
and never limited an operator. The shares are now correct, so operators can spill sooner, and an
88+
operator that cannot spill can fail when it exceeds its share. A size written with a unit, such
89+
as `16g`, is unaffected. To get the old behavior back, set
90+
`spark.comet.exec.memoryPool=greedy_unified`, which leaves every limit to Spark. See
91+
[Configuring Comet Memory](tuning/memory.md#configuring-comet-memory).
92+
93+
A malformed value of `spark.comet.maxTempDirectorySize` or `spark.comet.explain.native.enabled` now
94+
fails the query instead of being replaced by the default. `spark.comet.debug.enabled`,
95+
`spark.comet.explain.native.enabled` and `spark.comet.tracing.enabled` now also take effect in
96+
Comet's native code when they are written in upper case, such as `TRUE`.
97+
98+
### Conditions for Enabling Comet
99+
100+
Comet needs Spark's off-heap memory to be enabled. `CometPlugin` already disabled Comet when
101+
off-heap memory was disabled, but an application that registered `CometSparkSessionExtensions`
102+
directly with `spark.sql.extensions` skipped that check, and Comet ran in on-heap mode, which exists
103+
only for running tests. Comet `1.1.0` makes the same check on that path, and disables itself with a
104+
warning when off-heap memory is not enabled. To keep using Comet, set
105+
`spark.memory.offHeap.enabled=true` and `spark.memory.offHeap.size` when the application starts;
106+
see [Configuring Comet Memory](tuning/memory.md#configuring-comet-memory). Both checks read
107+
`spark.memory.offHeap.enabled` from the SparkContext, so setting it on a `SparkSession.builder`
108+
after the SparkContext exists has no effect.
109+
110+
Comet also now checks the shuffle manager that the application is running, rather than the
111+
session's `spark.shuffle.manager`. A session that named `CometShuffleManager` after the SparkContext
112+
had started with a different shuffle manager used to plan Comet shuffles that failed with a
113+
`ClassCastException`. Such a session now runs without Comet, with a warning.
114+
115+
### Deprecated and Removed Settings
116+
117+
`spark.comet.exec.memoryPool.fraction` is deprecated and will be removed in a future major release.
118+
It was documented as leaving room in `spark.memory.offHeap.size` for the native memory that Comet's
119+
memory pools do not track, but it cannot: Spark hands out the whole off-heap pool whatever it is set
120+
to. It keeps working as before, and the driver now logs a warning when it is set. Size
121+
`spark.executor.memoryOverhead` for that memory instead; see
122+
[Configuring Executor Memory Overhead](tuning/memory.md#configuring-executor-memory-overhead).
123+
124+
Comet `1.1.0` removes `spark.comet.memoryOverhead`, `spark.comet.exec.onHeap.memoryPool` and
125+
`spark.comet.shuffle.jvm.memoryFactor`, including its older name
126+
`spark.comet.columnar.shuffle.memory.factor`. They applied only to on-heap mode
127+
(`spark.comet.exec.onHeap.enabled`), which exists for running Spark's SQL tests against Comet and no
128+
longer tracks native memory at all. They were in the testing category, which the
129+
[versioning policy](../../about/versioning_policy.md#testing-and-internal-configurations-are-exempt)
130+
exempts, and Comet ignores them if they are still set.
131+
63132
## Upgrading to Comet 1.0.0
64133

65134
Comet `1.0.0` is the first release under the stable

‎spark/src/main/scala/org/apache/comet/CometConf.scala‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -336,7 +336,7 @@ object CometConf extends ShimCometConf {
336336
conf("spark.comet.exec.sortMergeJoinWithJoinFilter.enabled")
337337
.category(CATEGORY_ENABLE_EXEC)
338338
.doc("Support for Sort Merge Join with filter. " +
339-
"Deprecated: this config will be removed in a future release.")
339+
"Deprecated: this config will be removed in a future major release.")
340340
.booleanConf
341341
.createWithDefault(true)
342342

@@ -916,12 +916,12 @@ object CometConf extends ShimCometConf {
916916
conf("spark.comet.exec.memoryPool.fraction")
917917
.category(CATEGORY_TUNING)
918918
.doc(
919-
"Deprecated: this config will be removed in a future release. It does not leave room " +
920-
"in spark.memory.offHeap.size for native memory that Comet's memory pools do not " +
921-
"track, because Spark hands out the whole off-heap pool whatever this is set to. Size " +
922-
"spark.executor.memoryOverhead for that memory instead. Only applies to off-heap " +
923-
"mode, where the fair_unified pool limits each memory consumer in a task to this " +
924-
"fraction of the off-heap size divided by the task's consumers, and the " +
919+
"Deprecated: this config will be removed in a future major release. It does not " +
920+
"leave room in spark.memory.offHeap.size for native memory that Comet's memory " +
921+
"pools do not track, because Spark hands out the whole off-heap pool whatever this " +
922+
"is set to. Size spark.executor.memoryOverhead for that memory instead. Only applies " +
923+
"to off-heap mode, where the fair_unified pool limits each memory consumer in a task " +
924+
"to this fraction of the off-heap size divided by the task's consumers, and the " +
925925
s"greedy_unified pool ignores it. $TUNING_GUIDE.")
926926
.doubleConf
927927
.createWithDefault(1.0)

‎spark/src/main/scala/org/apache/spark/Plugins.scala‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -215,10 +215,10 @@ object CometDriverPlugin extends Logging {
215215
val key = CometConf.COMET_OFFHEAP_MEMORY_POOL_FRACTION.key
216216
conf.getOption(key).foreach { value =>
217217
logWarning(
218-
s"$key=$value is deprecated and will be removed in a future release. It does not leave " +
219-
"room in spark.memory.offHeap.size for native memory that Comet's memory pools do " +
220-
"not track, because Spark hands out the whole off-heap pool whatever it is set to. " +
221-
s"Size ${EXECUTOR_MEMORY_OVERHEAD.key} for that memory instead. " +
218+
s"$key=$value is deprecated and will be removed in a future major release. It does " +
219+
"not leave room in spark.memory.offHeap.size for native memory that Comet's memory " +
220+
"pools do not track, because Spark hands out the whole off-heap pool whatever it is " +
221+
s"set to. Size ${EXECUTOR_MEMORY_OVERHEAD.key} for that memory instead. " +
222222
s"${CometConf.TUNING_GUIDE}.")
223223
}
224224
}

0 commit comments

Comments
 (0)