@@ -27,7 +27,11 @@ A **behavior change** is one where the same query, run over the same data, with
2727set configuration, produces a different result or a different error than it did in the previous
2828release. Comet's [ versioning policy] ( ../../about/versioning_policy.md ) permits these in a minor
2929release only when a ` spark.comet.legacy.* ` configuration key restores the previous behavior, so
30- every entry below names such a key.
30+ every behavior change below names such a key.
31+
32+ A release's section can also list changes that need no legacy key but can still change what an
33+ existing deployment does, such as a fix that makes Comet apply a setting as documented, or a setting
34+ that was deprecated or removed. Each of these entries names the settings involved.
3135
3236Two kinds of change are deliberately absent from this guide:
3337
@@ -55,11 +59,76 @@ key is removed.
5559
5660## Upgrading to Comet 1.1.0
5761
58- Comet ` 1.1.0 ` makes no behavior changes that need a ` spark.comet.legacy.* ` key.
62+ Comet ` 1.1.0 ` makes no behavior changes that need a ` spark.comet.legacy.* ` key. The changes below
63+ need none either, but check whether any of them applies to your deployment.
5964
6065Comet ` 1.1.0 ` requires JDK 17 or later. JDK 11 is no longer supported. See
6166[ Installing Comet] ( installation.md ) for the supported Java, Scala, and Spark versions.
6267
68+ ### Settings That Now Take Effect as Documented
69+
70+ Comet ` 1.0.0 ` misread three size settings. Comet ` 1.1.0 ` reads each of them as documented, so a job
71+ that sets one of them can behave differently after the upgrade. In each case, a setting that
72+ already exists restores the old effect.
73+
74+ - ` spark.comet.shuffle.native.writeBufferSize ` was read in MiB but used as a number of bytes, so
75+ the native shuffle writer ran with a 1-byte write buffer by default, and a value of ` 64m ` gave it
76+ 64 bytes. The setting is now read in bytes with a default of 1 MiB, and a value with a unit means
77+ what it says. Each shuffle task holds a few of these buffers in native memory that no memory pool
78+ tracks, so check any value you set with a unit against ` spark.executor.memoryOverhead ` . A bare
79+ number keeps its old meaning, so writing the number without its unit restores the old buffer
80+ size. Values of 2 GiB or more are now rejected.
81+ - ` spark.comet.maxTempDirectorySize ` was ignored when it was written with a unit, such as ` 10g ` , and
82+ the 100 GB default applied instead. It is now enforced, so a query that spills more than that
83+ amount now fails. Remove the setting to keep the old limit. See
84+ [ Limiting Spill Disk Usage] ( tuning/memory.md#limiting-spill-disk-usage ) .
85+ - ` spark.memory.offHeap.size ` was read as MiB when it was written as a bare number of bytes. The
86+ ` fair_unified ` memory pool's per-operator shares were therefore about a million times too large,
87+ and never limited an operator. The shares are now correct, so operators can spill sooner, and an
88+ operator that cannot spill can fail when it exceeds its share. A size written with a unit, such
89+ as ` 16g ` , is unaffected. To get the old behavior back, set
90+ ` spark.comet.exec.memoryPool=greedy_unified ` , which leaves every limit to Spark. See
91+ [ Configuring Comet Memory] ( tuning/memory.md#configuring-comet-memory ) .
92+
93+ A malformed value of ` spark.comet.maxTempDirectorySize ` or ` spark.comet.explain.native.enabled ` now
94+ fails the query instead of being replaced by the default. ` spark.comet.debug.enabled ` ,
95+ ` spark.comet.explain.native.enabled ` and ` spark.comet.tracing.enabled ` now also take effect in
96+ Comet's native code when they are written in upper case, such as ` TRUE ` .
97+
98+ ### Conditions for Enabling Comet
99+
100+ Comet needs Spark's off-heap memory to be enabled. ` CometPlugin ` already disabled Comet when
101+ off-heap memory was disabled, but an application that registered ` CometSparkSessionExtensions `
102+ directly with ` spark.sql.extensions ` skipped that check, and Comet ran in on-heap mode, which exists
103+ only for running tests. Comet ` 1.1.0 ` makes the same check on that path, and disables itself with a
104+ warning when off-heap memory is not enabled. To keep using Comet, set
105+ ` spark.memory.offHeap.enabled=true ` and ` spark.memory.offHeap.size ` when the application starts;
106+ see [ Configuring Comet Memory] ( tuning/memory.md#configuring-comet-memory ) . Both checks read
107+ ` spark.memory.offHeap.enabled ` from the SparkContext, so setting it on a ` SparkSession.builder `
108+ after the SparkContext exists has no effect.
109+
110+ Comet also now checks the shuffle manager that the application is running, rather than the
111+ session's ` spark.shuffle.manager ` . A session that named ` CometShuffleManager ` after the SparkContext
112+ had started with a different shuffle manager used to plan Comet shuffles that failed with a
113+ ` ClassCastException ` . Such a session now runs without Comet, with a warning.
114+
115+ ### Deprecated and Removed Settings
116+
117+ ` spark.comet.exec.memoryPool.fraction ` is deprecated and will be removed in a future major release.
118+ It was documented as leaving room in ` spark.memory.offHeap.size ` for the native memory that Comet's
119+ memory pools do not track, but it cannot: Spark hands out the whole off-heap pool whatever it is set
120+ to. It keeps working as before, and the driver now logs a warning when it is set. Size
121+ ` spark.executor.memoryOverhead ` for that memory instead; see
122+ [ Configuring Executor Memory Overhead] ( tuning/memory.md#configuring-executor-memory-overhead ) .
123+
124+ Comet ` 1.1.0 ` removes ` spark.comet.memoryOverhead ` , ` spark.comet.exec.onHeap.memoryPool ` and
125+ ` spark.comet.shuffle.jvm.memoryFactor ` , including its older name
126+ ` spark.comet.columnar.shuffle.memory.factor ` . They applied only to on-heap mode
127+ (` spark.comet.exec.onHeap.enabled ` ), which exists for running Spark's SQL tests against Comet and no
128+ longer tracks native memory at all. They were in the testing category, which the
129+ [ versioning policy] ( ../../about/versioning_policy.md#testing-and-internal-configurations-are-exempt )
130+ exempts, and Comet ignores them if they are still set.
131+
63132## Upgrading to Comet 1.0.0
64133
65134Comet ` 1.0.0 ` is the first release under the stable
0 commit comments