Skip to content

Commit 3819297

Browse files
committed
address review: keep Scala 2.12 coverage, drop remaining Spark 3.4 references, add 1.3.0 upgrade note
1 parent 1a5f85e commit 3819297

9 files changed

Lines changed: 22 additions & 13 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -605,7 +605,7 @@ jobs:
605605
needs: changes
606606
# Queue-only by default; PRs need the `run-iceberg-tests` label. Iceberg
607607
# 1.11 is our only Spark 4.1 Iceberg coverage, which is why it is the one
608-
# Iceberg version the queue runs while 1.8/1.9/1.10 run nightly.
608+
# Iceberg version the queue runs while 1.9/1.10 run nightly.
609609
if: needs.changes.outputs.iceberg_1_11 == 'true'
610610
uses: ./.github/workflows/iceberg_spark_test_reusable.yml
611611
with:

‎dev/ci/linux-test-profiles.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@
4646
# `run-all-spark-profiles` label). The goal of the list is coverage of every
4747
# Java, Scala and Spark version without testing every combination.
4848
PROFILES = [
49+
{"name": "Spark 3.5, JDK 17, Scala 2.12", "java_version": "17", "maven_opts": "-Pspark-3.5 -Pscala-2.12", "tier": "nightly"},
4950
{"name": "Spark 3.5, JDK 17, Scala 2.13", "java_version": "17", "maven_opts": "-Pspark-3.5 -Pscala-2.13", "tier": "nightly"},
5051
{"name": "Spark 4.0, JDK 21", "java_version": "21", "maven_opts": "-Pspark-4.0", "tier": "nightly"},
5152
# The default build profile, and the one a contributor builds locally.

‎dev/regenerate-golden-files.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -128,7 +128,7 @@ main() {
128128

129129
# Validate target version if specified
130130
if [ -n "$target_version" ]; then
131-
if [[ ! "$target_version" =~ ^(3\.4|3\.5|4\.0|4\.1|4\.2)$ ]]; then
131+
if [[ ! "$target_version" =~ ^(3\.5|4\.0|4\.1|4\.2)$ ]]; then
132132
echo "[ERROR] Invalid Spark version: $target_version"
133133
echo "[ERROR] Supported versions: 3.5, 4.0, 4.1, 4.2"
134134
exit 1

‎docs/source/about/faq.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ not documented, please [file an issue](https://github.com/apache/datafusion-come
5252

5353
### Which Spark and Java versions does Comet support?
5454

55-
Spark 3.4, 3.5, 4.0, 4.1, and 4.2, on Java 17 or later. Spark 3.4 is deprecated. The
55+
Spark 3.5, 4.0, 4.1, and 4.2, on Java 17 or later. The
5656
[installation guide](../user-guide/latest/installation.md#supported-spark-versions) lists the exact
5757
Spark, Java, and Scala versions, and the
5858
[Spark Version Compatibility](../user-guide/latest/compatibility/spark-versions.md) page lists known

‎docs/source/about/gluten_comparison.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -84,7 +84,7 @@ the community behind each engine alongside the technical differences.
8484

8585
Both projects target a similar set of Spark releases.
8686

87-
Comet supports Spark 3.4, 3.5, 4.0, 4.1, and 4.2 in production builds. See the
87+
Comet supports Spark 3.5, 4.0, 4.1, and 4.2 in production builds. See the
8888
[Spark version compatibility guide] for the exact patch versions and JDK/Scala combinations.
8989

9090
[Spark version compatibility guide]: /user-guide/latest/compatibility/spark-versions.md

‎docs/source/contributor-guide/ci.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -298,7 +298,7 @@ re-armed automatically.
298298
`ci.yml` also runs on a schedule, at 06:00 UTC every day, against what landed on `main` since the
299299
last successful scheduled run. That run executes only the nightly tier: the Comet test suites against the Spark
300300
profiles other than 4.1, the Spark SQL suites for Spark 3.5, 4.0 and 4.2, the Iceberg suites for
301-
1.8, 1.9 and 1.10, and the Delta contrib suites for Spark 3.5, 4.0 and 4.1, each only when the day's
301+
1.9 and 1.10, and the Delta contrib suites for Spark 3.5, 4.0 and 4.1, each only when the day's
302302
changes touched files it covers. The queue already ran
303303
everything else against the same tree, so nothing in the queue tier is repeated. A day with no
304304
merges, or with only documentation changes, runs nothing.

‎docs/source/contributor-guide/iceberg-writes.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -454,8 +454,8 @@ the writer: run the write suites and the Iceberg Spark tests. The pin policy is
454454
| `CometIcebergWriteBenchmark` | Native versus iceberg-java for unpartitioned, clustered, fanout and copy-on-write delete writes. It checks each arm's plan before timing it. |
455455
| `CometIcebergEndToEndBenchmark` | An Iceberg read, hash repartition and Iceberg write against stock Spark, over primitive, struct, array, map and nested columns. It checks each arm's plan and the rows it wrote before timing it. |
456456

457-
The Comet suites run against the Iceberg version each Spark profile pins in `spark/pom.xml`: 1.5.2
458-
for Spark 3.4, 1.8.1 for 3.5, 1.10.0 for 4.0, and 1.11.0 for 4.1. No Iceberg runtime is published
457+
The Comet suites run against the Iceberg version each Spark profile pins in `spark/pom.xml`: 1.8.1
458+
for Spark 3.5, 1.10.0 for 4.0, and 1.11.0 for 4.1. No Iceberg runtime is published
459459
for Spark 4.2, so the 4.2 profile builds against the 4.0 runtime and the Iceberg suites skip there
460460
(`icebergAvailable` is false on Spark 4.2). Only the default profile runs on every pull request.
461461
Behaviour that differs between Iceberg versions, such as reflection targets, metrics conventions

‎docs/source/user-guide/latest/iceberg.md‎

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -651,9 +651,8 @@ normal `remove_orphan_files` maintenance.
651651
When one task fails, the tasks that had already completed leave committed-nothing data files
652652
too. The committer collects each task's commit message as that task finishes, so on a job
653653
failure it aborts with the completed messages and deletes their data files through the table
654-
`FileIO`. (From Iceberg 1.6 on, Iceberg's own `SparkWrite.abort` skips cleanup unless a commit
655-
failed with a cleanable error, so on the stock path those files are left for
656-
`remove_orphan_files`. Iceberg 1.5, which the Spark 3.4 profile pins, deletes them.) A
654+
`FileIO`. (Iceberg's own `SparkWrite.abort` skips cleanup unless a commit failed with a
655+
cleanable error, so on the stock path those files are left for `remove_orphan_files`.) A
657656
failure during the driver-side commit itself behaves exactly as on the stock path: the commit
658657
messages carry genuine `SparkWrite$TaskCommit` objects, so Iceberg's own `SparkWrite.abort`
659658
cleanup (which deletes the files listed in the commit messages for cleanable failures) applies
@@ -707,9 +706,8 @@ a data file but not what any reader computes from it:
707706
per task. The native writer zero-pads the task attempt id and counts from `00000`; iceberg-java
708707
leaves the id unpadded and counts from `00001`.
709708
- Partition directory names match iceberg-java 1.8+'s `PartitionSpec.partitionToPath` for every
710-
partition type the native writer accepts. On Iceberg 1.5.x,
711-
which the Spark 3.4 profile pins, iceberg-java itself spelled `timestamp` and `timestamptz`
712-
directories with `LocalDateTime.toString()` / `OffsetDateTime.toString()`
709+
partition type the native writer accepts. Earlier iceberg-java releases spelled `timestamp` and
710+
`timestamptz` directories with `LocalDateTime.toString()` / `OffsetDateTime.toString()`
713711
(`ts=1969-12-31T23:59:58.500Z`) and left the partition field name unescaped; Comet uses the
714712
1.8+ spelling on every profile. Distinct partition values still get distinct directories in all
715713
cases, and no reader parses these names: files are resolved through committed manifests.

‎docs/source/user-guide/latest/migration-guide.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,16 @@ Treat setting one of these keys as a temporary measure. If you find you cannot s
5757
legacy behavior, please open an issue describing your use case so it can be considered before the
5858
key is removed.
5959

60+
## Upgrading to Comet 1.3.0
61+
62+
Comet `1.3.0` makes no behavior changes that need a `spark.comet.legacy.*` key. The change below
63+
needs none either, but check whether it applies to your deployment.
64+
65+
Comet `1.3.0` no longer builds or publishes support for Spark 3.4. Spark 3.4 has been deprecated
66+
since Comet `1.0.0`. If you run Spark 3.4, stay on the last Comet release that publishes
67+
`comet-spark-spark3.4_*`. See [Installing Comet](installation.md) for the supported Java, Scala,
68+
and Spark versions.
69+
6070
## Upgrading to Comet 1.2.0
6171

6272
Comet `1.2.0` makes no behavior changes that need a `spark.comet.legacy.*` key. The changes below

0 commit comments

Comments
 (0)