Skip to content

Commit ae66818

Browse files
schenksjclaude
andcommitted
fix(build): pin Spark 4.1 to 4.1.1 for delta-spark 4.1.0 compatibility (apache#76 follow-up)
The spark-4.1 profile (and top-level default) pinned spark.version=4.1.2, but delta-spark 4.1.0 references org.apache.spark.sql.catalyst.plans.logical.IgnoreCachedData, which Spark removed in 4.1.2. So `-Pspark-4.1,contrib-delta` builds/tests failed at runtime with NoClassDefFoundError (the contrib-delta CI matrix already labelled the cell 4.1.1, but the build resolved the pom's 4.1.2). Comet itself works on either patch; the constraint is delta-spark. Pin to 4.1.1 in both spots so CI and local agree and the contrib suites run. Verified: full contrib package 152/0 with no -Dspark.version override. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
1 parent a4ae76d commit ae66818

1 file changed

Lines changed: 8 additions & 2 deletions

File tree

‎pom.xml‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,11 @@ under the License.
7777
<scala.plugin.version>4.9.6</scala.plugin.version>
7878
<scalatest.version>3.2.16</scalatest.version>
7979
<scalatest-maven-plugin.version>2.2.0</scalatest-maven-plugin.version>
80-
<spark.version>4.1.2</spark.version>
80+
<!-- Spark 4.1 is pinned to 4.1.1 (not 4.1.2): delta-spark 4.1.0 (delta.version above) references
81+
org.apache.spark.sql.catalyst.plans.logical.IgnoreCachedData, which Spark removed in 4.1.2,
82+
so 4.1.2 + delta-spark 4.1.0 fails at runtime (NoClassDefFoundError). Comet itself works on
83+
either patch; the constraint is delta-spark. Bump together with delta.version. -->
84+
<spark.version>4.1.1</spark.version>
8185
<spark.version.short>4.1</spark.version.short>
8286
<spark.maven.scope>provided</spark.maven.scope>
8387
<protobuf.version>3.25.5</protobuf.version>
@@ -718,7 +722,9 @@ under the License.
718722
yet published, so the -Psemanticdb / scalafix lint job is skipped for spark-4.1. -->
719723
<scala.version>2.13.17</scala.version>
720724
<scala.binary.version>2.13</scala.binary.version>
721-
<spark.version>4.1.2</spark.version>
725+
<!-- 4.1.1, not 4.1.2: delta-spark 4.1.0 needs IgnoreCachedData, which 4.1.2 removed.
726+
See the top-level spark.version comment. -->
727+
<spark.version>4.1.1</spark.version>
722728
<spark.version.short>4.1</spark.version.short>
723729
<parquet.version>1.16.0</parquet.version>
724730
<semanticdb.version>4.13.6</semanticdb.version>

0 commit comments

Comments
 (0)