[SPARK-45144][BUILD] Downgrade `scala-maven-plugin` to 4.7.1 #42899

LuciferYang · 2023-09-13T05:56:00Z

What changes were proposed in this pull request?

This pr downgrade scala-maven-plugin to version 4.7.1 to avoid it automatically adding the -release option as a Scala compilation argument.

Why are the changes needed?

The scala-maven-plugin versions 4.7.2 and later will try to automatically append the -release option as a Scala compilation argument when it is not specified by the user:

4.7.2 and 4.8.0: try to add the -release option for Scala versions 2.13.9 and higher.
4.8.1: try to append the -release option for Scala versions 2.12.x/2.13.x/3.1.1, and append -java-output-version for Scala 3.1.2.

The addition of the -release option has caused issues mentioned in SPARK-44376 | #41943 and #40442 (comment). This is because the -release option has stronger compilation restrictions than -target, ensuring not only bytecode format, but also that the API used in the code is compatible with the specified version of Java. However, many APIs in the sun.* package are not exports in Java 11, 17, and 21, such as sun.nio.ch.DirectBuffer, sun.util.calendar.ZoneInfo, and sun.nio.cs.StreamDecoder, making them invisible when compiling across different versions.

For discussions within the Scala community, see scala/bug#12643, scala/bug#12824, scala/bug#12866, but this is not a bug.

I have also submitted an issue to the scala-maven-plugin community to discuss the possibility of adding additional settings to control the addition of the -release option: davidB/scala-maven-plugin#722.

For Apache Spark 4.0, in the short term, I suggest downgrading scala-maven-plugin to version 4.7.1 to avoid it automatic adding the -release option as a Scala compilation argument. In the long term, we should reduce use of APIs that are not exports for compatibility with the -release compilation option due to -target already deprecated after Scala 2.13.9.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass GitHub Actions
Manual check

run git revert 656bf36363c466b60d00452399994ccaaa654ed8 to revert to using Scala 2.13.11 and run dev/change-scala-version.sh 2.13 to change Scala to 2.13

Run build/mvn clean install -DskipTests -Pscala-2.13 -X to check the Scala compilation arguments.

Before

[[DEBUG] [zinc] Running cached compiler 1992eaf4 for Scala compiler version 2.13.11
[DEBUG] [zinc] The Scala compiler is invoked with:
  -unchecked
  -deprecation
  -feature
  -explaintypes
  -target:jvm-1.8
  -Wconf:cat=deprecation:wv,any:e
  -Wunused:imports
  -Wconf:cat=scaladoc:wv
  -Wconf:cat=lint-multiarg-infix:wv
  -Wconf:cat=other-nullary-override:wv
  -Wconf:cat=other-match-analysis&site=org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction.catalogFunction:wv
  -Wconf:cat=other-pure-statement&site=org.apache.spark.streaming.util.FileBasedWriteAheadLog.readAll.readFile:wv
  -Wconf:cat=other-pure-statement&site=org.apache.spark.scheduler.OutputCommitCoordinatorSuite.<local OutputCommitCoordinatorSuite>.futureAction:wv
  -Wconf:msg=^(?=.*?method|value|type|object|trait|inheritance)(?=.*?deprecated)(?=.*?since 2.13).+$:s
  -Wconf:msg=^(?=.*?Widening conversion from)(?=.*?is deprecated because it loses precision).+$:s
  -Wconf:msg=Auto-application to \`\(\)\` is deprecated:s
  -Wconf:msg=method with a single empty parameter list overrides method without any parameter list:s
  -Wconf:msg=method without a parameter list overrides a method with a single empty one:s
  -Wconf:cat=deprecation&msg=procedure syntax is deprecated:e
  -Wconf:cat=unchecked&msg=outer reference:s
  -Wconf:cat=unchecked&msg=eliminated by erasure:s
  -Wconf:msg=^(?=.*?a value of type)(?=.*?cannot also be).+$:s
  -Wconf:cat=unused-imports&src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBase.scala:s
  -Wconf:cat=unused-imports&src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBaseOps.scala:s
  -Wconf:msg=Implicit definition should have explicit type:s
  -release
  8
  -bootclasspath
...

After

[DEBUG] [zinc] Running cached compiler 72dd4888 for Scala compiler version 2.13.11
[DEBUG] [zinc] The Scala compiler is invoked with:
  -unchecked
  -deprecation
  -feature
  -explaintypes
  -target:jvm-1.8
  -Wconf:cat=deprecation:wv,any:e
  -Wunused:imports
  -Wconf:cat=scaladoc:wv
  -Wconf:cat=lint-multiarg-infix:wv
  -Wconf:cat=other-nullary-override:wv
  -Wconf:cat=other-match-analysis&site=org.apache.spark.sql.catalyst.catalog.SessionCatalog.lookupFunction.catalogFunction:wv
  -Wconf:cat=other-pure-statement&site=org.apache.spark.streaming.util.FileBasedWriteAheadLog.readAll.readFile:wv
  -Wconf:cat=other-pure-statement&site=org.apache.spark.scheduler.OutputCommitCoordinatorSuite.<local OutputCommitCoordinatorSuite>.futureAction:wv
  -Wconf:msg=^(?=.*?method|value|type|object|trait|inheritance)(?=.*?deprecated)(?=.*?since 2.13).+$:s
  -Wconf:msg=^(?=.*?Widening conversion from)(?=.*?is deprecated because it loses precision).+$:s
  -Wconf:msg=Auto-application to \`\(\)\` is deprecated:s
  -Wconf:msg=method with a single empty parameter list overrides method without any parameter list:s
  -Wconf:msg=method without a parameter list overrides a method with a single empty one:s
  -Wconf:cat=deprecation&msg=procedure syntax is deprecated:e
  -Wconf:cat=unchecked&msg=outer reference:s
  -Wconf:cat=unchecked&msg=eliminated by erasure:s
  -Wconf:msg=^(?=.*?a value of type)(?=.*?cannot also be).+$:s
  -Wconf:cat=unused-imports&src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBase.scala:s
  -Wconf:cat=unused-imports&src=org\/apache\/spark\/graphx\/impl\/VertexPartitionBaseOps.scala:s
  -Wconf:msg=Implicit definition should have explicit type:s
  -target:8
  -bootclasspath
  ...

After downgrading the version, the -release option should no longer appear in the compilation arguments.

Maven can build the project with Java 17 without the issue described in [SPARK-44376][BUILD] Fix maven build using scala 2.13 and Java 11 or later #41943. And after this pr, we can re-upgrade Scala 2.13 to Scala 2.13.11.

Was this patch authored or co-authored using generative AI tooling?

No

LuciferYang · 2023-09-13T06:36:47Z

I am writing the PR description, will be updated later

LuciferYang · 2023-09-13T07:03:38Z

updated

dongjoon-hyun

+1, LGTM.

srowen · 2023-09-13T13:54:01Z

Is the idea that we can update this again, along with Scala to 2.13.11, after we drop Java 8 support?
I'm still a little bit confused in that this is reporting an error building vs Java 11, but we do not do that or support that, because we support Java 8 and thus builds need to be compiled with Java 8. They then should run on Java 11, 17, etc.
Is this a problem now, given that?

LuciferYang · 2023-09-13T14:19:10Z

@srowen After this downgrade, we can upgrade to Scala 2.13.11 again without waiting for dropping Java 8, because the lower version of the plugin will not automatically adding the -release option anymore. We can still build with Java 8, and then test with Java 11/17/21.

srowen · 2023-09-13T14:26:53Z

OK that's useful if so.

dongjoon-hyun · 2023-09-13T15:43:30Z

Ya, I agree with @LuciferYang and hope we can have Scala 2.13.11 back after this PR.

dongjoon-hyun · 2023-09-13T15:45:27Z

The UT failure is irrelevant one which is tracked here (#42908).
Merged to master for Apache Spark 4.0.0. Thank you, @LuciferYang and all.

LuciferYang · 2023-09-13T15:50:47Z

Thanks @dongjoon-hyun @HyukjinKwon @srowen , I will re-upgrade Scala 2.13.11 later

init

0094e70

LuciferYang marked this pull request as draft September 13, 2023 05:56

github-actions bot added the BUILD label Sep 13, 2023

HyukjinKwon approved these changes Sep 13, 2023

View reviewed changes

LuciferYang marked this pull request as ready for review September 13, 2023 07:03

dongjoon-hyun approved these changes Sep 13, 2023

View reviewed changes

Merge branch 'upmaster' into SPARK-45144

48c3d33

dongjoon-hyun closed this in c12eb2c Sep 13, 2023

nchammas mentioned this pull request Dec 17, 2023

[SPARK-45993][INFRA] scalac: remove deprecated option -target #44387

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-45144][BUILD] Downgrade `scala-maven-plugin` to 4.7.1 #42899

[SPARK-45144][BUILD] Downgrade `scala-maven-plugin` to 4.7.1 #42899

Uh oh!

LuciferYang commented Sep 13, 2023 •

edited

Loading

Uh oh!

LuciferYang commented Sep 13, 2023

Uh oh!

LuciferYang commented Sep 13, 2023

Uh oh!

dongjoon-hyun left a comment

Uh oh!

srowen commented Sep 13, 2023

Uh oh!

LuciferYang commented Sep 13, 2023

Uh oh!

srowen commented Sep 13, 2023

Uh oh!

dongjoon-hyun commented Sep 13, 2023

Uh oh!

dongjoon-hyun commented Sep 13, 2023

Uh oh!

LuciferYang commented Sep 13, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[SPARK-45144][BUILD] Downgrade scala-maven-plugin to 4.7.1 #42899

[SPARK-45144][BUILD] Downgrade scala-maven-plugin to 4.7.1 #42899

Uh oh!

Conversation

LuciferYang commented Sep 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

LuciferYang commented Sep 13, 2023

Uh oh!

LuciferYang commented Sep 13, 2023

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

srowen commented Sep 13, 2023

Uh oh!

LuciferYang commented Sep 13, 2023

Uh oh!

srowen commented Sep 13, 2023

Uh oh!

dongjoon-hyun commented Sep 13, 2023

Uh oh!

dongjoon-hyun commented Sep 13, 2023

Uh oh!

LuciferYang commented Sep 13, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[SPARK-45144][BUILD] Downgrade `scala-maven-plugin` to 4.7.1 #42899

[SPARK-45144][BUILD] Downgrade `scala-maven-plugin` to 4.7.1 #42899

LuciferYang commented Sep 13, 2023 •

edited

Loading