Add basic support for the Pants build tool by Sohamr · Pull Request #935 · scalameta/metals

Sohamr · 2019-09-19T15:15:37Z

Previously, Metals didn't work with the Pants build tool https://www.pantsbuild.org/. This PR adds support for Pants that works similarly to the build tool support with sbt, Gradle, Maven and Mill.

the user needs to configure the pants-targets setting to specify what parts of their Pants build should be exported to Metals/Bloop.
the user is prompted to import a Pants workspace on first startup
when BUILD files change, Metals will prompt to the user to "import changes"

This PR is still a WIP. There are several issues related to handling large number of individual source files that belong to a Pants target.

mzarnowski

Thanks for the contribution! I couldn't resist, so I have looked around.
I have left some questions and suggestions but I suspect @tgodzik (our expert on handling build tools) will probably do another review once this is ready.

metals/src/main/scala/scala/meta/internal/builds/BuildTool.scala

metals/src/main/scala/scala/meta/internal/builds/PantsDigest.scala

metals/src/main/scala/scala/meta/internal/metals/UserConfiguration.scala

tests/unit/src/test/scala/tests/PantsDigestSuite.scala

metals/src/main/scala/scala/meta/internal/builds/PantsDigest.scala

tgodzik

Reviewed most of the files, only the strictly pants related stuff eludes me a bit, so that was not reviewed that well.

It's mostly some doubts I have and few suggestions on what to improve.

metals/src/main/scala/scala/meta/internal/pantsbuild/BloopPants.scala

tgodzik · 2019-09-24T11:36:05Z

tests/slow/src/test/scala/tests/pants/PantsSlowSuite.scala

+      _ = assertStatus(_.isInstalled)
+    } yield assertNoDiff(client.workspaceMessageRequests, "")
+  }
+//


tgodzik · 2019-09-24T11:40:40Z

metals/src/main/scala/scala/meta/internal/builds/BuildTools.scala

 }
+
+object BuildTools {
+  def all: List[BuildTool] = {


This is only used in tests and there is an already existing all method. Could we remove it?

The other all method doesn't work the same way. In the tests we enumerated the whole list in several different places. I have moved this method into the BuildTools class and converted this into a default method to get a default BuildTools instance, which we can use for testing purposes.

tgodzik · 2019-09-24T11:43:57Z

tests/slow/src/test/scala/tests/pants/PantsSlowSuite.scala

+    pants.toFile().setExecutable(true)
+    import scala.sys.process._
+    val exit = List(pants.toString(), "generate-pants-ini").!
+    require(exit == 0, "failed to generate pants.ini")


Can we have a static pants.ini defined with rest of the files. This will make the tests run much slower.

build.sbt

mzarnowski · 2019-09-25T07:40:04Z

metals/src/main/scala/scala/meta/internal/builds/BuildTools.scala

+    val userConfig: () => UserConfiguration = () => UserConfiguration()
+    val config = MetalsServerConfig.default
+    List(
+      SbtBuildTool.apply("", userConfig, config),


do we need an explicit apply here?

Suggested change

SbtBuildTool.apply("", userConfig, config),

SbtBuildTool("", userConfig, config),

Nope, removed.

metals/src/main/scala/scala/meta/internal/builds/PantsBuildTool.scala

metals/src/main/scala/scala/meta/internal/builds/PantsDigest.scala

metals/src/main/scala/scala/meta/internal/pantsbuild/BloopPants.scala

metals/src/main/scala/scala/meta/internal/pantsbuild/Tarjans.scala

mzarnowski · 2019-09-25T09:06:15Z

tests/slow/src/test/scala/tests/pants/PantsSlowSuite.scala

+
+  private def preInitialized = {
+    val pants_targets_config =
+      s"""


why not just """{ "pants-targets": "src::" } """ ?

mzarnowski · 2019-09-25T09:07:37Z

tests/slow/src/test/scala/tests/pants/PantsSlowSuite.scala

+      workspace: AbsolutePath
+  ): Option[String] = {
+    new PantsDigest(
+      () => UserConfiguration(pantsTargets = Option(s"src::"))


why s"src::" is an interpolation ?

olafurpg

Thank you for the detailed review @tgodzik and @marek1840. A lot of great comments 🙏

build.sbt

olafurpg · 2019-09-26T14:58:44Z

metals/src/main/scala/scala/meta/internal/builds/BuildTools.scala

 }
+
+object BuildTools {
+  def all: List[BuildTool] = {


The other all method doesn't work the same way. In the tests we enumerated the whole list in several different places. I have moved this method into the BuildTools class and converted this into a default method to get a default BuildTools instance, which we can use for testing purposes.

olafurpg · 2019-09-26T14:58:58Z

metals/src/main/scala/scala/meta/internal/builds/BuildTools.scala

+    val userConfig: () => UserConfiguration = () => UserConfiguration()
+    val config = MetalsServerConfig.default
+    List(
+      SbtBuildTool.apply("", userConfig, config),


Nope, removed.

metals/src/main/scala/scala/meta/internal/builds/PantsBuildTool.scala

metals/src/main/scala/scala/meta/internal/pantsbuild/BloopPants.scala

metals/src/main/scala/scala/meta/internal/pantsbuild/Tarjans.scala

### Problem We would like to support the Metals VSCode plugin -- see associated PR at scalameta/metals#935. Pants currently does not have any fields which specifically provide the scala version and the compiler jars, so that PR currently iterates over the `libraries` keys, which doesn't provide all the scala compiler jars. ### Solution Add a `scala_platform` key to the `./pants export` output containing the `scala_version` and `compiler_classpath`: ```json { ..., "scala_platform": { "scala_version": "2.12", "compiler_classpath": [ "/path/to/scala-compiler-2.12.8.jar", "/path/to/scala-library-2.12.8.jar", ... ] } ``` ### Result Using the branch at https://github.com/cosmicexplorer/language-server/tree/sohamr/add-pants-build-tool (which is based off of scalameta/metals#935), I have been able to show that metals can extract the `scala_platform` from the json!

jvican

This looks really cool. Good job! 💯

metals/src/main/scala/scala/meta/internal/builds/PantsBuildTool.scala

metals/src/main/scala/scala/meta/internal/metals/FileWatcher.scala

metals/src/main/scala/scala/meta/internal/metals/MetalsEnrichments.scala

olafurpg · 2019-11-02T23:10:03Z

A lot of progress in the latest commits! This PR is not yet ready for review but I think it's getting close to being feature complete

BUILD file changes are detected pretty quickly now
creation of new source files automatically triggers a quick regeneration of bloop files and re-indexing
sources of workspace "binary" dependencies now treated just like normal external *-sources.jar files
basic test infrastructure for running metals in fresh pants workspaces
"import build" with pants shows a nice progress bar and cancellation works

To fix the CI errors, we're blocked by #1030

Sohamr · 2019-11-03T17:55:13Z

A lot of progress in the latest commits! This PR is not yet ready for review but I think it's getting close to being feature complete

BUILD file changes are detected pretty quickly now

creation of new source files automatically triggers a quick regeneration of bloop files and re-indexing

sources of workspace "binary" dependencies now treated just like normal external *-sources.jar files

basic test infrastructure for running metals in fresh pants workspaces

"import build" with pants shows a nice progress bar and cancellation works

To fix the CI errors, we're blocked by #1030

This is Awesome! Thanks a lot, Olaf!!!

Previously, the `./pants export` output did not include information whether a target has enabled strict dependencies or not. This information is needed in order to faithfully reproduce the compilation outside of Pants, for example with the IntelliJ compiler or with Bloop see (scalameta/metals#935).

olafurpg · 2019-11-29T16:29:11Z

This PR is still not ready for review. The latest batch of changes makes several improvements

use pants filemap instead of custom glob expansion
use the full classpath produced by pants export-classpath

These changes fix several compile errors that I hit on in the wild.

Previously, the `./pants export` output did not include information whether a target has enabled strict dependencies or not. This information is needed in order to faithfully reproduce the compilation outside of Pants, for example with the IntelliJ compiler or with Bloop see (scalameta/metals#935).

olafurpg

I broke up the massive mono-commit into smaller commits and opened a separate PR adding all the non-Pants related changes #1145

We should get that PR merged first before continuing here.

olafurpg · 2019-12-03T16:22:10Z

metals/src/main/scala/scala/meta/internal/metals/UserConfiguration.scala

    errors ++= symbolPrefixes.keys.flatMap { sym =>
      Symbol.validated(sym).left.toOption
    }
+    val worksheetScreenWidth =


These changes got introduces in this PR to enforce consistency in UserConfiguration. Every field should be configurable

olafurpg · 2019-12-03T16:27:10Z

tests/unit/src/test/scala/tests/DetectionSuite.scala

@@ -22,7 +24,16 @@ object DetectionSuite extends BaseSuite {
  }
  def checkSbt(name: String, layout: String, isTrue: Boolean = true): Unit = {


Previously, it was not possible to use Metals with the Pants build tool. This commit implements all the functionality to use Metals with Pants, along with a lot smaller fixes that are necessary to work with Metals in a larger workspace (encountered while developing the Pants integration). This is commit is large because it was difficult to rebase the individual commits on top of the Metals master.

olafurpg · 2019-12-05T13:50:38Z

I believe this PR is finally ready for merge! The Pants->Bloop conversion is still not perfect, I expect it to evolve as we iterate on the Pants integration and gain more experience with it.

tgodzik

Looks good already! I just had a couple of questions really.

tgodzik · 2019-12-05T15:33:33Z

metals/src/main/scala/scala/meta/internal/builds/PantsBuildTool.scala

+  private def pantsTargets(): List[String] =
+    userConfig().pantsTargets match {
+      case None => Nil
+      case Some(target) => target.split(" ").toList


Targets will be separate by a space? Will that be consistent with the VS Code representation? It would be nice if it was represented as an array inside the settings.

tgodzik · 2019-12-05T15:37:03Z

metals/src/main/scala/scala/meta/internal/builds/PantsBuildTool.scala

+        BloopPants
+          .pantsOwnerOf(
+            workspace,
+            source.resolveSibling(_.stripSuffix(".sc") + ".scala")


Not sure I understand the logic here? Why for .sc files we are looking for the same but with .scala?

I added a comment

// Convert Scala script name into a `*.scala` filename to find out what // target it should belong to. Pants doesn't support Scala scripts so // using the script name unchanged would return no targets.

tgodzik · 2019-12-05T15:42:32Z

metals/src/main/scala/scala/meta/internal/builds/PantsBuildTool.scala

+  def bloopInstall(
+      workspace: AbsolutePath,
+      languageClient: MetalsLanguageClient,
+      systemProcess: List[String] => Future[BloopInstallResult]


systemProcess doesn't seem to be used anywhere here

Renamed

// Not used: we call metals/slowTask directly _unused: List[String] => Future[BloopInstallResult]

tgodzik · 2019-12-05T15:47:16Z

metals/src/main/scala/scala/meta/internal/builds/PantsDigest.scala

+    } {
+      isOk &= Digest.digestFile(buildFile, digest)
+    }
+    isOk


We could do this with forAll instead of a var. yield Digest.digestFile(buildFile, digest) and then results.forAll(_)

tgodzik · 2019-12-05T16:08:55Z

metals/src/main/scala/scala/meta/internal/metals/debug/DebugProxy.scala

      outputTerminated = true
      server.consume(message)
-
+    case null =>


Can we pack it into an Option like:

{ message => Option(message) match { case _ if cancelled.get() => // ignore case Some(OutputNotification()) if outputTerminated => // ignore. When restarting, the output keeps getting printed for a short while after the // output window gets refreshed resulting in stale messages being printed on top, before // any actual logs from the restarted process case Some(msg) => client.consume(msg) case None => } }

This change is unrelated to the PR, let's refactor it separately. I just wanted to fix a NPE

tgodzik · 2019-12-05T16:10:25Z

metals/src/main/scala/scala/meta/internal/pantsbuild/Args.scala

+       |  --[no-]cache (default=false)
+       |    If enabled, cache the result from `./pants export`
+       |  --[no-]compile (default=$isCompile)
+       |    If ena, do not run `./pants compile`


ena ? Looks like a typo

tgodzik · 2019-12-05T16:13:45Z

metals/src/main/scala/scala/meta/internal/pantsbuild/BloopPants.scala

+        val sourceStr = Value.Str(source.toString())
+        if (!sources.contains(sourceStr)) {
+          sources += sourceStr
+          jsonFile.writeText(ujson.write(json, indent = 4))


I believe Jorge was strongly opposed to anything than bloop modifying the config file while it's working 🤔

The config file can be modified while it's working, but it must be updated atomically. I don't remember being strongly opposed to this, but I do remember mentioning that tools that don't own these configuration files should never update them, that's something that only the original build tool can do. Let me know if this clarifies your thinking @tgodzik 😄

@jvican Thanks! That makes sense, I must have misunderstood previously.

Sounds great! I will follow up with a PR to make our file write operations atomic.

We use _root_.bloop.config.write(json, out) to write most of the JSON files in the Pants export. This step here happens only when the user creates a new file and we need to re-expand the file globs.

tgodzik · 2019-12-05T16:27:53Z

metals/src/main/scala/scala/meta/internal/pantsbuild/BloopPants.scala

+      s"export"
+    ) ++ args.targets
+    val shortName = "pants export-classpath export"
+    SystemProcess.run(


Shouldn't we use the process runner from the bloopInstall function?

No, since that one launches metals/slowTask and we call several system processes in this step.

olafurpg

@tgodzik Thank you for the review!

olafurpg · 2019-12-05T19:24:11Z

metals/src/main/scala/scala/meta/internal/pantsbuild/BloopPants.scala

+      s"export"
+    ) ++ args.targets
+    val shortName = "pants export-classpath export"
+    SystemProcess.run(


No, since that one launches metals/slowTask and we call several system processes in this step.

olafurpg · 2019-12-05T19:24:31Z

metals/src/main/scala/scala/meta/internal/builds/PantsBuildTool.scala

+        BloopPants
+          .pantsOwnerOf(
+            workspace,
+            source.resolveSibling(_.stripSuffix(".sc") + ".scala")


I added a comment

// Convert Scala script name into a `*.scala` filename to find out what // target it should belong to. Pants doesn't support Scala scripts so // using the script name unchanged would return no targets.

olafurpg · 2019-12-05T19:25:01Z

metals/src/main/scala/scala/meta/internal/builds/PantsBuildTool.scala

+  def bloopInstall(
+      workspace: AbsolutePath,
+      languageClient: MetalsLanguageClient,
+      systemProcess: List[String] => Future[BloopInstallResult]


Renamed

// Not used: we call metals/slowTask directly _unused: List[String] => Future[BloopInstallResult]

olafurpg · 2019-12-05T19:27:50Z

metals/src/main/scala/scala/meta/internal/builds/PantsDigest.scala

+    } {
+      isOk &= Digest.digestFile(buildFile, digest)
+    }
+    isOk


olafurpg · 2019-12-05T19:28:14Z

metals/src/main/scala/scala/meta/internal/metals/debug/DebugProxy.scala

      outputTerminated = true
      server.consume(message)
-
+    case null =>


This change is unrelated to the PR, let's refactor it separately. I just wanted to fix a NPE

tgodzik

Looks good! Excited to have 5! build tools supported out of the box 😄

olafurpg · 2019-12-05T20:24:51Z

🎉

Following up on scalameta#935 (comment)

Previously, the `./pants export` output did not include information whether a target has enabled strict dependencies or not. This information is needed in order to faithfully reproduce the compilation outside of Pants, for example with the IntelliJ compiler or with Bloop see (scalameta/metals#935).

mzarnowski requested changes Sep 20, 2019

View reviewed changes

olafurpg force-pushed the sohamr/add-pants-build-tool branch from 0705006 to 79171d6 Compare September 20, 2019 15:40

olafurpg changed the title ~~Sohamr/add pants build tool~~ Add basic support for the Pants build tool Sep 20, 2019

cosmicexplorer mentioned this pull request Sep 20, 2019

add bloop contrib package for BSP support in VSCode and IntelliJ pantsbuild/pants#7961

Closed

5 tasks

tgodzik requested changes Sep 24, 2019

View reviewed changes

mzarnowski requested changes Sep 25, 2019

View reviewed changes

olafurpg reviewed Sep 27, 2019

View reviewed changes

olafurpg force-pushed the sohamr/add-pants-build-tool branch from 6e2adf8 to fb029ef Compare September 27, 2019 15:09

cosmicexplorer mentioned this pull request Oct 1, 2019

add scala_platform to export output for metals! pantsbuild/pants#8370

Merged

olafurpg mentioned this pull request Oct 2, 2019

Warn when a source jar is corrupted #954

Merged

olafurpg force-pushed the sohamr/add-pants-build-tool branch from 1ca17fa to e970d01 Compare October 3, 2019 17:45

This was referenced Oct 10, 2019

Expose the hash of a given target pantsbuild/pants#8445

Open

add --output-format=json output option to v2 list pantsbuild/pants#8450

Closed

jvican reviewed Oct 12, 2019

View reviewed changes

metals/src/main/scala/scala/meta/internal/builds/PantsBuildTool.scala Show resolved Hide resolved

mzarnowski reviewed Oct 14, 2019

View reviewed changes

metals/src/main/scala/scala/meta/internal/metals/FileWatcher.scala Outdated Show resolved Hide resolved

metals/src/main/scala/scala/meta/internal/metals/MetalsEnrichments.scala Outdated Show resolved Hide resolved

olafurpg force-pushed the sohamr/add-pants-build-tool branch from e970d01 to d85a0b0 Compare October 18, 2019 13:40

olafurpg force-pushed the sohamr/add-pants-build-tool branch 2 times, most recently from 4302e71 to 40b318a Compare November 2, 2019 22:37

olafurpg force-pushed the sohamr/add-pants-build-tool branch 3 times, most recently from cb01c98 to b7d6f0a Compare November 18, 2019 13:47

olafurpg force-pushed the sohamr/add-pants-build-tool branch from ce438d8 to 5b1d179 Compare November 20, 2019 12:17

olafurpg force-pushed the sohamr/add-pants-build-tool branch from 5b1d179 to d2435af Compare November 28, 2019 16:46

olafurpg mentioned this pull request Nov 29, 2019

Include transitive target dependencies in export-dep-as-jar pantsbuild/pants#8736

Closed

olafurpg force-pushed the sohamr/add-pants-build-tool branch from 1a61ae3 to f693969 Compare November 29, 2019 16:26

olafurpg force-pushed the sohamr/add-pants-build-tool branch 4 times, most recently from ae29e8d to ee87336 Compare December 4, 2019 15:16

olafurpg mentioned this pull request Dec 4, 2019

Internal refactorings to prepare for Pants support #1145

Merged

olafurpg reviewed Dec 4, 2019

View reviewed changes

olafurpg force-pushed the sohamr/add-pants-build-tool branch 2 times, most recently from c4d8aeb to b816d21 Compare December 5, 2019 11:39

olafurpg mentioned this pull request Dec 5, 2019

NullPointerException in DebugProxy #1154

Closed

olafurpg force-pushed the sohamr/add-pants-build-tool branch from 2287c8c to 35b4286 Compare December 5, 2019 13:21

olafurpg requested a review from tgodzik December 5, 2019 13:45

tgodzik reviewed Dec 5, 2019

View reviewed changes

Olafur Pall Geirsson added 5 commits December 5, 2019 19:20

Support string or list of strings in pants-targets configuration.

f0cea29

Address review feedback

63a3094

Address review feedback

70c43ce

Move PantsDigestSuite to the same package as other digest tests

312bdd3

Fix compile error

a95ae87

olafurpg reviewed Dec 5, 2019

View reviewed changes

tgodzik approved these changes Dec 5, 2019

View reviewed changes

olafurpg merged commit b0fcf30 into scalameta:master Dec 5, 2019

olafurpg added a commit to olafurpg/metals that referenced this pull request Dec 5, 2019

Use atomic file move operations.

b68fb8c

Following up on scalameta#935 (comment)

olafurpg mentioned this pull request Dec 5, 2019

Use atomic file move operations. #1155

Merged

gabro mentioned this pull request Dec 31, 2019

support Pants build system scalameta/metals-feature-requests#55

Closed

murph mentioned this pull request Feb 17, 2020

Add support for metals.pantsTargets setting scalameta/metals-vscode#214

Closed

	SbtBuildTool.apply("", userConfig, config),
	SbtBuildTool("", userConfig, config),

		@@ -22,7 +24,16 @@ object DetectionSuite extends BaseSuite {
		}
		def checkSbt(name: String, layout: String, isTrue: Boolean = true): Unit = {

Conversation

Sohamr commented Sep 19, 2019 • edited by olafurpg Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mzarnowski left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tgodzik left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

olafurpg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jvican left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

olafurpg commented Nov 2, 2019

Uh oh!

Sohamr commented Nov 3, 2019

Uh oh!

olafurpg commented Nov 29, 2019

Uh oh!

olafurpg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Sohamr commented Sep 19, 2019 •

edited by olafurpg

Loading