[2.x] Implement worker + forked console by eed3si9n · Pull Request #8018 · sbt/sbt

eed3si9n · 2025-01-27T05:26:45Z

Ref #1918

Problem

See https://eed3si9n.com/rfc-4-persistent-worker/ for the general motivation. We want to move long-running tasks out of the command loop, and potentially from the sbt process entirely.
console / fork issue was created a long time ago since it's closely related to running and testing, which already can fork.

Solution

This implements a pure Java app called worker, which can run arbitrary application. For now I'm using property files to communicate. I'll likely switch to using JSON. The worker currently implements run command, which can run a program specified in the properties file.
This also implements ConsoleMain, which is a command line program that wraps Zinc's console function.

Caveat

One major limitation of the Compile / console / fork := true implementation at this time is that I wasn't able to configure JLine terminal to xterm-256color, which uses a sequence of characters to send up and down keys. So while the color works, up and down keys do not, which is inconvenient for Scala repl. Nonetheless, this is still a useful demonstration of the forked task running in a worker process that's able to accept ordinary key inputs.

**Problem** 1. See https://eed3si9n.com/rfc-4-persistent-worker/ for the general motivation. We want to move long-running tasks out of the command loop, and potentially from the sbt process entirely. 2. `console / fork` issue was created a long time ago since it's closely related to running and testing, which already can fork. **Solution** 1. This implements a pure Java app called worker, which can run arbitrary application. For now I'm using property files to communicate. I'll likely switch to using JSON. The worker currently implements `run` command, which can run a program specified in the properties file. 2. This also implements `ConsoleMain`, which is a command line program that wraps Zinc's `console` function.

adpi2

Some comments +

I tried console / fork := true and I got this:

$ sbtn console       
[info] entering *experimental* thin client - BEEP WHIRR
[info] server was not detected. starting an instance
[info] welcome to sbt 2.0.0-SNAPSHOT (Eclipse Adoptium Java 17)
[info] loading project definition from /home/piquerez/github/sbt/sbt-worker/project
[info] set current project to sbt-worker (in build file:/home/piquerez/github/sbt/sbt-worker/)
[info] sbt server started at local:///home/piquerez/.sbt/2.0.0-SNAPSHOT/server/5ae139875a5ddd586664/sock
[info][info] terminate the server with `shutdown`
> console
Welcome to Scala 3.6.3 (17, Java OpenJDK 64-Bit Server VM).
Type in expressions for evaluation. Or try :help.
~null null>....java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at sbt.internal.worker.WorkerMain.run(WorkerMain.java:66)
        at sbt.internal.worker.WorkerMain.run(WorkerMain.java:91)
        at sbt.internal.worker.WorkerMain.run(WorkerMain.java:99)
        at sbt.internal.worker.WorkerMain.main(WorkerMain.java:37)
Caused by: java.io.IOError: java.io.IOException: Bad file descriptor
        at org.jline.keymap.BindingReader.readCharacter(BindingReader.java:169)
        at org.jline.keymap.BindingReader.readBinding(BindingReader.java:109)
        at org.jline.keymap.BindingReader.readBinding(BindingReader.java:61)
        at org.jline.reader.impl.LineReaderImpl.doReadBinding(LineReaderImpl.java:974)
        at org.jline.reader.impl.LineReaderImpl.readBinding(LineReaderImpl.java:1007)
        at org.jline.reader.impl.LineReaderImpl.readLine(LineReaderImpl.java:690)
        at org.jline.reader.impl.LineReaderImpl.readLine(LineReaderImpl.java:512)
        at dotty.tools.repl.JLineTerminal.readLine(JLineTerminal.scala:80)
        at dotty.tools.repl.ReplDriver.readLine$1(ReplDriver.scala:197)
        at dotty.tools.repl.ReplDriver.loop$1(ReplDriver.scala:207)
        at dotty.tools.repl.ReplDriver.runUntilQuit$$anonfun$1(ReplDriver.scala:212)
        at dotty.tools.repl.ReplDriver.withRedirectedOutput(ReplDriver.scala:251)
        at dotty.tools.repl.ReplDriver.runBody$$anonfun$1(ReplDriver.scala:225)
        at dotty.tools.runner.ScalaClassLoader$.asContext(ScalaClassLoader.scala:80)
        at dotty.tools.repl.ReplDriver.runBody(ReplDriver.scala:225)
        at dotty.tools.repl.ReplDriver.runUntilQuit(ReplDriver.scala:212)
        at xsbt.ConsoleInterface.run(ConsoleInterface.java:52)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at sbt.internal.inc.AnalyzingCompiler.invoke(AnalyzingCompiler.scala:329)
        at sbt.internal.inc.AnalyzingCompiler.console(AnalyzingCompiler.scala:233)
        at sbt.Console.console0$1(Console.scala:73)
        at sbt.Console.apply$$anonfun$1$$anonfun$1$$anonfun$1(Console.scala:83)
        at sbt.Console.apply$$anonfun$1$$anonfun$1$$anonfun$adapted$1(Console.scala:83)
        at scala.Function0.apply$mcV$sp(Function0.scala:42)
        at sbt.Run$.executeSuccess(Run.scala:203)
        at sbt.Console.apply$$anonfun$1$$anonfun$1(Console.scala:83)
        at sbt.internal.util.Terminal.withRawInput(Terminal.scala:159)
        at sbt.internal.util.Terminal.withRawInput$(Terminal.scala:23)
        at sbt.internal.util.Terminal$ProxyTerminal$.withRawInput(Terminal.scala:455)
        at sbt.Console.apply$$anonfun$1(Console.scala:83)
        at sbt.internal.util.Terminal$DefaultTerminal.withRawOutput(Terminal.scala:1114)
        at sbt.internal.util.Terminal$ProxyTerminal$.withRawOutput(Terminal.scala:494)
        at sbt.Console.apply(Console.scala:84)
        at sbt.Console.apply(Console.scala:51)
        at sbt.Console.apply(Console.scala:43)
        at sbt.internal.ConsoleMain.run$$anonfun$1(ConsoleMain.scala:32)
        at sbt.io.IO$.withTemporaryDirectory(IO.scala:496)
        at sbt.io.IO$.withTemporaryDirectory(IO.scala:506)
        at sbt.internal.ConsoleMain.run(ConsoleMain.scala:23)
        at sbt.internal.ConsoleMain$.main(ConsoleMain.scala:100)
        at sbt.internal.ConsoleMain.main(ConsoleMain.scala)
        ... 8 more
Caused by: java.io.IOException: Bad file descriptor
        at java.base/java.io.FileInputStream.readBytes(Native Method)
        at java.base/java.io.FileInputStream.read(FileInputStream.java:276)
        at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
        at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:263)
        at sbt.internal.util.JLine3$.decodeInput(JLine3.scala:52)
        at sbt.internal.util.JLine3$$anon$4.fillBuffer(JLine3.scala:135)
        at sbt.internal.util.JLine3$$anon$4.read(JLine3.scala:143)
        at org.jline.utils.NonBlockingReader.read(NonBlockingReader.java:56)
        at org.jline.keymap.BindingReader.readCharacter(BindingReader.java:159)
        ... 51 more
[success] Total time: 2 s, completed Feb 6, 2025, 3:24:59 PM

adpi2 · 2025-02-06T12:33:23Z

main/src/main/scala/sbt/Defaults.scala

    }

  def makeProducts: Initialize[Task[Seq[File]]] = Def.task {
+    val s = streams.value


Why do we need to sprinkle those unused val s = streams.value?

Maybe I used it for logging. I should delete this one.

adpi2 · 2025-02-06T13:58:26Z

main/src/main/scala/sbt/Defaults.scala

+      val config = ConsoleConfig(
+        scalaInstanceConfig = sic,
+        bridgeJar = bridge.toString(),
+        externalDependencyJars = depsJars.map(_.toString()),
+      )


Why don't we have the full classpath in here?

consoleTask is defined as:

sbt/main/src/main/scala/sbt/Defaults.scala

Line 2198 in 613eb86

def consoleTask: Initialize[Task[Unit]] = consoleTask(fullClasspath, console)

Yea. I was going to eventually include the user code, but I have to move them into a sandbox like bgRun.

Yes it makes sense. I hope you'll have time to work on those next steps because I am afraid I won't be available to help.

adpi2 · 2025-02-06T14:01:15Z

main-actions/src/main/scala/sbt/internal/ForkWorker.scala

+// Used for currentClasspath
+private[sbt] class ForkWorker


Or you can do classOf[ForkWorker.type]

adpi2 · 2025-02-06T14:11:20Z

main-actions/src/main/scala/sbt/internal/ConsoleMain.scala

+import xsbti.compile.ClasspathOptionsUtil
+// import sbt.internal.util.{ DeprecatedJLine } // Terminal as ITerminal
+
+class ConsoleMain:


object ConsoleMain ?

eed3si9n · 2025-02-07T04:45:10Z

@adpi2 Yea. I've thus far tested it using it sbt --server:

$ sbt --server
[info] welcome to sbt 2.0.0-SNAPSHOT (Azul Systems, Inc. Java 17.0.12)
....
sbt:foo> console
Welcome to Scala 3.6.3 (17.0.12, Java OpenJDK 64-Bit Server VM).
Type in expressions for evaluation. Or try :help.

scala> 1 + 1
val res0: Int = 2

scala> sys.exit
[success] elapsed time: 10 s

Next step would be to let sbtn start the fork process into the worker.

adpi2

Overall LGTM

xuwei-k · 2025-03-16T10:23:15Z

main-actions/src/main/scala/sbt/internal/ConsoleMain.scala

+      case arg :: Nil if arg.startsWith("@") =>
+        import sbt.internal.worker.codec.JsonProtocol.given
+        val arg1 = arg.drop(1)
+        val s = IO.read(File(arg1))


Suggested change

case arg :: Nil if arg.startsWith("@") =>

import sbt.internal.worker.codec.JsonProtocol.given

val arg1 = arg.drop(1)

val s = IO.read(File(arg1))

case s"@${arg}" :: Nil =>

import sbt.internal.worker.codec.JsonProtocol.given

val s = IO.read(File(arg))

eed3si9n requested review from BillyAutrey and adpi2 January 27, 2025 05:26

eed3si9n force-pushed the wip/worker branch 3 times, most recently from 9722c23 to f8b9d9c Compare January 29, 2025 12:00

eed3si9n mentioned this pull request Jan 29, 2025

[2.x] test: Bump 2.12 versions in scripted #8023

Merged

eed3si9n force-pushed the wip/worker branch from f8b9d9c to 7fe8fdd Compare January 29, 2025 13:16

adpi2 reviewed Feb 6, 2025

View reviewed changes

adpi2 approved these changes Feb 7, 2025

View reviewed changes

adpi2 mentioned this pull request Feb 17, 2025

[1.x] Implement client-side run #8040

Merged

xuwei-k reviewed Mar 16, 2025

View reviewed changes

xuwei-k mentioned this pull request Sep 23, 2025

[1.x][2.x] sbt console does not work with Scala 2.13.x nightly #8294

Closed

eed3si9n marked this pull request as draft January 21, 2026 06:50

This was referenced Jan 21, 2026

Fork in 'console' task #1918

Closed

[2.x] refactor: scalaInstanceConfig key #8648

Merged

This was referenced Feb 7, 2026

Find a way forward for JLine updates scala/scala-dev#912

Open

[2.x] Minimalist Scala repl wrapper #8704

Closed

eed3si9n closed this Mar 4, 2026

eed3si9n deleted the wip/worker branch March 4, 2026 05:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[2.x] Implement worker + forked console#8018

[2.x] Implement worker + forked console#8018
eed3si9n wants to merge 1 commit intosbt:developfrom
eed3si9n:wip/worker

eed3si9n commented Jan 27, 2025

Uh oh!

adpi2 left a comment

Uh oh!

adpi2 Feb 6, 2025

Uh oh!

eed3si9n Feb 7, 2025

Uh oh!

adpi2 Feb 6, 2025

Uh oh!

eed3si9n Feb 7, 2025

Uh oh!

adpi2 Feb 7, 2025

Uh oh!

adpi2 Feb 6, 2025

Uh oh!

adpi2 Feb 6, 2025

Uh oh!

eed3si9n commented Feb 7, 2025

Uh oh!

adpi2 left a comment

Uh oh!

xuwei-k Mar 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

eed3si9n commented Jan 27, 2025

Problem

Solution

Caveat

Uh oh!

adpi2 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eed3si9n commented Feb 7, 2025

Uh oh!

adpi2 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants