WIP, mostly submitted to enable running the benchmarks

Benchmarks are flat except for "scala", which seems to get a ~2% speedup.

I'd like to hear opinions how / if to move this forward. The change as proposed uglifies the build. It will also require a few fixes, as the "quick" classfiles of the library are now split up in build/quick/{library|libraryJava}, which breaks partest and mima.

Benchmarks are flat except for "scala", which seems to get a ~2% speedup.

I thnk we're seeing some jitter in that, we probably need more forks :(

Certain Java sources cannot be moved (JFunctionN$mc...) as they depend on Scala sources in the library.

Another approach would be to define target/shims/scala/FunctionN.java containing (empty) interfaces for FunctionN. We could them invoke javac with -sourcepath target/shims -implicit:none.

We could them revert to JavaThenScala compilation order.

Nice idea! I was trying to come up with something along these lines. There are two other files that currently require mixed compilation (LambdaDeserialize and StructuralCallSite), they can probably be done with shims as well.

I think we're seeing some jitter in that, we probably need more forks :(

I'll keep an eye on that and experiment a bit

Pushed a WIP with shims. It works when I do it on the command line

$> javac -d . $(find ../src/library -name '*.java') -sourcepath /Users/luc/scala/scala/src/shims -implicit:none

but sbt crashes because of the missing classfiles for the shims

Caused by: java.lang.ClassNotFoundException: scala.runtime.LambdaDeserializer$
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.getDeclaredFields0(Native Method)
	at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
	at java.lang.Class.getDeclaredFields(Class.java:1916)
	at sbt.ClassToAPI$.structure(ClassToAPI.scala:78)
	at sbt.ClassToAPI$.x$2$lzycompute$1(ClassToAPI.scala:66)
	at sbt.ClassToAPI$.x$2$1(ClassToAPI.scala:66)
	at sbt.ClassToAPI$.instance$lzycompute$1(ClassToAPI.scala:66)
	at sbt.ClassToAPI$.sbt$ClassToAPI$$instance$1(ClassToAPI.scala:66)
	at sbt.ClassToAPI$$anonfun$5.apply(ClassToAPI.scala:67)
	at sbt.ClassToAPI$$anonfun$5.apply(ClassToAPI.scala:67)
	at xsbti.SafeLazy$Impl._t$lzycompute(SafeLazy.scala:18)
	at xsbti.SafeLazy$Impl._t(SafeLazy.scala:16)
	at xsbti.SafeLazy$Impl.get(SafeLazy.scala:22)
	at sbt.ClassToAPI$$anonfun$process$1.apply(ClassToAPI.scala:22)
	at sbt.ClassToAPI$$anonfun$process$1.apply(ClassToAPI.scala:22)
	at scala.collection.immutable.List.foreach(List.scala:318)
	at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
	at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
	at sbt.ClassToAPI$.process(ClassToAPI.scala:22)
	at sbt.compiler.javac.AnalyzingJavaCompiler.sbt$compiler$javac$AnalyzingJavaCompiler$$readAPI$1(AnalyzingJavaCompiler.scala:76)
	at sbt.compiler.javac.AnalyzingJavaCompiler$$anonfun$compile$2$$anonfun$apply$mcV$sp$2$$anonfun$apply$5.apply(AnalyzingJavaCompiler.scala:84)
	at sbt.compiler.javac.AnalyzingJavaCompiler$$anonfun$compile$2$$anonfun$apply$mcV$sp$2$$anonfun$apply$5.apply(AnalyzingJavaCompiler.scala:84)
...

@dwijnand ideas? Can we disable dependency analysis for these classes?

(It works if I remove the -implicit:none flag, then hopefully compiling the Scala classes would overwrite the classfiles, but it would be safer to avoid this)

Can we disable dependency analysis for these classes?

I had a look and couldn't find any way..

https://github.com/sbt/sbt/blob/v0.13.16/compile/integration/src/main/scala/sbt/compiler/javac/AnalyzingJavaCompiler.scala#L74-L86

(It works if I remove the -implicit:none flag, then hopefully compiling the Scala classes would overwrite the classfiles, but it would be safer to avoid this)

If the compilation is JavaThenScala then isn't this safe?

Thank you for checking, Dale! I guess it's safe to rely on overwriting the classfiles. In case it doesn't happen for some reason and we end up running with the stubs, things should immediately crash at run / test time.

This works, but leaves some new warnings in the sbt log. It's because sources are pulled in with javac's -sourcepath, so sbt doesn't know where to find them.

[warn] Could not determine source for class scala.Function1
[warn] Could not determine source for class scala.runtime.LambdaDeserializer$
[warn] Could not determine source for class scala.runtime.EmptyMethodCache
[warn] Could not determine source for class scala.Function2
[warn] Could not determine source for class scala.Function0
[warn] Could not determine source for class scala.runtime.MethodCache
[warn] there were two deprecation warnings (since 2.11.0); re-run with -deprecation for details

Is there a way to add sources to library/sources but only for the javac compilation, not the scalac (in JavaThenScala)?

From my reading in JavaThenScala java sources aren't passed to scalac: https://github.com/sbt/sbt/blob/v0.13.16/compile/integration/src/main/scala/sbt/compiler/MixedAnalyzingCompiler.scala#L47

so just move them into src/library? Alternatively you can add src/shims to unmanagedSourceDirectories in Compile.

Having them in src/library would give double def errors for people doing scalac $(find src/library -name '*.java' -or -name '*.scala'). I think I did that (very rarely), but it's probably exotic enough? I'm fine with any solution, even the current one with warnings.

Personally I've not done that, but I'm equally fine with any solution.

@lrytz Finally getting back to this one.

How about we just cut the compile time dependency from Java sources to Scala sources?

Seems doable: https://github.com/scala/scala/compare/2.13.x...retronym:review/6085?expand=1

I've implemented JFunctionN in Scala (we can do this now that we're bootstrapping on 2.12+). The other dependencies can be broken with reflection (which if done correctly, using method handles stored in static finals, is no slower after JIT). We could make this slightly cleaner by using Java only for the static bootstrap method (and any static final method handles), and pushing the other logic into Scala.

Doing this work on 2.13 will make life easier, as we can move things around without binary compat breakages.

Doing this work on 2.13 will make life easier, as we can move things around without binary compat breakages.

I've added some commits to this end. We can actually define the bootstrap methods in Scala if we rely on the static forwarders.

Thanks @retronym, that looks like a better way forward.