should I run a community build on this one?

@SethTisue yes, that'd be great!

For reference, I just tried this change in Bloop. The frontend of bloop is the biggest module in our build, and it uses Shapeless via caseapp. Our compile times have been unusually slow because of Lazy. I'm seeing a 2x speedups in cold compilation, which is a great number. We're not even big users of Shapeless (only one file using Shapeless in the whole repository), so I would expect Shapeless-based codebases (many in our ecosystem) to benefit from this change.

Some numbers, on a cold JVM, post sbt clean update:

• ensime-server, api/compile (heavy user of shapeless)
  • 2.12.5: 47s
  • this: 47s (no change)
  • https://github.com/fommil/ensime-server/tree/experiment-implicit-pruning <= has semanticdb / scalafix disabled
• scalaz-deriving, test:compile
  • 2.12.5: 19s
  • this: 19s
  • the ADTs are very small, so I wouldn't expect much change
• my work project, (~500 files, 30k LOC) compile in main and all test configurations
  • 2.12.5: 167s
  • this: 146s (but sbt-assembly failed due to duplicate classfile entries, seems there is an sbt bug about pruning the scala-library.jar when using a custom scala dist)
  • just compile: 49s vs 45s (a small speedup)

All these projects are heavy users of shapeless generic derivation. However, I am using "semiauto" style, i.e. def gen is not implicit, and therefore not recursive. It is more boilerplate, but I feel it addresses all the problems that were highlighted in export-hook whilst being much faster to compile (I use the @scalaz.deriving annotation to generate the boilerplate).

I suspect semiauto is equivalent to the automated polymorphic pruning. Perhaps it would be feasible to disable semiauto and just derive the top level of the ADT, but I don't have time to do that refactor at the moment. My work project also has some scanamo and pureconfig uses, which are doing full auto derivation at the point of use, so that's where I guess the benefits are coming from.

sbt outputs this which gives me reason to believe it is working as expected:

> ++2.12.5=/home/fommil/Projects/scala/build/pack!
[info] Using Scala home /home/fommil/Projects/scala/build/pack with actual version 2.12.7-20180430-184100-af4ffa8
[info] Forcing Scala version to 2.12.5 on all projects.

I suspect semiauto is equivalent to the automated polymorphic pruning. Perhaps it would be feasible to disable semiauto and just derive the top level of the ADT

That would be a very useful comparison to make: semiauto style trades more boilerplate for lower compilation times, so if with this change fully automatic derivation is on a par with semiauto wrt compile times that's a big win.

That would be a very useful comparison to make: semiauto style trades more boilerplate for lower compilation times, so if with this change fully automatic derivation is on a par with semiauto wrt compile times that's a big win.

Note that there's still value in having semiauto over fully automatic derivation: runtime performance and code size. Semiauto is more boilerplate-y but allows programmers to control better where the instances live, and even ensure there's only one instance per type in the codebase.

@jvican I have a plan up my sleeve to solve the boilerplate problem and improve the runtime performance even further, but I don't want to hijack this thread 😉

@jvican the missing part of the picture is here IMO: #6139.

results from https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-community-build/2789/consoleFull. a lot of projects were green, but specs2 crashed in typer:

[specs2] scala.MatchError: ?T (of class scala.reflect.internal.Types$TypeVar$$anon$4)
[specs2] 	at scala.reflect.internal.Variances.inType$1(Variances.scala:211)
[specs2] 	at scala.reflect.internal.Variances.$anonfun$varianceInType$1(Variances.scala:206)
[specs2] 	at scala.reflect.internal.Variances.inArgs$1(Variances.scala:206)
[specs2] 	at scala.reflect.internal.Variances.inType$1(Variances.scala:219)
[specs2] 	at scala.reflect.internal.Variances.$anonfun$varianceInType$1(Variances.scala:206)
[specs2] 	at scala.reflect.internal.Variances.inArgs$1(Variances.scala:206)
[specs2] 	at scala.reflect.internal.Variances.inType$1(Variances.scala:219)
[specs2] 	at scala.reflect.internal.Variances.varianceInType(Variances.scala:228)
[specs2] 	at scala.reflect.internal.Variances.varianceInType$(Variances.scala:205)
[specs2] 	at scala.reflect.internal.SymbolTable.varianceInType(SymbolTable.scala:18)
[specs2] 	at scala.tools.nsc.typechecker.Implicits$ImplicitSearch.$anonfun$matchesPtInst$3(Implicits.scala:582)
[specs2] 	at scala.tools.nsc.typechecker.Implicits$ImplicitSearch.scala$tools$nsc$typechecker$Implicits$ImplicitSearch$$matchesPtInst(Implicits.scala:582)
[specs2] 	at scala.tools.nsc.typechecker.Implicits$ImplicitSearch$ImplicitComputation.rankImplicits(Implicits.scala:1008)
[specs2] 	at scala.tools.nsc.typechecker.Implicits$ImplicitSearch$ImplicitComputation.findBest(Implicits.scala:1038)
[specs2] 	at scala.tools.nsc.typechecker.Implicits$ImplicitSearch.searchImplicit(Implicits.scala:1095)

with lots more stack trace where that came from.

losing specs2 knocks out many downstream projects, of course.

Hmm, https://blog.stephenn.com/2017/07/circe-argonaut-shapless-play-json-compile-time.html suggests that semiauto is no faster than full auto in batch compile (it is, of course, significantly faster on a per case class basis, and hence better in the incremental compiler and presentation compiler).

I don't know how to explain this. I would really like to see @stephennancekivell redo the benchmarks. It would be very interesting if full auto was somehow faster than semi-auto with this patch applied! Perhaps because it finds things in the import scope and never has to check companions and nested package objects.

hey @fommil, Im happy to help.

I updated those json codec deriving benchmarks, but unfortunately the results aren’t very promising. Maybe my benchmarks are more impacted by the case class to Hlist macro, than the amount of implicits looked up.

Here are some results.

Here building codecs for 300 case classes with up to 15 values.
circe
71s
circe.topic
81s
circe-derivation
31s
circe-derivation.topic
30s

More at https://gist.github.com/stephennancekivell/82b481eda1653f5c0922a229fa4577a1

I tried increasing the number of case class values to 100, but typer crashed with a stack overflow error. At 50x50 it wasn’t much different.

Edit: To be clear, my benchmark just creates a bunch of codes like this. https://github.com/stephennancekivell/circe-argonaut-compile-times/blob/master/circe/src/main/scala/Codecs.scala

Rebased, incorporated feedback from #6580 and hopefully fixed the Specs2 issue seen in the community build.

@SethTisue I had a tough time reproducing the issue you saw, but I think it should be fixed now. If you could run another community build when this goes green that would be awesome :-)

@milessabin you might prefer to just do it yourself, rather than wait for me? you can just hit "rebuild" on the last one and change the SHA in the Scala version parameter.

the disk space issues we used to have on the Jenkins workers are fixed now, so it's fine to launch community build jobs without worrying about what else is going on

Rerun here ... a failure in http4s which I think is probably spurious and a failure in scala-refactoring, which I'll investigate, otherwise all green :-)

the http4s failure looks spurious to you in what sense...?

@SethTisue my mistake ... I was confusing the http4s output with the blaze output.

I'll investigate.

Rebased, incorporated #6612, and fix for http4s issue seen in the community build.

I'll rerun the community build when this goes green.

Failed community build run here: https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-community-build/3010/

@SethTisue the failures here look pretty random and mostly IO and environment related. I find it fairly hard to believe that they're related to this PR. What's the current expected state of the 2.12.x community build?

@milessabin

• scala-js: known failure, pursuing it at compiler crash in Scala 2.12 community build scala-js/scala-js#3381
• akka-more: looks like a transient failure to resolve a dependency, should go away on re-run
• http4s, breeze, eff, classutil: idk what these are, I have not seen these in any other runs, so I think you have to suspect your PR first, I'm afraid. but also, just try re-running the whole thing (it shouldn't take 13 hours this time since dbuild will have previous results cached)

once https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-community-build/3014/ finishes, you can take it as a representative recent run. (in other recent runs specs2 failed which took out many downstream projects, but 3014 has a workaround for that)

What's the current expected state of the 2.12.x community build?

(if you're a William Pène du Bois fan, if not, don't worry about it)

I hit "rebuild" for you, it'll be https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-community-build/3015/ (404 til Jenkins queue clears)

I'm still puzzled by the results. I'll wait for a green build and then try again from there.

Another unsuccessful run here: https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-integrate-community-build/3030/.

We now have breeze, classutil, specs2-scalaz, scala-debugger, http4s failing. I think this is due to implicit conversions being resolved differently ... I'll investigate.

@SethTisue the classutil failure is reproducible, but unrelated to this PR ... one of the tests is expecting a non-None result from scala.util.Properties.releaseVersion. Do you have something similar to the below for the main 2.12.x community build? Or is there something missing in the configuration for this this build?

diff --git src/test/scala/org/clapper/classutil/ClassFinderSpec.scala src/test/scala/org/clapper/classutil/ClassFinderSpec.scala
index cb8849f..a5cef52 100644
--- src/test/scala/org/clapper/classutil/ClassFinderSpec.scala
+++ src/test/scala/org/clapper/classutil/ClassFinderSpec.scala
@@ -66,7 +66,7 @@ class ClassFinderSpec extends BaseSpec {
 
   private val (runtimeClassFiles, runtimeClassFinder) = {
     import scala.util.Properties
-    val version = Properties.releaseVersion.get
+    val version = Properties.releaseVersion.getOrElse("2.12.6")
     val shortVersion = version.split("""\.""").take(2).mkString(".")
 
     val targetDirectory: Option[File] = Array(

@SethTisue @adriaanm @etorreborre ...

The failure in specs2-scalaz is a bit of a weird one. It turns out that the backport of #6612 is the culprit. It seems that the the failing line in TaskMatchers was relying on side-effects in the unseen and now suppressed TypeDiagnostics.findReqMsg.

Partially reverting the definition of ContextErrors.typeErrorMsg to,

def typeErrorMsg(context: Context, found: Type, req: Type) =
  // if (context.openImplicits.nonEmpty && !settings.XlogImplicits.value) "type mismatch"
  // else "type mismatch" + foundReqMsg(found, req)
  "type mismatch" + foundReqMsg(found, req)

fixes the problem but defeats the object of #6612.

#6612 is present in 2.13.0-M4 which spec2-scalaz builds with, so what's happening there? It turns out that the problem has been worked around, I guess more or less incidentally as part of the process of updating for the new collections,

--- scalaz/shared/src/main/scala/org/specs2/matcher/TaskMatchers.scala
+++ scalaz/shared/src/main/scala/org/specs2/matcher/TaskMatchers.scala
@@ -26,8 +26,8 @@ trait TaskMatchers {
   def returnBefore[T](duration: Duration): TaskMatcher[T] =
     attemptRun(ValueCheck.alwaysOk, Some(duration))
 
-  def failWith[T <: Throwable : ClassTag]: Matcher[Task[_]] =
-    returnValue(be_-\/(haveClass[T])) ^^ { t: Task[_] => t.attempt }
+  def failWith[T <: Throwable : ClassTag]: Matcher[Task[Any]] =
+    returnValue(be_-\/[Throwable](haveClass(implicitly[ClassTag[T]]))) ^^ { t: Task[Any] => t.attempt }
 
   private[specs2] def attemptRun[T](check: ValueCheck[T], duration: Option[Duration]): TaskMatcher[T] =
     TaskMatcher(check, duration)

Applying the same change to specs2 for the 2.12.x build also fixes the problem with the backport of #6612 here too.

@SethTisue I'd like to push on with this community build ... would you be happy to fork specs2 with the above change? Is there some way it can be done without disrupting the normal 2.12.x community build?

The change in specs2 can be minimized slightly: the following,

--- scalaz/shared/src/main/scala/org/specs2/matcher/TaskMatchers.scala
+++ scalaz/shared/src/main/scala/org/specs2/matcher/TaskMatchers.scala
@@ -27,7 +27,7 @@ trait TaskMatchers {
     attemptRun(ValueCheck.alwaysOk, Some(duration))
 
   def failWith[T <: Throwable : ClassTag]: Matcher[Task[_]] =
-    returnValue(be_-\/(haveClass[T])) ^^ { t: Task[_] => t.attempt }
+    returnValue(be_-\/[Throwable](haveClass[T])) ^^ { t: Task[_] => t.attempt }
 
   private[specs2] def attemptRun[T](check: ValueCheck[T], duration: Option[Duration]): TaskMatcher[T] =
     TaskMatcher(check, duration)

appears to do the trick. @etorreborre ... would you be happy to make that change in specs2 for 2.12+?

This also appears to be responsible for the http4s failure in dsl/test:compile.

The scala-debugger failure appears to be a spurious runtime glitch.

I noticed one of our larger Scala projects fails to build using this patch (assuming I correctly built Scala using this branch).

The code that fails is basically this:

implicit def toOption[T](v: T): Option[T] = Option(v)
val a: Int = 123
val b: Option[Long] = a // Works under 2.12.6 but not with the implicit-poly-prune-2.12.x PR

I'm not really sure if this was intended to work in the first place but this pattern
showed up in a large Scala project we have. I'll probably remove the implicit from
our code and make all the conversions explicit since that is not a very good implicit
to have in the first place. However I'm not sure if the PR intended to change this
behavior.

Here is an SBT project that demonstrates the issue: https://github.com/tpunder/implicit-poly-prune-2.12.x-failure

@tpunder that looks similar to the breeze failure ... I'm working on a fix at the moment.

@milessabin my understanding of this change

returnValue(be_-\/[Throwable](haveClass[T])) ^^ { t: Task[_] => t.attempt }

is that [Throwable] has to be added to help type inference later on with ^^. Is that right? This use of ^^ (or adapt) is generally localized, meaning it is only used when defining matchers, not using them. If on the other hand every time people wanted to write be_-\/(haveClass[T]) they would have to write be_-\/[Throwable](haveClass[T]) that would be breaking a lot more code.

@etorreborre yes, I believe so.

I've tracked down and fixed the issue with breeze now so, given that this isn't going to compile cleanly with #6612 I'm going to declare victory on this and wrap up #6580.

I got ~8% improvement on my current work project 👍 (just tried it out though, no benchmarks or anything reliable)

@joroKr21 comments on #6580 would be helpful :-)