Thanks! 128 is a nice round number, but maybe we can increase with enough of a margin under docker/encrypted/Windows file systems?

An alternative solution would be to build straight to jar on those systems.

Compiling to jar also wouldn't be prone to eg https://issues.scala-lang.org/browse/SI-8199

The limit for ecryptfs is 140.

as @mkotsbak said, the ecryptfs limit is 140, but I believe that's in bytes. So a 128 character filename with, say, 12 2-byte unicode points would hit that limit. So that was my justification for using such a nice round number =p

Can you give me a pointer to "compiling to a jar"? Obviously the problem doesn't occur if you're compiling downstream projects with references to too-long files in jars, but I assume you're referring to some sort of compilation that doesn't involve temporary files?

Yes, scalac supports a jar as the output directory (scalac -d /tmp/output.jar). There's also an sbt plugin that supports mixed java/scala compilation: https://github.com/typesafehub/sbt-tojar

I would prefer working on making compile-straight-to-jar suitable for use on CI (if anything needs to be done) to changing the filename length limits. Likely, compiling to jars is also faster on those FSs.

My concern with the tojar solution is I've never heard of it. The out of the box experience suffers as things stand currently, and I'm not sure how to even provide a path to the tojar solution assuming it works. @adriaanm can you provide some detail why you'd rather not change the limit? Aside from the risk of unknown breakages, it seems to me that having filenames that can fit on a single line would be a benefit with little downside.

I can be convinced otherwise, but my default position is that exceptional circumstances (compiling on docker -- even though I agree they're becoming more common) should not affect the defaults for the common case. With the tojar solution, it's opt-in. Changing how we shorten names affects all users and the whole eco-system. That's a lot of filenames to shorten. Makes me a bit nervous :-) We have time to fix and battle test this in M5, so it's good you bring this up now!

You're right that tojar hasn't been used much, but we'd like to that to change and we're committed to improve it because of the performance implications in huge builds on slower FSs (corporate environments). Like compiling to jars, we also haven't tested the compiler much in the scenario where it has to shorten names more. How does it affect binary compatibility? Incremental compilation? Open questions -- not saying it's necessarily a problem, but we'll have to investigate.

From my perspective, if there's a path to toJar becoming a default or at least a fully supported member of the sbt ecosystem, I'm happy to drop this. I'm trying to get ToJar working with my problematic project now...

Otherwise, while theres some pain and risk (potentially a lot?) in changing the defaults, I think a default length of 255 is needlessly aggressive without bringing a benefit. Using the compile flag is currently useless as I understand it because it has to be the same across the entire classpath. Therefore the fact that changing the default changes everything, is, unfortunately, the point.

Bear in mind that Ubuntu is shipping ecryptfs encrypted home directories with a single checkbox on install as well.

As you said, now is the time to get such changes in!

As an aside, it seems like the community build would be really useful to see if this destroyed everything or nothing. Is there a way to kick it off from a branch or something?

@easel Did you have a look how many of these long names are still generated with 2.12? (I could imagine that the new lambda encoding takes care of quite some cases.)

/cc @SethTisue on trying out a community build with the classname length limit set to 128

@soc I haven't. I'm sure you're right, many of the worst offenders will be cleared up with native lambdas. My thought is to resolve the issue once and for all though. The good news is that it also means this change will affect far fewer actual files =p

In other news, building scalajs-react blew up trying to do something with javadocs with toJar. I will pursue that further elsewhere.

Tojar has an obvious showstopper for most users:

It should also be noted that straight-to-jar compilation will disable the usual incremental compilation mechanisms of sbt: if any source file in a straight-to-jar subproject is modified in a way that makes recompilation necessary, then all of the source files of that subproject will be recompiled.

During development I guess most developers want incremental compilation. The Scala compiler is not known for being lightning fast.

@soc : At least in our case with 2.11.8, we seem to be more likely to hit length limits with -Ybackend:GenBCode -Ydelambdafy:method -target:jvm-1.8 than we were before. cc @adriaanm

anonfun -> lambda is fine, but in some other cases it seems that things have gotten longer. Will report back.

EDIT: It looks like the case where we're more likely to hit this limit is when the compiler chooses to repeat the entire classname (including the package) in the classname. Haven't tried isolating it (this is codegen'd code), but an example generated class is:

com/twitter/ads/db/service/thriftscala/ConversionLiftStudyAttributionWindowsScanContinuation$class$lambda$$com$twitter$ads$db$service$thriftscala$ConversionLiftStudyAttributionWindowsScanContinuation$class$$$nestedInAnonfun$2$1.class

EDIT2: And it looks like -Xmax-classfile-name is not actually respected in this case... even with the value set to 127, you'll get a classname like the above.

Scala 2.12 will improve the situation here by virtue of the fact that we no longer need to emit classes at all for lambdas.

class Test {
  def test(l: List[String]) = {
    for {_ <- l; _ <- l; _ <- l; _ <- l; _ <- l; _ <- l; _ <- l; _ <- l; _ <- l} yield 42
  }
}

% mkdir -p /tmp/out; ~/scala/2.11/bin/scalac -d /tmp/out sandbox/test1.scala && ls /tmp/out

Test$$anonfun$test$1$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6$$anonfun$apply$7$$anonfun$apply$8.class
Test$$anonfun$test$1$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6$$anonfun$apply$7.class
Test$$anonfun$test$1$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6.class
Test$$anonfun$test$1$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3$$anonfun$apply$4$$anonfun$apply$5.class
Test$$anonfun$test$1$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3$$anonfun$apply$4.class
Test$$anonfun$test$1$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3.class
Test$$anonfun$test$1$$anonfun$apply$1$$anonfun$apply$2.class
Test$$anonfun$test$1$$anonfun$apply$1.class
Test$$anonfun$test$1.class
Test.class

$ mkdir -p /tmp/out; rm /tmp/out/*.class; ~/scala/2.12/bin/scalac -d /tmp/out sandbox/test1.scala && ls /tmp/out
Test.class

The long names will still be generated sometimes when using deeply nested non-functions, though.

package com.acme.wizzle.wozzle.wibble

class Test {
  private def test(l: List[String]) = {
    new { new { new { new { new { new { new { new { new { new {}}}}}}}}}}
  }
}
object Test {
  def makeJVMPublic(t: Test) = t.test(Nil)
}

⚡ ⚡ mkdir -p /tmp/out; rm /tmp/out/**/*.class; ~/scala/2.12/bin/scalac -d /tmp/out sandbox/test1.scala && find /tmp/out
/tmp/out
/tmp/out/com
/tmp/out/com/acme
/tmp/out/com/acme/wizzle
/tmp/out/com/acme/wizzle/wozzle
/tmp/out/com/acme/wizzle/wozzle/wibble
/tmp/out/com/acme/wizzle/wozzle/wibble/Test$$anon$1$$anon$2$$anon$3$$anon$4$$anon$5$$anon$6$$anon$7$$anon$8$$anon$9$$anon$10.class
/tmp/out/com/acme/wizzle/wozzle/wibble/Test$$anon$1$$anon$2$$anon$3$$anon$4$$anon$5$$anon$6$$anon$7$$anon$8$$anon$9.class
/tmp/out/com/acme/wizzle/wozzle/wibble/Test$$anon$1$$anon$2$$anon$3$$anon$4$$anon$5$$anon$6$$anon$7$$anon$8.class
/tmp/out/com/acme/wizzle/wozzle/wibble/Test$$anon$1$$anon$2$$anon$3$$anon$4$$anon$5$$anon$6$$anon$7.class
/tmp/out/com/acme/wizzle/wozzle/wibble/Test$$anon$1$$anon$2$$anon$3$$anon$4$$anon$5$$anon$6.class
/tmp/out/com/acme/wizzle/wozzle/wibble/Test$$anon$1$$anon$2$$anon$3$$anon$4$$anon$5.class
/tmp/out/com/acme/wizzle/wozzle/wibble/Test$$anon$1$$anon$2$$anon$3$$anon$4.class
/tmp/out/com/acme/wizzle/wozzle/wibble/Test$$anon$1$$anon$2$$anon$3.class
/tmp/out/com/acme/wizzle/wozzle/wibble/Test$$anon$1$$anon$2.class
/tmp/out/com/acme/wizzle/wozzle/wibble/Test$$anon$1.class
/tmp/out/com/acme/wizzle/wozzle/wibble/Test$.class
/tmp/out/com/acme/wizzle/wozzle/wibble/Test.class

package com.acme.wizzle.wozzle.wibble

trait Sam {
  def apply(): Sam
  val field = 42
}
class Test {
  private def test(l: List[String]): Sam = {
    {() => () => () => () => () => () => () => () => () => (null: Sam)}
  }
}
object Test {
  def makeJVMPublic(t: Test) = t.test(Nil)
}

⚡ mkdir -p /tmp/out; rm /tmp/out/**/*.class; ~/scala/2.12/bin/scalac -d /tmp/out sandbox/test1.scala && find /tmp/out
/tmp/out
/tmp/out/com
/tmp/out/com/acme
/tmp/out/com/acme/wizzle
/tmp/out/com/acme/wizzle/wozzle
/tmp/out/com/acme/wizzle/wozzle/wibble
/tmp/out/com/acme/wizzle/wozzle/wibble/Sam.class
/tmp/out/com/acme/wizzle/wozzle/wibble/Test$$anonfun$com$acme$wizzle$wozzle$wibble$Test$$$nestedInanonfun$1$1.class
/tmp/out/com/acme/wizzle/wozzle/wibble/Test$$anonfun$com$acme$wizzle$wozzle$wibble$Test$$$nestedInanonfun$2$1.class
/tmp/out/com/acme/wizzle/wozzle/wibble/Test$$anonfun$com$acme$wizzle$wozzle$wibble$Test$$$nestedInanonfun$3$1.class
/tmp/out/com/acme/wizzle/wozzle/wibble/Test$$anonfun$com$acme$wizzle$wozzle$wibble$Test$$$nestedInanonfun$4$1.class
/tmp/out/com/acme/wizzle/wozzle/wibble/Test$$anonfun$com$acme$wizzle$wozzle$wibble$Test$$$nestedInanonfun$5$1.class
/tmp/out/com/acme/wizzle/wozzle/wibble/Test$$anonfun$com$acme$wizzle$wozzle$wibble$Test$$$nestedInanonfun$6$1.class
/tmp/out/com/acme/wizzle/wozzle/wibble/Test$$anonfun$com$acme$wizzle$wozzle$wibble$Test$$$nestedInanonfun$7$1.class
/tmp/out/com/acme/wizzle/wozzle/wibble/Test$$anonfun$com$acme$wizzle$wozzle$wibble$Test$$$nestedInanonfun$8$1.class
/tmp/out/com/acme/wizzle/wozzle/wibble/Test$$anonfun$com$acme$wizzle$wozzle$wibble$Test$$test$1.class
/tmp/out/com/acme/wizzle/wozzle/wibble/Test$.class
/tmp/out/com/acme/wizzle/wozzle/wibble/Test.class

The problem with 2.11.8 -Ydelambdafy:method not respecting the max classfile length is https://issues.scala-lang.org/browse/SI-9319. I found an unmerged commit in my fork where I attempted to fix this, I can't remember why i didn't submit this: retronym@41ac572.

I also remember proposing a change to make the full names of nested classes shorter by eliding anonymous owners in #852.

We've discussed this more and we do not feel comfortable changing the length of class file names on 2.12. There's a real danger of causing other problems with this, the flag itself being untested and incomplete. Luckily, it should no longer be needed thanks to 2.12's lambda encoding. We could still consider generating shorter names where breakage still occurs (for example by no longer repeating $anon over and over in these names, as in the commit linked below).

For 2.11, we'd be happy to accept a fix based on retronym@41ac572.

Thanks for the feedback @adriaanm. I will see if I can make sense of back porting retronym/scala@41ac572.

Ran into this on https://github.com/cdiniz/slick-akka-http-oauth2 just now

@wsargent but not with Scala 2.12?

@mkotsbak 2.11.7 on Cinnamon Mint, but not inside of a VM. Could not reproduce on MacOS. Have not tried upgrading to 2.12.

I ran into this while running zinc tests on Drone in Scala 2.11.1. I fixed it with -Xmax-classfile-name 240.

Note that you cannot safely use -Xmax-classfile-name in combination with any other library, it's not binary compatible: https://groups.google.com/d/msg/scala-internals/hNWuwWBJCOg/CrvPhnqJVaMJ

Thanks for the heads up. Fortunately, I'm only using it for one test and I don't need binary compatibility.

Any chance we could revive this for scala 2.13 or 2.14?

@stewSquared Adriaan said, "There's a real danger of causing other problems with this, the flag itself being untested and incomplete", so it's probably already too late for 2.13 and would require someone to seriously look at it to get in 2.14. What kind of code are you writing that hits this? Maybe the problem can be mitigated at the source.

I also wonder if it wouldn't make more sense to fix the damn file system to accept longer file names :).

@smarter I did read that; I figured 2.14 might be a good opportunity to fend off future filesystem trouble preemptively, since 2.14 will already break binary compatibility. We're already hashing long (> 255) classfile names, so hashing slightly less long classfile names so that dockerized CI doesn't break seems like a small risk with good reward (limit 240?).

Anyway, I discovered this with a project built for scala 2.11 with -Ydelambdafy:method, so scala/bug#9319 is relevant, and the limit would be ignored anyway. I'm removing the compiler flag for now, but this also shouldn't affect me after I migrate the project to scala 2.12.

see also #7497, anyone who commented on this ticket might be interested in reading/commenting there as well

for anyone trying to compile to jar the biggest issue that we had was the tool chain

e.g. incremental compilation - you need to remove the old jar entries and append to the jar, not create a new one. None of this was that hard, but without support from your build tool and IDE it is a bit fiddly

well worth it for the speedup that we got though :-)
best of all is a clean of your workspace on windows deletes a few jars, not 000s of files very slowly