Merge 2.11 to 2.12 [ci: last-only] #5237

lrytz · 2016-06-19T12:25:27Z

$ git log --decorate --oneline -1 2.11.x | cat
aaf7bc0 (upstream/2.11.x, 2.11.x) Merge pull request #5190 from szeiger/wip/validate-test-use-sbt

$ git log --decorate --oneline -1 2.12.x | cat
79e7334 (HEAD -> 2.12.x, upstream/2.12.x, origin/2.12.x) Merge pull request #5202 from lrytz/lineNumbers

$ export MB=$(git merge-base 2.12.x 2.11.x)

$ git checkout 2.12.x
$ git checkout -b merge-2.11-to-2.12-june-19

$ git log --graph --oneline --decorate $MB...2.11.x | cat
*   aaf7bc0 (upstream/2.11.x, 2.11.x) Merge pull request #5190 from szeiger/wip/validate-test-use-sbt
|\
| * c2c08a4 Remove bnd warnings from log output
| * f2d0f1e Use sbt for PR validation
|/
*   91b6944 Merge pull request #5219 from som-snytt/issue/9245
|\
| * 3f3a3bb SI-9245 Fresher name in Try and test
*   ab06ce8 Merge pull request #5213 from som-snytt/issue/stringof-pariterable
|\
| * be9c0fe SI-9737 [no-merge] Backport stringOf ParIterable
* f775e1f Merge pull request #5223 from FelAl/typo_fix
* c7e2b28 Typo fix

$ git merge f775e1f
CONFLICT (content): Merge conflict in build.sbt
  >> small conflict, easy fix

$ git merge ab06ce8 -s ours
Merge made by the 'ours' strategy.

$ git merge 91b6944

$ git merge aaf7bc0
CONFLICT (content): Merge conflict in versions.properties
  >> partest.version.number

CONFLICT (rename/delete): test/disabled/run/t7933.scala deleted in HEAD and renamed in aaf7bc0.
CONFLICT (rename/delete): test/disabled/run/t7933.check deleted in HEAD and renamed in aaf7bc0.
CONFLICT (rename/delete): test/disabled/run/t7843-jsr223-service.scala deleted in HEAD and renamed in aaf7bc0.
CONFLICT (rename/delete): test/disabled/run/t7843-jsr223-service.check deleted in HEAD and renamed in aaf7bc0.
  >> removed (those tests were removed in 2.12.x in 3cddeaa)

CONFLICT (content): Merge conflict in scripts/jobs/validate/test
  >> nothing complicated (2.12 passed -opt:l:classpath instead of -optimise)

CONFLICT (content): Merge conflict in build.sbt
  >> some conflicts in dependencies (2.12 doesn't have akka actors, continuations, but JOL)
  >> a conflict because 2.12 doesn't build forkjoin
  >> a conflict because 2.12 hase a new command to generated product/function/.. sources (bfb7a74)

Cherry-picked c5f3d3f Edited comment: in stringOf, let GenIterable subsume both Iterable and ParIterable. This change is required for Scala.js compatibility as it does not support parallel collections. Conflicts: src/library/scala/runtime/ScalaRunTime.scala

Fresh name for catcher gets a dollar. "Here, have a dollar." Test due to retronym demonstrates possible conflict. Over the lifetime of the universe, surely at least one code monkey would type in that identifier to catch a banana.

Typo fix build.sbt

SI-9737 [no-merge] Backport stringOf ParIterable

SI-9245 Fresher name in Try and test

- Support directories in `-doc-external-doc`: It is documented as accepting a “classpath_entry_path” for the keys but this only worked for JARs and not for individual class files. When checking for external-doc mappings for a Symbol, we now find the root directory relative to a class file instead of using the full class file path. The corresponding tests for SI-191 and SI8557 are also fixed to support individual class files instead of JARs in partest. This is required for the sbt build which runs partest on “quick” instead of “pack”. - Fix version and repository handling for bootstrapping. The bootstrap `scalaInstance` can now be resolved from any repository added to the project (not just the bootstrap repositories) by using a different workaround for sbt/sbt#1872. - Workaround for sbt/sbt#2640 (putting the wrong `scalaInstance` on partest’s classpath). The required `ScalaInstance` constructor is deprecated, so we have to disable deprecation warnings and fatal warnings until there is a better fix. - Add MiMa to the sbt build (port of the old `test.bc` ant task). The sbt-mima plugin doesn’t have all the features we need, so we do it manually in a similar way to what the plugin does. Checks are done in both directions for the `library` and `compiler` projects. The base version has to be set in `build.sbt`. When set to `None`, MiMa checks are skipped. MiMa checks are run sequentially to avoid spurious errors (see lightbend-labs/mima#115). - Port the OSGi tests to the sbt build. The set of JARs that gets copied into build/osgi as bundles is a bit different from the ant build. We omit the source JARs but add additional modules that are part of the Scala distribution, which seems more correct. - Get rid up `pull-binary-libs.sh` for the sbt build. Add artifacts are resolved from the special bootstrap repository through Ivy. The special `code.jar` and `instrumented.jar` artifacts are copied to the location where partest expects them (because these paths are hardcoded in partest). Other extra JARs for partest in `test/files/lib` are referenced directly from the Ivy cache. - Move common settings that should be available with unqualified names in local `.sbt` files and on the command line into an auto-plugin. - Add an `antStyle` setting to sbt to allow users to easily enable ant-style incremental compilation instead of sbt’s standard name hashing with `set antStyle := true`. - Disable verbose `info`-level logging during sbt startup for both, `validate/test` and `validate/publish-core` jobs. Update logging is no longer disabled when running locally (where it is useful and doesn’t generate excessive output). - Pass optimization flags for scalac down to partest, using the new partest version 1.0.15\6. - Call the new sbt-based PR validation from `scripts/jobs/validate/test`. - Disable the tests `run/t7843-jsr223-service` and `run/t7933` from scala#4959 for now. We need to set up a new test project (either partest or junit) that can run them on a packaged version of Scala, or possibly move them into a separate project that would naturally run from a packaged Scala as part of the community build.

Use sbt for PR validation [ci: last-only]

lrytz · 2016-06-19T12:26:06Z

review by @szeiger

szeiger · 2016-06-20T08:42:31Z

scripts/jobs/validate/test

+       "test" \
+       "partest run pos neg jvm" \
+       "partest res scalap specialized scalacheck" \
+       "partest instrumented presentation" \


I've only seen PermGen errors on 2.11 when running too many partest tests in one go, but the heap error here looks similar. Splitting this up into two runs may work better:

"partest instrumented" \ "partest presentation" \

lrytz · 2016-06-20T10:26:31Z

hmm, no, that didn't help..

lrytz · 2016-06-20T13:40:40Z

I can reproduce the OOME locally, when running the presentation tests with -opt:l:classpath (the stack trace shows that the error happens in the optimizer).

I'm not sure that partest correctly picks up the java options defined here: f2d0f1e#diff-fdc3abdfd754eeb24090dbd90aeec2ceR691. They do show up in the log output:

Java options are:    -Xmx1024M -Xms64M -XX:MaxPermSize=128M

but when I change them to something larger (-Xmx4096M), the process that crashes with OOME does not seem to allocate more memory than before.

lrytz · 2016-06-20T13:49:47Z

to reproduce, run

sbt -Dstarr.version="2.12.0-6be982e-SNAPSHOT" 'setupValidateTest https://scala-ci.typesafe.com/artifactory/scala-pr-validation-snapshots/' 'partest presentation'

note that setupValidateTest also enables the optimizer.

retronym · 2016-06-20T17:02:08Z

@lrytz perhaps try passing invalid JVM options to make sure they are passed down?

lrytz · 2016-06-21T09:21:29Z

Good idea, I tried that. However the result is the same, we still get OOMEs while compiling the first tests. After that error, the other tests are compiled an run, and they do fail because of the invalid flag.

So it seems that the jvm flags are only passed when running the test, but not to the JVM that compiles the tests.

lrytz · 2016-06-21T10:08:24Z

just saw that there's a separate setting for the java options of the JVM that's running partest, changing this helped locally. added a commit 62035e9.

szeiger · 2016-06-21T10:39:36Z

If it's the actual scalac command that doesn't get the options, I don't think this will help. I'm still trying to figure out how they are supposed to be passed down in partest. Maybe there's a problem with DirectCompiler similar to the one we had for scalacOpts.

lrytz · 2016-06-21T11:29:26Z

IRC, partest runs the compiler (actually multiple instances of it) in its own JVM

szeiger · 2016-06-21T11:35:44Z

Ah, I think you're right. It still uses scalacOpts because it creates a new Global with these settings but it shouldn't need separate javaOpts. We should know for sure in a few minutes :)

lrytz · 2016-06-21T12:14:02Z

it helped, all partest tests passed. now we just need to fix osgi :)

[error] Test tools.test.osgi.libonly.BasicLibraryTest.everythingLoads failed: java.lang.ClassNotFoundException: scala.collection.GenTraversableOnce, took 0.537 sec

szeiger · 2016-06-21T12:31:57Z

I haven't cross-checked with 2.11.x yet but this looks suspicious:

*** (o.o.p.e.n.i.NativeTestContainer) Bundle [org.scala-lang.scala-library_2.12.0.v20160621-120706-62035e9 [14]] is not resolved

Maybe we broke something in the bundle creation with an earlier merge of the OSGi functionality into 2.12.x?

szeiger · 2016-06-21T13:06:42Z

So far unable to reproduce this locally. I had some other errors (indicating duplicate bundles) because I didn't run osgiTestFelix/clean before osgiTestFelix/test. Now that I did, the tests passed.

lrytz · 2016-06-21T14:33:39Z

same here, after osgiTestFelix/clean it passed locally, but not before. i don't have any intuition what could cause this (on the ci). should we just add the clean command to the script?

szeiger · 2016-06-21T15:39:05Z

It shouldn't be needed on CI. There shouldn't be anything in the folder that we don't want. The error message when this occured locally was also pretty much the opposite from what we've seen on CI (multiple instances of a bundle instead of ClassNotFoundException)

szeiger · 2016-06-21T17:01:48Z

/rebuild

Partest failures in:

# Failed test paths (this command will update checkfiles)
partest --update-check \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/annotatedRetyping.scala \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/class-symbol-contravariant.scala \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/compiler-asSeenFrom.scala \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/constant-type.scala \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/constrained-types.scala \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/ctries-new \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/ctries-old \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/existential-rangepos.scala \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/existentials-in-compiler.scala \
  /home/jenkins/workspace/scala-2.12.x-validate-test/test/files/run/existentials.scala

Can't reproduce locally. Spurious errors?

lrytz · 2016-06-21T17:04:53Z

it would be great if the jenkins output would show the partest log output of failed tests (see https://scala-ci.typesafe.com/job/scala-2.12.x-validate-test/2389/consoleFull), it used to be the case with ant. not sure what flags we need to pass to partest for that..

som-snytt · 2016-06-21T20:38:58Z

@lrytz --show-diff is the diff, but somewhere I recently restored --show-log, which will show log of crashed tests, where you want the stack trace.

szeiger · 2016-06-22T08:29:16Z

Okay, now we got a totally different set of failing tests...

szeiger · 2016-06-22T08:44:20Z

@lrytz I'm trying to figure out why ant shows more output. I'll push a commit to enable it in sbt, too. Maybe this will give us an idea of what the failures are.

szeiger · 2016-06-22T11:47:28Z

Now that I've added the failure logging in #5242, partest succeeds (as expected). But I was finally able to reproduce the OSGi failure locally. I'm trying to narrow it down.

szeiger · 2016-06-22T15:18:12Z

I was able to reproduce the failure on every test run locally. Upgrading to newer versions of the OSGi test frameworks (same as in the ant build on 2.12.x) seems to fix it. Let's see what Jenkins has to say...

szeiger · 2016-06-22T16:40:47Z

OSGi tests working again! Now disabling MiMa and hoping for a green build (or some of those spurious errors, now with more debug output)

lrytz · 2016-06-22T18:43:47Z

cool! closing this one in favor of #5242

nicolasstucki and others added 12 commits June 3, 2016 10:41

SI-9245 Fresher name in Try and test

3f3a3bb

Fresh name for catcher gets a dollar. "Here, have a dollar." Test due to retronym demonstrates possible conflict. Over the lifetime of the universe, surely at least one code monkey would type in that identifier to catch a banana.

Typo fix

c7e2b28

Merge pull request scala#5223 from FelAl/typo_fix

f775e1f

Typo fix build.sbt

Merge pull request scala#5213 from som-snytt/issue/stringof-pariterable

ab06ce8

SI-9737 [no-merge] Backport stringOf ParIterable

Merge pull request scala#5219 from som-snytt/issue/9245

91b6944

SI-9245 Fresher name in Try and test

Remove bnd warnings from log output

c2c08a4

Merge pull request scala#5190 from szeiger/wip/validate-test-use-sbt

aaf7bc0

Use sbt for PR validation [ci: last-only]

Merge commit 'f775e1f' into merge-2.11-to-2.12-june-19

f2d9748

Merge commit 'ab06ce8' into merge-2.11-to-2.12-june-19

e9afc2d

Merge commit '91b6944' into merge-2.11-to-2.12-june-19

b328366

scala-jenkins added this to the 2.12.0-RC1 milestone Jun 19, 2016

lrytz modified the milestones: 2.12.0-M5, 2.12.0-RC1 Jun 19, 2016

lrytz force-pushed the merge-2.11-to-2.12-june-19 branch from 3302852 to badeb34 Compare June 19, 2016 13:03

szeiger reviewed Jun 20, 2016
View reviewed changes

lrytz force-pushed the merge-2.11-to-2.12-june-19 branch from badeb34 to 6be982e Compare June 20, 2016 08:51

Merge commit 'aaf7bc0' into merge-2.11-to-2.12-june-19

e4c2d68

lrytz force-pushed the merge-2.11-to-2.12-june-19 branch from 6be982e to e4c2d68 Compare June 21, 2016 09:23

more memory for compiling partest tests

62035e9

lrytz mentioned this pull request Jun 21, 2016

reduce the optimizer's heap requirements scala/scala-dev#160

Open

osgiTest/clean first

7958b3d

szeiger mentioned this pull request Jun 22, 2016

Merge 2.11 to 2.12 [ci: last-only] #5242

Merged

lrytz closed this Jun 22, 2016

adriaanm added 2.12 and removed 2.12 labels Oct 29, 2016

Merge 2.11 to 2.12 [ci: last-only] #5237

Merge 2.11 to 2.12 [ci: last-only] #5237

Uh oh!

Conversation

lrytz commented Jun 19, 2016

Uh oh!

lrytz commented Jun 19, 2016

Uh oh!

szeiger Jun 20, 2016

Choose a reason for hiding this comment

Uh oh!

lrytz commented Jun 20, 2016

Uh oh!

lrytz commented Jun 20, 2016

Uh oh!

lrytz commented Jun 20, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

retronym commented Jun 20, 2016

Uh oh!

lrytz commented Jun 21, 2016

Uh oh!

lrytz commented Jun 21, 2016

Uh oh!

szeiger commented Jun 21, 2016

Uh oh!

lrytz commented Jun 21, 2016

Uh oh!

szeiger commented Jun 21, 2016

Uh oh!

lrytz commented Jun 21, 2016

Uh oh!

szeiger commented Jun 21, 2016

Uh oh!

szeiger commented Jun 21, 2016

Uh oh!

lrytz commented Jun 21, 2016

Uh oh!

szeiger commented Jun 21, 2016

Uh oh!

szeiger commented Jun 21, 2016

Uh oh!

lrytz commented Jun 21, 2016

Uh oh!

som-snytt commented Jun 21, 2016

Uh oh!

szeiger commented Jun 22, 2016

Uh oh!

szeiger commented Jun 22, 2016

Uh oh!

szeiger commented Jun 22, 2016

Uh oh!

szeiger commented Jun 22, 2016

Uh oh!

szeiger commented Jun 22, 2016

Uh oh!

lrytz commented Jun 22, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

lrytz commented Jun 20, 2016 •

edited

Loading