Authors: @dwijnand, @eed3si9n

This proposal seeks to work with the community, particularly the authors and maintainers of testing libraries, in order to introduce a basic, zero-dependency unit testing library to the Scala project. Such a library would live in the scala/scala repo and be a part of the Scala distribution, sharing the same Maven organisation id (org.scala-lang), version, release cadence, and bidirection binary compatibility as the rest of the Scala distribution.

This library would, therefore, be able to be used in projects that currently must fallback on using JUnit, such as the Scala project itself and the Scala.js project.

Additionally, we hope that by working with the community we may find a common path forward to overcome some of the fragmentation in testing styles and libraries that is currently present in the Scala community.

Background

When Scala started gaining popularity there was a heavy emphasis on its extensibility via DSL. As such, it attracted a cultural import from other language communities, including the notion of behavior-driven development (BDD). The two most used Scala test frameworks today, ScalaTest and specs2, were created under this trend creating English-like DSLs.

As another cultural import, the notion of property-based testing has been gaining traction over the years, with ScalaCheck as the front runner. Most projects that use ScalaCheck use it through ScalaTest or specs2, possibly via Discipline, with a small number of projects using ScalaCheck alone or ScalaCheck with Claimant. There are also scalaprops and Hedgehog which are alternatives to ScalaCheck, but not as popular.

In recent years, Scala has started to shed some of these older influences and begun to form its own culture, and there's been a swing-back movement towards minimalistic unit test frameworks that do not employ "should"-DSLs. This demand is partly due to our diverse need to cross-build across multiple Scala versions, as well as
multiple platforms (JVM, Scala.JS, and Scala Native). Some projects (particularly the Scala, Scala.js and Scala Native projects themselves) use JUnit for this purpose. uTest and Minitest, as well as the revival of Expecty for power asserts, are examples from this post-BDD, neo-unit-test era, however these aren't as popular.

Conform how Scala code is unit tested

One of the hopes of this proposal is that a standard unit testing library would conform how Scala code is tested, both in documentation (on the official docs, books, blogs/articles/forums/websites) and in projects.

Where books (or other forms of eduction) teaching Scala previously had to either (a) choose which testing library to document and/or use, (b) document and/or use multiple options, or maybe (c) chosen to avoid the topic, with this proposal they could choose to document and/or use the official unit testing library, and perhaps just mention the other options in the ecosystem.

Also, projects that previously had limited choices (e.g. only JUnit) would be able to use the same library the rest of the ecosystem uses.

Provide a unit testing library to zero dependency libraries and for every Scala version

When bootstrapping the Scala ecosystem there is a problem of inter-dependencies between core Scala projects and testing libraries, with ScalaCheck requiring Scala.js and Scala Native, and ScalaTest and specs2 requring ScalaCheck. This puts a few projects into the critical path (the ecosystem depends on the release of Scala.js, Scala Native, and ScalaCheck before it can get its first, non-JUnit, unit testing library) and in awkward positions (for example scala-xml cannot test itself with ScalaTest because its a dependency of ScalaTest). So to avoid cyclical dependencies when bootstrapping some projects choose to use JUnit.

But the use of JUnit for unit testing isn't free, as both Scala.js and Scala Native had to provide special support for it in order for it to actually work, a support that will need maintaince as JUnit continues to evolve.

Some libraries, often to avoid problems when bootstrapping, choose to (or must) have zero dependencies, and that means that they either choose to use JUnit or they create their own testing library, such as the Scala, Scala.js and Scala Native projects using JUnit, which isn't what most of the rest of the ecosystem uses.

The inter-dependency between Scala projects also means that some may consider every non-release, published version of Scala for upcoming major versions to be effectively unusable as there are no non-JUnit unit testing libraries available for it. Specifically this includes every artifact publish from branch builds and pull
request validation. This is why the Scala, Scala.js, and Scala Native (and Dotty) projects use JUnit (and partest) for their testing needs.

Remove any remaining blockers for not testing Scala code

The availability of such a library would also remove any remaining blockers a user might have for not testing their Scala code. These include:

• a greater reduction in concern about the continuing availability of the testing library in new major versions of Scala
• build tools (such as sbt) would be preconfigured to use this library for unit testing
• the Scala project seeds, templates and guides (as well as the seeds, templates and guides of other projects) would more likely contain a unit testing example

Design Decisions

Here are a few design decisions that I think should discussed and actioned:

• should test definition execute in the constructor or in a method?
• should the library require extending a class or trait? use an annotation? manually register? Scala.js </3 annotations and wants extends
• what is the shared glossary of terms? ("test", "example", "test suite", etc)
• matchers or no matchers? how many matchers?
• 1 style, or multiple styles?
• tests as syntax or with values?
• test results with exceptions or with values?
• artifactId? scala-testkit (kind of taken now), testkit, scala-testlib, scala-unit-test?
• package name? scala.testlib?
• no property-based testing? (let's fix unit testing first)
• support in build tools, such as current versions of sbt
• target Scala: 2.14? 3.0? 2.13.1? 3.1?
  • create a backport library for past versions of Scala (2.12, maybe 2.11), like threetenbp?
• stable in 2.14? I think so, with it published pre-2.14 as non-stable (0.x) for experimentation
• let's not repeat history: i.e. scala.testing.UnitTest and scala.testing.SUnit, a short history:
  • May 2003 (Scala 1.0.0) introduction of scala.testing.UnitTest (initially scala.UnitTest) scala/scala@f6ca275
  • Oct 2004 (Scala 1.4.0) introduction of scala.testing.SUnit scala/scala@6c23d94
  • Mar 2007 (Scala 2.4.0) deprecation of UnitTest scala/scala@61b2deb
  • Dec 2007 (Scala 2.7.1) removal of deprecated UnitTest scala/scala@8ad4310
  • Aug 2008 (Scala 2.7.2) deprecation of SUnit scala/scala@e1bca64
  • May 2011 (Scala 2.10.0) removal of deprecated SUnit scala/scala@4afa092
  • July 2019 (Scala 2.14.0) proposal for a standard scala-testlib module?
• (stretch) support in-file unit tests, using the Scala pre-processor? E.g. Rust's #[cfg(test)]
• Future support? Scala.js needs it to support asynchronous tests, i.e., tests returning a Future that will decide whether or not they succeeded

Participants

I propose we involve the creators and maintainers of existing testing libraries in the community, so we can get buy-in and consensus on this proposal and the design decisions in them. So a working group.

Lead: Eugene Yokota (Scala Team, sbt maintainer, and maintainer of the revived Expecty)

Here are the people I think it would be lovely if they were able to participate:

• Bill Venners - ScalaTest
• Eric Torreborre - specs2
• Alexandru Nedelcu - Minitest
• Li Haoyi - uTest (aka "µTest")
• David Barri - Scala Test-State
• Rob Norris - ffstest
• Rickard Nilsson - ScalaCheck
  • or Erik Osheim
  • or Aaron Hawley
• Kenji Yoshida - scalaprops
• Lars Hupel - Discipline
• Erik Osheim - Claimant
  • or Travis Brown
• Charles O'Farrell - Hedgehog