All Your Tests Belong to You: Maintaining Mixed JUnit 4/JUnit 5 and Testng/JUnit 5 Test Suites

If you are seasoned Java developer who practices test-driven development (hopefully, everyone does it), it is very likely JUnit 4 has been your one-stop-shop testing toolbox. Personally, I truly loved it and still love: simple, minimal, non-intrusive and intuitive. Along with terrific libraries like Assertj and Hamcrest it makes writing test cases a pleasure.

But time passes by, Java has evolved a lot as a language, however JUnit 4 was not really up for a ride. Around 2015 the development of JUnit 5 has started with ambitious goal to become a next generation of the programmer-friendly testing framework for Java and the JVM. And, to be fair, I think this goal has been reached: many new projects adopt JUnit 5 from the get-go whereas the old ones are already in the process of migration (or at least are thinking about it).

For existing projects, the migration to JUnit 5 will not happen overnight and would probably take some time. In today's post we are going to talk about the ways to maintain mixed JUnit 4 / JUnit 5 and TestNG / JUnit 5 test suites with a help of Apache Maven and Apache Maven Surefire plugin.

To have an example a bit more realistic, we are going to test a UploadDestination class, which basically just provides a single method which says if a particular destination scheme is supported or not:

import java.net.URI;

public class UploadDestination {
    public boolean supports(String location) {
        final String scheme = URI.create(location).getScheme();
        return scheme.equals("http") || scheme.equals("s3") || scheme.equals("sftp");
    }
}

The implementer was kind enough to create a suite of JUnit 4 unit tests to verify that all expected destination schemes are indeed supported.

import static org.junit.Assert.assertTrue;

import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameters;

@RunWith(Parameterized.class)
public class JUnit4TestCase {
    private UploadDestination destination;
    private final String location;

    public JUnit4TestCase(String location) {
        this.location = location;
    }

    @Before
    public void setUp() {
        destination = new UploadDestination();
    }

    @Parameters(name= "{index}: location {0} is supported")
    public static Object[] locations() {
        return new Object[] { "s3://test", "http://host:9000", "sftp://host/tmp" };
    }

    @Test
    public void testLocationIsSupported() {
        assertTrue(destination.supports(location));
    }
}

In the project build, at very least, you need to add JUnit 4 dependency along with Apache Maven Surefire plugin and, optionally Apache Maven Surefire Reporter plugin, to your pom.xml, the snippet below illustrates that.

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.13.1</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.0.0-M5</version>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-report-plugin</artifactId>
                <version>3.0.0-M5</version>
            </plugin>
        </plugins>
    </build>

No magic here, triggering Apache Maven build would normally run all unit test suites every time.

...
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.example.JUnit4TestCase
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 s - in com.example.JUnit4TestCase
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
...

Awesome, let us imagine at some point another teammate happens to work on the project and noticed there are no unit tests verifying the unsupported destination schemes so she adds some using JUnit 5.

package com.example;

import static org.junit.jupiter.api.Assertions.assertFalse;

import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;

class JUnit5TestCase {
    private UploadDestination destination;

    @BeforeEach
    void setUp() {
        destination = new UploadDestination();
    }

    @ParameterizedTest(name = "{index}: location {0} is supported")
    @ValueSource(strings = { "s3a://test", "https://host:9000", "ftp://host/tmp" } )
    public void testLocationIsNotSupported(String location) {
        assertFalse(destination.supports(location));
    }
}

Consequently, another dependency appears in the project's pom.xml to bring JUnit 5 in (since its API is not compatible with JUnit 4).

    <dependencies>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <version>5.7.0</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

Looks quite legitimate, isn't it? But there is a catch ... the test run results would surprise this time.

...
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.example.JUnit5TestCase
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.076 s - in com.example.JUnit5TestCase
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
...

The JUnit 4 test suites are gone and such a behavior is actually well documentented by Apache Maven Surefire team in the Provider Selection section of the official documentation. So how we could get them back? There are a few possible options but the simplest one by far is to use JUnit Vintage engine in order to run JUnit 4 test suites using JUnit 5 platform.

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.0.0-M5</version>
                <dependencies>
                    <dependency>
                        <groupId>org.junit.vintage</groupId>
                        <artifactId>junit-vintage-engine</artifactId>
                        <version>5.7.0</version>
                    </dependency>
                </dependencies>
            </plugin>

With that, both JUnit 4 and JUnit 5 test suites are going to be executed side by side.

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.example.JUnit5TestCase
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.079 s - in com.example.JUnit5TestCase
[INFO] Running com.example.JUnit4TestCase
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 s - in com.example.JUnit4TestCase
[INFO] 
[INFO] Results:
[INFO]
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------

The lesson to learn here: please watch carefully that all your test suites are being executed (the CI/CD usually keeps track of such trends and warns you right away). Especially, be extra careful when migrating to latest Spring Boot or Apache Maven Surefire plugin versions.

Another quite common use case you may run into is mixing the TestNG and JUnit 5 test suites in the scope of one project. The symptoms are pretty much the same, you are going to wonder why only JUnit 5 test suites are being run. The treatment in this case is a bit different and one of the options which seems to work pretty well is to enumerate the test engine providers explicitly.

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.0.0-M5</version>
                <dependencies>
                    <dependency>                                      
                        <groupId>org.apache.maven.surefire</groupId>  
                        <artifactId>surefire-junit-platform</artifactId>      
                        <version>3.0.0-M5</version>                   
                    </dependency>
                    <dependency>                                      
                        <groupId>org.apache.maven.surefire</groupId>  
                        <artifactId>surefire-testng</artifactId>      
                        <version>3.0.0-M5</version>                   
                    </dependency>                          
                </dependencies>
            </plugin>

The somewhat undesired effect in this case is the fact that the test suites are run separately (there are other ways though to try out), for example:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.example.JUnit5TestCase
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.074 s - in com.example.JUnit5TestCase
[INFO] 
[INFO] Results:
[INFO]
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running TestSuite
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.315 s - in TestSuite
[INFO] 
[INFO] Results:
[INFO]
INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] 
[INFO] ------------------------------------------------------------------------

To be fair, I think JUnit 5 is a huge step forward towards having modern and concise test suites for your Java (and in general, JVM) projects. These days there are seamless integrations available with mostly any other test framework or library (Mockito, TestContainers, ... ) and the migration path is not that difficult in most cases. Plus, as you have seen, co-existence of JUnit 5 with older test engines is totally feasible.

As always, the complete project samples are available on Github: JUnit 4/JUnit 5, TestNG / JUnit 5.

Testing Spring Boot conditionals the sane way

If you are more or less experienced Spring Boot user, it is very luckily that at some point you may need to run into the situation when the particular beans or configurations have to be injected conditionally. The mechanics of it is well understood but sometimes the testing such conditions (and their combinations) could get messy. In this post we are going to talk about some possible (arguably, sane) ways to approach that.

Since Spring Boot 1.5.x is still widely used (nonetheless it is racing towards the EOL this August), we would include it along with Spring Boot 2.1.x, both with JUnit 4.x and JUnit 5.x. The techniques we are about to cover are equally applicable to the regular configuration classes as well as auto-configurations classes.

The example we will be playing with would be related to our home-made logging. Let us assume our Spring Boot application requires some bean for dedicated logger with name "sample". In certain circumstances however this logger has to be disabled (or become effectively a noop), so the property logging.enabled serves like a kill switch here. We use Slf4j and Logback in this example, but it is not really important. The LoggingConfiguration snippet below reflects this idea.

@Configuration
public class LoggingConfiguration {
    @Configuration
    @ConditionalOnProperty(name = "logging.enabled", matchIfMissing = true)
    public static class Slf4jConfiguration {
        @Bean
        Logger logger() {
            return LoggerFactory.getLogger("sample");
        }
    }
    
    @Bean
    @ConditionalOnMissingBean
    Logger logger() {
        return new NOPLoggerFactory().getLogger("sample"); 
    }
}

So how would we test that? Spring Boot (and Spring Framework in general) has always offered the outstanding test scaffolding support. The @SpringBootTest and @TestPropertySource annotations allow to quickly bootstrap the application context with the customized properties. There is one issue though: they are applied per test class level, not a per test method. It certainly makes sense but basically requires you to create a test class per combination of conditionals.

If you are still with JUnit 4.x, there is one trick you may found useful which exploits Enclosed runner, the hidden gem of the framework.

@RunWith(Enclosed.class)
public class LoggingConfigurationTest {
    @RunWith(SpringRunner.class)
    @SpringBootTest
    public static class LoggerEnabledTest {
        @Autowired private Logger logger;
        
        @Test
        public void loggerShouldBeSlf4j() {
            assertThat(logger).isInstanceOf(ch.qos.logback.classic.Logger.class);
        }
    }
    
    @RunWith(SpringRunner.class)
    @SpringBootTest
    @TestPropertySource(properties = "logging.enabled=false")
    public static class LoggerDisabledTest {
        @Autowired private Logger logger;
        
        @Test
        public void loggerShouldBeNoop() {
            assertThat(logger).isSameAs(NOPLogger.NOP_LOGGER);
        }
    }
}

You still have the class per condition but at least they are all in the same nest. With JUnit 5.x, some things got easier but not to the level as one might expect. Unfortunately, Spring Boot 1.5.x does not support JUnit 5.x natively, so we have to rely on extension provided by spring-test-junit5 community module. Here are the relevant changes in pom.xml, please notice that junit is explicitly excluded from the spring-boot-starter-test dependencies graph.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-test</artifactId>
    <scope>test</scope>
    <exclusions>
        <exclusion>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
        </exclusion>
    </exclusions>
</dependency>

<dependency>
    <groupId>com.github.sbrannen</groupId>
    <artifactId>spring-test-junit5</artifactId>
    <version>1.5.0</version>
    <scope>test</scope>
</dependency>

<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-api</artifactId>
    <version>5.5.0</version>
    <scope>test</scope>
</dependency>

<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-engine</artifactId>
    <version>5.5.0</version>
    <scope>test</scope>
</dependency>

The test case itself is not very different besides usage of the @Nested annotation, which comes from JUnit 5.x to support tests as inner classes.

public class LoggingConfigurationTest {
    @Nested
    @ExtendWith(SpringExtension.class)
    @SpringBootTest
    @DisplayName("Logging is enabled, expecting Slf4j logger")
    public static class LoggerEnabledTest {
        @Autowired private Logger logger;
        
        @Test
        public void loggerShouldBeSlf4j() {
            assertThat(logger).isInstanceOf(ch.qos.logback.classic.Logger.class);
        }
    }
    
    @Nested
    @ExtendWith(SpringExtension.class)
    @SpringBootTest
    @TestPropertySource(properties = "logging.enabled=false")
    @DisplayName("Logging is disabled, expecting NOOP logger")
    public static class LoggerDisabledTest {
        @Autowired private Logger logger;
        
        @Test
        public void loggerShouldBeNoop() {
            assertThat(logger).isSameAs(NOPLogger.NOP_LOGGER);
        }
    }
}

If you try to run the tests from the command line using Apache Maven and Maven Surefire plugin, you might be surprised to see that none of them were executed during the build. The issue is that ... all nested classes are excluded ... so we need to put in place another workaround.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.22.2</version>
    <configuration>
        <excludes>
            <exclude />
        </excludes>
    </configuration>
</plugin>

With that, things should be rolling smoothly. But enough about legacy, the Spring Boot 2.1.x comes as the complete game changer. The family of the context runners, ApplicationContextRunner, ReactiveWebApplicationContextRunner and WebApplicationContextRunner, provide an easy and straightforward way to tailor the context on per test method level, keeping the test executions incredibly fast.

public class LoggingConfigurationTest {
    private final ApplicationContextRunner runner = new ApplicationContextRunner()
        .withConfiguration(UserConfigurations.of(LoggingConfiguration.class));
    
    @Test
    public void loggerShouldBeSlf4j() {
        runner
            .run(ctx -> 
                assertThat(ctx.getBean(Logger.class)).isInstanceOf(Logger.class)
            );
    }
    
    @Test
    public void loggerShouldBeNoop() {
        runner
            .withPropertyValues("logging.enabled=false")
            .run(ctx -> 
                assertThat(ctx.getBean(Logger.class)).isSameAs(NOPLogger.NOP_LOGGER)
            );
    }
}

It looks really great. The JUnit 5.x support in Spring Boot 2.1.x is much better and with the the upcoming 2.2 release, JUnit 5.x will be the default engine (not to worry, the old JUnit 4.x will still be supported). As of now, the switch to JUnit 5.x needs a bit of work on dependencies side.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-test</artifactId>
    <scope>test</scope>
    <exclusions>
        <exclusion>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
        </exclusion>
    </exclusions>
</dependency>

<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-api</artifactId>
    <scope>test</scope>
</dependency>

<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-engine</artifactId>
    <scope>test</scope>
</dependency>

As an additional step, you may need to use recent Maven Surefire plugin, 2.22.0 or above, with out-of-the box JUnit 5.x support. The the snippet below illustrates that.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.22.2</version>
</plugin>

The sample configuration we have worked with is pretty naive, many of the real-world applications would end up with quite complex contexts built out of many conditionals. The flexibility and enormous opportunities that come out of the context runners, the invaluable addition to the Spring Boot 2.x test scaffolding, are just the live savers, please keep them in mind.

The complete project sources are available on Github.