Overview

In the test suite for the Spring Framework, we have a TestExecutionListenersNestedTests test class that is only intended to be run as a top-level test class. Specifically, if you run a @Test in a @Nested test class directly, that test will fail, because the @Test methods in the enclosing test classes have not been run. Although that is perhaps questionable behavior, it is actually by design: this particular test fixture tests the interactions between JUnit Jupiter and the Spring TestContext Framework.

We also have SpringJUnitJupiterTestSuite, which selects TestExecutionListenersNestedTests via @SelectPackages("org.springframework.test.context.junit.jupiter") and @IncludeClassNamePatterns(".*Tests$").

Historically SpringJUnitJupiterTestSuite succeeded (with JUnit Jupiter versions prior to 5.13), but it now fails.

Since that setup is a bit complex, I have reduced this scenario to just use the following JUnit Jupiter extensions.

abstract class BaseExtension implements BeforeAllCallback {

	@Override
	public void beforeAll(ExtensionContext context) throws Exception {
		ExecutionOrderForNestedTests.extensions.add(getClass().getSimpleName());
	}
}

class Foo extends BaseExtension {
}

class Bar extends BaseExtension {
}

class Baz extends BaseExtension {
}

class Qux extends BaseExtension {
}

And the simplified test class looks like this.

package example;

// imports...

@ExtendWith(Foo.class)
class ExecutionOrderForNestedTests {

	static final String FOO = "Foo";
	static final String BAR = "Bar";
	static final String BAZ = "Baz";
	static final String QUX = "Qux";

	static final List<String> extensions = new ArrayList<>();

	@AfterEach
	void resetExtensions() {
		extensions.clear();
	}

	@Test
	void test() {
		assertThat(extensions).containsExactly(FOO);
	}

	@Nested
	class InheritedExtensionsTests {

		@Test
		void test() {
			assertThat(extensions).containsExactly(FOO);
		}
	}

	@Nested
	@ExtendWith(Bar.class)
	class InheritedAndLocalExtensionsTests {

		@Test
		void test() {
			assertThat(extensions).containsExactly(FOO, BAR);
		}

		@Nested
		@ExtendWith(Baz.class)
		class DoubleNestedWithInheritedAndLocalExtensionsTests {

			@Test
			void test() {
				assertThat(extensions).containsExactly(FOO, BAR, BAZ);
			}

			@Nested
			@ExtendWith(Qux.class)
			class TripleNestedWithInheritedAndLocalExtensionsTests {

				@Test
				void test() {
					assertThat(extensions).containsExactly(FOO, BAR, BAZ, QUX);
				}
			}
		}
	}
}

If you run ExecutionOrderForNestedTests by itself, all of the tests (including the tests in the @Nested test classes) pass.

As I mentioned above, if you run an individual @Test in a @Nested test class directly, that will fail, but this is by design. So, that is not a regression.

The Regression

The regression is apparent only when running ExecutionOrderForNestedTests via a @Suite.

✅ This suite passes:

@Suite
@SelectPackages("example")
@IncludeClassNamePatterns(".+ExecutionOrderForNestedTests$")
class TestSuite {
}

✅ This suite also passes:

@Suite
@SelectClasses(ExecutionOrderForNestedTests.class)
class TestSuite {
}

❌ This suite fails:

@Suite
@SelectPackages("example")
class TestSuite {
}

❌ This suite also fails:

@Suite
@SelectPackages("example")
@IncludeClassNamePatterns(".+Tests$")
class TestSuite {
}

For example, TripleNestedWithInheritedAndLocalExtensionsTests.test() fails as follows when run in one of those failing @Suite scenarios.

org.opentest4j.AssertionFailedError: 
Expecting actual:
  ["Foo", "Bar", "Foo", "Bar", "Baz", "Foo", "Bar", "Baz", "Qux"]
to contain exactly (and in same order):
  ["Foo", "Bar", "Baz", "Qux"]
but some elements were not expected:
  ["Foo", "Bar", "Foo", "Bar", "Baz"]

Analysis

If ExecutionOrderForNestedTests is selected explicitly in the IDE or in a @Suite (without discovering the individual @Nested test classes in ExecutionOrderForNestedTests), all tests pass.

If the @Nested test classes in ExecutionOrderForNestedTests are discovered in a @Suite using classpath scanning via @SelectPackages, the @Nested test methods get executed before (or "independently of") the test methods in enclosing classes, which was not previously the case.