Skip to content

Commit 6ecdc91

Browse files
knutwannhedenknutwannheden
authored
Add withExceptions to ExpectedToFail (#769 / #774)
Adds a new `withExceptions` attribute to `@ExpectedToFail`, which allows to limit the scope of the extension to only consider the test as successful if the thrown exception matches one of the given types. Closes: #769 PR: #774
1 parent 1251593 commit 6ecdc91

6 files changed

Lines changed: 129 additions & 1 deletion

File tree

‎README.adoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,7 @@ The least we can do is to thank them and list some of their accomplishments here
130130
* https://github.com/eeverman[Eric Everman] added `@RestoreSystemProperties` and `@RestoreEnvironmentVariables` annotations to the https://junit-pioneer.org/docs/system-properties/[System Properties] and https://junit-pioneer.org/docs/environment-variables/[Environment Variables] extensions (#574 / #700)
131131
* https://github.com/meredrica[Florian Westreicher] contributed to the JSON argument source extension (#704 / #724)
132132
* https://github.com/IlyasYOY[Ilya Ilyinykh] found unused demo tests (#791)
133+
* https://github.com/knutwannheden[Knut Wannheden] contributed the `withExceptions` attribute of the https://junit-pioneer.org/docs/expected-to-fail-tests/[`@ExpectedToFail` extension] (#769 / #774)
133134
* https://github.com/petrandreev[Pёtr Andreev] added back support for NULL values to `@CartesianTestExtension` (#764 / #765)
134135

135136
==== 2022

‎docs/expected-to-fail-tests.adoc‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,25 @@ A custom message can be provided, explaining why the tested code is not working
4444
include::{demo}[tag=expected_to_fail_message]
4545
----
4646

47+
=== Only Abort on Expected Exceptions
48+
49+
A test that is `@ExpectedToFail` will change its behavior by starting to actually fail, once the code under test behaves correctly.
50+
If the underlying failure changes, though, for example from an assertion error to an exception or from an `UnsupportedOperationException` of a formerly missing implementation to a runtime exception of buggy implementation, the `@ExpectedToFail`-test will keep passing.
51+
52+
To better react to such changes, `@ExpectedToFail` has an attribute `withExceptions` that can be used to enumerate the exceptions which when thrown will result in an aborted (and thus passing) test.
53+
Any other exception thrown by the code under test will result in a failing test.
54+
55+
In the following example a test case for `productionFeature()` has been implemented.
56+
While the test is fully implemented, the production code has been stubbed and throws an `UnsupportedOperationException` to indicate this.
57+
58+
[source,java,indent=0]
59+
----
60+
include::{demo}[tag=expected_to_fail_withexception]
61+
----
62+
63+
Once `productionFeature()` is implemented, `@ExpectedToFail` will fail the test, as no `UnsupportedOperationException` is thrown anymore.
64+
By using `withExceptions` you can thus prevent "masking" a faulty implementation (e.g. when a value other rather than `10` is returned) with an aborted test (which would be the result when no `withExceptions` is set).
65+
4766
== Thread-Safety
4867

4968
This extension is safe to use during https://junit.org/junit5/docs/current/user-guide/#writing-tests-parallel-execution[parallel test execution].

‎src/demo/java/org/junitpioneer/jupiter/ExpectedToFailExtensionDemo.java‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,4 +38,17 @@ private int brokenMethod() {
3838
return 0;
3939
}
4040

41+
// tag::expected_to_fail_withexception[]
42+
@Test
43+
@ExpectedToFail(withExceptions = UnsupportedOperationException.class)
44+
void testProductionFeature() {
45+
int actual = productionFeature();
46+
assertThat(actual).isEqualTo(10);
47+
}
48+
49+
private int productionFeature() {
50+
throw new UnsupportedOperationException("productionFeature() is not yet implemented");
51+
}
52+
// end::expected_to_fail_withexception[]
53+
4154
}

‎src/main/java/org/junitpioneer/jupiter/ExpectedToFail.java‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,13 @@
3232
* This helps to avoid creating duplicate tests by accident and counteracts the accumulation
3333
* of disabled tests over time.</p>
3434
*
35+
* <p>Further, the {@link #withExceptions()} attribute can be used to restrict the extension's behavior
36+
* to specific exceptions. That is, only if the test method ends up throwing one of the specified exceptions
37+
* will the test be aborted. This can, for example, be used when the production code temporarily throws
38+
* an {@link UnsupportedOperationException} because some feature has not been implemented yet, but the
39+
* test method is already implemented and should not fail on a failing assertion.
40+
* </p>
41+
*
3542
* <p>The annotation can only be used on methods and as meta-annotation on other annotation types.
3643
* Similar to {@code @Disabled}, it has to be used in addition to a "testable" annotation, such
3744
* as {@link org.junit.jupiter.api.Test @Test}. Otherwise the annotation has no effect.</p>
@@ -68,4 +75,11 @@
6875
*/
6976
String value() default "";
7077

78+
/**
79+
* Specifies which exceptions are expected to be thrown and will cause the test to be aborted rather than fail.
80+
* An empty array is considered a configuration error and will cause the test to fail. Instead, consider leaving
81+
* the attribute set to the default value when any exception should cause the test to be aborted.
82+
*/
83+
Class<? extends Throwable>[] withExceptions() default { Throwable.class };
84+
7185
}

‎src/main/java/org/junitpioneer/jupiter/ExpectedToFailExtension.java‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,10 @@
1111
package org.junitpioneer.jupiter;
1212

1313
import java.lang.reflect.Method;
14+
import java.util.stream.Stream;
1415

1516
import org.junit.jupiter.api.extension.Extension;
17+
import org.junit.jupiter.api.extension.ExtensionConfigurationException;
1618
import org.junit.jupiter.api.extension.ExtensionContext;
1719
import org.junit.jupiter.api.extension.InvocationInterceptor;
1820
import org.junit.jupiter.api.extension.ReflectiveInvocationContext;
@@ -30,6 +32,11 @@ public void interceptTestMethod(Invocation<Void> invocation, ReflectiveInvocatio
3032

3133
private static void invokeAndInvertResult(Invocation<Void> invocation, ExtensionContext extensionContext)
3234
throws Throwable {
35+
ExpectedToFail expectedToFail = getExpectedToFailAnnotation(extensionContext);
36+
if (expectedToFail.withExceptions().length == 0) {
37+
throw new ExtensionConfigurationException("@ExpectedToFail withExceptions must not be empty");
38+
}
39+
3340
try {
3441
invocation.proceed();
3542
// at this point, the invocation succeeded, so we'd want to call `fail(...)`,
@@ -41,7 +48,12 @@ private static void invokeAndInvertResult(Invocation<Void> invocation, Extension
4148
throw t;
4249
}
4350

44-
String message = getExpectedToFailAnnotation(extensionContext).value();
51+
if (Stream.of(expectedToFail.withExceptions()).noneMatch(clazz -> clazz.isInstance(t))) {
52+
throw new AssertionFailedError(
53+
"Test marked as temporarily 'expected to fail' failed with an unexpected exception", t);
54+
}
55+
56+
String message = expectedToFail.value();
4557
if (message.isEmpty()) {
4658
message = "Test marked as temporarily 'expected to fail' failed as expected";
4759
}

‎src/test/java/org/junitpioneer/jupiter/ExpectedToFailExtensionTests.java‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
import org.junit.jupiter.api.BeforeEach;
2727
import org.junit.jupiter.api.DisplayName;
2828
import org.junit.jupiter.api.Test;
29+
import org.junit.jupiter.api.extension.ExtensionConfigurationException;
2930
import org.junitpioneer.testkit.ExecutionResults;
3031
import org.junitpioneer.testkit.PioneerTestKit;
3132
import org.opentest4j.TestAbortedException;
@@ -100,6 +101,52 @@ void failsOnWorkingTest() {
100101
.hasMessage("Test marked as 'expected to fail' succeeded; remove @ExpectedToFail from it");
101102
}
102103

104+
@Test
105+
void doesNotAbortOnTestThrowingExpectedException() {
106+
ExecutionResults results = PioneerTestKit
107+
.executeTestMethod(ExpectedToFailTestCases.class, "withExceptionsExpected");
108+
assertThat(results)
109+
.hasSingleStartedTest()
110+
.whichAborted()
111+
.withExceptionInstanceOf(TestAbortedException.class)
112+
.hasMessage("Test marked as temporarily 'expected to fail' failed as expected")
113+
.hasCauseInstanceOf(UnsupportedOperationException.class);
114+
}
115+
116+
@Test
117+
void failsOnTestThrowingUnexpectedException() {
118+
ExecutionResults results = PioneerTestKit
119+
.executeTestMethod(ExpectedToFailTestCases.class, "withExceptionsUnexpected");
120+
assertThat(results)
121+
.hasSingleStartedTest()
122+
.whichFailed()
123+
.withExceptionInstanceOf(AssertionError.class)
124+
.hasMessage("Test marked as temporarily 'expected to fail' failed with an unexpected exception")
125+
.hasCauseInstanceOf(IllegalStateException.class);
126+
}
127+
128+
@Test
129+
void failsOnWorkingTestWithExpectedException() {
130+
ExecutionResults results = PioneerTestKit
131+
.executeTestMethod(ExpectedToFailTestCases.class, "withExceptionsWorking");
132+
assertThat(results)
133+
.hasSingleStartedTest()
134+
.whichFailed()
135+
.withExceptionInstanceOf(AssertionError.class)
136+
.hasMessage("Test marked as 'expected to fail' succeeded; remove @ExpectedToFail from it");
137+
}
138+
139+
@Test
140+
void failsOnWorkingTestWithEmptyExpectedExceptions() {
141+
ExecutionResults results = PioneerTestKit
142+
.executeTestMethod(ExpectedToFailTestCases.class, "withExceptionsEmpty");
143+
assertThat(results)
144+
.hasSingleStartedTest()
145+
.whichFailed()
146+
.withExceptionInstanceOf(ExtensionConfigurationException.class)
147+
.hasMessage("@ExpectedToFail withExceptions must not be empty");
148+
}
149+
103150
@Test
104151
void doesNotAbortOnBeforeEachTestFailure() {
105152
ExecutionResults results = PioneerTestKit
@@ -218,6 +265,28 @@ void working() {
218265
// Does not cause failure or error
219266
}
220267

268+
@Test
269+
@ExpectedToFail(withExceptions = { IllegalStateException.class, UnsupportedOperationException.class })
270+
void withExceptionsExpected() {
271+
throw new UnsupportedOperationException();
272+
}
273+
274+
@Test
275+
@ExpectedToFail(withExceptions = UnsupportedOperationException.class)
276+
void withExceptionsUnexpected() {
277+
throw new IllegalStateException();
278+
}
279+
280+
@Test
281+
@ExpectedToFail(withExceptions = UnsupportedOperationException.class)
282+
void withExceptionsWorking() {
283+
}
284+
285+
@Test
286+
@ExpectedToFail(withExceptions = {})
287+
void withExceptionsEmpty() {
288+
}
289+
221290
}
222291

223292
/**

0 commit comments

Comments
 (0)