Skip to content

fix: warn when matching test classes have no provider - #3567

Merged
olamy merged 2 commits into
apache:masterfrom
OKAN-SOFTWARE:fix/3566-pojo-missing-provider-warning
Nov 14, 2026
Merged

olamy merged 2 commits into
apache:masterfrom
OKAN-SOFTWARE:fix/3566-pojo-missing-provider-warning

Conversation

@OKAN-SOFTWARE

@OKAN-SOFTWARE OKAN-SOFTWARE commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #3566.

Since Surefire 3.6.0, projects containing plain Java test classes (POJO tests) can complete successfully without executing those tests or displaying a warning.

I added a warning to make this behavior more visible and help developers identify missing test framework dependencies.

Changes

  • Added a warning when matching test classes are discovered but no compatible test provider is available.
  • Explained that POJO tests are no longer supported and recommended adding JUnit or TestNG.
  • Added integration test coverage using a POJO-only project without test framework dependencies.
  • Added a regression test confirming that the warning is not displayed when no test classes match.

The warning is only displayed when test classes are discovered but no provider is available. It is skipped when the test scan is empty.

Behavior Verification

I reproduced the behavior using the same POJO test fixture across different Surefire versions.

Surefire Version Result
3.5.4 Executes the POJO test through JUnit3Provider and PojoTestSetExecutor. The build fails as expected because the fixture contains a failing assertion.
3.6.1 Completes successfully without executing the test or displaying a warning.
3.6.2-SNAPSHOT (with fix) Displays the missing-provider warning when matching test classes are found.

These results confirm the behavior reported in #3566.

Testing

Regression Tests

Before the fix:

  • Confirmed that the warning regression test failed because the expected warning was missing.

After the fix:

  • 2 integration tests passed
  • 0 failures
  • 0 errors
  • 0 skipped

The integration tests verify:

  1. A warning is displayed when matching POJO test classes are found without an available test provider.
  2. No warning is displayed when the test scan finds no matching classes.

Test Command

mvn.cmd -B -pl surefire-its -am -P run-its "-Dit.test=PojoWithoutProviderIT" verify

Additional Validation

  • Previously validated the Maven reactor with 15 modules building successfully.
  • Compiled and executed both updated integration tests against the PR branch.
  • Confirmed that the changes pass Git whitespace validation.
  • The full integration-test suite has not been executed.
  • The local cloud environment required a Java compiler workaround, so its Java 8 API signature check was skipped. Standard GitHub CI validation remains pending.

Contribution Declaration

Contribution Declaration


TestClassPath testClasspath = generateTestClasspath();
List<ProviderInfo> providers = createProviders(testClasspath);
if (providers.isEmpty()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this skipped if there are no test classes?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. When the test scan is empty, Surefire returns before reaching the provider check, so this warning is skipped.

I added an integration test using -Dtest=DoesNotExist with surefire.failIfNoSpecifiedTests=false to verify this behavior. Both integration tests pass.

*/
package demo;

public class CalculatorTest {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did this ever work? I'm surprised if we ever ran something like this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I reproduced this behavior using the same POJO test fixture.

Surefire 3.5.4 discovers and executes the test through JUnit3Provider and PojoTestSetExecutor, resulting in one expected test failure.

With Surefire 3.6.1, the same project completes successfully without executing the test.

This confirms the behavior reported in #3566 and explains why the warning is useful.

@github-actions

Copy link
Copy Markdown

⚠️ License declaration missing

The pull request description must contain at least one of the following lines with a checked box ([x]):

- [ ] I hereby declare this contribution to be licensed under the [Apache License Version 2.0, January 2004](https://www.apache.org/licenses/LICENSE-2.0)
- [ ] I hereby declare that I have filed an [Apache Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

This comment will be hidden automatically once the description is fixed.

@olamy olamy added this to the 3.6.2 milestone Nov 12, 2026
@olamy olamy self-assigned this Nov 12, 2026
@olamy
olamy merged commit 1601a77 into apache:master Nov 14, 2026
11 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tests in POJO test classes pass silently since 3.6.0

3 participants