Skip to content

CTest (using catch_discover_tests) treats skipped tests as failed #2873

@jonathanverner

Description

@jonathanverner

Describe the bug
When using CMake integration to auto-discover tests and run them with CTest, tests which are skipped are reported as failed.

Expected behavior
Skipped tests are reported as skipped.

Platform information:

  • OS: Linux (Neon 22.04, based on Ubuntu 22.04)
  • Compiler+version: 17.0.6
  • Catch version: v3.3.2

Additional context
I think the reason is that ctest runs the tests one-by-one and if a test is skipped, this actually means that no test was run so, since --allow-running-no-tests is not provided, the return code is non-zero.

Now, CTest allows marking skipped tests via a special return code by setting the SKIP_RETURN_CODE property on the test. So, it seems to me, this could be fixed by replacing the

set(properties ${TEST_PROPERTIES})

line in CatchAddTests.cmake with

set(properties "${TEST_PROPERTIES};SKIP_RETURN_CODE;4")

This works for me as does passing SKIP_RETURN_CODE 4 to catch_discover_tests PROPETIES argument, e.g. like so:

catch_discover_tests(middleware_tests
  PROPERTIES
    SKIP_RETURN_CODE 4
)

Metadata

Metadata

Assignees

No one assigned

    Labels

    ExtrasTouches utility scripts outside of Catch2 proper, e.g. CMake integration.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions