Skip to content

tests/unittests: allow out-of-tree tests#21328

Merged
crasbe merged 1 commit intoRIOT-OS:masterfrom
maribu:tests/unittests/out-of-tree-tests
Nov 8, 2025
Merged

tests/unittests: allow out-of-tree tests#21328
crasbe merged 1 commit intoRIOT-OS:masterfrom
maribu:tests/unittests/out-of-tree-tests

Conversation

@maribu
Copy link
Copy Markdown
Member

@maribu maribu commented Mar 26, 2025

Contribution description

This adds and documents the new EXTERNAL_UNIT_TEST_DIRS environment variable that allows including out-of-tree unit tests. The intention is to allow users of EXTERNAL_MODULE_DIRS to also provide corresponding unit tests and run them all with a single test app.

Testing procedure

Export both EXTERNAL_MDOULE_DIRS and EXTERNAL_UNIT_TEST_DIRS, then build and run the unit tests. The external tests should now be included.

Run again with both variables unset, and only the internal tests should be build and run as before.

Issues/PRs references

None

@maribu maribu added Type: new feature The issue requests / The PR implemements a new feature for RIOT CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Mar 26, 2025
@maribu maribu requested a review from miri64 as a code owner March 26, 2025 23:37
@github-actions github-actions bot added Area: doc Area: Documentation Area: tests Area: tests and testing framework labels Mar 26, 2025
@riot-ci
Copy link
Copy Markdown

riot-ci commented Mar 26, 2025 •
edited
Loading

Murdock results

✔️ PASSED

eff6816 tests/unittests: allow out-of-tree tests

Success Failures Total Runtime
10936 0 10936 12m:55s

Artifacts

@maribu maribu requested a review from benpicco March 26, 2025 23:51
mguetschow
mguetschow reviewed Mar 27, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@mguetschow mguetschow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the benefit of reusing tests/unittests rather than writing your own (out-of-tree) application that contains the unittest?

In any case, I'd like to have a corresponding test akin to https://github.com/RIOT-OS/RIOT/tree/master/tests/build_system/external_board_dirs

tests/unittests/README.md
same naming convention (each folder in `EXTERNAL_UNIT_TEST_DIRS` should have
`tests-<name>` folders containing the unit tests).

This feature works best with `EXTERNAL_MODULE_DIRS` that contain the code the
Copy link
Copy Markdown
Contributor

@mguetschow mguetschow Mar 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is the envisioned use-case, would you mind adding a pointer to it the documentation over at https://doc.riot-os.org/creating-modules.html?

Copy link
Copy Markdown
Contributor

@mguetschow mguetschow Mar 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would also work nicely with other EXTERNAL_*_DIRS, right?

Copy link
Copy Markdown
Member Author

@maribu maribu Mar 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it would work with about any folder containing extra unit tests. So if you external board has unit tests, this would also work here.

@maribu
Copy link
Copy Markdown
Member Author

maribu commented Mar 27, 2025

What's the benefit of reusing tests/unittests rather than writing your own (out-of-tree) application that contains the unittest?

The motivation is two-fold:

  1. The upstream unit tests are pretty good at detecting issues with the toolchain / build setup. So running them along with the downstream tests is useful. And doing so with a single app can speed up time (especially when running the tests on the MCU and the MCU fits both upstream and downstream unit tests: It safes a full build and flash cycle.)
  2. The downstream unit test app would be very much a copy-paste of the upstream unit test app. Allowing the upstream app to run the downstream tests instead provides some incentive to upstream fixes/improvements to the unit test app :)

@github-actions github-actions bot added the Area: sys Area: System label Mar 27, 2025
@maribu
Copy link
Copy Markdown
Member Author

maribu commented Mar 27, 2025

I added a small test app that has both an internal and an external unit test. The python runner checks that two tests are run.

@maribu maribu force-pushed the tests/unittests/out-of-tree-tests branch from 6cdd155 to 908789b Compare April 1, 2025 06:03
@maribu maribu force-pushed the tests/unittests/out-of-tree-tests branch from 908789b to 48141bd Compare April 25, 2025 06:32
@github-actions github-actions bot removed the Area: sys Area: System label Apr 25, 2025
@benpicco
Copy link
Copy Markdown
Contributor

benpicco commented Nov 6, 2025

This needs a rebase.

@crasbe crasbe added the State: needs rebase State: The codebase was changed since the creation of the PR, making a rebase necessary label Nov 6, 2025
@maribu maribu force-pushed the tests/unittests/out-of-tree-tests branch from 48141bd to f4f583f Compare November 6, 2025 16:16
@benpicco benpicco removed the State: needs rebase State: The codebase was changed since the creation of the PR, making a rebase necessary label Nov 6, 2025
@benpicco benpicco enabled auto-merge November 6, 2025 16:31
@crasbe crasbe disabled auto-merge November 6, 2025 21:12
@crasbe
Copy link
Copy Markdown
Contributor

crasbe commented Nov 6, 2025

I think the build system considers the out-of-tree test folder as a dedicated application:

Screenshot 2025-11-06 221127 
buechse@skyleaf:~/RIOTstuff/riot-vanillaice/RIOT$ make info-applications | grep tests/build_system/external_unittests
tests/build_system/external_unittests
tests/build_system/external_unittests/external_tests_dir/tests-out_of_tree
tests/build_system/external_unittests/tests-in_tree

And since $(RIOTBASE) is undefined there, the compilation fails:

buechse@skyleaf:~/RIOTstuff/riot-vanillaice/RIOT$ make -C tests/build_system/external_unittests/external_tests_dir/tests-out_of_tree
make: Entering directory '/home/buechse/RIOTstuff/riot-vanillaice/RIOT/tests/build_system/external_unittests/external_tests_dir/tests-out_of_tree'
Makefile:1: /Makefile.base: No such file or directory
make: *** No rule to make target '/Makefile.base'.  Stop.
make: Leaving directory '/home/buechse/RIOTstuff/riot-vanillaice/RIOT/tests/build_system/external_unittests/external_tests_dir/tests-out_of_tree'

You'll probably have to add the folders to makefiles/app_dirs.blacklist.

@maribu maribu force-pushed the tests/unittests/out-of-tree-tests branch from f4f583f to cecc1c6 Compare November 8, 2025 16:34
@github-actions github-actions bot added the Area: build system Area: Build system label Nov 8, 2025
@maribu
Copy link
Copy Markdown
Member Author

maribu commented Nov 8, 2025

Good catch. Let's see if this does the trick

@maribu @crasbe
tests/unittests: allow out-of-tree tests
eff6816 
This adds and documents the new `EXTERNAL_UNIT_TEST_DIRS` environment
variable that allows including out-of-tree unit tests. The intention is
to allow users of `EXTERNAL_MODULE_DIRS` to also provide corresponding
unit tests and run them all with a single test app.

Co-authored-by: crasbe <[email protected]>
@maribu maribu force-pushed the tests/unittests/out-of-tree-tests branch from cecc1c6 to eff6816 Compare November 8, 2025 16:56
@crasbe crasbe added this pull request to the merge queue Nov 8, 2025
Merged via the queue into RIOT-OS:master with commit bc3c992 Nov 8, 2025
26 checks passed
@leandrolanzieri leandrolanzieri added this to the Release 2026.01 milestone Jan 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mguetschow mguetschow mguetschow left review comments

@benpicco benpicco benpicco approved these changes

@miri64 miri64 Awaiting requested review from miri64 miri64 is a code owner

@leandrolanzieri leandrolanzieri Awaiting requested review from leandrolanzieri leandrolanzieri is a code owner

@aabadie aabadie Awaiting requested review from aabadie aabadie is a code owner

@MichelRottleuthner MichelRottleuthner Awaiting requested review from MichelRottleuthner MichelRottleuthner is a code owner

@jia200x jia200x Awaiting requested review from jia200x jia200x is a code owner

@Teufelchen1 Teufelchen1 Awaiting requested review from Teufelchen1 Teufelchen1 is a code owner

@AnnsAnns AnnsAnns Awaiting requested review from AnnsAnns AnnsAnns is a code owner

Assignees

No one assigned

Labels

Area: build system Area: Build system Area: doc Area: Documentation Area: tests Area: tests and testing framework CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR Type: new feature The issue requests / The PR implemements a new feature for RIOT

Projects

None yet

Milestone

Release 2026.01

Development

Successfully merging this pull request may close these issues.

6 participants

@maribu @riot-ci @benpicco @crasbe @mguetschow @leandrolanzieri