Enable compile_and_test_for_board to skip if nothing changed by MrKevinWeiss · Pull Request #19546 · RIOT-OS/RIOT

MrKevinWeiss · 2023-05-04T11:05:10Z

Contribution description

The overall goal of this PR is to be able to keep running the compile_and_test_for_board.py script without destroying ones boards. Useful if you are a board owner that wants to keep testing on master (say with nightlies, say in a CI).

For example, I could now just run and rerun the following:

./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . <MY_BOARD> -c

and monitor the results, even after updating a branch.

This is because the hashes now get stored with the results (thanks to the RIOT_TEST_HASH_DIR var) which means cleaning will not wipe them out. Specifically in (<results_base>/<board>/hashes/<app_dir>/test-input-hash.sha1)

I tried to do as much as I could with make and only alter the python script when needed.

Testing procedure

Clear results folder and run:

./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . native --applications tests/shell -c

Details

INFO:native:Saving toolchain
INFO:native.tests/shell:Board supported: True
INFO:native.tests/shell:Board has enough memory: True
INFO:native.tests/shell:Application has test: True
INFO:native.tests/shell:Run compilation
**INFO:native.tests/shell:Hashes match: False**
INFO:native.tests/shell:Run test
INFO:native.tests/shell:Run test.flash
INFO:native.tests/shell:Success
INFO:native:Tests successful

it should execute the test...

Then the same command again should skip it.

Details

INFO:native:Saving toolchain
INFO:native.tests/shell:Board supported: True
INFO:native.tests/shell:Board has enough memory: True
INFO:native.tests/shell:Application has test: True
INFO:native.tests/shell:Run compilation
**INFO:native.tests/shell:Hashes match: True**
INFO:native.tests/shell:Success
INFO:native:Tests successful

Try changing something (say the tests/shell/01-run.py) and rerun, it should trigger the test again.

Details

INFO:native:Saving toolchain
INFO:native.tests/shell:Board supported: True
INFO:native.tests/shell:Board has enough memory: True
INFO:native.tests/shell:Application has test: True
INFO:native.tests/shell:Run compilation
**INFO:native.tests/shell:Hashes match: False**
INFO:native.tests/shell:Run test
INFO:native.tests/shell:Run test.flash
INFO:native.tests/shell:Success
INFO:native:Tests successful

...finally, running without the changes check should run the test as usual, even if nothing changes:

./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . native --applications tests/shell

Details

INFO:native:Saving toolchain
INFO:native.tests/shell:Board supported: True
INFO:native.tests/shell:Board has enough memory: True
INFO:native.tests/shell:Application has test: True
INFO:native.tests/shell:Run compilation
INFO:native.tests/shell:Run test
INFO:native.tests/shell:Run test.flash
INFO:native.tests/shell:Success
INFO:native:Tests successful

Issues/PRs references

Might help the surprises we got with #19469

MrKevinWeiss · 2023-05-04T11:13:10Z

Maybe @Teufelchen1 or @kaspar030 would be interested in this.

Some open questions are:

Is the test-input-hash-changed target implemented OK, for example, should I make the full dir if missing?
Will this affect the current .murdock way of doing things, or can we simplify the script now?
Does the RIOT_TEST_HASH_DIR need to be exported to docker as well (currently the test_like_murdock script does not work with docker since the test-input-hash gets exported to a strange place)?
Maybe the result of the make command would be better as a status code or something rather than a string, or at least a better formatted string?
The the <results_base>/<board>/hashes/<app_dir>/test-input-hash.sha1 to keep the hashes persistently (since the typical results dir gets cleaned before) or I rather filter the clean function to ignore .sha1 files?
- Should there be a different flag to remove the hashes if desired?

Follow-up ideas (not for this PR) would be:

Export some metadata as well beside the test results (ie, git commit)
Better recording of the current environment
Recording of the args used to run the script
A runner to read in a list of boards and possibly full board environments so that multiple boards can be run at the same time

Teufelchen1

Looks fine and works. Please add the documentation.

dist/tools/compile_and_test_for_board/compile_and_test_for_board.py

MrKevinWeiss · 2023-05-10T09:35:03Z

Looks fine and works. Please add the documentation.

I added a bit more info in a comment, I don't want to clutter the help string.

Teufelchen1 · 2023-05-10T10:04:32Z

Some open questions are:
[..]

Maybe the result of the make command would be better as a status code or something rather than a string, or at least a better formatted string?

I also thought about that but couldn't come up with an ideal solution. Have you tried a status code?

[..]

The the <results_base>/<board>/hashes/<app_dir>/test-input-hash.sha1 to keep the hashes persistently (since the typical results dir gets cleaned before) or I rather filter the clean function to ignore .sha1 files?

Should there be a different flag to remove the hashes if desired?

I wouldn't add complexity to a clean. It is probably fine to just remove the .sha1 when the build is cleaned. After all, as I understand this PR, it is intended to ease the burden of repeatedly building applications after small changes? That's not a task where I would clean in between - and if I do, I want a fresh restart including removing the cached files.
Misunderstood the question. Could you elaborate? Your typo is putting me off.

Follow-up ideas (not for this PR) would be:

Export some metadata as well beside the test results (ie, git commit)

Better recording of the current environment

Recording of the args used to run the script

A runner to read in a list of boards and possibly full board environments so that multiple boards can be run at the same time

👍

MrKevinWeiss · 2023-05-10T10:22:57Z

Misunderstood the question. Could you elaborate? Your typo is putting me off.

Currently the hashes are stored in the following format:

<results_base>/<board>/hashes/<app_dir>/test-input-hash.sha1

The results are stored in a different format:

<results_base>/<board>/<app_dir>/*

By default, the script will clean the results if they exist by removing the <results_base>/<board>/<app_dir>/ directory.

My solution for this was to namespace the hashes directory so it does not get removed between runs (which would defeat the point).

The first question would be, is this a good way to preserve the hashes, by namespacing the hashes folder, or should I keep the hashes in the hashes next to the results, for example, <results_base>/<board>/<app_dir>/test-input-hash.sha1 and carefully remove the remove everything but the hashes during the typical clean.

The second question would be to add a command (arg flag) to clean the hashes since they will persist until either they are manually removed or the results directory changes.

MrKevinWeiss · 2023-06-29T09:54:58Z

ping

MrKevinWeiss · 2023-07-11T11:55:26Z

ping

Teufelchen1 · 2023-09-05T10:54:51Z

The first question would be, is this a good way to preserve the hashes, by namespacing the hashes folder, or should I keep the hashes in the hashes next to the results, for example, <results_base>/<board>/<app_dir>/test-input-hash.sha1 and carefully remove the remove everything but the hashes during the typical clean.

I think namespacing is fine. I would rather have a different folder than messing with clean. I would be confused when I clean something and some arbitrary hashes remain.

The second question would be to add a command (arg flag) to clean the hashes since they will persist until either they are manually removed or the results directory changes.

Hm voting against that. If you don't care about the hashes, don't pass -c to the tool, done. I do not see a use-case for deleting the hashes otherwise?

On a site note, this tool suffers from the common RIOT issue of not being exposed to user. Ideally, this PR would add a change to a handbook somewhere to explain its abilities to future RIOT developers...

MrKevinWeiss · 2023-09-06T15:36:22Z

On a site note, this tool suffers from the common RIOT issue of not being exposed to user. Ideally, this PR would add a change to a handbook somewhere to explain its abilities to future RIOT developers...

True, maybe an fairly obvious entrypoint in the docs for all the tools, why solve this problem when you can try to tackle all the problems.

benpicco · 2023-10-27T12:04:27Z

bors merge

bors · 2023-10-27T12:04:31Z

🕐 Waiting for PR status (GitHub check) to be set, probably by CI. Bors will automatically try to run when all required PR statuses are set.

riot-ci · 2023-10-27T12:06:27Z

Murdock results

✔️ PASSED

ceec795 compile_and_test_for_board: Add only-if-changed feature

Success	Failures	Total	Runtime
8629	0	8629	09m:03s

Artifacts

Documentation preview

benpicco · 2023-10-27T14:33:03Z

bors merge

19546: Enable compile_and_test_for_board to skip if nothing changed r=benpicco a=MrKevinWeiss ### Contribution description The overall goal of this PR is to be able to keep running the `compile_and_test_for_board.py` script without destroying ones boards. Useful if you are a board owner that wants to keep testing on master (say with nightlies, say in a CI). For example, I could now just run and rerun the following: ``` ./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . <MY_BOARD> -c ``` and monitor the results, even after updating a branch. This is because the hashes now get stored with the results (thanks to the `RIOT_TEST_HASH_DIR` var) which means cleaning will not wipe them out. Specifically in (`<results_base>/<board>/hashes/<app_dir>/test-input-hash.sha1`) I tried to do as much as I could with make and only alter the python script when needed. ### Testing procedure Clear results folder and run: ``` ./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . native --applications tests/shell -c ``` <details> ``` INFO:native:Saving toolchain INFO:native.tests/shell:Board supported: True INFO:native.tests/shell:Board has enough memory: True INFO:native.tests/shell:Application has test: True INFO:native.tests/shell:Run compilation **INFO:native.tests/shell:Hashes match: False** INFO:native.tests/shell:Run test INFO:native.tests/shell:Run test.flash INFO:native.tests/shell:Success INFO:native:Tests successful ``` </details> it should execute the test... Then the same command again should skip it. <details> ``` INFO:native:Saving toolchain INFO:native.tests/shell:Board supported: True INFO:native.tests/shell:Board has enough memory: True INFO:native.tests/shell:Application has test: True INFO:native.tests/shell:Run compilation **INFO:native.tests/shell:Hashes match: True** INFO:native.tests/shell:Success INFO:native:Tests successful ``` </details> Try changing something (say the `tests/shell/01-run.py`) and rerun, it should trigger the test again. <details> ``` INFO:native:Saving toolchain INFO:native.tests/shell:Board supported: True INFO:native.tests/shell:Board has enough memory: True INFO:native.tests/shell:Application has test: True INFO:native.tests/shell:Run compilation **INFO:native.tests/shell:Hashes match: False** INFO:native.tests/shell:Run test INFO:native.tests/shell:Run test.flash INFO:native.tests/shell:Success INFO:native:Tests successful ``` </details> ...finally, running without the changes check should run the test as usual, even if nothing changes: ``` ./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . native --applications tests/shell ``` <details> ``` INFO:native:Saving toolchain INFO:native.tests/shell:Board supported: True INFO:native.tests/shell:Board has enough memory: True INFO:native.tests/shell:Application has test: True INFO:native.tests/shell:Run compilation INFO:native.tests/shell:Run test INFO:native.tests/shell:Run test.flash INFO:native.tests/shell:Success INFO:native:Tests successful ``` </details> ### Issues/PRs references Might help the surprises we got with #19469 20022: pkg/lwip: add support for slipdev r=benpicco a=benpicco 20025: tests/drivers/at: fix device table overflow r=benpicco a=krzysztof-cabaj ### Contribution description This PR fix device table overflow in `tests/driver/at`, which could lead to device crash. ### Testing procedure PR was tested on two nucleo boards with 2 and 3 UARTs (nucleo-l476rg and nucleo-l496zg). Flash `tests/driver/at` with and without this PR. Output with this PR: ``` > main(): This is RIOT! (Version: 2022.07-devel-5083-g2b9e8-tests-drivers-at) AT command test app > init 5 9600 Wrong UART device number - should by in range 0-2. > ``` Output without this PR: ``` > main(): This is RIOT! (Version: 2022.07-devel-5083-g2b9e8) AT command test app > init 5 9600 8001afd *** RIOT kernel panic: FAILED ASSERTION. *** halted. Context before hardfault: r0: 0x0000000a r1: 0x00000000 . . . ``` ### Issues/PRs references None Co-authored-by: MrKevinWeiss <[email protected]> Co-authored-by: Benjamin Valentin <[email protected]> Co-authored-by: krzysztof-cabaj <[email protected]>

bors · 2023-10-27T14:54:34Z

Build failed (retrying...):

Murdock

19546: Enable compile_and_test_for_board to skip if nothing changed r=benpicco a=MrKevinWeiss ### Contribution description The overall goal of this PR is to be able to keep running the `compile_and_test_for_board.py` script without destroying ones boards. Useful if you are a board owner that wants to keep testing on master (say with nightlies, say in a CI). For example, I could now just run and rerun the following: ``` ./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . <MY_BOARD> -c ``` and monitor the results, even after updating a branch. This is because the hashes now get stored with the results (thanks to the `RIOT_TEST_HASH_DIR` var) which means cleaning will not wipe them out. Specifically in (`<results_base>/<board>/hashes/<app_dir>/test-input-hash.sha1`) I tried to do as much as I could with make and only alter the python script when needed. ### Testing procedure Clear results folder and run: ``` ./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . native --applications tests/shell -c ``` <details> ``` INFO:native:Saving toolchain INFO:native.tests/shell:Board supported: True INFO:native.tests/shell:Board has enough memory: True INFO:native.tests/shell:Application has test: True INFO:native.tests/shell:Run compilation **INFO:native.tests/shell:Hashes match: False** INFO:native.tests/shell:Run test INFO:native.tests/shell:Run test.flash INFO:native.tests/shell:Success INFO:native:Tests successful ``` </details> it should execute the test... Then the same command again should skip it. <details> ``` INFO:native:Saving toolchain INFO:native.tests/shell:Board supported: True INFO:native.tests/shell:Board has enough memory: True INFO:native.tests/shell:Application has test: True INFO:native.tests/shell:Run compilation **INFO:native.tests/shell:Hashes match: True** INFO:native.tests/shell:Success INFO:native:Tests successful ``` </details> Try changing something (say the `tests/shell/01-run.py`) and rerun, it should trigger the test again. <details> ``` INFO:native:Saving toolchain INFO:native.tests/shell:Board supported: True INFO:native.tests/shell:Board has enough memory: True INFO:native.tests/shell:Application has test: True INFO:native.tests/shell:Run compilation **INFO:native.tests/shell:Hashes match: False** INFO:native.tests/shell:Run test INFO:native.tests/shell:Run test.flash INFO:native.tests/shell:Success INFO:native:Tests successful ``` </details> ...finally, running without the changes check should run the test as usual, even if nothing changes: ``` ./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . native --applications tests/shell ``` <details> ``` INFO:native:Saving toolchain INFO:native.tests/shell:Board supported: True INFO:native.tests/shell:Board has enough memory: True INFO:native.tests/shell:Application has test: True INFO:native.tests/shell:Run compilation INFO:native.tests/shell:Run test INFO:native.tests/shell:Run test.flash INFO:native.tests/shell:Success INFO:native:Tests successful ``` </details> ### Issues/PRs references Might help the surprises we got with #19469 Co-authored-by: MrKevinWeiss <[email protected]>

bors · 2023-10-27T19:55:02Z

Build failed:

Murdock

MrKevinWeiss · 2023-11-01T14:03:33Z

Well that seems unrelated...

MrKevinWeiss · 2023-11-01T14:03:54Z

bors merge

19546: Enable compile_and_test_for_board to skip if nothing changed r=MrKevinWeiss a=MrKevinWeiss ### Contribution description The overall goal of this PR is to be able to keep running the `compile_and_test_for_board.py` script without destroying ones boards. Useful if you are a board owner that wants to keep testing on master (say with nightlies, say in a CI). For example, I could now just run and rerun the following: ``` ./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . <MY_BOARD> -c ``` and monitor the results, even after updating a branch. This is because the hashes now get stored with the results (thanks to the `RIOT_TEST_HASH_DIR` var) which means cleaning will not wipe them out. Specifically in (`<results_base>/<board>/hashes/<app_dir>/test-input-hash.sha1`) I tried to do as much as I could with make and only alter the python script when needed. ### Testing procedure Clear results folder and run: ``` ./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . native --applications tests/shell -c ``` <details> ``` INFO:native:Saving toolchain INFO:native.tests/shell:Board supported: True INFO:native.tests/shell:Board has enough memory: True INFO:native.tests/shell:Application has test: True INFO:native.tests/shell:Run compilation **INFO:native.tests/shell:Hashes match: False** INFO:native.tests/shell:Run test INFO:native.tests/shell:Run test.flash INFO:native.tests/shell:Success INFO:native:Tests successful ``` </details> it should execute the test... Then the same command again should skip it. <details> ``` INFO:native:Saving toolchain INFO:native.tests/shell:Board supported: True INFO:native.tests/shell:Board has enough memory: True INFO:native.tests/shell:Application has test: True INFO:native.tests/shell:Run compilation **INFO:native.tests/shell:Hashes match: True** INFO:native.tests/shell:Success INFO:native:Tests successful ``` </details> Try changing something (say the `tests/shell/01-run.py`) and rerun, it should trigger the test again. <details> ``` INFO:native:Saving toolchain INFO:native.tests/shell:Board supported: True INFO:native.tests/shell:Board has enough memory: True INFO:native.tests/shell:Application has test: True INFO:native.tests/shell:Run compilation **INFO:native.tests/shell:Hashes match: False** INFO:native.tests/shell:Run test INFO:native.tests/shell:Run test.flash INFO:native.tests/shell:Success INFO:native:Tests successful ``` </details> ...finally, running without the changes check should run the test as usual, even if nothing changes: ``` ./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . native --applications tests/shell ``` <details> ``` INFO:native:Saving toolchain INFO:native.tests/shell:Board supported: True INFO:native.tests/shell:Board has enough memory: True INFO:native.tests/shell:Application has test: True INFO:native.tests/shell:Run compilation INFO:native.tests/shell:Run test INFO:native.tests/shell:Run test.flash INFO:native.tests/shell:Success INFO:native:Tests successful ``` </details> ### Issues/PRs references Might help the surprises we got with #19469 Co-authored-by: MrKevinWeiss <[email protected]>

bors · 2023-11-01T14:09:19Z

Build failed:

Murdock

MrKevinWeiss · 2023-11-01T14:13:18Z

I am starting to think that it is me...

benpicco · 2023-11-04T17:44:41Z

bors merge

19546: Enable compile_and_test_for_board to skip if nothing changed r=benpicco a=MrKevinWeiss ### Contribution description The overall goal of this PR is to be able to keep running the `compile_and_test_for_board.py` script without destroying ones boards. Useful if you are a board owner that wants to keep testing on master (say with nightlies, say in a CI). For example, I could now just run and rerun the following: ``` ./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . <MY_BOARD> -c ``` and monitor the results, even after updating a branch. This is because the hashes now get stored with the results (thanks to the `RIOT_TEST_HASH_DIR` var) which means cleaning will not wipe them out. Specifically in (`<results_base>/<board>/hashes/<app_dir>/test-input-hash.sha1`) I tried to do as much as I could with make and only alter the python script when needed. ### Testing procedure Clear results folder and run: ``` ./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . native --applications tests/shell -c ``` <details> ``` INFO:native:Saving toolchain INFO:native.tests/shell:Board supported: True INFO:native.tests/shell:Board has enough memory: True INFO:native.tests/shell:Application has test: True INFO:native.tests/shell:Run compilation **INFO:native.tests/shell:Hashes match: False** INFO:native.tests/shell:Run test INFO:native.tests/shell:Run test.flash INFO:native.tests/shell:Success INFO:native:Tests successful ``` </details> it should execute the test... Then the same command again should skip it. <details> ``` INFO:native:Saving toolchain INFO:native.tests/shell:Board supported: True INFO:native.tests/shell:Board has enough memory: True INFO:native.tests/shell:Application has test: True INFO:native.tests/shell:Run compilation **INFO:native.tests/shell:Hashes match: True** INFO:native.tests/shell:Success INFO:native:Tests successful ``` </details> Try changing something (say the `tests/shell/01-run.py`) and rerun, it should trigger the test again. <details> ``` INFO:native:Saving toolchain INFO:native.tests/shell:Board supported: True INFO:native.tests/shell:Board has enough memory: True INFO:native.tests/shell:Application has test: True INFO:native.tests/shell:Run compilation **INFO:native.tests/shell:Hashes match: False** INFO:native.tests/shell:Run test INFO:native.tests/shell:Run test.flash INFO:native.tests/shell:Success INFO:native:Tests successful ``` </details> ...finally, running without the changes check should run the test as usual, even if nothing changes: ``` ./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . native --applications tests/shell ``` <details> ``` INFO:native:Saving toolchain INFO:native.tests/shell:Board supported: True INFO:native.tests/shell:Board has enough memory: True INFO:native.tests/shell:Application has test: True INFO:native.tests/shell:Run compilation INFO:native.tests/shell:Run test INFO:native.tests/shell:Run test.flash INFO:native.tests/shell:Success INFO:native:Tests successful ``` </details> ### Issues/PRs references Might help the surprises we got with #19469 19944: cpu/esp32: add low-level LCD parallel interface using LCD peripheral r=benpicco a=gschorcht ### Contribution description This PR the implementation of the LCD low-level MCU 8080 parallel interface using the LCD peripheral for ESP32, ESP32-S2 and ESP32-S3 or `periph_gpio_ll` for the ESP32-C3. ### Testing procedure ``` BOARD=esp32s3-wt32-sc01-plus make -C tests/drivers/st77xx flash ``` should work on top of PR #19941. Drawing operations should be much faster. ### Issues/PRs references Co-authored-by: MrKevinWeiss <[email protected]> Co-authored-by: Gunar Schorcht <[email protected]>

bors · 2023-11-04T18:03:26Z

Build failed (retrying...):

Murdock

19546: Enable compile_and_test_for_board to skip if nothing changed r=benpicco a=MrKevinWeiss ### Contribution description The overall goal of this PR is to be able to keep running the `compile_and_test_for_board.py` script without destroying ones boards. Useful if you are a board owner that wants to keep testing on master (say with nightlies, say in a CI). For example, I could now just run and rerun the following: ``` ./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . <MY_BOARD> -c ``` and monitor the results, even after updating a branch. This is because the hashes now get stored with the results (thanks to the `RIOT_TEST_HASH_DIR` var) which means cleaning will not wipe them out. Specifically in (`<results_base>/<board>/hashes/<app_dir>/test-input-hash.sha1`) I tried to do as much as I could with make and only alter the python script when needed. ### Testing procedure Clear results folder and run: ``` ./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . native --applications tests/shell -c ``` <details> ``` INFO:native:Saving toolchain INFO:native.tests/shell:Board supported: True INFO:native.tests/shell:Board has enough memory: True INFO:native.tests/shell:Application has test: True INFO:native.tests/shell:Run compilation **INFO:native.tests/shell:Hashes match: False** INFO:native.tests/shell:Run test INFO:native.tests/shell:Run test.flash INFO:native.tests/shell:Success INFO:native:Tests successful ``` </details> it should execute the test... Then the same command again should skip it. <details> ``` INFO:native:Saving toolchain INFO:native.tests/shell:Board supported: True INFO:native.tests/shell:Board has enough memory: True INFO:native.tests/shell:Application has test: True INFO:native.tests/shell:Run compilation **INFO:native.tests/shell:Hashes match: True** INFO:native.tests/shell:Success INFO:native:Tests successful ``` </details> Try changing something (say the `tests/shell/01-run.py`) and rerun, it should trigger the test again. <details> ``` INFO:native:Saving toolchain INFO:native.tests/shell:Board supported: True INFO:native.tests/shell:Board has enough memory: True INFO:native.tests/shell:Application has test: True INFO:native.tests/shell:Run compilation **INFO:native.tests/shell:Hashes match: False** INFO:native.tests/shell:Run test INFO:native.tests/shell:Run test.flash INFO:native.tests/shell:Success INFO:native:Tests successful ``` </details> ...finally, running without the changes check should run the test as usual, even if nothing changes: ``` ./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . native --applications tests/shell ``` <details> ``` INFO:native:Saving toolchain INFO:native.tests/shell:Board supported: True INFO:native.tests/shell:Board has enough memory: True INFO:native.tests/shell:Application has test: True INFO:native.tests/shell:Run compilation INFO:native.tests/shell:Run test INFO:native.tests/shell:Run test.flash INFO:native.tests/shell:Success INFO:native:Tests successful ``` </details> ### Issues/PRs references Might help the surprises we got with #19469 Co-authored-by: MrKevinWeiss <[email protected]>

bors · 2023-11-04T18:10:05Z

Build failed:

Murdock

MrKevinWeiss · 2023-11-07T08:35:39Z

It was just a copy pasta error (RIOT_TEST_RIOT_TEST_HASH_DIR instead of RIOT_TEST_HASH_DIR). Let's see if it is happy now.

RIOT_TEST_HASH_DIR represents the dir to generate the test-input-hash.sha1 file for checking if a test has changed, uses BINDIR by default. Since it defaults to BINDIR it should have no effect on running systems. This allows a bit more fine grain control, especially when using the compile_and_test_for_board.py script.

This target allows one to check if a test-input-hash is there and if it differs from a new one, adding basic support to skip tests if nothing needs to be run.

This will use the make test-input-hash-changed feature to save the test hashes with the results and optionally skip running the test if nothing has changed. Murdock already has this feature but it is not easily accessible. This should prevent unneeded flash cycles as well as speeding up constant rerunning of tests for boards.

MrKevinWeiss · 2024-02-01T12:50:14Z

rebased

MrKevinWeiss · 2024-02-06T09:50:43Z

Thanks!

github-actions bot added Area: build system Area: Build system Area: tests Area: tests and testing framework Area: tools Area: Supplementary tools labels May 4, 2023

MrKevinWeiss force-pushed the pr/reruntests branch from f17daf3 to 940dac4 Compare May 5, 2023 08:54

Teufelchen1 requested changes May 9, 2023

View reviewed changes

dist/tools/compile_and_test_for_board/compile_and_test_for_board.py Show resolved Hide resolved

MrKevinWeiss force-pushed the pr/reruntests branch from 940dac4 to 38d8cd4 Compare May 10, 2023 09:34

MrKevinWeiss force-pushed the pr/reruntests branch from 38d8cd4 to 0465b6d Compare May 10, 2023 09:37

MrKevinWeiss force-pushed the pr/reruntests branch from 0465b6d to c0c0623 Compare May 11, 2023 08:43

MrKevinWeiss added the CI: skip compile test If set, CI server will run only non-compile jobs, but no compile jobs or their dependent jobs label Jul 11, 2023

MrKevinWeiss force-pushed the pr/reruntests branch 2 times, most recently from 1d8eb0b to a439442 Compare September 4, 2023 13:56

Teufelchen1 approved these changes Sep 5, 2023

View reviewed changes

benpicco added the CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR label Oct 27, 2023

MrKevinWeiss force-pushed the pr/reruntests branch from a439442 to 650354c Compare November 1, 2023 14:12

MrKevinWeiss force-pushed the pr/reruntests branch from 650354c to 328487c Compare November 7, 2023 08:34

benpicco enabled auto-merge January 3, 2024 14:31

MrKevinWeiss added 3 commits February 1, 2024 13:49

makefiles/tests: Add test-input-hash-changed target

66d75f1

This target allows one to check if a test-input-hash is there and if it differs from a new one, adding basic support to skip tests if nothing needs to be run.

MrKevinWeiss force-pushed the pr/reruntests branch from 328487c to ceec795 Compare February 1, 2024 12:50

MrKevinWeiss disabled auto-merge February 1, 2024 13:06

Teufelchen1 enabled auto-merge February 1, 2024 13:34

Teufelchen1 added this pull request to the merge queue Feb 1, 2024

Merged via the queue into RIOT-OS:master with commit ed44a43 Feb 1, 2024

MrKevinWeiss deleted the pr/reruntests branch February 6, 2024 09:50

MrKevinWeiss added this to the Release 2024.04 milestone Apr 30, 2024

Conversation

MrKevinWeiss commented May 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution description

Testing procedure

Issues/PRs references

Uh oh!

MrKevinWeiss commented May 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Teufelchen1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MrKevinWeiss commented May 10, 2023

Uh oh!

Teufelchen1 commented May 10, 2023

Uh oh!

MrKevinWeiss commented May 10, 2023

Uh oh!

MrKevinWeiss commented Jun 29, 2023

Uh oh!

MrKevinWeiss commented Jul 11, 2023

Uh oh!

Teufelchen1 commented Sep 5, 2023

Uh oh!

MrKevinWeiss commented Sep 6, 2023

Uh oh!

benpicco commented Oct 27, 2023

Uh oh!

bors bot commented Oct 27, 2023

Uh oh!

riot-ci commented Oct 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Murdock results

Artifacts

Uh oh!

benpicco commented Oct 27, 2023

Uh oh!

bors bot commented Oct 27, 2023

Uh oh!

bors bot commented Oct 27, 2023

Uh oh!

MrKevinWeiss commented Nov 1, 2023

Uh oh!

MrKevinWeiss commented Nov 1, 2023

Uh oh!

bors bot commented Nov 1, 2023

Uh oh!

MrKevinWeiss commented Nov 1, 2023

Uh oh!

benpicco commented Nov 4, 2023

Uh oh!

bors bot commented Nov 4, 2023

Uh oh!

bors bot commented Nov 4, 2023

Uh oh!

MrKevinWeiss commented Nov 7, 2023

Uh oh!

MrKevinWeiss commented Feb 1, 2024

Uh oh!

MrKevinWeiss commented Feb 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

MrKevinWeiss commented May 4, 2023 •

edited

Loading

MrKevinWeiss commented May 4, 2023 •

edited

Loading

riot-ci commented Oct 27, 2023 •

edited

Loading