Skip to content

tests/riotboot: add automatic test#11707

Merged
fjmolinas merged 3 commits intoRIOT-OS:masterfrom
kaspar030:add_riotboot_test_script
Jul 15, 2019
Merged

tests/riotboot: add automatic test#11707
fjmolinas merged 3 commits intoRIOT-OS:masterfrom
kaspar030:add_riotboot_test_script

Conversation

@kaspar030
Copy link
Copy Markdown
Contributor

@kaspar030 kaspar030 commented Jun 17, 2019 •
edited
Loading

Contribution description

This PR adds a script that automatically tests riotboot's image selection.
It expects slot0, APP_VER=0 be pre-flashed on the device (and verifies that). This is what happens when "make flash" is run, and also happens on the CI.

A following "make test" will:

  1. verify slot0, APP_VER=0
  2. flash slot1, APP_VER=1, verify it is booting
  3. flash slot0, APP_VER=2, verify it is booting

Testing procedure

  1. look at CI output
  2. try local make -Ctests/riotboot flash test

Issues/PRs references

Needs #11697 and #11690.

@kaspar030 kaspar030 added Area: tests Area: tests and testing framework State: waiting for other PR State: The PR requires another PR to be merged first CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR CI: run tests If set, CI server will run tests on hardware for the labeled PR labels Jun 17, 2019
@kaspar030 kaspar030 mentioned this pull request Jun 17, 2019
Merged
@kaspar030
Copy link
Copy Markdown
Contributor Author

kaspar030 commented Jun 17, 2019

See e.g., https://ci.riot-os.org/RIOT-OS/RIOT/11707/372fb0b6aead9398987ec43cfd4098bb8ceefde3/output/run_test/tests/riotboot/nrf52dk:gnu.txt or https://ci.riot-os.org/RIOT-OS/RIOT/11707/372fb0b6aead9398987ec43cfd4098bb8ceefde3/output/run_test/tests/riotboot/samr21-xpro:gnu.txt for the test's output.

@kaspar030 kaspar030 requested a review from fjmolinas June 17, 2019 15:40
fjmolinas
fjmolinas requested changes Jun 17, 2019
View reviewed changes
Copy link
Copy Markdown
Contributor

@fjmolinas fjmolinas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested locally, it works after slot1 is compiled separetly (see comment below). Sticking to the changes of this PR related to the tests it looks good, just need to address my single comment. Commits might need to be fixed by there is still much noise from the other PR's, so let's wait for them.

@kaspar030 kaspar030 force-pushed the add_riotboot_test_script branch from 837bec4 to fbdfffe Compare June 18, 2019 09:27
@kaspar030 kaspar030 force-pushed the add_riotboot_test_script branch from b31accb to 55bf6af Compare June 27, 2019 13:22
@kaspar030 kaspar030 mentioned this pull request Jul 5, 2019
Merged
@fjmolinas fjmolinas mentioned this pull request Jul 5, 2019
Closed
kaspar030 added a commit to kaspar030/RIOT that referenced this pull request Jul 8, 2019
@kaspar030
DEPENDENCY MERGE: add_riotboot_test_script (RIOT-OS#11707)
a79eb60
@kaspar030 kaspar030 mentioned this pull request Jul 8, 2019
Merged
kaspar030 added a commit to kaspar030/RIOT that referenced this pull request Jul 10, 2019
@kaspar030
DEPENDENCY MERGE: add_riotboot_test_script (RIOT-OS#11707)
6a029f2
@kaspar030 kaspar030 force-pushed the add_riotboot_test_script branch from 55bf6af to 1d73c87 Compare July 12, 2019 08:12
kaspar030 added a commit to kaspar030/RIOT that referenced this pull request Jul 12, 2019
@kaspar030
DEPENDENCY MERGE: add_riotboot_test_script (RIOT-OS#11707)
c907174
@fjmolinas
Copy link
Copy Markdown
Contributor

fjmolinas commented Jul 14, 2019

@kaspar030 please rebase so I can give a fresh round of testing on monday.

@kaspar030 kaspar030 force-pushed the add_riotboot_test_script branch from fee339f to dc33043 Compare July 15, 2019 08:31
@kaspar030 kaspar030 removed the State: waiting for other PR State: The PR requires another PR to be merged first label Jul 15, 2019
@kaspar030
Copy link
Copy Markdown
Contributor Author

kaspar030 commented Jul 15, 2019

  • rebased, not waiting for other PR's anymore!

kaspar030 added a commit to kaspar030/RIOT that referenced this pull request Jul 15, 2019
@kaspar030
DEPENDENCY MERGE: add_riotboot_test_script (RIOT-OS#11707)
ef6bc21
fjmolinas
fjmolinas reviewed Jul 15, 2019
View reviewed changes
Copy link
Copy Markdown
Contributor

@fjmolinas fjmolinas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kaspar030 tested locally and it is working fine, please squash so we can re-lunch the ci, and I will check the output.

kaspar030 added 3 commits July 15, 2019 11:30
@kaspar030
tests/riotboot: add re-flashing and image selection test
43c3e43
@kaspar030
make/riotboot: allow slot image creation without compiling
6b2727c
@kaspar030
riotboot: always link both slot .elf when linking
aaa187e 
This enables "make test" and "make test-murdock" to rely on both images being
present and thus execute without re-compiling.
@kaspar030 kaspar030 force-pushed the add_riotboot_test_script branch from dc33043 to aaa187e Compare July 15, 2019 09:31
@kaspar030 kaspar030 added CI: disable test cache If set, CI will always run all tests regardless of whether they have been run successfully before CI: run tests If set, CI server will run tests on hardware for the labeled PR and removed CI: run tests If set, CI server will run tests on hardware for the labeled PR labels Jul 15, 2019
@fjmolinas
Copy link
Copy Markdown
Contributor

fjmolinas commented Jul 15, 2019

See e.g., https://ci.riot-os.org/RIOT-OS/RIOT/11707/372fb0b6aead9398987ec43cfd4098bb8ceefde3/output/run_test/tests/riotboot/nrf52dk:gnu.txt or https://ci.riot-os.org/RIOT-OS/RIOT/11707/372fb0b6aead9398987ec43cfd4098bb8ceefde3/output/run_test/tests/riotboot/samr21-xpro:gnu.txt for the test's output.

@kaspar030 how can I view the same outputs for the new build?

@kaspar030 kaspar030 added CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR and removed CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Jul 15, 2019
@kaspar030
Copy link
Copy Markdown
Contributor Author

kaspar030 commented Jul 15, 2019

@kaspar030 how can I view the same outputs for the new build?

I've triggered a re-build with "disable test cache", that should produce the output.

@fjmolinas
Copy link
Copy Markdown
Contributor

fjmolinas commented Jul 15, 2019

@kaspar030 out of this PR topic, but does the CI progression bar take into account the time taken to run the tests?

fjmolinas
fjmolinas approved these changes Jul 15, 2019
View reviewed changes
Copy link
Copy Markdown
Contributor

@fjmolinas fjmolinas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was able to tests locally and verified it ran correctly in the CI on hardware. I think the tests scripts could all be implemented in python, but I leave this for another PR. ACK!

@fjmolinas fjmolinas merged commit 23c9188 into RIOT-OS:master Jul 15, 2019
@kaspar030 kaspar030 deleted the add_riotboot_test_script branch July 15, 2019 11:38
@kaspar030
Copy link
Copy Markdown
Contributor Author

kaspar030 commented Jul 15, 2019

Thanks for reviewing!

@fjmolinas fjmolinas mentioned this pull request Jul 15, 2019
Merged
@kb2ma kb2ma added this to the Release 2019.10 milestone Sep 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fjmolinas fjmolinas fjmolinas approved these changes

Assignees

No one assigned

Labels

Area: tests Area: tests and testing framework CI: disable test cache If set, CI will always run all tests regardless of whether they have been run successfully before CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR CI: run tests If set, CI server will run tests on hardware for the labeled PR

Projects

None yet

Milestone

Release 2019.10

Development

Successfully merging this pull request may close these issues.

3 participants

@kaspar030 @fjmolinas @kb2ma