overview

At the RIOT summit 2018 @MrKevinWeiss and I were hosting a breakout session on hardware in the loop (HIL) testing for RIOT. Apart from the hardware and tools Kevin developed, we also introduced a new style/way of writing tests for RIOT utilising the RobotFramework, and showed some examples on existing tests adapted to RF but also a completely new test for the I2C peripheral bus.

However, there was no consensus on how to actually move forward with using the RobotFramework to write tests for RIOT. We agreed on opening the discussion on this to a wider audience by providing initial PRs as proof-of-concept. These PRs can be found in #10095 as the initial introduction of RF-based tests to RIOT, and #10147 providing the I2C tests. The current implementation does not replace any existing tests nor does it change the test behaviour, but rather introduces a new make target, i.e., robot-test to run RF based tests, so both PRs are non-intrusive.

The purpose of this issue is to move the general discussion around using RobotFramework to write and run RIOT tests from mentioned PRs here and get more people involved. But also collect all the different arguments in one place, because some commented on the PRs, some voiced opinions in eMails, telcos and f2f meetings.

more details

As mentioned above, currently the integration of RobotFramework is non-intrusive and parallel to all testrunner scripts. Also RF-based tests are not run for every PR via Murdock, but we already hooked it into the nightlies - currently the test report only contains I2C tests.

But why consider using RobotFramework anyway? And what about testrunner?

RobotFramework is an active open source test framework that is around for several years and widely used by other projects. Under the hood it is based on Python and can be extended using Python or Java.

Tests are written in a RobotFramework specific format and syntax - that is obviously different from our current way of writing test with testrunner, which is basically Python+pexpect, right?
Certainly, if someone is used (and loves) to write Python code, RF-tests look and are (very) different. While plain Python allows for a larger degree of freedom to implement tests, it does not provide or enforce any structure to adhere to. In RobotFramework there are a limited number of central building blocks and concepts that are part of every test and are used repeatedly.

For instance, each test-script in RF defines a test-suite which contains one or more test cases, and each test case is a collection of keywords processed and evaluated sequentially. The test file itself typically contains several sections, like settings, variables, and most importantly test cases. Test suites and cases can have documentation and tags, the latter allowing for test selection during execution but also for evaluation. As an example look at a simple tests such as xtimer_hang.robot, which does the same as the corresponding Python script for testrunner.

In the aforementioned PRs I already started to create custom RIOT specific keywords (see riot_util.keywords.txt) that combine several existing keywords. Also there is a Python wrapper around the I2C HIL testing API which easily integrates with RobotFramework. This can and should be extended in the future to ease writing new tests and also keep them readable.

RobotFramework generates XML output for every test script, which can be stored and processed further. Therefore it provides extensive tooling around this output such as generating HTML from one XML file, but also allowing to combine XML output from many tests into a single HTML report or select test results (from multiple XMLs) eg. by tags to generate a smaller test report. Further, it allows to do process the test output from a previous run and rerun all failed tests.

discussion

Unsorted, incomplete list of issues raised so far:

• From what was discussed and commented so far via different channels, one major issue with RF-style tests is its format and syntax of writing actual tests. (see details above). But then, is I like Python code better a good and valid argument in this discussion?

• Another issue is, that the output format of RobotFramework is not that special, there are other standards too and such could easily be added to testrunner. That may be true, but in RF its already there, no work needed.

Comparison with PyTest (NEW)

links

• riot_util.keywords.txt
• xtimer_hang.robot
• RF README for RIOT

finally

please comment, ask questions ...