RIOT-OS · smlng · Jan 15, 2019 · Jan 22, 2019 · Jan 23, 2019 · MrKevinWeiss
@@ -0,0 +1,53 @@
+# Testing RIOT using the RobotFramework
+
+## Requirements
+
+RobotFramework is based on Python and is available as a Python package that
+provides tools (binaries) and builtin libraries to run tests. Before running
+or writing any test install the package by calling:
+
+```
+pip install robotframework
+```
+
+There is also a `requirements.txt` which also installs the `riot_pal` package
+for HIL tests, to install all at once, run:
+
+```
+pip install -r dist/robotframework/requirements.txt
+```
+
+That is all for the initial setup. If you don't want to install RobotFramework
+or any other Python dependency to your system environment you may consider to
+use [virtualenv](https://virtualenv.pypa.io).
+
+## Running Tests
+
+The RobotFramework tests are integrated into RIOTs build and make system by
+a distinct target, i.e., `make robot-test` to run tests and create the XML
+and HTML output files. For the latter, you can also specify a custom output
+folder by setting `RFOUTPATH` in your system environment - or directly when
+calling `make robot-test`. By default all test results are store in:
+
+```
+<RIOTBASE>/build/robot/<BOARD>/<APPLICATION>/
+```
+
+A typical test invocation is to run the following from the RIOT root folder:
+
+```
+BOARD=samr21-xpro make -C tests/<test-name> flash robot-test
+```
+
+## Extending and Writing Tests
+
+To write your own tests with the RobotFramework you only need to create a
+`.robot` file in the `tests` subfolder of the test application, that way it
+will be automatically found and executed by RIOTs build system. You may also
+extend an existing robot test script in the mentioned directory - these files
+also serve as examples for RIOT tests.
+
+A general overview and examples on how to write robot tests can be found on
+the [RobotFramework website](https://robotframework.org). Nevertheless, there
+are a number of RIOT specific keywords that should be used when writing tests.
+Please refer to the `*.keyword.txt` files in `dist/robotframework/res/`.
diff --git a/dist/robotframework/lib/PhilipAPI.py b/dist/robotframework/lib/PhilipAPI.py
@@ -0,0 +1,21 @@
+from riot_pal import LLMemMapIf, PHILIP_MEM_MAP_PATH
+from robot.version import get_version
+from time import sleep
+
+class PhilipAPI(LLMemMapIf):
+
+    ROBOT_LIBRARY_SCOPE = 'TEST SUITE'
+    ROBOT_LIBRARY_VERSION = get_version()
+
+    def __init__(self, port, baudrate):
+        super(PhilipAPI, self).__init__(PHILIP_MEM_MAP_PATH, 'serial', port, baudrate)
+
+    def reset_dut(self):
+        ret = list()
+        ret.append(self.write_reg('sys.cr', 0xff))
+        ret.append(self.execute_changes())
+        sleep(1)
+        ret.append(self.write_reg('sys.cr', 0x00))
+        ret.append(self.execute_changes())
+        sleep(1)
+        return ret
diff --git a/dist/robotframework/requirements.txt b/dist/robotframework/requirements.txt
@@ -0,0 +1,3 @@
+robotframework
+riot_pal==0.2.2
+
diff --git a/dist/robotframework/res/api_shell.keywords.txt b/dist/robotframework/res/api_shell.keywords.txt
@@ -0,0 +1,45 @@
+*** Settings ***
+Library     String
+Library     Collections
+
+*** Keywords ***
+API Call Expect
+    [Documentation]     Fails if the result of the given ``call`` does not
+    ...                 match the expected outcome.
+    [Arguments]         ${expect}  ${call}  @{args}  &{kwargs}
+    ${RESULT}=          Run Keyword  ${call}  @{args}  &{kwargs}
+    Set Suite Variable  ${RESULT}
+    Should Contain      ${RESULT['result']}  ${expect}
+
+API Call Should Succeed
+    [Documentation]     Fails if the given API ``call`` does not succeed.
+    [Arguments]         ${call}  @{args}  &{kwargs}
+    API Call Expect     Success  ${call}  @{args}  &{kwargs}
+
+API Call Should Timeout
+    [Documentation]     Fails if the given API ``call`` does not timeout.
+    [Arguments]         ${call}  @{args}  &{kwargs}
+    API Call Expect     Timeout  ${call}  @{args}  &{kwargs}
+
+API Call Should Error
+    [Documentation]     Fails if the given API ``call`` does not fail.
+    [Arguments]         ${call}  @{args}  &{kwargs}
+    API Call Expect     Error  ${call}  @{args}  &{kwargs}
+
+API Call Get Result As Integer
+    [Documentation]     Return result of last API call as an integer
+    ${ret}=             Convert to Integer  ${RESULT['data'][0]}
+    [Return]            ${ret}
+
+Repeat API Call on Timeout
+    [Documentation]     Repeats the given API ``call`` up to 5 times on timeout.
+    [Arguments]         ${call}  @{args}  &{kwargs}
+    :FOR    ${i}    IN RANGE  0  5
+    \   Run Keyword And Ignore Error  API Call Should Timeout  ${call}  @{args}  &{kwargs}
+    \   Run Keyword If  "${RESULT['result']}"!="Timeout"  Exit For Loop
+    Should Contain      ${RESULT['result']}   Success
+
+DUT Must Have API Firmware
+    [Documentation]     Verify that the DUT runs the required API test firmware
+    API Call Should Succeed  Get Metadata
+    Should Contain      ${RESULT['msg']}  %{APPLICATION}
diff --git a/dist/robotframework/res/philip.keywords.txt b/dist/robotframework/res/philip.keywords.txt
@@ -0,0 +1,11 @@
+*** Settings ***
+Library             PhilipAPI  port=%{PHILIP_PORT}  baudrate=${115200}  WITH NAME  PHILIP
+
+Resource            riot_base.keywords.txt
+
+*** Keywords ***
+Reset DUT and PHILIP
+    [Documentation]     Reset the device under test and the PHILIP tester.
+    Reset Application
+    PHILIP.Reset MCU
+    PHILIP.Reset DuT
diff --git a/dist/robotframework/res/riot_base.keywords.txt b/dist/robotframework/res/riot_base.keywords.txt
@@ -0,0 +1,16 @@
+*** Settings ***
+Library     Process
+Library     OperatingSystem
+Library     String
+Library     Collections
+
+*** Variables ***
+${RIOTTOOLS}            %{RIOTBASE}/dist/tools
+
+*** Keywords ***
+Reset Application
+    [Documentation]     Reset the test application
+    [Arguments]         ${sleep_before}=3  ${sleep_after}=3
+    Sleep               ${sleep_before}
+    Run Process         make reset  shell=True  cwd=%{APPDIR}
+    Sleep               ${sleep_after}
diff --git a/makefiles/robotframework.inc.mk b/makefiles/robotframework.inc.mk
@@ -0,0 +1,24 @@
+RFBASE ?= $(RIOTBASE)/dist/robotframework
+RFPYPATH ?= $(APPDIR)/tests:$(RFBASE)/lib:$(RFBASE)/res:$(RIOTBASE)/dist/tests
+RFOUTPATH ?= $(BUILD_DIR)/robot/$(BOARD)/$(APPLICATION)/
+
+ROBOT_FILES ?= $(wildcard tests/*.robot)
+
+robot-test:
+	$(call check_cmd,robot,RobotFramework tool)
+ifneq (,$(ROBOT_FILES))
+	python3 -m robot.run \
+				--name $(APPLICATION) \
+				--noncritical warn-if-failed \
+				--settag "APP_$(APPLICATION)" \
+				--settag "BOARD_$(BOARD)" \
+				--metadata RIOT-Version:$(RIOT_VERSION) \
+				--metadata RIOT-Board:$(BOARD) \
+				--metadata RIOT-Application:$(APPLICATION) \
+				--xunit xunit.xml \
+				-P "$(RFPYPATH)" \
+				-d $(RFOUTPATH) \
+				$(ROBOT_FILES)
+else
+	@echo No RobotFramework test scripts found, no worries.
+endif
@@ -7,3 +7,5 @@ RIOTBASE ?= $(CURDIR)/../..
 QUIET ?= 1
 # DEVELHELP enabled by default for all tests, set 0 to disable
 DEVELHELP ?= 1
+
+include $(RIOTBASE)/makefiles/robotframework.inc.mk
@@ -0,0 +1,8 @@
+include ../Makefile.tests_common
+
+USEMODULE += shell
+USEMODULE += xtimer
+
+CFLAGS += -DRIOT_APPLICATION=\"$(APPLICATION)\"
+
+include $(RIOTBASE)/Makefile.include
@@ -0,0 +1,16 @@
+# Xtimer API test
+
+This application enables tests of xtimer API calls using the RIOT shell.
+Consult the 'help' shell command for available actions.
+
+## Requirements
+
+The automated tests are based on RobotFramework and thus depend on additional
+Python packages, namely `robotframework` and `riot_pal`.
+See also the RobotFramework [README.md](../../dist/robotframework/README.md)
+for more details.
+
+## Background
+
+As this application provides a wrapper for (basic) xtimer API calls, it allows
+to implement various test cases by utilising and combining different calls.
@@ -0,0 +1,62 @@
+/*
+ * Copyright (C) 2019 HAW Hamburg
+ *
+ * This file is subject to the terms and conditions of the GNU Lesser
+ * General Public License v2.1. See the file LICENSE in the top level
+ * directory for more details.
+ */
+
+/**
+ * @ingroup tests
+ * @{
+ *
+ * @file
+ * @brief       Test application for xtimer API
+ *
+ * @author      Sebastian Meiling <[email protected]>
+ *
+ * @}
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+
+#include "shell.h"
+#include "xtimer.h"
+
+
+int cmd_xtimer_now(int argc, char **argv)
+{
+    (void)argv;
+    (void)argc;
+
+    uint32_t now = xtimer_now().ticks32;
+    printf("Success: xtimer_now(): [%"PRIu32"]\n", now);
+    return 0;
+}
+
+int cmd_get_metadata(int argc, char **argv)
+{
+    (void)argv;
+    (void)argc;
+
+    printf("Success: [%s, %s]\n", RIOT_BOARD, RIOT_APPLICATION);
+
+    return 0;
+}
+
+static const shell_command_t shell_commands[] = {
+    { "xtimer_now", "Get number of ticks (32Bit) from xtimer", cmd_xtimer_now },
+    { "get_metadata", "Get the metadata of the test firmware", cmd_get_metadata },
+    { NULL, NULL, NULL }
+};
+
+int main(void)
+{
+    puts("Start: Test for the xtimer API");
+
+    char line_buf[SHELL_DEFAULT_BUFSIZE];
+    shell_run(shell_commands, line_buf, SHELL_DEFAULT_BUFSIZE);
+
+    return 0;
+}
@@ -0,0 +1,31 @@
+*** Settings ***
+Documentation       Basic tests to verify functionality of the Xtimer API.
+
+# reset application and check DUT has correct firmware, skip all tests on error
+Suite Setup         Run Keywords    Reset Application
+...                                 DUT Must Have API Firmware
+# reset application and check DUT is up again befor every test case
+Test Setup          Run Keywords    Reset Application
+...                                 DUT Must Have API Firmware
+
+# import libs and keywords
+Library             Xtimer  port=%{PORT}  baudrate=%{BAUD}  timeout=${10}
+Resource            api_shell.keywords.txt
+Resource            riot_base.keywords.txt
+
+# add default tags to all tests
+Force Tags          xtimer
+
+*** Test Cases ***
+Xtimer Now Should Succeed
+    [Documentation]             Verify xtimer_now() API call.
+    API Call Should Succeed     Xtimer Now
+
+Xtimer Values Should Increase
+    [Documentation]             Compare two xtimer values (t1, t2) and verify
+    ...                         that they increase (t2 > t1).
+    API Call Should Succeed     Xtimer Now
+    ${t1}=                      API Call Get Result As Integer
+    API Call Should Succeed     Xtimer Now
+    ${t2}=                      API Call Get Result As Integer
+    Should Be True              ${t2} > ${t1}
@@ -0,0 +1,63 @@
+# Copyright (C) 2018 Kevin Weiss <[email protected]>
+#
+# This file is subject to the terms and conditions of the GNU Lesser
+# General Public License v2.1. See the file LICENSE in the top level
+# directory for more details.
+"""@package PyToAPI
+This module handles parsing of information from RIOT xtimer_cli test.
+"""
+import logging
+
+from riot_pal import DutShell
+from robot.version import get_version
+
+
+class Xtimer(DutShell):
+    """Interface to the a node with xtimer_cli firmware."""
+
+    ROBOT_LIBRARY_SCOPE = 'TEST SUITE'
+    ROBOT_LIBRARY_VERSION = get_version()
+
+    FW_ID = 'xtimer_cli'
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+
+    def is_connected_to_board(self):
+        """Checks if board is connected."""
+        return self.i2c_get_id()["data"] == [self.FW_ID]
+
+    def xtimer_now(self):
+        """Get current timer ticks."""
+        return self.send_cmd('xtimer_now')
+
+    def get_metadata(self):
+        """Get the metadata of the firmware."""
+        return self.send_cmd('get_metadata')
+
+    def get_command_list(self):
+        """List of all commands."""
+        cmds = list()
+        cmds.append(self.get_metadata)
+        cmds.append(self.xtimer_now)
+        return cmds
+
+
+def main():
+    """Test for Xtimer."""
+
+    logging.getLogger().setLevel(logging.DEBUG)
+    try:
+        xtimer = Xtimer()
+        cmds = xtimer.get_command_list()
+        logging.debug("======================================================")
+        for cmd in cmds:
+            cmd()
+            logging.debug("--------------------------------------------------")
+        logging.debug("======================================================")
+    except Exception as exc:
+        logging.debug(exc)
+
+
+if __name__ == "__main__":
+    main()