@kYc0o how did you test this? Which board revision and which test? On my side, make test sill doesn't work. Pressing the reset button after calling make test and before pyterm is started, triggers a reset whereas this seems defect after pyterm started. However, on master the board takes button resets always. Ideas?

I'm investigating, strangely I notice the same behaviour as you in a raspberry, but with in my OS X everything works...

What happens if you try make reset? With @kaspar030 we also noticed that when the script is called without arguments, the script ends with exit(2), which makes the rule make test fail.

make reset works if there is no serial connection open. make test fails due to timeout.

make reset works if there is no serial connection open

Which is completely normal since the serial line cannot be opened by two processes.

make test fails due to timeout.

Yes, but this is because the target doesn't succeed to reset.

In OS X succeeds as follows:

snake:RIOT facosta$ BOARD=remote-revb make -C tests/pkg_tinycrypt/ flash-only test
make: Entering directory '/Users/facosta/git/RIOT-OS/RIOT/tests/pkg_tinycrypt'
/Users/facosta/git/RIOT-OS/RIOT/dist/tools/cc2538-bsl/cc2538-bsl.py -p "/dev/tty.SLAB_USBtoUART" -e -w -v -b 115200 /Users/facosta/git/RIOT-OS/RIOT/tests/pkg_tinycrypt/bin/remote-revb/tests_pkg_tinycrypt.bin
Opening port /dev/tty.SLAB_USBtoUART, baud 115200
Reading data from /Users/facosta/git/RIOT-OS/RIOT/tests/pkg_tinycrypt/bin/remote-revb/tests_pkg_tinycrypt.bin
Cannot auto-detect firmware filetype: Assuming .bin
Connecting to target...
CC2538 PG2.0: 512KB Flash, 32KB SRAM, CCFG at 0x0027FFD4
Primary IEEE Address: 00:12:4B:00:14:D5:2D:91
Erasing 524288 bytes starting at address 0x00200000
    Erase done
Writing 524288 bytes starting at address 0x00200000
Write 16 bytes at 0x0027FFF0F8
    Write done                                
Verifying by comparing CRC32 calculations.
    Verified (match: 0xcac73e7a)
tests/01-run.py
/Users/facosta/git/RIOT-OS/RIOT/dist/tools/pyterm/pyterm -p "/dev/tty.SLAB_USBtoUART" -b "115200"
No handlers could be found for logger "root"
2018-06-28 13:33:38,964 - INFO # Connect to serial port /dev/tty.SLAB_USBtoUART
Welcome to pyterm!
Type '/exit' to exit.
2018-06-28 13:33:41,765 - INFO # "&��2 0x99 0xB3 0x1A 0x02 0xD7 0x3A ]
2018-06-28 13:33:41,769 - INFO # 
2018-06-28 13:33:41,771 - INFO # And now decrypt the cipher again:
2018-06-28 13:33:41,773 - INFO # decrypted: [ 0x54 0x77 0x6F 0x20 0x4F 0x6E 0x65 0x20 0x4E 0x69 0x6E 0x65 0x20 0x54 0x77 0x6F ]
2018-06-28 13:33:41,774 - INFO #     ASCII: Two One Nine Two
2018-06-28 13:33:41,774 - INFO # 
2018-06-28 13:33:41,775 - INFO # [SUCCESS]

make: Leaving directory '/Users/facosta/git/RIOT-OS/RIOT/tests/pkg_tinycrypt'

I'm investigating

Did you already figure something out?

Did you already figure something out?

Not yet. Tomorrow I'll have time.

I've added the reset command to the script directly, it's cleaner but I don't know why the exit message still 2 and it's not what testrunner expects. I'll check further.

For actual HIL testing in a fixed setup I would rather connect/wire the reset pin to a GPIO of the Raspberry Pi and trigger a reset there.

@kYc0o still not working here. I don't quite understand why it is cleaner to adapt the cc2538-bsl script. Generally, I would try to avoid that - if possible. Please ping me once ready for testing.

@smlng the need for a pin-based reset for HIL shouldn't obsolete make test, does it? AFAIK the RE-Mote has a reset pin anyway (compare pinout)

@PeterKietzmann I'm aware of the pinout and the reset pin, that's why I mentioned it - and b/c I'd rather not change the bsl script, too. However, I looked into it, and I guess there is no why of resetting the remote without either changing the bsl script or write a separate script coping parts from the bsl just for the reset command.

Just in case, the original code on github has moved a bit during the last months, so modifying the bsl script will make it harder to backport it again into the RIOT code base.

IMHO the script has changed a lot compared to the "original" one, which was just copied from the contiki repo. @gebart did some modifications to make it cleaner and thus, I'd say adding 3 lines of code won't make it worse to maintain, if ever there's actually a "need" to maintain it (with maintain I mean to "update" it according to the original script).

Sorry, it was not @gebart who modified it, but several people. I just looked at the history.

With this diff for the testrunner.py:

diff --git a/dist/tools/testrunner/testrunner.py b/dist/tools/testrunner/testrunner.py
index 8678b90..51a55b1 100755
--- a/dist/tools/testrunner/testrunner.py
+++ b/dist/tools/testrunner/testrunner.py
@@ -46,11 +46,16 @@ def run(testfunc, timeout=10, echo=True, traceback=False):
         child.logfile = sys.stdout
 
     try:
-        subprocess.check_output(('make', 'reset'), env=env,
+        print("resetting...")
+        out = subprocess.check_output(('make', 'reset'), env=env,
                                 stderr=subprocess.PIPE)
-    except subprocess.CalledProcessError:
+        print(out)
+
+    except subprocess.CalledProcessError as e:
         # make reset yields error on some boards even if successful
+        print(e, e.output)
         pass

In OS X it works like a charm (python 3.6.5), but in Linux (python 3.5.2) I have the following:

vagrant@vagrant:~/RIOT/tests/pkg_tinycrypt$ BOARD=remote-revb make test
tests/01-run.py
resetting...
Command '('make', 'reset')' returned non-zero exit status 2 b'/home/vagrant/RIOT/dist/tools/cc2538-bsl/cc2538-bsl.py -p "/dev/ttyUSB0" --reset\n/home/vagrant/RIOT/Makefile.include:478: recipe for target \'reset\' failed\n'
/home/vagrant/RIOT/dist/tools/pyterm/pyterm -p "/dev/ttyUSB0" -b "115200"

However make reset outside the python script works like a charm both in Linux and OS X.

Any ideas?

So, modifying the cc2538-bsl.py script doesn't help at all so I removed the commit. What I think is the problem is that since the serial line is required for both reset and test, for some reason Linux is slow enough to not free the resource and thus it makes it fail, while in OS X is freed soon enough and thus succeeds.

Then, the scope for this PR is just to add the reset command and not to make make test work.

IMHO that should be handled either differently or by modifying testrunner.

I can confirm that make reset works for the remote-revb and make test still does not. I'm using Ubuntu 18.04.

I've updated the above description because this PRdoes not fix make test but make reset

I agree. Thanks!