tools/testrunner: make interactive test sync retries/delay configurable#12891
Merged
fjmolinas merged 1 commit intoRIOT-OS:masterfrom Dec 13, 2019
Merged
Conversation
aabadie
added
Area: tests
aabadie
added
CI: ready for build
aabadie
force-pushed
the
pr/boards/hifive1b_fix_autotest
branch
from
9718696 to
1648a19
Compare
This was referenced Dec 9, 2019
aabadie
force-pushed
the
pr/boards/hifive1b_fix_autotest
branch
from
1648a19 to
55ef046
Compare
Contributor
Author
|
I dropped the commit related to hifive1b because I found another way (which I think is better) to fix the failing tests. |
aabadie
changed the title
fjmolinas
approved these changes
Dec 13, 2019
Contributor
fjmolinas
left a comment
fjmolinas
left a comment
There was a problem hiding this comment.
Although I don't see a clear use case for setting the variables yet, it causes no harm and could be used down the line. Tested that they can be set through the environment:
BOARD=iotlab-m3 TEST_INTERACTIVE_RETRIES=0 make -C tests/xtimer_usleep test --no-print-directory
Details
Connect to serial port /dev/riot/tty-iotlab-m3
Welcome to pyterm!
Type '/exit' to exit.
�Help: Press s to start test, r to print it is ready
Traceback (most recent call last):
File "/home/francisco/.local/lib/python3.6/site-packages/pexpect/expect.py", line 109, in expect_loop
return self.timeout()
File "/home/francisco/.local/lib/python3.6/site-packages/pexpect/expect.py", line 82, in timeout
raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: <pexpect.pty_spawn.spawn object at 0x7f96955c9c50>
command: /usr/bin/make
args: [b'/usr/bin/make', b'cleanterm']
buffer (last 100 chars): "3\r\nWelcome to pyterm!\r\nType '/exit' to exit.\r\n�Help: Press s to start test, r to print it is ready\r\n"
before (last 100 chars): "3\r\nWelcome to pyterm!\r\nType '/exit' to exit.\r\n�Help: Press s to start test, r to print it is ready\r\n"
after: <class 'pexpect.exceptions.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 7943
child_fd: 15
closed: False
timeout: 10
delimiter: <class 'pexpect.exceptions.EOF'>
logfile: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1
searcher: searcher_string:
0: 'READY'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/francisco/workspace/RIOT-test/tests/xtimer_usleep/tests/01-run.py", line 59, in <module>
sys.exit(run(testfunc))
File "/home/francisco/workspace/RIOT-test/dist/pythonlibs/testrunner/__init__.py", line 27, in run
logfile=sys.stdout if echo else None)
File "/home/francisco/workspace/RIOT-test/dist/pythonlibs/testrunner/spawn.py", line 60, in setup_child
sync_child(child)
File "/home/francisco/workspace/RIOT-test/dist/pythonlibs/testrunner/spawn.py", line 83, in sync_child
_test_utils_interactive_sync(child, modules)
File "/home/francisco/workspace/RIOT-test/dist/pythonlibs/testrunner/spawn.py", line 92, in _test_utils_interactive_sync
TEST_INTERACTIVE_DELAY)
File "/home/francisco/workspace/RIOT-test/dist/pythonlibs/testrunner/utils.py", line 24, in test_utils_interactive_sync
child.expect_exact('READY', timeout=0)
File "/home/francisco/.local/lib/python3.6/site-packages/pexpect/spawnbase.py", line 418, in expect_exact
return exp.expect_loop(timeout)
File "/home/francisco/.local/lib/python3.6/site-packages/pexpect/expect.py", line 119, in expect_loop
return self.timeout(e)
File "/home/francisco/.local/lib/python3.6/site-packages/pexpect/expect.py", line 82, in timeout
raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: <pexpect.pty_spawn.spawn object at 0x7f96955c9c50>
command: /usr/bin/make
args: [b'/usr/bin/make', b'cleanterm']
buffer (last 100 chars): "3\r\nWelcome to pyterm!\r\nType '/exit' to exit.\r\n�Help: Press s to start test, r to print it is ready\r\n"
before (last 100 chars): "3\r\nWelcome to pyterm!\r\nType '/exit' to exit.\r\n�Help: Press s to start test, r to print it is ready\r\n"
after: <class 'pexpect.exceptions.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 7943
child_fd: 15
closed: False
timeout: 10
delimiter: <class 'pexpect.exceptions.EOF'>
logfile: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1
searcher: searcher_string:
0: 'READY'
<pexpect.pty_spawn.spawn object at 0x7f96955c9c50>
command: /usr/bin/make
args: [b'/usr/bin/make', b'cleanterm']
buffer (last 100 chars): "3\r\nWelcome to pyterm!\r\nType '/exit' to exit.\r\n�Help: Press s to start test, r to print it is ready\r\n"
before (last 100 chars): "3\r\nWelcome to pyterm!\r\nType '/exit' to exit.\r\n�Help: Press s to start test, r to print it is ready\r\n"
after: <class 'pexpect.exceptions.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 7943
child_fd: 15
closed: False
timeout: 10
delimiter: <class 'pexpect.exceptions.EOF'>
logfile: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1
searcher: searcher_string:
0: 'READY'
/home/francisco/workspace/RIOT-test/tests/xtimer_usleep/../../Makefile.include:704: recipe for target 'test' failed
make: *** [test] Error 1
Contributor
|
ACK and GO! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Contribution description
This PR is an attempt to fix (most of) the automatic tests on hifive1b.
On this board, with the new test_interactive_sync feature, sending too much characters on UART before the firmware is ready can lead to a crash. Example on master:
The proposed solution is to make the test_interactive_sync delay/retries configurable from environment variables and use a large enough delay value for hifive1b. 3 seconds is enough.
I also tried to find a fix by looking at the fe310 cpu initialization but couldn't find it and I have no idea how this could be fixed from there.
Testing procedure
Run
compile_and_test_for_board.pyfor hifive1b: on master almost all tests are failing with the above message. With this PR they are almost all passing. Some tests are failing because of a bug on fe310, some tests can randomly fail because the flasher fails or because some initialization sequence with the on-board ESP32 radio fails.Here I give the final results (no output) with this PR (on master they all fail):
Details
Issues/PRs references
None