Description

Opening this issue for tracking the problems with tests found in #12649. I remove the problems related to interaction with the UART because they should be fixed #12461 and the problem related to network setup with GNRC tests.

Here is the list of failing tests:

• tests/bitarithm_timings: looks like a context switching issue a bug in the test fixed by tests/bitarithm_timings: Fixed bugs #12714

  Details

  make RIOT_CI_BUILD=1 CC_NOCOLOR=1 --no-print-directory -C ./tests/bitarithm_timings test
  /work/riot/RIOT/dist/tools/pyterm/pyterm -p "/dev/ttyACM0" -b "115200" --noprefix --no-repeat-command-on-empty-line
  Connect to serial port /dev/ttyACM0
  Welcome to pyterm!
  Type '/exit' to exit.
  
  
  main(): This is RIOT! (Version: buildtest)
  Start.
  + bitarithm_msb: 102096 iterations per second
  Timeout in expect script at "child.expect('\+ bitarithm_lsb: \d+ iterations per second')" (tests/bitarithm_timings/tests/01-run.py:16)
  

• tests/event_wait_timeout: Fixed in tests/event_wait_timeout: Fix for 8bit platforms #12776

  Details

  make RIOT_CI_BUILD=1 CC_NOCOLOR=1 --no-print-directory -C ./tests/event_wait_timeout test
  /work/riot/RIOT/dist/tools/pyterm/pyterm -p "/dev/ttyACM0" -b "115200" --noprefix --no-repeat-command-on-empty-line
  Connect to serial port /dev/ttyACM0
  Welcome to pyterm!
  Type '/exit' to exit.
  finished: 4/4 events and 1/4 timeouts recorded
  [FAILED]
  
  
  main(): This is RIOT! (Version: buildtest)
  [START] event_wait_timeout test application.
  
  finished: 4/4 events and 1/4 timeouts recorded
  [FAILED]
  Timeout in expect script at "child.expect_exact(u"[SUCCESS]")" (tests/event_wait_timeout/tests/01-run.py:15)
  

• tests/gnrc_ipv6_fwd_w_sub: timing issue => fixed by tests: use test_interactive_test_util when possible #12461

  Details

  make RIOT_CI_BUILD=1 CC_NOCOLOR=1 --no-print-directory -C ./tests/gnrc_ipv6_fwd_w_sub test
  run_test
  /work/riot/RIOT/dist/tools/pyterm/pyterm -p "/dev/ttyACM0" -b "115200" --noprefix --no-repeat-command-on-empty-line
  Connect to serial port /dev/ttyACM0
  Welcome to pyterm!
  Type '/exit' to exit.
  
  
  main(): This is RIOT! (Version: buildtest)
  > Timeout in expect script at "child.expect(r"Forwarded Ethernet frame:")" (tests/gnrc_ipv6_fwd_w_sub/tests/01-run.py:16)
  

• tests/gnrc_ndp: Fixed in tests/gnrc_ndp: don't guess size of gnrc_pktsnip_t in pktbuf #16051

  Details

  make RIOT_CI_BUILD=1 CC_NOCOLOR=1 --no-print-directory -C ./tests/gnrc_ndp test
  /work/riot/RIOT/dist/tools/pyterm/pyterm -p "/dev/ttyACM0" -b "115200" --noprefix --no-repeat-command-on-empty-line
  Connect to serial port /dev/ttyACM0
  Welcome to pyterm!
  Type '/exit' to exit.
  
  
  main(): This is RIOT! (Version: buildtest)
  ............................
  tests.test_nbr_sol_send__pktbuf_full4 (tests/gnrc_ndp/main.c 552) gnrc_pktbuf_is_empty()
  .
  tests.test_nbr_adv_send__foreign_tgt_unspecified_dst_no_supply_tl2a_no_ext_opts (tests/gnrc_ndp/main.c 571) src != pkt->src
  .
  tests.test_nbr_adv_send__foreign_tgt_unspecified_dst_no_supply_tl2a_ext_opts (tests/gnrc_ndp/main.c 568) exp -1 was 0
  .
  tests.test_nbr_adv_send__foreign_tgt_unspecified_dst_supply_tl2a_no_ext_opts (tests/gnrc_ndp/main.c 568) exp 5 was 0
  .
  tests.test_nbr_adv_send__foreign_tgt_unspecified_dst_supply_tl2a_ext_opts (tests/gnrc_ndp/main.c 568) exp -1 was 0
  .
  tests.test_nbr_adv_send__foreign_tgt_specified_dst_no_supply_tl2a_no_ext_opts (tests/gnrc_ndp/main.c 574) exp 2 was 8
  .
  tests.test_nbr_adv_send__foreign_tgt_specified_dst_no_supply_tl2a_ext_opts (tests/gnrc_ndp/main.c 568) exp -1 was 0
  .
  tests.test_nbr_adv_send__foreign_tgt_specified_dst_supply_tl2a_no_ext_opts (tests/gnrc_ndp/main.c 568) exp -1 was 104
  .
  tests.test_nbr_adv_send__foreign_tgt_specified_dst_supply_tl2a_ext_opts (tests/gnrc_ndp/main.c 568) exp -1 was 0
  .
  tests.test_nbr_adv_send__src_tgt_unspecified_dst_no_supply_tl2a_no_ext_opts (tests/gnrc_ndp/main.c 568) exp 5 was 0
  .
  tests.test_nbr_adv_send__src_tgt_unspecified_dst_no_supply_tl2a_ext_opts (tests/gnrc_ndp/main.c 568) exp -1 was 0
  .
  tests.test_nbr_adv_send__src_tgt_unspecified_dst_supply_tl2a_no_ext_opts (tests/gnrc_ndp/main.c 568) exp 5 was 0
  .
  tests.test_nbr_adv_send__src_tgt_unspecified_dst_supply_tl2a_ext_opts (tests/gnrc_ndp/main.c 568) exp -1 was 0
  .
  tests.test_nbr_adv_send__src_tgt_specified_dst_no_supply_tl2a_no_ext_opts (tests/gnrc_ndp/main.c 574) exp 2 was 8
  .
  tests.test_nbr_adv_send__src_tgt_specified_dst_no_supply_tl2a_ext_opts (tests/gnrc_ndp/main.c 568) exp -1 was 0
  .
  tests.test_nbr_adv_send__src_tgt_specified_dst_supply_tl2a_no_ext_opts (tests/gnrc_ndp/main.c 568) exp -1 was 104
  .
  tests.test_nbr_adv_send__src_tgt_specified_dst_supply_tl2a_ext_opts (tests/gnrc_ndp/main.c 568) exp -1 was 0
  ....
  tests.test_nbr_adv_send__pktbuf_full4 (tests/gnrc_ndp/main.c 763) gnrc_pktbuf_is_empty()
  .
  tests.test_rtr_sol_send__dst_NULL (tests/gnrc_ndp/main.c 784) exp 2 was 0
  .
  tests.test_rtr_sol_send__dst_local (tests/gnrc_ndp/main.c 780) src != pkt->src
  .....
  tests.test_rtr_sol_send__pktbuf_full4 (tests/gnrc_ndp/main.c 886) gnrc_pktbuf_is_empty()
  .
  tests.test_rtr_adv_send__src_NULL_dst_NULL_no_fin_no_ext_opts (tests/gnrc_ndp/main.c 903) exp -1 was 0
  .
  tests.test_rtr_adv_send__src_NULL_dst_NULL_no_fin_ext_opts (tests/gnrc_ndp/main.c 911) dst != pkt->dst
  .
  tests.test_rtr_adv_send__src_NULL_dst_NULL_fin_no_ext_opts (tests/gnrc_ndp/main.c 903) exp 5 was 8647
  .
  tests.test_rtr_adv_send__src_NULL_dst_NULL_fin_ext_opts (tests/gnrc_ndp/main.c 903) exp -1 was 0
  .
  tests.test_rtr_adv_send__src_NULL_dst_no_fin_no_ext_opts (tests/gnrc_ndp/main.c 903) exp 5 was 8647
  .
  tests.test_rtr_adv_send__src_NULL_dst_no_fin_ext_opts (tests/gnrc_ndp/main.c 903) exp -1 was 0
  .
  tests.test_rtr_adv_send__src_NULL_dst_fin_no_ext_opts (tests/gnrc_ndp/main.c 903) exp 5 was 8647
  .
  tests.test_rtr_adv_send__src_NULL_dst_fin_ext_opts (tests/gnrc_ndp/main.c 903) exp -1 was 0
  .
  tests.test_rtr_adv_send__src_dst_NULL_no_fin_no_ext_opts (tests/gnrc_ndp/main.c 903) exp 5 was 8647
  .
  tests.test_rtr_adv_send__src_dst_NULL_no_fin_ext_opts (tests/gnrc_ndp/main.c 903) exp -1 was 0
  .
  tests.test_rtr_adv_send__src_dst_NULL_fin_no_ext_opts (tests/gnrc_ndp/main.c 903) exp 5 was 8647
  .
  tests.test_rtr_adv_send__src_dst_NULL_fin_ext_opts (tests/gnrc_ndp/main.c 903) exp -1 was 0
  .
  tests.test_rtr_adv_send__src_dst_no_fin_no_ext_opts (tests/gnrc_ndp/main.c 903) exp 5 was 8647
  .
  tests.test_rtr_adv_send__src_dst_no_fin_ext_opts (tests/gnrc_ndp/main.c 903) exp -1 was 0
  .
  tests.test_rtr_adv_send__src_dst_fin_no_ext_opts (tests/gnrc_ndp/main.c 903) exp 5 was 8647
  .
  tests.test_rtr_adv_send__src_dst_fin_ext_opts (tests/gnrc_ndp/main.c 903) exp -1 was 0
  ....
  tests.test_rtr_adv_send__pktbuf_full4 (tests/gnrc_ndp/main.c 1085) gnrc_pktbuf_is_empty()
  
  run 75 failures 38
  Timeout in expect script at "child.expect(r"OK \(\d+ tests\)")" (tests/gnrc_ndp/tests/01-run.py:16)
  

• tests/isr_yield_higher: Fixed in cpu/atmega_common #12790

  Details

  make RIOT_CI_BUILD=1 CC_NOCOLOR=1 --no-print-directory -C ./tests/isr_yield_higher test
  /work/riot/RIOT/dist/tools/pyterm/pyterm -p "/dev/ttyACM0" -b "115200" --noprefix --no-repeat-command-on-empty-line
  Connect to serial port /dev/ttyACM0
  Welcome to pyterm!
  Type '/exit' to exit.
  first thread done
  
  
  main(): This is RIOT! (Version: buildtest)
  first thread started
  timer triggered
  TEST FAILED
  first thread done
  Timeout in expect script at "child.expect('TEST SUCCESSFUL')" (tests/isr_yield_higher/tests/test.py:17)
  
  

• tests/libfixmath: fixed by pkg/libfixmath: several improvements and 8bit fix #12687

  Details

  make RIOT_CI_BUILD=1 CC_NOCOLOR=1 --no-print-directory -C ./tests/libfixmath test
  r
  /work/riot/RIOT/dist/tools/pyterm/pyterm -p "/dev/ttyACM0" -b "115200" --noprefix --no-repeat-command-on-empty-line
  Connect to serial port /dev/ttyACM0
  Welcome to pyterm!
  Type '/exit' to exit.
  
  
  main(): This is RIOT! (Version: buildtest)
  Help: Press s to start test, r to print it is ready
  r
  READY
  s
  START
  Unary.
  abs(-10.0000) = 10.0000
  sq(-10.0000) = 100.0000
  atan(-10.0000) = -1.4989
  
  [...]
  
  min(-4.2500, 1.2500) = -4.2500
  max(-4.2500, 1.2500) = 1.2500
  add(-4.2500, 1.5000) = -2.7500
  sub(-4.2500, 1.5000) = -5.7500
  mul(-4.2500, 1.5000) = -5.8750
  Timeout in expect script at "child.expect('{}\(-?\d+.\d+\) = -?\d+.\d+'.format(op_name))" (tests/libfixmath/tests/01-run.py:14)
  

• tests/pipe: fixed in tests/pipe: cleanup application #12792

  Details

  make RIOT_CI_BUILD=1 CC_NOCOLOR=1 --no-print-directory -C ./tests/pipe test
  /work/riot/RIOT/dist/tools/pyterm/pyterm -p "/dev/ttyACM0" -b "115200" --noprefix --no-repeat-command-on-empty-line
  Connect to serial port /dev/ttyACM0
  Welcome to pyterm!
  Type '/exit' to exit.
  
  
  main(): This is RIOT! (Version: buildtest)
  Start.
  Middle read: <Middle read: <End read: <End read: <Middle read: <Middle read: <End read: <End read: <Middle read: <Middle read: <End read: <End read: <Middle read: <Middle read: <Main done.
  End read: <End read: <Middle read: <Middle done.
  End read: <End done.
  Timeout in expect script at "child.expect_exact('Middle read: <ABCD> [0:4]')" (tests/pipe/tests/01-run.py:9)
  

• tests/pkg_jsmn: Fixed in tests/pkg_jsmn: replace use of "%.*s" string formatter #12795

  Details

  make RIOT_CI_BUILD=1 CC_NOCOLOR=1 --no-print-directory -C ./tests/pkg_jsmn test
  /work/riot/RIOT/dist/tools/pyterm/pyterm -p "/dev/ttyACM0" -b "115200" --noprefix --no-repeat-command-on-empty-line
  Connect to serial port /dev/ttyACM0
  Welcome to pyterm!
  Type '/exit' to exit.
  
  
  main(): This is RIOT! (Version: buildtest)
  - User: - Admin: - UID: - Groups:
  Timeout in expect script at "child.expect_exact('- User: johndoe')" (tests/pkg_jsmn/tests/01-run.py:8)
  

• tests/pkg_lora-serialization: seems like a conversion issue. Maybe this package is not compatible with 8bit architecture. Fixed by pkg/lora-serialization: bump to latest version and update test for AVR #12689

  Details

  make RIOT_CI_BUILD=1 CC_NOCOLOR=1 --no-print-directory -C ./tests/pkg_lora-serialization test
  /work/riot/RIOT/dist/tools/pyterm/pyterm -p "/dev/ttyACM0" -b "115200" --noprefix --no-repeat-command-on-empty-line
  Connect to serial port /dev/ttyACM0
  Welcome to pyterm!
  Type '/exit' to exit.
  
  
  main(): This is RIOT! (Version: buildtest)
  Lora Serialization test
  
  Test 1
  Temperature and humidity
  ---------------------------------
  - Writing temperature: 80.12
  - Writing humidity: 99.99
  - Encoded:  1f 4c 0f 27
  - Expected: 1f 4c 0f 27
  ---------------------------------
  SUCCESS
  
  Test 2
  Coordinates and unix time
  ---------------------------------
  - Writing coordinates: -33.905052, 151.26641
  - Writing unix time: 1467632413
  - Encoded:  64 a6 ff ff 60 24 00 00 1d 4b 00 00
  - Expected: 64 a6 fa fd 6a 24 04 09 1d 4b 7a 57
  ---------------------------------
  FAILED
  Timeout in expect script at "child.expect('SUCCESS')" (tests/pkg_lora-serialization/tests/01-run.py:12)
  

• tests/pkg_micro-ecc: fixed in tests/pkg_micro-ecc: Cleanup and AVR fixes #12797

  Details

  make RIOT_CI_BUILD=1 CC_NOCOLOR=1 --no-print-directory -C ./tests/pkg_micro-ecc test
  /work/riot/RIOT/dist/tools/pyterm/pyterm -p "/dev/ttyACM0" -b "115200" --noprefix --no-repeat-command-on-empty-line
  Connect to serial port /dev/ttyACM0
  Welcome to pyterm!
  Type '/exit' to exit.main(): This is RIOT! (Version: buildtest)
  micro-ecc compiled!
  Testing 16 random private key pairs and signature without using HWRNG
  .
  
  main(): This is RIOT! (Version: buildtest)
  micro-ecc compiled!
  Timeout in expect script at "child.expect_exact('................ done with 0 error(s)')" (tests/pkg_micro-ecc/tests/01-run.py:16)
  

• tests/ps_schedstatistics: too much load in background threads, fixed by tests/ps_schedstatistics: fix test on AVR + improve Python test script #12832

  Details

  make RIOT_CI_BUILD=1 CC_NOCOLOR=1 --no-print-directory -C ./tests/ps_schedstatistics test
  /work/riot/RIOT/dist/tools/pyterm/pyterm -p "/dev/ttyACM0" -b "115200" --noprefix --no-repeat-command-on-empty-line
  Connect to serial port /dev/ttyACM0
  Welcome to pyterm!
  Type '/exit' to exit.
  
  
  main(): This is RIOT! (Version: buildtest)
  Creating thread #0, next=1
  Creating thread #1, next=2
  Creating thread #2, next=3
  Creating thread #3, next=4
  Creating thread #4, next=0
  
  Timeout in expect script at "child.expect('>')" (tests/ps_schedstatistics/tests/01-run.py:43)
  

• tests/thread_msg_bus/

  Details

  Help: Press s to start test, r to print it is ready
  READY
  s
  START
  main(): This is RIOT! (Version: 2020.10-devel-18-gab072f-HEAD)
  THREAD 1 start
  THREAD 2 start
  THREAD 3 start
  THREADS CREATED
  Posted event 22 to 0 threads
  Posted event 23 to 0 threads
  Posted event 24 to 0 threads
  SUCCESS
  Timeout in expect script at "child.expect_exact('T1 recv: Hello Threads! (type=23)')" (tests/thread_msg_bus/tests/01-run.py:15)
  
  

• tests/pkg_libb2/

  Details

  2020-07-22 15:07:31,824 # START
  2020-07-22 15:07:31,830 # main(): This is RIOT! (Version: 2020.10-devel-18-gab072f-HEAD)
  2020-07-22 15:07:31,837 # .
  2020-07-22 15:07:31,845 # blake2_tests.test_blake2s (tests/pkg_libb2/main.c 56) memcmp(b2s, hash, sizeof hash) == 0
  2020-07-22 15:07:31,865 # .
  
  2020-07-22 15:07:31,870 # Help: Press s to start test, r to print it is ready
  
  

Steps to reproduce the issue

Run the following command:

$ BUILD_IN_DOCKER=1 ./dist/tools/compile_and_test_for_board/compile_and_test_for_board.py . atmega256rfr2-xpro --jobs=4 --with-test-only

Expected results

All tests succeed. Unsupported tests (for good reasons) are skipped.

Actual results

Some tests are failing.

Versions

Versions from riotbuild docker image.