tests/sys/shell: make test script more robust by maribu · Pull Request #21036 · RIOT-OS/RIOT

maribu · 2024-11-25T10:53:09Z

Contribution description

This changes the behavior of the test script for verifying the help command: It no longer assumes a specific order for the list of commands.

Making the test robust is a bit tricky, as the module shell_cmds_default that is used here may add commands specific to a set of board. We use help_json to get the list of commands actually provided, so that we know how many rows the command table printed by help need to be parsed.

A minimum set of commands that must be present for all boards is hard-coded in the test script and the actually provided commands are tested against this. Otherwise e.g. an empty list of commands presented by help and help_json would also pass.

Testing procedure

make BOARD=samr21-xpro -C tests/sys/shell flash test

make: Entering directory '/home/[email protected]/Repos/software/RIOT/master/tests/sys/shell'
Building application "tests_shell" for "samr21-xpro" with CPU "samd21".

"make" -C /home/[email protected]/Repos/software/RIOT/master/pkg/cmsis/ 
"make" -C /home/[email protected]/Repos/software/RIOT/master/boards/common/init
"make" -C /home/[email protected]/Repos/software/RIOT/master/boards/samr21-xpro
"make" -C /home/[email protected]/Repos/software/RIOT/master/core
"make" -C /home/[email protected]/Repos/software/RIOT/master/core/lib
"make" -C /home/[email protected]/Repos/software/RIOT/master/cpu/samd21
"make" -C /home/[email protected]/Repos/software/RIOT/master/cpu/cortexm_common
"make" -C /home/[email protected]/Repos/software/RIOT/master/cpu/cortexm_common/periph
"make" -C /home/[email protected]/Repos/software/RIOT/master/cpu/sam0_common
"make" -C /home/[email protected]/Repos/software/RIOT/master/cpu/sam0_common/periph
"make" -C /home/[email protected]/Repos/software/RIOT/master/cpu/samd21/periph
"make" -C /home/[email protected]/Repos/software/RIOT/master/cpu/samd21/vectors
"make" -C /home/[email protected]/Repos/software/RIOT/master/drivers
"make" -C /home/[email protected]/Repos/software/RIOT/master/drivers/periph_common
"make" -C /home/[email protected]/Repos/software/RIOT/master/sys
"make" -C /home/[email protected]/Repos/software/RIOT/master/sys/app_metadata
"make" -C /home/[email protected]/Repos/software/RIOT/master/sys/auto_init
"make" -C /home/[email protected]/Repos/software/RIOT/master/sys/div
"make" -C /home/[email protected]/Repos/software/RIOT/master/sys/frac
"make" -C /home/[email protected]/Repos/software/RIOT/master/sys/isrpipe
"make" -C /home/[email protected]/Repos/software/RIOT/master/sys/libc
"make" -C /home/[email protected]/Repos/software/RIOT/master/sys/malloc_thread_safe
"make" -C /home/[email protected]/Repos/software/RIOT/master/sys/newlib_syscalls_default
"make" -C /home/[email protected]/Repos/software/RIOT/master/sys/pm_layered
"make" -C /home/[email protected]/Repos/software/RIOT/master/sys/preprocessor
"make" -C /home/[email protected]/Repos/software/RIOT/master/sys/ps
"make" -C /home/[email protected]/Repos/software/RIOT/master/sys/shell
"make" -C /home/[email protected]/Repos/software/RIOT/master/sys/shell/cmds
"make" -C /home/[email protected]/Repos/software/RIOT/master/sys/stdio
"make" -C /home/[email protected]/Repos/software/RIOT/master/sys/stdio_uart
"make" -C /home/[email protected]/Repos/software/RIOT/master/sys/test_utils/interactive_sync
"make" -C /home/[email protected]/Repos/software/RIOT/master/sys/test_utils/print_stack_usage
"make" -C /home/[email protected]/Repos/software/RIOT/master/sys/tsrb
"make" -C /home/[email protected]/Repos/software/RIOT/master/sys/ztimer
   text	  data	   bss	   dec	   hex	filename
  18928	   168	  2832	 21928	  55a8	/home/[email protected]/Repos/software/RIOT/master/tests/sys/shell/bin/samr21-xpro/tests_shell.elf
/home/[email protected]/Repos/software/RIOT/master/dist/tools/edbg/edbg.sh flash /home/[email protected]/Repos/software/RIOT/master/tests/sys/shell/bin/samr21-xpro/tests_shell.bin
### Flashing Target ###
Debugger: ATMEL EDBG CMSIS-DAP ATML2127031800009225 03.25.01B6 (S)
Clock frequency: 16.0 MHz
Target: SAM R21G18 (Rev C)
Verification....
at address 0x168 expected 0xc4, read 0xc0
Error: verification failed
Debugger: ATMEL EDBG CMSIS-DAP ATML2127031800009225 03.25.01B6 (S)
Clock frequency: 16.0 MHz
Target: SAM R21G18 (Rev C)
Programming...... done.
Verification...... done.
Done flashing
/home/[email protected]/Repos/software/RIOT/master/tests/sys/shell/tests/01-run.py:88: SyntaxWarning: invalid escape sequence '\ '
  ("""echo "t\\e st" "\\"" '\\'' a\ b""", '"echo""te st"""""\'""a b"'),  # noqa: W605


socat - open:/dev/ttyACM0,b115200,echo=0,raw,cs8,parenb=0,cstopb=0 

> bufsize

> bufsize
60
____________________________________________________verylong
> ____________________________________________________verylong
shell: maximum line length exceeded
> garbage1234
garbage1234

> 
> shell exited

> 
> periodic 5
> 
> periodic 5
> test
test
test
test
test


> start_test
start_test
[TEST_START]
> 


> 
> echo a string
echo a string
"echo""a""string"
> echo   multiple   spaces   between   argv
echo   multiple   spaces   between   argv
"echo""multiple""spaces""between""argv"
> echo 	tabs		processed 		like			spaces
echo 	tabs		processed 		like			spaces
"echo""tabs""processed""like""spaces"
> unknown_command
unknown_command
shell: command not found: unknown_command
>      echo leading spaces
     echo leading spaces
"echo""leading""spaces"
> 					echo leading tabs
					echo leading tabs
"echo""leading""tabs"
> echo trailing spaces     
echo trailing spaces     
"echo""trailing""spaces"
> echo trailing tabs					
echo trailing tabs					
"echo""trailing""tabs"
> hello-world
hello-world
shell: command not found: hello-world                                                                                                                                   echo
echo
"echo"
> echo \'
echo \'
"echo""'"
> echo \"
echo \"
"echo""""
> echo escaped\ space
echo escaped\ space
"echo""escaped space"
> echo escape within '\s\i\n\g\l\e\q\u\o\t\e'
echo escape within '\s\i\n\g\l\e\q\u\o\t\e'
"echo""escape""within""singlequote"
> echo escape within "\d\o\u\b\l\e\q\u\o\t\e"
echo escape within "\d\o\u\b\l\e\q\u\o\t\e"
"echo""escape""within""doublequote"
> echo "t\e st" "\"" '\'' a\ b
echo "t\e st" "\"" '\'' a\ b
"echo""te st"""""'""a b"
> echo "hello"world
echo "hello"world
"echo""helloworld"
> echo hel"lowo"rld
echo hel"lowo"rld
"echo""helloworld"
> echo hello"world"
echo hello"world"
"echo""helloworld"
> echo quoted space " "
echo quoted space " "
"echo""quoted""space"" "
> echo abc"def'ghijk"lmn
echo abc"def'ghijk"lmn
"echo""abcdef'ghijklmn"
> echo abc'def"ghijk'lmn
echo abc'def"ghijk'lmn
"echo""abcdef"ghijklmn"
> echo "'" '"'
echo "'" '"'
"echo""'""""
> echo a\
echo a\
shell: incorrect quoting
> echo "
echo "
shell: incorrect quoting
> echo '
echo '
shell: incorrect quoting
> echo abcdef"ghijklmn
echo abcdef"ghijklmn
shell: incorrect quoting
> echo abcdef'ghijklmn
echo abcdef'ghijklmn
shell: incorrect quoting
> ps
ps
	pid | state    Q | pri 
	 1 | running  Q |   7
> xfa_test1
xfa_test1
[XFA TEST 1 OK]
> xfa_test2
xfa_test2
[XFA TEST 2 OK]
> reboot
reboot
main(): This is RIOT! (Version: 2025.01-devel-189-g3e08d6-tests/sys/shell/fix-test)
test_shell.
> end_test
end_test
[TEST_END]
> help_json
help_json
{"cmds": [{"cmd": "bufsize", "desc": "Get the shell's buffer size"}, {"cmd": "start_test", "desc": "starts a test"}, {"cmd": "end_test", "desc": "ends a test"}, {"cmd": "echo", "desc": "prints the input command"}, {"cmd": "empty", "desc": "print nothing on command"}, {"cmd": "periodic", "desc": "periodically print command"}, {"cmd": "xfa_test1", "desc": "xfa test command 1"}, {"cmd": "xfa_test2", "desc": "xfa test command 2"}, {"cmd": "app_metadata", "desc": "Returns application metadata"}, {"cmd": "pm", "desc": "interact with layered PM subsystem"}, {"cmd": "ps", "desc": "Prints information about running threads."}, {"cmd": "version", "desc": "Prints current RIOT_VERSION"}, {"cmd": "reboot", "desc": "Reboot the node"}]}
> help
help
Command              Description
---------------------------------------
bufsize              Get the shell's buffer size
start_test           starts a test
end_test             ends a test
echo                 prints the input command
empty                print nothing on command
periodic             periodically print command
xfa_test1            xfa test command 1
xfa_test2            xfa test command 2
app_metadata         Returns application metadata
pm                   interact with layered PM subsystem
ps                   Prints information about running threads.
version              Prints current RIOT_VERSION
reboot               Reboot the node
> 

> 
make: Leaving directory '/home/[email protected]/Repos/software/RIOT/master/tests/sys/shell'

Issues/PRs references

Fixes regression from #20958 that caused failing nightlies

This fixes: /home/[email protected]/Repos/software/RIOT/master/tests/sys/shell/tests/01-run.py:86: SyntaxWarning: invalid escape sequence '\e' Python still assumes `\\<char>` if `\<char>` is not valid, but started to warn. Chances are good that this will be elevated to an error in future python releases.

Teufelchen1

Great! Nit: check_help() is a bit hard to read, but still fine for this test case.

maribu · 2024-11-25T12:57:27Z

check_help() is a bit hard to read, but still fine for this test case.

I sprinkled some comments over the code.

tests/sys/shell/tests/01-run.py

riot-ci · 2024-11-25T14:34:23Z

Murdock results

✔️ PASSED

b858614 tests/sys/shell: make test more robust

Success	Failures	Total	Runtime
81	0	85	01m:53s

Artifacts

Documentation preview

This changes the behavior of the test script for verifying the `help` command: It no longer assumes a specific order for the list of commands. Making the test robust is a bit tricky, as the module `shell_cmds_default` that is used here may add commands specific to a set of board. We use `help_json` to get the list of commands actually provided, so that we know how many rows the command table printed by `help` need to be parsed. A minimum set of commands that *must* be present for all boards is hard-coded in the test script and the actually provided commands are tested against this. Otherwise e.g. an empty list of commands presented by `help` and `help_json` would also pass. Co-authored-by: benpicco <[email protected]>

maribu · 2024-11-27T15:11:47Z

🎉 and thx :)

maribu requested review from MichelRottleuthner, aabadie and leandrolanzieri as code owners November 25, 2024 10:53

maribu added Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors) Area: tests Area: tests and testing framework CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Nov 25, 2024

github-actions bot removed Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors) CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Nov 25, 2024

maribu added Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors) CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Nov 25, 2024

maribu changed the title ~~Tests/sys/shell/fix test~~ tests/sys/shell: make test script more robust Nov 25, 2024

Teufelchen1 approved these changes Nov 25, 2024

View reviewed changes

maribu force-pushed the tests/sys/shell/fix-test branch from 3e08d6c to 4ed8197 Compare November 25, 2024 12:56

maribu force-pushed the tests/sys/shell/fix-test branch from 4ed8197 to ad4206d Compare November 25, 2024 12:58

benpicco reviewed Nov 25, 2024

View reviewed changes

tests/sys/shell/tests/01-run.py Outdated Show resolved Hide resolved

maribu force-pushed the tests/sys/shell/fix-test branch from a6c50c9 to b858614 Compare November 25, 2024 16:23

mguetschow mentioned this pull request Nov 26, 2024

boards/seeedstudio-xiao-nrf52840: initial board support #20980

Merged

Teufelchen1 added this pull request to the merge queue Nov 27, 2024

Merged via the queue into RIOT-OS:master with commit 5fa8f65 Nov 27, 2024

maribu deleted the tests/sys/shell/fix-test branch November 27, 2024 15:11

MrKevinWeiss added this to the Release 2025.01 milestone Jan 20, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests/sys/shell: make test script more robust#21036

tests/sys/shell: make test script more robust#21036
Teufelchen1 merged 2 commits intoRIOT-OS:masterfrom
maribu:tests/sys/shell/fix-test

maribu commented Nov 25, 2024

Uh oh!

Teufelchen1 left a comment

Uh oh!

maribu commented Nov 25, 2024

Uh oh!

Uh oh!

riot-ci commented Nov 25, 2024 •

edited

Loading

Uh oh!

maribu commented Nov 27, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

maribu commented Nov 25, 2024

Contribution description

Testing procedure

Issues/PRs references

Uh oh!

Teufelchen1 left a comment

Choose a reason for hiding this comment

Uh oh!

maribu commented Nov 25, 2024

Uh oh!

Uh oh!

riot-ci commented Nov 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Murdock results

Artifacts

Uh oh!

maribu commented Nov 27, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

riot-ci commented Nov 25, 2024 •

edited

Loading