Skip failing tests on ARM to get red/green feedback from Jenkins#19412
Skip failing tests on ARM to get red/green feedback from Jenkins#19412jessfraz merged 1 commit intomoby:masterfrom
Conversation
Signed-off-by: Stefan Scherer <[email protected]>
icecrime
added
status/2-code-review
and removed
platform/windows
status/0-triage
labels
|
Triggered an ARM build for this PR, thanks. |
|
@icecrime Thanks! |
|
For comparison later on, here is our Jenkins console log https://gist.github.com/StefanScherer/df97b69f584736696694 |
|
|
|
Yes, the first 💚 arm build. So cool! |
|
Woot! 😍 |
|
@StefanScherer Yeah! |
|
LGTM |
Skip failing tests on ARM to get red/green feedback from Jenkins
|
Awesome! |
There was a problem hiding this comment.
@StefanScherer out of pure curiosity, is the problem with Linux or something else in Docker? Why wouldn't this work on ARM as-is?
There was a problem hiding this comment.
@tiborvass Good catch. The problem on ARM is in TestRunSysWritableInPrivilegedContainers. This one should work indeed. Transferring all these failing tests from the logs into the code with a long feedback loop where testing each change individually would take "days" this can happen. So fixing this TestRunSysNotWritableInNonPrivilegedContainers for ARM is a really low hanging fruit.
There was a problem hiding this comment.
@StefanScherer then what's the problem with TestRunSysWritableInPrivilegedContainers on ARM?
There was a problem hiding this comment.
02:10:12.794
02:10:12.794 ----------------------------------------------------------------------
02:10:12.795 FAIL: docker_cli_run_test.go:942: DockerSuite.TestRunSysWritableInPrivilegedContainers
02:10:12.796
02:10:12.796 docker_cli_run_test.go:946:
02:10:12.796 c.Fatalf("sys should be writable in privileged container")
02:10:12.797 ... Error: sys should be writable in privileged container
02:10:12.797
02:10:14.436
02:10:14.436 ----------------------------------------------------------------------
It's reproducible on ARM (good to dig deeper into that), but no idea at the moment.
There was a problem hiding this comment.
Running the test manually on a Scaleway machine
root@scw-04:~# docker run armhf/busybox touch /sys/kernel/profiling
touch: /sys/kernel/profiling: Read-only file system
root@scw-04:~# docker run --privileged armhf/busybox touch /sys/kernel/profiling
touch: /sys/kernel/profiling: Permission denied
What are the best practices to investigate such issues?
7 participants
With this PR the Jenkins arm build should go to green as our internal Jenkins build already had shown this to our team. We would like to skip some of the integration tests for now for the ARM build and work against an "always green" CI system to bring back the tests step by step as far as they make sense on ARM. (Similar to Windows CI #19195)
The advantage is that the arm build also can show the status for new incoming PR's and whether there are problems introduced with it.
Our latest Jenkins build now shows
OK: 1117 passed, 67 skippedI had collected all the failing tests from the https://jenkins.dockerproject.org/job/Docker-PRs-arm/8/console output and from our Jenkins builds. As our Jenkins build seems to pass some more tests than the arm build we now can send PR's to get a better feedback from official CI.
Everybody can count the skipped tests with a
grep -r NotArmin the source tree from time to time.