.murdock: print hash input when hashes mismatch by maribu · Pull Request #18236 · RIOT-OS/RIOT

maribu · 2022-06-21T10:14:09Z

Contribution description

This prints the hash input when KConfig and non-KConfig builds differ in hash. Hopefully, this helps debugging the issue.

Testing procedure

Murdock will do so.

Issues/PRs references

None

maribu · 2022-06-21T10:14:57Z

I am not sure if we really want to merge this. If this leads to the issue being traced down and fixed, maybe we don't need this after all.

maribu · 2022-06-21T10:16:20Z

On the other hand, if the issue is not triggered on the first try, it would likely be better to just merge, rather than having this PR getting multiple CI spins.

miri64 · 2022-06-21T10:50:14Z

Hopefully, this helps debugging the issue.

Which issue?

gschorcht · 2022-06-21T11:04:37Z

I am not sure if we really want to merge this. If this leads to the issue being traced down and fixed, maybe we don't need this after all.

I would suggest merging it and reverting it later when it is no longer needed.

maribu · 2022-06-21T11:28:00Z

Which issue?

hash mismatches between KConfig and non-KConfig builds that cannot be reproduced outside of Murdock. My wild guess is that the output of sha1sum is read before the hash is fully written occasionally.

I kind of have the bad feeling that because the output of sha1sum is now read twice (once for the debug message and once again for the actual comparison), that if it is indeed a short read, the additional read will cause the second read to no longer being short. Thus, it will fix the issue.

kaspar030 · 2022-06-21T11:41:18Z

.murdock

@@ -469,12 +470,17 @@ compile() {
        make -C${appdir} clean all test-input-hash -j${JOBS:-4}


maybe the "-j4" here causes an issue? the "test-input-hash" target doesn't depend on ${CLEAN}, so it might pick up an old hashed file before clean cleans it and all causes a rebuild?
(to fix, split "clean", or "test-input-hash", in a seperate line?

maribu · 2022-06-21T14:38:47Z

For future reference:

-- running on worker breeze thread 5, build number 82723.
make: Entering directory '/tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/tests/pkg_tflite-micro'
rm -rf /tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/build/pkg-build/flatbuffers
rm -rf /tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/build/pkg-build/gemmlowp
rm -rf /tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/build/pkg-build/ruy
rm -rf /tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/build/pkg-build/tflite-micro
make: Leaving directory '/tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/tests/pkg_tflite-micro'
make: Entering directory '/tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/tests/pkg_tflite-micro'
git-cache: cloning from cache. tag=commit26ed4846f831f730d852507e178061053e522daf-220874
git-cache: cloning from cache. tag=commita9a295fecf3fbd5a4f571f53b01f63202a3e2113-220931
Building application "tests_pkg_tflite-micro" for "nucleo-l433rc" with MCU "stm32".

git-cache: cloning from cache. tag=commite844ffd17118c1e17d94e1ba4354c075a4577b88-222442
git-cache: cloning from cache. tag=commit7ef39c5745a61f43071e699c6a96da41701ae59f-223618
git-cache: cloning from cache. tag=commit1501b574b74fd7877aba30aa9d8b667f41b139c3-226629
/opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: /tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/build/tests_pkg_tflite-micro.elf section `.ARM.exidx' will not fit in region `rom'
/opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: region `rom' overflowed by 1768 bytes
collect2: error: ld returned 1 exit status
make: *** [/tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/tests/pkg_tflite-micro/../../Makefile.include:728: /tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/build/tests_pkg_tflite-micro.elf] Error 1
make: Leaving directory '/tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/tests/pkg_tflite-micro'
An error occurred while compiling using Kconfig
make: Entering directory '/tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/tests/pkg_tflite-micro'
rm -rf /tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/build/pkg-build/flatbuffers
rm -rf /tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/build/pkg-build/gemmlowp
Building application "tests_pkg_tflite-micro" for "nucleo-l433rc" with MCU "stm32".
rm -rf /tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/build/pkg-build/ruy

rm -rf /tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/build/pkg-build/tflite-micro
git-cache: cloning from cache. tag=commit26ed4846f831f730d852507e178061053e522daf-290967
git-cache: cloning from cache. tag=commita9a295fecf3fbd5a4f571f53b01f63202a3e2113-291053
git-cache: cloning from cache. tag=commite844ffd17118c1e17d94e1ba4354c075a4577b88-297031
git-cache: cloning from cache. tag=commit7ef39c5745a61f43071e699c6a96da41701ae59f-302362
git-cache: cloning from cache. tag=commit1501b574b74fd7877aba30aa9d8b667f41b139c3-310760
/opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: /tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/build/tests_pkg_tflite-micro.elf section `.ARM.exidx' will not fit in region `rom'
/opt/gcc-arm-none-eabi-10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: region `rom' overflowed by 1192 bytes
collect2: error: ld returned 1 exit status
make: *** [/tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/tests/pkg_tflite-micro/../../Makefile.include:728: /tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/build/tests_pkg_tflite-micro.elf] Error 1
make: Leaving directory '/tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/tests/pkg_tflite-micro'
cat: /tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/build/test-input-hash.sha1: No such file or directory
cat: /tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/build/test-input-hash.sha1: No such file or directory
Hashes of binaries with and without Kconfig mismatch for tests/pkg_tflite-micro with nucleo-l433rc
Please check that all used modules are modelled in Kconfig and enabled
Input without KConfig:

Input with KConfig:

rm -rf /tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/build/pkg-build/flatbuffers
rm -rf /tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/build/pkg-build/gemmlowp
rm -rf /tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/build/pkg-build/ruy
rm -rf /tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/build/pkg-build/tflite-micro
rm -rf /tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/build/pkg-build/flatbuffers
rm -rf /tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/build/pkg-build/gemmlowp
rm -rf /tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/build/pkg-build/ruy
rm -rf /tmp/dwq.0.09030777032063919/0c3bd3522c42b1ddcd236879f918e65b/build/pkg-build/tflite-micro
{"build/": 116}

maribu · 2022-06-22T08:12:04Z

There seems to be no issue with comparing strings that contain whitespace:

$ cat test.sh 
#!/bin/sh

a="foo 1337\nbar 42\n"
b="foo 1337\nbar 42\n"
c="foo 1337\nbar 43\n"

if [ "$a" == "$b" ]; then
    echo "a == b"
else
    echo "a != b"
fi

if [ "$a" == "$c" ]; then
    echo "a == c"
else
    echo "a != c"
fi
$ sh test.sh 
a == b
a != c

And I have the feeling that this is actually faster than first launching sha1sum on the hash files and then comparing the outputs. Those hash files won't get that large.

kaspar030 · 2022-06-22T10:09:57Z

.murdock

        CCACHE_BASEDIR="$(pwd)" BOARD=${board} TOOLCHAIN=${toolchain} RIOT_CI_BUILD=1 TEST_KCONFIG=1 \
-                        make -C${appdir} all test-input-hash -j${JOBS:-4}
+                        make -C${appdir} all -j${JOBS:-4}
+        CCACHE_BASEDIR="$(pwd)" BOARD=${board} TOOLCHAIN=${toolchain} RIOT_CI_BUILD=1 TEST_KCONFIG=1 \


Is the split needed here? the clean is already in an extra line.

kaspar030 · 2022-06-22T10:10:57Z

.murdock

    CCACHE_BASEDIR="$(pwd)" BOARD=$board TOOLCHAIN=$toolchain RIOT_CI_BUILD=1 \
-        make -C${appdir} clean all test-input-hash -j${JOBS:-4}
+        make -C${appdir} clean
+        make -C${appdir} all -j${JOBS:-4}


as above, It should suffice to split the "clean" run.

.murdock

gschorcht · 2022-06-28T05:40:17Z

The problem with the hashes seems to become more serious again. 8 of the last 10 compilations failed due to these hash mismatches.

maribu · 2022-06-28T06:38:46Z

Indeed. I guess this will be the prio 1 topic for today's sprint

kaspar030 · 2022-06-28T19:18:47Z

This needs a rebase & squash. Otherwise it looks good to me!

maribu · 2022-07-01T10:34:15Z

${BINDIR} does not contain the application and board name. Could it be, that this is not exported from the build system to the .murdock script?

E.g.

cat: /tmp/dwq.0.09030777032063919/3850afa3686b98cd686e252b2dcbe064/build/test-input-hash.sha1: No such file or directory

I would expect something like /tmp/dwq.0.09030777032063919/3850afa3686b98cd686e252b2dcbe064/build/examples/lua_REPL/bin/nucleo-g071rb/test-input-hash.sha1 as path.

kaspar030 · 2022-07-01T19:34:19Z

${BINDIR} does not contain the application and board name.
Could it be, that this is not exported from the build system to the .murdock script?

That's expected, .murdock must hard sets it to ./build.

Looking at the build failures, it seems like they only happen on builds that don't actually link (RIOT_NO_LINK=1)?

kaspar030 · 2022-07-01T19:46:18Z

Looking at the build failures, it seems like they only happen on builds that don't actually link (RIOT_NO_LINK=1)?

yeah, no test-input-hash without .elf:

RIOT/makefiles/tests/tests.inc.mk

Line 74 in 932c58d

# this target only makes sense if an ELFFILE is actually created, thus guard by

benpicco · 2022-07-10T21:22:15Z

Is there anything speaking against merging this?

maribu · 2022-07-11T15:18:08Z

I think only squashing was still missing to get this merged. Just did so.

It seems like RIOT-OS#18236 introduced some errors in the kconfig hash check. Running locally I get a `.murdock: line 482: [: too many arguments` message. On the CI it looks like `./.murdock: 480: [: 4994c097e49decc77a99c2254e46c710238dc4f2: unexpected operator`. I noticed that the path names are now included in the comparison. It seems surrounding them in quotes helps but this also compares whitespaces and newlines...

maribu requested a review from kaspar030 as a code owner June 21, 2022 10:14

maribu added the CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR label Jun 21, 2022

github-actions bot added the Area: CI Area: Continuous Integration of RIOT components label Jun 21, 2022

maribu mentioned this pull request Jun 21, 2022

makefiles/tests: use BINFILE for hash comparision instead of ELFFILE #18216

Merged

kaspar030 reviewed Jun 21, 2022

View reviewed changes

kaspar030 reviewed Jun 22, 2022

View reviewed changes

maribu added CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR and removed CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR labels Jun 23, 2022

kaspar030 mentioned this pull request Jun 23, 2022

murdock: don't do both "clean" and "test-input-hash" targets #18254

Merged

kaspar030 reviewed Jun 23, 2022

View reviewed changes

.murdock Outdated Show resolved Hide resolved

chrysn mentioned this pull request Jun 30, 2022

Add some Rust library building infrastructure #16833

Merged

2 tasks

maribu force-pushed the murdock branch from db58bc6 to bd68023 Compare June 30, 2022 17:29

github-actions bot added Area: build system Area: Build system Area: tests Area: tests and testing framework labels Jul 2, 2022

maribu removed the Area: tests Area: tests and testing framework label Jul 2, 2022

github-actions bot added the Area: tests Area: tests and testing framework label Jul 2, 2022

.murdock: print hash input when hashes mismatch

de74c6a

maribu force-pushed the murdock branch from 6721481 to de74c6a Compare July 11, 2022 15:17

benpicco requested a review from leandrolanzieri July 11, 2022 20:39

benpicco approved these changes Jul 11, 2022

View reviewed changes

gschorcht merged commit a88cb32 into RIOT-OS:master Jul 12, 2022

maribu deleted the murdock branch July 22, 2022 16:16

MrKevinWeiss mentioned this pull request Jul 25, 2022

.murdock: Fix kconfig hash check #18364

Merged

chrysn added this to the Release 2022.07 milestone Aug 25, 2022

		@@ -469,12 +470,17 @@ compile() {
		make -C${appdir} clean all test-input-hash -j${JOBS:-4}

Conversation

maribu commented Jun 21, 2022

Contribution description

Testing procedure

Issues/PRs references

Uh oh!

maribu commented Jun 21, 2022

Uh oh!

maribu commented Jun 21, 2022

Uh oh!

miri64 commented Jun 21, 2022

Uh oh!

gschorcht commented Jun 21, 2022

Uh oh!

maribu commented Jun 21, 2022

Uh oh!

kaspar030 Jun 21, 2022

Choose a reason for hiding this comment

Uh oh!

maribu commented Jun 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maribu commented Jun 22, 2022

Uh oh!

kaspar030 Jun 22, 2022

Choose a reason for hiding this comment

Uh oh!

kaspar030 Jun 22, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gschorcht commented Jun 28, 2022

Uh oh!

maribu commented Jun 28, 2022

Uh oh!

kaspar030 commented Jun 28, 2022

Uh oh!

maribu commented Jul 1, 2022

Uh oh!

kaspar030 commented Jul 1, 2022

Uh oh!

kaspar030 commented Jul 1, 2022

Uh oh!

benpicco commented Jul 10, 2022

Uh oh!

maribu commented Jul 11, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

maribu commented Jun 21, 2022 •

edited

Loading