run-make-check: remove write permission on libceph-common.so during ctest by cbodley · Pull Request #62832 · ceph/ceph

cbodley · 2025-04-15T16:59:10Z

https://tracker.ceph.com/issues/70700 tracks a ton of recent failures during ctest of the form:

loading shared libraries: /path/to/lib/libceph-common.so.2: file too short

it seems that something is corrupting this file, so this removes write permissions for the duration. ideally this would cause a different test failure if a test actually is trying to modify it

Show available Jenkins commands

jenkins test classic perf Jenkins Job | Jenkins Job Definition
jenkins test crimson perf Jenkins Job | Jenkins Job Definition
jenkins test signed Jenkins Job | Jenkins Job Definition
jenkins test make check Jenkins Job | Jenkins Job Definition
jenkins test make check arm64 Jenkins Job | Jenkins Job Definition
jenkins test submodules Jenkins Job | Jenkins Job Definition
jenkins test dashboard Jenkins Job | Jenkins Job Definition
jenkins test dashboard cephadm Jenkins Job | Jenkins Job Definition
jenkins test api Jenkins Job | Jenkins Job Definition
jenkins test docs ReadTheDocs | Github Workflow Definition
jenkins test ceph-volume all Jenkins Jobs | Jenkins Jobs Definition
jenkins test windows Jenkins Job | Jenkins Job Definition
jenkins test rook e2e Jenkins Job | Jenkins Job Definition

cbodley · 2025-04-15T17:58:57Z

https://jenkins.ceph.com/job/ceph-pull-requests/156235/ and https://jenkins.ceph.com/job/ceph-pull-requests-arm64/72334/ both show the same "file too short" failures

cbodley · 2025-04-15T18:34:09Z

did not reproduce in
https://jenkins.ceph.com/job/ceph-pull-requests/156239/
https://jenkins.ceph.com/job/ceph-pull-requests-arm64/72337/

cbodley · 2025-04-15T18:34:13Z

jenkins test make check

cbodley · 2025-04-15T18:34:17Z

jenkins test make check arm64

cbodley · 2025-04-15T18:48:42Z

https://jenkins.ceph.com/job/ceph-pull-requests/156240/ green

cbodley · 2025-04-15T18:48:46Z

jenkins test make check

cbodley · 2025-04-15T18:59:05Z

https://jenkins.ceph.com/job/ceph-pull-requests/156241/ green

cbodley · 2025-04-15T18:59:09Z

jenkins test make check

cbodley · 2025-04-15T19:12:37Z

https://jenkins.ceph.com/job/ceph-pull-requests/156242/ green
https://jenkins.ceph.com/job/ceph-pull-requests-arm64/72338/ green

cbodley · 2025-04-15T19:12:40Z

jenkins test make check

cbodley · 2025-04-15T19:12:44Z

jenkins test make check arm64

cbodley · 2025-04-15T19:29:27Z

https://jenkins.ceph.com/job/ceph-pull-requests/156245/ green
https://jenkins.ceph.com/job/ceph-pull-requests-arm64/72340/ green

cbodley · 2025-04-15T19:43:45Z

https://jenkins.ceph.com/job/ceph-pull-requests/156246/ green
https://jenkins.ceph.com/job/ceph-pull-requests-arm64/72341/ green

cbodley · 2025-04-15T19:43:50Z

jenkins test make check

cbodley · 2025-04-15T19:43:59Z

jenkins test make check arm64

cbodley · 2025-04-15T19:56:48Z

https://jenkins.ceph.com/job/ceph-pull-requests/156248/ green

cbodley · 2025-04-15T19:56:52Z

jenkins test make check

cbodley · 2025-04-15T20:07:25Z

https://jenkins.ceph.com/job/ceph-pull-requests/156251/ green

cbodley · 2025-04-15T20:07:30Z

jenkins test make check

cbodley · 2025-04-15T20:14:33Z

https://jenkins.ceph.com/job/ceph-pull-requests-arm64/72343/ green

cbodley · 2025-04-15T20:14:38Z

jenkins test make check arm64

cbodley · 2025-04-15T20:17:06Z

https://jenkins.ceph.com/job/ceph-pull-requests/156252/ green

cbodley · 2025-04-15T20:17:14Z

jenkins test make check

cbodley · 2025-04-15T21:17:26Z

https://jenkins.ceph.com/job/ceph-pull-requests/156254/ green
https://jenkins.ceph.com/job/ceph-pull-requests-arm64/72346/ green

…test https://tracker.ceph.com/issues/70700 tracks a ton of recent failures during ctest of the form: > loading shared libraries: /path/to/lib/libceph-common.so.2: file too short it seems that something is corrupting this file, so this removes write permissions for the duration. ideally this would cause a different test failure if a test actually is trying to modify it Signed-off-by: Casey Bodley <[email protected]>

cbodley · 2025-04-16T12:28:13Z

https://jenkins.ceph.com/job/ceph-pull-requests/156255/ reproduced

before ctest:

make check: successful build on 2214d2fae929551011975eb3da58e9199727f346
lrwxrwxrwx 1 jenkins-build users        19 Apr 15 21:27 lib/libceph-common.so -> libceph-common.so.2
-rwxr-xr-x 1 jenkins-build users 190751192 Apr 15 21:27 lib/libceph-common.so.2

after ctest:

Errors while running CTest
lrwxrwxrwx 1 jenkins-build users        19 Apr 15 21:27 lib/libceph-common.so -> libceph-common.so.2
-rwxr-xr-x 1 jenkins-build users 190751192 Apr 15 21:33 lib/libceph-common.so.2

ctest was green until this:

106/317 Test #156: unittest_decode_start_v_checker ..................***Failed    0.38 sec
[1/6] Building CXX object src/test/common/CMakeFiles/unittest_decode_start_v_checker.dir/test_decode_start_v_checker.cpp.o
[2/6] Building CXX object src/extblkdev/CMakeFiles/extblkdev.dir/ExtBlkDevPlugin.cc.o
[3/6] Linking CXX static library lib/libextblkdev.a
[4/6] Linking CXX shared library lib/libceph-common.so.2
FAILED: lib/libceph-common.so.2 
: && /usr/bin/clang++-19 -fPIC -g -Werror  -Wl,-Bsymbolic -Wl,-Bsymbolic-functions -shared -Wl,-soname,libceph-common.so.2 -o lib/libceph-common.so.2 [...]
clang++-19: error: no such file or directory: 'lib/libextblkdev.a'
ninja: build stopped: subcommand failed.

cbodley · 2025-04-16T12:36:46Z

https://jenkins.ceph.com/job/ceph-pull-requests-arm64/72347/ reproduced

before ctest:

make check: successful build on 2214d2fae929551011975eb3da58e9199727f346
lrwxrwxrwx 1 jenkins-build users        19 Apr 15 21:33 lib/libceph-common.so -> libceph-common.so.2
-rwxr-xr-x 1 jenkins-build users 194071304 Apr 15 21:33 lib/libceph-common.so.2

after ctest:

Errors while running CTest
lrwxrwxrwx 1 jenkins-build users        19 Apr 15 21:49 lib/libceph-common.so -> libceph-common.so.2
-rwxr-xr-x 1 jenkins-build users 194071304 Apr 15 21:49 lib/libceph-common.so.2

ctest was green until this:

115/314 Test #103: unittest_throttle ................................   Passed   15.38 sec
        Start 157: unittest_decode_start_v_checker_expect_failure
116/314 Test  #35: run-cli-tests ....................................***Failed   25.14 sec

unlike the above, that test case passed:

264/314 Test #157: unittest_decode_start_v_checker_expect_failure ...   Passed   35.96 sec

cbodley · 2025-04-16T13:13:17Z

cc @rzarzynski #60159 did merge ~3 weeks ago which is when we started seeing libceph-common.so.2: file too short errors in https://tracker.ceph.com/issues/70700

dmick · 2025-04-18T16:07:18Z

I marked the two most recent failing builds as "keep forever" so that Jenkins won't reap them

github-actions bot added the build/ops label Apr 15, 2025

cbodley force-pushed the wip-70700-chmod branch from 31a1a0f to c6dcd51 Compare April 15, 2025 17:58

cbodley force-pushed the wip-70700-chmod branch from 8a9da2a to 66e9652 Compare April 15, 2025 21:23

cbodley closed this Apr 25, 2025

Comments

Conversation

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 15, 2025

Uh oh!

cbodley commented Apr 16, 2025

Uh oh!

cbodley commented Apr 16, 2025

Uh oh!

cbodley commented Apr 16, 2025

Uh oh!

dmick commented Apr 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants