Skip to content

ci: Work around podman stop intermittent failure #28547

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
fanquake merged 4 commits into from
Oct 13, 2023
Merged

ci: Work around podman stop intermittent failure #28547

fanquake merged 4 commits into from
Oct 13, 2023

Conversation

@maflcko
Copy link
Member

@maflcko maflcko commented Sep 28, 2023 •
edited
Loading

Sometimes, it seems that podman stop does not work. Presumably, it falls back to podman kill, which is async.

Try to work around this intermittent issue by using the rm --force over stop.

Example failing log https://cirrus-ci.com/task/4549784611061760?logs=ci#L238:

Restart docker before run to stop and clear all containers started with --rm
++ podman container stop --all
e4eca0766f87864d89fc230aa884a238c214cfbcd44cf76a4dbdb2a30c982009
++ echo 'Prune all dangling images'
Prune all dangling images
++ docker image prune --force
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
+++ docker run --cap-add LINUX_IMMUTABLE --rm --interactive --detach --tty --mount type=bind,src=/tmp/cirrus-build-1970593815,dst=/tmp/cirrus-build-1970593815,readonly --mount type=volume,src=ci_macos_cross_ccache,dst=/tmp/ccache_dir --mount type=volume,src=ci_macos_cross_depends,dst=/ci_container_base/depends --mount type=volume,src=ci_macos_cross_previous_releases,dst=/ci_container_base/prev_releases --env-file /tmp/env --name ci_macos_cross ci_macos_cross
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
time="2023-09-27T20:55:39Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
Error: creating container storage: the container name "ci_macos_cross" is already in use by e4eca0766f87864d89fc230aa884a238c214cfbcd44cf76a4dbdb2a30c982009. You have to remove that container to be able to reuse that name: that name is already in use

@DrahtBot
Copy link
Contributor

DrahtBot commented Sep 28, 2023 •
edited
Loading

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type Reviewers
ACK hebasto

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #27793 (ci: label docker images and prune dangling images selectively by stickies-v)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@DrahtBot DrahtBot changed the title ci: Work around podman stop bug ci: Work around podman stop bug Sep 28, 2023
@DrahtBot DrahtBot added the Tests label Sep 28, 2023
@maflcko maflcko changed the title ci: Work around podman stop bug ci: Work around podman stop intermittent failure Sep 28, 2023
@maflcko
Copy link
Member Author

maflcko commented Sep 28, 2023

Looks like this is working, except that the container id contains the name when the issue happens https://cirrus-ci.com/task/6582091933024256?logs=ci#L250:

Restart docker before run to stop and clear all containers started with --rm
++ podman container stop --all
e4eca0766f87864d89fc230aa884a238c214cfbcd44cf76a4dbdb2a30c982009
++ export MAYBE_REPLACE=--replace
++ MAYBE_REPLACE=--replace
++ echo 'Prune all dangling images'
Prune all dangling images
++ docker image prune --force
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
+++ docker run --cap-add LINUX_IMMUTABLE --rm --interactive --detach --tty --replace --mount type=bind,src=/tmp/cirrus-build-2529187836,dst=/tmp/cirrus-build-2529187836,readonly --mount type=volume,src=ci_macos_cross_ccache,dst=/tmp/ccache_dir --mount type=volume,src=ci_macos_cross_depends,dst=/ci_container_base/depends --mount type=volume,src=ci_macos_cross_previous_releases,dst=/ci_container_base/prev_releases --env-file /tmp/env --name ci_macos_cross ci_macos_cross
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
time="2023-09-28T10:48:22Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
++ CI_CONTAINER_ID='ci_macos_cross
be2dfa413d1bb2912b6258f1fa3232497efbb004ddcdee79673a242f53db5265'
++ export CI_CONTAINER_ID
++ export 'CI_EXEC_CMD_PREFIX=docker exec ci_macos_cross
be2dfa413d1bb2912b6258f1fa3232497efbb004ddcdee79673a242f53db5265'
++ CI_EXEC_CMD_PREFIX='docker exec ci_macos_cross
be2dfa413d1bb2912b6258f1fa3232497efbb004ddcdee79673a242f53db5265'
++ export -f CI_EXEC
++ CI_EXEC rsync --archive --stats --human-readable /tmp/cirrus-build-2529187836/ /ci_container_base
++ docker exec ci_macos_cross be2dfa413d1bb2912b6258f1fa3232497efbb004ddcdee79673a242f53db5265 bash -c 'export PATH=/ci_container_base/ci/scratch/bins/:/ci_container_base/ci/retry:$PATH && cd "/ci_container_base" && rsync --archive --stats --human-readable /tmp/cirrus-build-2529187836/ /ci_container_base'
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
Error: crun: executable file `be2dfa413d1bb2912b6258f1fa3232497efbb004ddcdee79673a242f53db5265` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found
++ echo 'Nothing to copy from /tmp/cirrus-build-2529187836/'
Nothing to copy from /tmp/cirrus-build-2529187836/
++ CI_EXEC /ci_container_base/ci/test/01_base_install.sh
++ docker exec ci_macos_cross be2dfa413d1bb2912b6258f1fa3232497efbb004ddcdee79673a242f53db5265 bash -c 'export PATH=/ci_container_base/ci/scratch/bins/:/ci_container_base/ci/retry:$PATH && cd "/ci_container_base" && /ci_container_base/ci/test/01_base_install.sh'
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
Error: crun: executable file `be2dfa413d1bb2912b6258f1fa3232497efbb004ddcdee79673a242f53db5265` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found
Exit status: 127

@DrahtBot DrahtBot mentioned this pull request Sep 28, 2023
Merged
@maflcko
Copy link
Member Author

maflcko commented Oct 4, 2023

ok, pushed the complete set of changes (please refer to the commit messages for details)

@DrahtBot DrahtBot removed the CI failed label Oct 5, 2023
@maflcko maflcko added this to the 26.0 milestone Oct 5, 2023
MarcoFalke added 4 commits October 9, 2023 16:16
ci: Rename 04_install to 02_run_container
fac9abb 
This reflects what the script does (docker run ...).
ci: Add set -ex to 02_run_container.sh
fa09a03 
The same is done by the 06 script.
ci: Work around podman stop bug
fa695b4 
Force remove any containers, pontentially leaving dangling processes,
which should be fine.
ci: move-only CI_CONTAINER_ID to 02_run_container.sh
fa2c894 
This limits the scope of the CI_CONTAINER_ID symbol.

Can be reviewed with --color-moved=dimmed-zebra
@maflcko maflcko requested a review from hebasto October 11, 2023 14:19
hebasto
hebasto approved these changes Oct 12, 2023
View reviewed changes
Copy link
Member

@hebasto hebasto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK fa2c894, I have reviewed the code and tested it locally.

@fanquake fanquake merged commit 504f1f3 into bitcoin:master Oct 13, 2023
@maflcko maflcko deleted the 2309-ci-stop- branch October 13, 2023 08:30
@hebasto hebasto mentioned this pull request Oct 13, 2023
Merged
Frank-GER pushed a commit to syscoin/syscoin that referenced this pull request Oct 13, 2023
@fanquake
Merge bitcoin#28547: ci: Work around podman stop intermittent failure
7859bdc 
fa2c894 ci: move-only CI_CONTAINER_ID to 02_run_container.sh (MarcoFalke)
fa695b4 ci: Work around podman stop bug (MarcoFalke)
fa09a03 ci: Add set -ex to 02_run_container.sh (MarcoFalke)
fac9abb ci: Rename 04_install to 02_run_container (MarcoFalke)

Pull request description:

  Sometimes, it seems that `podman stop` does not work. Presumably, it falls back to `podman kill`, which is async.

  Try to work around this intermittent issue by using the `rm --force` over `stop`.

  Example failing log https://cirrus-ci.com/task/4549784611061760?logs=ci#L238:

  ```
  Restart docker before run to stop and clear all containers started with --rm
  ++ podman container stop --all
  e4eca0766f87864d89fc230aa884a238c214cfbcd44cf76a4dbdb2a30c982009
  ++ echo 'Prune all dangling images'
  Prune all dangling images
  ++ docker image prune --force
  Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
  +++ docker run --cap-add LINUX_IMMUTABLE --rm --interactive --detach --tty --mount type=bind,src=/tmp/cirrus-build-1970593815,dst=/tmp/cirrus-build-1970593815,readonly --mount type=volume,src=ci_macos_cross_ccache,dst=/tmp/ccache_dir --mount type=volume,src=ci_macos_cross_depends,dst=/ci_container_base/depends --mount type=volume,src=ci_macos_cross_previous_releases,dst=/ci_container_base/prev_releases --env-file /tmp/env --name ci_macos_cross ci_macos_cross
  Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
  time="2023-09-27T20:55:39Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
  Error: creating container storage: the container name "ci_macos_cross" is already in use by e4eca0766f87864d89fc230aa884a238c214cfbcd44cf76a4dbdb2a30c982009. You have to remove that container to be able to reuse that name: that name is already in use

ACKs for top commit:
  hebasto:
    ACK fa2c894, I have reviewed the code and tested it locally.

Tree-SHA512: 31fca340c6bedaadf4dd51fa745d9b3969042cebc0c7c904ef18af3f2f986039ec4354ccdff1422fbf77cf223e4423857368dce53cfa67ef15c76b78d007eace
@maflcko
Copy link
Member Author

maflcko commented Oct 23, 2023

If the issue persists, I guess cherry-picking 4444a11 can be another attempt.

@bitcoin bitcoin locked and limited conversation to collaborators Dec 3, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@hebasto hebasto hebasto approved these changes

Assignees

No one assigned

Labels

Tests

Projects

None yet

Milestone

26.0

Development

Successfully merging this pull request may close these issues.

4 participants

@maflcko @DrahtBot @hebasto @fanquake