Description
In my Github self hosted runners, I have a script to clean up any resources in Docker before a job is started:
docker ps --all --quiet | xargs -r docker stop
docker system prune --volumes --force --all
When I started creating ephemeral containers in workflow runs, when the next run is enqueued, this script runs and sometimes fails at docker system prune:
+ xargs -r docker stop
c9823b1e6360
+ docker system prune --volumes --force --all
Error response from daemon: rw layer snapshot not found for container c9823b1e63604612e348a6734062597d133956cc067b5f88a3817306b5871439
I suspect this may be a race condition where the Docker daemon tries to delete the ephemeral container at the same time as docker system prune. This is an issue because it can cause my jobs to fail and I have to wait for the workflow run to finish before I can restart those jobs.
Reproduce
Locally:
docker run --rm <image-name>
docker ps --all --quiet | xargs -r docker stop
docker system prune --volumes --force --all
Expected behavior
docker system prune should not delete an ephemeral container that is already handled by the Docker daemon in the background.
docker version
Client: Docker Engine - Community
Version: 28.4.0
API version: 1.51
Go version: go1.25.0
Git commit: d8eb465f86
Built: Wed Sep 3 20:52:20 2025
OS/Arch: darwin/arm64
Context: colima
Server: Docker Engine - Community
Engine:
Version: 28.3.3
API version: 1.51 (minimum version 1.24)
Go version: go1.24.5
Git commit: bea959c
Built: Fri Jul 25 11:35:18 2025
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.7.27
GitCommit: 05044ec0a9a75232cad458027ca83437aae3f4da
runc:
Version: 1.2.5
GitCommit: v1.2.5-0-g59923ef
docker-init:
Version: 0.19.0
GitCommit: de40ad0
docker info
Client: Docker Engine - Community
Version: 28.4.0
Context: colima
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.28.0
Path: /Users/<redacted>/.docker/cli-plugins/docker-buildx
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 4
Server Version: 28.3.3
Storage Driver: overlayfs
driver-type: io.containerd.snapshotter.v1
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
CDI spec directories:
/etc/cdi
/var/run/cdi
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
runc version: v1.2.5-0-g59923ef
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.8.0-64-generic
Operating System: Ubuntu 24.04.2 LTS
OSType: linux
Architecture: aarch64
CPUs: 2
Total Memory: 1.914GiB
Name: colima
ID: ba19cd5a-35cf-4052-a69a-6a5f04589f2c
Docker Root Dir: /var/lib/docker
Debug Mode: false
Username: <redacted>
Experimental: false
Insecure Registries:
::1/128
127.0.0.0/8
Live Restore Enabled: falseAdditional Info
Colima is running on this macOS version:
sw_vers
ProductName: macOS
ProductVersion: 14.8.2
BuildVersion: 23J126
Description
In my Github self hosted runners, I have a script to clean up any resources in Docker before a job is started:
When I started creating ephemeral containers in workflow runs, when the next run is enqueued, this script runs and sometimes fails at
docker system prune:I suspect this may be a race condition where the Docker daemon tries to delete the ephemeral container at the same time as
docker system prune. This is an issue because it can cause my jobs to fail and I have to wait for the workflow run to finish before I can restart those jobs.Reproduce
Locally:
Expected behavior
docker system pruneshould not delete an ephemeral container that is already handled by the Docker daemon in the background.docker version
Client: Docker Engine - Community Version: 28.4.0 API version: 1.51 Go version: go1.25.0 Git commit: d8eb465f86 Built: Wed Sep 3 20:52:20 2025 OS/Arch: darwin/arm64 Context: colima Server: Docker Engine - Community Engine: Version: 28.3.3 API version: 1.51 (minimum version 1.24) Go version: go1.24.5 Git commit: bea959c Built: Fri Jul 25 11:35:18 2025 OS/Arch: linux/arm64 Experimental: false containerd: Version: 1.7.27 GitCommit: 05044ec0a9a75232cad458027ca83437aae3f4da runc: Version: 1.2.5 GitCommit: v1.2.5-0-g59923ef docker-init: Version: 0.19.0 GitCommit: de40ad0docker info
Additional Info
Colima is running on this macOS version: