Description
ContainerStop has the potential to deadlock here:
|
if status := <-container.Wait(ctx, containerpkg.WaitConditionNotRunning); status.Err() != nil { |
|
logrus.Infof("Container %v failed to exit within %d seconds of signal %d - using the force", container.ID, seconds, stopSignal) |
|
// 3. If it doesn't, then send SIGKILL |
|
if err := daemon.Kill(container); err != nil { |
|
// Wait without a timeout, ignore result. |
|
<-container.Wait(context.Background(), containerpkg.WaitConditionNotRunning) |
|
logrus.Warn(err) // Don't return error because we only care that container is stopped, not what function stopped it |
|
} |
|
} |
If container stop (signal 15) did not work to kill the container, dockerd tries to do a SIGKILL (9). However, if daemon.Kill returns an error, then dockerd calls the container.Wait function without a timeout. Inside the wait function there are numerous code-paths that obtain a lock on both the container and state objects. So if the SIGKILL also did not work, dockerd will essentially be waiting forever. Subsequent calls to ContainerStop will also hang forever, and potentially deadlock on one of the locks that a previous ContainerStop call is holding.
Steps to reproduce the issue:
- add
return fmt.Errorf("foo error") here:
- Build a docker container that captures SIGTERM:
cat << EOF > Dockerfile
FROM bash:5.0
COPY killtrapper.sh /
ENTRYPOINT ["/killtrapper.sh"]
EOF
cat << EOF > killtrapper.sh
#!/usr/bin/env bash
sigterm() {
echo "caught a signal"
}
trap 'sigterm' QUIT
trap 'sigterm' INT
trap 'sigterm' HUP
trap 'sigterm' TERM
echo "test script started. My PID is $$"
while :; do
sleep 0.5
done
EOF
- build and run the container
chmod +x killtrapper.sh && docker build -t "killtrapper" . && docker run -it --rm --name "kt" killtrapper
- attempt to stop the container:
docker stop kt
- observe that the command hangs forever and this is logged into docker logs:
Oct 23 04:17:23 ip-10-0-0-176.us-west-2.compute.internal dockerd[29371]: time="2020-10-23T04:17:23.883792164Z" level=info msg="Container e0d10e9ca9d06815a903de8bab96c98b7b7d2435f6afc41bbe54e188f112564c failed to exit within 10 seconds of signal 15 - using the force"
Describe the results you expected:
If docker stop fails, log the error and return
Output of docker version:
dockerd engine is current head of master:
Client:
Version: 19.03.6-ce
API version: 1.40
Go version: go1.13.4
Git commit: 369ce74
Built: Fri May 29 04:01:26 2020
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: dev
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: 8a4671fb1f
Built: Fri Oct 23 04:07:23 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.3.2
GitCommit: ff48f57fc83a8c44cf4ad5d672424a98ba37ded6
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
Output of docker info:
Client:
Debug Mode: false
Server:
Containers: 2
Running: 1
Paused: 0
Stopped: 1
Images: 7
Server Version: dev
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: ff48f57fc83a8c44cf4ad5d672424a98ba37ded6
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683 (expected: de40ad0)
Security Options:
seccomp
Profile: default
Kernel Version: 4.14.198-152.320.amzn2.x86_64
Operating System: Amazon Linux 2
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 30.96GiB
Name: ip-10-0-0-176.us-west-2.compute.internal
ID: J7BA:BTKN:ZD5S:FWGT:2A2Y:NKTF:DDAJ:QSNA:UE23:L6S2:W7RG:BXBK
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.):
this test was run on an AWS m5.xlarge instance
Description
ContainerStop has the potential to deadlock here:
moby/daemon/stop.go
Lines 77 to 85 in 7cf6dfc
If container stop (signal 15) did not work to kill the container, dockerd tries to do a SIGKILL (9). However, if daemon.Kill returns an error, then dockerd calls the container.Wait function without a timeout. Inside the wait function there are numerous code-paths that obtain a lock on both the container and state objects. So if the SIGKILL also did not work, dockerd will essentially be waiting forever. Subsequent calls to ContainerStop will also hang forever, and potentially deadlock on one of the locks that a previous ContainerStop call is holding.
Steps to reproduce the issue:
return fmt.Errorf("foo error")here:moby/daemon/kill.go
Line 139 in 7cf6dfc
chmod +x killtrapper.sh && docker build -t "killtrapper" . && docker run -it --rm --name "kt" killtrapperdocker stop ktDescribe the results you expected:
If docker stop fails, log the error and return
Output of
docker version:Output of
docker info:Additional environment details (AWS, VirtualBox, physical, etc.):
this test was run on an AWS m5.xlarge instance