Restarting containerd leaves dockerd stopped #1155
Description
- This is a bug report
- This is a feature request
- I searched existing issues before opening this one
Expected behavior
After a containerd upgrade, the docker daemon should be running.
Actual behavior
If you restart containerd, dockerd remains down.
Steps to reproduce the behavior
root@vm-11:/home/bmitch# systemctl stop containerd
root@vm-11:/home/bmitch# systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/docker.service.d
└─override.conf
Active: inactive (dead) since Tue 2020-12-01 14:08:55 EST; 11s ago
Docs: https://docs.docker.com
Process: 467 ExecStart=/usr/bin/dockerd (code=exited, status=0/SUCCESS)
Main PID: 467 (code=exited, status=0/SUCCESS)
Dec 01 14:07:30 vm-11 dockerd[467]: time="2020-12-01T14:07:30.747469349-05:00" level=info msg="Loading containers: done."
Dec 01 14:07:31 vm-11 dockerd[467]: time="2020-12-01T14:07:31.281078096-05:00" level=info msg="Docker daemon" commit=633a0ea838 graphdriver(s)=overlay
Dec 01 14:07:31 vm-11 dockerd[467]: time="2020-12-01T14:07:31.281966418-05:00" level=info msg="Daemon has completed initialization"
Dec 01 14:07:31 vm-11 systemd[1]: Started Docker Application Container Engine.
Dec 01 14:07:31 vm-11 dockerd[467]: time="2020-12-01T14:07:31.325684247-05:00" level=info msg="API listen on /var/run/docker.sock"
Dec 01 14:08:55 vm-11 systemd[1]: Stopping Docker Application Container Engine...
Dec 01 14:08:55 vm-11 dockerd[467]: time="2020-12-01T14:08:55.080820926-05:00" level=info msg="Processing signal 'terminated'"
Dec 01 14:08:55 vm-11 dockerd[467]: time="2020-12-01T14:08:55.081301227-05:00" level=info msg="Daemon shutdown complete"
Dec 01 14:08:55 vm-11 systemd[1]: docker.service: Succeeded.
Dec 01 14:08:55 vm-11 systemd[1]: Stopped Docker Application Container Engine.
root@vm-11:/home/bmitch# systemctl start containerd
root@vm-11:/home/bmitch# systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/docker.service.d
└─override.conf
Active: inactive (dead) since Tue 2020-12-01 14:08:55 EST; 22s ago
Docs: https://docs.docker.com
Process: 467 ExecStart=/usr/bin/dockerd (code=exited, status=0/SUCCESS)
Main PID: 467 (code=exited, status=0/SUCCESS)
Dec 01 14:07:30 vm-11 dockerd[467]: time="2020-12-01T14:07:30.747469349-05:00" level=info msg="Loading containers: done."
Dec 01 14:07:31 vm-11 dockerd[467]: time="2020-12-01T14:07:31.281078096-05:00" level=info msg="Docker daemon" commit=633a0ea838 graphdriver(s)=overlay
Dec 01 14:07:31 vm-11 dockerd[467]: time="2020-12-01T14:07:31.281966418-05:00" level=info msg="Daemon has completed initialization"
Dec 01 14:07:31 vm-11 systemd[1]: Started Docker Application Container Engine.
Dec 01 14:07:31 vm-11 dockerd[467]: time="2020-12-01T14:07:31.325684247-05:00" level=info msg="API listen on /var/run/docker.sock"
Dec 01 14:08:55 vm-11 systemd[1]: Stopping Docker Application Container Engine...
Dec 01 14:08:55 vm-11 dockerd[467]: time="2020-12-01T14:08:55.080820926-05:00" level=info msg="Processing signal 'terminated'"
Dec 01 14:08:55 vm-11 dockerd[467]: time="2020-12-01T14:08:55.081301227-05:00" level=info msg="Daemon shutdown complete"
Dec 01 14:08:55 vm-11 systemd[1]: docker.service: Succeeded.
Dec 01 14:08:55 vm-11 systemd[1]: Stopped Docker Application Container Engine.
Output of docker version:
Client: Docker Engine - Community
Version: 19.03.13
API version: 1.40
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 17:02:55 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.13
API version: 1.40 (minimum version 1.12)
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 17:01:25 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.3.9
GitCommit: ea765aba0d05254012b0b9e595e995c09186427f
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
Output of docker info:
Client:
Debug Mode: false
Server:
Containers: 4
Running: 0
Paused: 0
Stopped: 4
Images: 2
Server Version: 19.03.13
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: ea765aba0d05254012b0b9e595e995c09186427f
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
userns
Kernel Version: 4.19.0-5-amd64
Operating System: Debian GNU/Linux 10 (buster)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 987.4MiB
Name: vm-11
ID: LMZU:DXKC:DD7P:4K45:XPTD:4BKB:DKYU:N2O3:WS2Q:RMPR:6FF2:ZMBW
Docker Root Dir: /var/lib/docker/231072.231072
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
from_ansible=true
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Additional environment details (AWS, VirtualBox, physical, etc.)
Above is from a small test VM. I'm posting this after seeing:
- https://www.reddit.com/r/docker/comments/k4jiyp/woke_up_to_my_docker_daemon_being_stopped_no_logs/
- https://unix.stackexchange.com/questions/622355/systemd-stopped-docker-daemon-for-no-obvious-reason
- plus a 2nd hand report from a client, and some posts in the docker community slack this morning
The cause from the above I suspect to be an update to containerd without a corresponding update to the docker daemon that would restart both services.