Docker 20.10.6: all containers stopped and cannot start if ipv6 is disabled on host #42288
Description
Related to the release notes here: https://docs.docker.com/engine/release-notes/#20106
Possibly related bug: moby/libnetwork#2629
Description
Since upgrading (automatically) to docker-ce 20.10.06, all our containers fail to start.
The error says: failed to start container" container=[number removed] error="driver failed programming external connectivity on endpoint tvheadend ([number removed]): Error starting userland proxy: listen tcp6 [::]:9982: socket: address family not supported by protocol"
Our docker machines have ipv6 disabled in the kernel with the commandline ipv6.disable=1
Steps to reproduce the issue:
- Have ipv6 disabled
- Update docker to 20.10.06
- All your containers fail to start
Describe the results you received:
All your containers fail to start
Describe the results you expected:
Containers restart normally?
Additional information you deem important (e.g. issue happens only occasionally):
This is the log we get at docker start:
avril 13 07:26:20 apigateway1 systemd[1]: Starting Docker Application Container Engine...
avril 13 07:26:25 apigateway1 dockerd[775]: time="2021-04-13T07:26:25.939440237Z" level=info msg="Starting up"
avril 13 07:26:26 apigateway1 dockerd[775]: time="2021-04-13T07:26:26.248057733Z" level=info msg="parsed scheme: \"unix\"" module=grpc
avril 13 07:26:26 apigateway1 dockerd[775]: time="2021-04-13T07:26:26.248153074Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
avril 13 07:26:26 apigateway1 dockerd[775]: time="2021-04-13T07:26:26.248233851Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc
avril 13 07:26:26 apigateway1 dockerd[775]: time="2021-04-13T07:26:26.248307947Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
avril 13 07:26:26 apigateway1 dockerd[775]: time="2021-04-13T07:26:26.265810303Z" level=info msg="parsed scheme: \"unix\"" module=grpc
avril 13 07:26:26 apigateway1 dockerd[775]: time="2021-04-13T07:26:26.265857230Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
avril 13 07:26:26 apigateway1 dockerd[775]: time="2021-04-13T07:26:26.265891159Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc
avril 13 07:26:26 apigateway1 dockerd[775]: time="2021-04-13T07:26:26.265910772Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
avril 13 07:26:26 apigateway1 dockerd[775]: time="2021-04-13T07:26:26.636639243Z" level=info msg="[graphdriver] using prior storage driver: overlay2"
avril 13 07:26:26 apigateway1 dockerd[775]: time="2021-04-13T07:26:26.915413961Z" level=warning msg="Your kernel does not support swap memory limit"
avril 13 07:26:26 apigateway1 dockerd[775]: time="2021-04-13T07:26:26.915450962Z" level=warning msg="Your kernel does not support CPU realtime scheduler"
avril 13 07:26:26 apigateway1 dockerd[775]: time="2021-04-13T07:26:26.915919011Z" level=info msg="Loading containers: start."
avril 13 07:26:27 apigateway1 dockerd[775]: time="2021-04-13T07:26:27.817745243Z" level=info msg="failed to read ipv6 net.ipv6.conf.<bridge>.accept_ra" bridge=br-3458afb7e0fb syspath=/proc/sys/net/ipv6/conf/br-3458afb7e0fb/accept_ra
avril 13 07:26:27 apigateway1 dockerd[775]: time="2021-04-13T07:26:27.884334406Z" level=info msg="failed to read ipv6 net.ipv6.conf.<bridge>.accept_ra" bridge=docker0 syspath=/proc/sys/net/ipv6/conf/docker0/accept_ra
avril 13 07:26:27 apigateway1 dockerd[775]: time="2021-04-13T07:26:27.884919514Z" level=info msg="failed to read ipv6 net.ipv6.conf.<bridge>.accept_ra" bridge=docker0 syspath=/proc/sys/net/ipv6/conf/docker0/accept_ra
avril 13 07:26:28 apigateway1 dockerd[775]: time="2021-04-13T07:26:28.169815521Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
avril 13 07:26:28 apigateway1 dockerd[775]: time="2021-04-13T07:26:28.170524023Z" level=info msg="failed to read ipv6 net.ipv6.conf.<bridge>.accept_ra" bridge=docker0 syspath=/proc/sys/net/ipv6/conf/docker0/accept_ra
avril 13 07:26:28 apigateway1 dockerd[775]: time="2021-04-13T07:26:28.461643528Z" level=warning msg="Failed to allocate and map port 1337-1337: Error starting userland proxy: listen tcp6 [::]:1337: socket: address family not supported by protocol"
avril 13 07:26:28 apigateway1 dockerd[775]: time="2021-04-13T07:26:28.489205748Z" level=warning msg="Failed to allocate and map port 8080-8080: Error starting userland proxy: listen tcp6 [::]:8080: socket: address family not supported by protocol"
avril 13 07:26:28 apigateway1 dockerd[775]: time="2021-04-13T07:26:28.629822309Z" level=warning msg="Failed to allocate and map port 8000-8000: Error starting userland proxy: listen tcp6 [::]:8000: socket: address family not supported by protocol"
avril 13 07:26:28 apigateway1 dockerd[775]: time="2021-04-13T07:26:28.670531244Z" level=error msg="791cd702dbdc24094aa27be1a6bf21e3b008b545d297b1e1181d5d278ae0ef62 cleanup: failed to delete container from containerd: no such container"
avril 13 07:26:28 apigateway1 dockerd[775]: time="2021-04-13T07:26:28.670913826Z" level=error msg="failed to start container" container=791cd702dbdc24094aa27be1a6bf21e3b008b545d297b1e1181d5d278ae0ef62 error="driver failed programming external connectivity on endpoint kong_konga_1 (f1cafeb16e89b42fb4f418add1abee0b44dc7c776b1dd5d5a97fbc04dda863de): Error starting userland proxy: listen tcp6 [::]:1337: socket: address family not supported by protocol"
avril 13 07:26:28 apigateway1 dockerd[775]: time="2021-04-13T07:26:28.722145131Z" level=warning msg="Failed to allocate and map port 9042-9042: Error starting userland proxy: listen tcp6 [::]:9042: socket: address family not supported by protocol"
avril 13 07:26:28 apigateway1 dockerd[775]: time="2021-04-13T07:26:28.772404127Z" level=error msg="c997c4c1b097c34a2688d30a9c178f297866098ce86cfd437e20689b0631fa11 cleanup: failed to delete container from containerd: no such container"
avril 13 07:26:28 apigateway1 dockerd[775]: time="2021-04-13T07:26:28.772911179Z" level=error msg="failed to start container" container=c997c4c1b097c34a2688d30a9c178f297866098ce86cfd437e20689b0631fa11 error="driver failed programming external connectivity on endpoint kong_kong-sidecar_1 (d54f401742450b01cb649b428f91ef76b820e4722a43b77406d24c5f8a888e0a): Error starting userland proxy: listen tcp6 [::]:8080: socket: address family not supported by protocol"
avril 13 07:26:28 apigateway1 dockerd[775]: time="2021-04-13T07:26:28.884089786Z" level=error msg="1dd07dc91b27d7a57c7c3becbb1519fee76bbe79ff79569270a4731735a4861f cleanup: failed to delete container from containerd: no such container"
avril 13 07:26:28 apigateway1 dockerd[775]: time="2021-04-13T07:26:28.884841529Z" level=error msg="failed to start container" container=1dd07dc91b27d7a57c7c3becbb1519fee76bbe79ff79569270a4731735a4861f error="driver failed programming external connectivity on endpoint kong_kong_1 (7d0441a5219f56147a10ce269edb8b7cd07391846a25db3cc445a30e79b03a45): Error starting userland proxy: listen tcp6 [::]:8000: socket: address family not supported by protocol"
avril 13 07:26:28 apigateway1 dockerd[775]: time="2021-04-13T07:26:28.972249864Z" level=error msg="c77ee0c3833f73b747ed9ad7affa3a692123b909bf8fce8107402d0c7e3f4ba5 cleanup: failed to delete container from containerd: no such container"
avril 13 07:26:28 apigateway1 dockerd[775]: time="2021-04-13T07:26:28.972310664Z" level=error msg="failed to start container" container=c77ee0c3833f73b747ed9ad7affa3a692123b909bf8fce8107402d0c7e3f4ba5 error="driver failed programming external connectivity on endpoint kong_db_1 (a33664b1825441bef099df18cbd46470f449907ae8a16ba76f528b68d4abb26c): Error starting userland proxy: listen tcp6 [::]:9042: socket: address family not supported by protocol"
avril 13 07:26:28 apigateway1 dockerd[775]: time="2021-04-13T07:26:28.972369960Z" level=info msg="Loading containers: done."
avril 13 07:26:29 apigateway1 dockerd[775]: time="2021-04-13T07:26:29.208780479Z" level=info msg="Docker daemon" commit=8728dd2 graphdriver(s)=overlay2 version=20.10.6
avril 13 07:26:29 apigateway1 dockerd[775]: time="2021-04-13T07:26:29.209351109Z" level=info msg="Daemon has completed initialization"
avril 13 07:26:29 apigateway1 systemd[1]: Started Docker Application Container Engine.
Output of docker version:
Docker version 20.10.6, build 370c289
Output of docker info:
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
scan: Docker Scan (Docker Inc., v0.7.0)
Server:
Containers: 6
Running: 4
Paused: 0
Stopped: 2
Images: 8
Server Version: 20.10.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.19.0-16-amd64
Operating System: Debian GNU/Linux 10 (buster)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.854GiB
Name: apigateway1
ID: ZTQA:TV6R:PYGF:SJ6W:TEDY:74TF:LT4T:DUHS:UATR:LUJY:FSH7:HL53
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Additional environment details (AWS, VirtualBox, physical, etc.):
The docker machines run on KVM, and are on Debian 9 or 10 depending on freshness of install.