Description
Upon upgrading to Docker 20.10.1 any container which has a port published to a hosts IPv6 literal address cannot clean up after itself. Initial launch of the container is successful but any subsequent restarts will result in an error.
Restarting the Docker daemon will cause the docker-proxy instance holding the port open to be released however the issue will reoccur upon attempts to restart the container again.
Steps to reproduce the issue:
docker run -d --name nginx-test -p [fc00:1337:1337:1337:1337:1337:1337:1337]:80:80 nginxdocker restart nginx-test
Describe the results you received:
Once issuing the restart the following error is received:
Error response from daemon: Cannot restart container nginx-test: driver failed programming external connectivity on endpoint nginx-test (9a6f7a503369a88aa1d200050e4a82ba09e755e48588f4000086c5abae6c3710): Bind for fc00:1337:1337:1337:1337:1337:1337:1337:80 failed: port is already allocated
Upon checking the status of the docker daemon via systemctl I can see there's an instance of docker-proxy holding on to the port:
systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2020-12-17 13:04:12 AEDT; 1 day 23h ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 384483 (dockerd)
Tasks: 41 (limit: 38458)
Memory: 265.4M
CGroup: /system.slice/docker.service
├─384483 /usr/bin/dockerd -H fd://
├─384495 containerd --config /var/run/docker/containerd/containerd.toml --log-level info
└─634622 /usr/bin/docker-proxy -proto tcp -host-ip fc00:1337:1337:1337:1337:1337:1337:1337 -host-port 80 -container-ip 172.17.0.2 -container-port 80
The logs for the Docker daemon also provide the following output:
Dec 19 12:04:11 mana dockerd[384483]: time="2020-12-19T12:04:11.366530647+11:00" level=info msg="ignoring event" container=ed24834d6f45bbaf301c8243f51a942bb82d81675c70481f68d64cf14f8f005e module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Dec 19 12:04:11 mana dockerd[384495]: time="2020-12-19T12:04:11.366851423+11:00" level=info msg="shim disconnected" id=ed24834d6f45bbaf301c8243f51a942bb82d81675c70481f68d64cf14f8f005e
Dec 19 12:04:11 mana dockerd[384483]: time="2020-12-19T12:04:11.368999767+11:00" level=warning msg="\ncould not release {tcp 172.17.0.2 80 fc00:1337:1337:1337:1337:1337:1337:1337 80 80} because of port is not mapped"
Dec 19 12:04:11 mana dockerd[384483]: time="2020-12-19T12:04:11.582843296+11:00" level=warning msg="Failed to allocate and map port 80-80: Bind for fc00:1337:1337:1337:1337:1337:1337:1337:80 failed: port is already allocated"
Dec 19 12:04:11 mana dockerd[384483]: time="2020-12-19T12:04:11.682239315+11:00" level=error msg="ed24834d6f45bbaf301c8243f51a942bb82d81675c70481f68d64cf14f8f005e cleanup: failed to delete container from containerd: no such container"
Dec 19 12:04:11 mana dockerd[384483]: time="2020-12-19T12:04:11.693431409+11:00" level=error msg="Handler for POST /v1.41/containers/nginx-test/restart returned error: Cannot restart container nginx-test: driver failed programming external connectivity on endpoint nginx-test (9a6f7a503369a88aa1d200050e4a82ba09e755e48588f4000086c5abae6c3710): Bind for fc00:1337:1337:1337:1337:1337:1337:1337:80 failed: port is already allocated"
Describe the results you expected:
I expect that the container would have restarted successfully with the appropriate port being published on the specified IPv6 address.
Additional information you deem important (e.g. issue happens only occasionally):
I cannot reproduce the issue if publishing a port to a hosts IPv4 address, for example:
docker run -d --name nginx-test -p 10.10.10.4:80:80 nginxdocker restart nginx-test
The above commands will run successfully.
Downgrading to 19.03.14 provides a temporary workaround as this issue does not exist in the older version.
While my default runtime is set to crun I also went back to the Docker defaults with runc and I was able to reproduce the issue.
Output of docker version:
Client:
Version: 20.10.1
API version: 1.41
Go version: go1.15.6
Git commit: 831ebeae96
Built: Tue Dec 15 22:25:01 2020
OS/Arch: linux/amd64
Context: default
Experimental: true
Server:
Engine:
Version: 20.10.1
API version: 1.41 (minimum version 1.12)
Go version: go1.15.6
Git commit: f0014860c1
Built: Tue Dec 15 22:24:28 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.4.3
GitCommit: 269548fa27e0089a8b8278fc4fc781d7f65a939b.m
crun:
Version: 0.16
GitCommit: eb0145e5ad4d8207e84a327248af76663d4e50dd
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Output of docker info:
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-tp-docker)
Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 32
Server Version: 20.10.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: crun io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: crun
Init Binary: docker-init
containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b.m
runc version: eb0145e5ad4d8207e84a327248af76663d4e50dd
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 5.9.14-arch1-1
Operating System: Archcraft
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.32GiB
Name: mana
ID: WSAG:GWTA:H6MH:IIOR:M5TO:DMEQ:6HMW:BJ25:VB3W:ERGY:ORNA:3RGQ
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Registry Mirrors:
http://nerv:5000/
Live Restore Enabled: false
WARNING: No blkio weight support
WARNING: No blkio weight_device support
Additional environment details (AWS, VirtualBox, physical, etc.):
Not applicable.
Description
Upon upgrading to Docker
20.10.1any container which has a port published to a hosts IPv6 literal address cannot clean up after itself. Initial launch of the container is successful but any subsequent restarts will result in an error.Restarting the Docker daemon will cause the
docker-proxyinstance holding the port open to be released however the issue will reoccur upon attempts to restart the container again.Steps to reproduce the issue:
docker run -d --name nginx-test -p [fc00:1337:1337:1337:1337:1337:1337:1337]:80:80 nginxdocker restart nginx-testDescribe the results you received:
Once issuing the restart the following error is received:
Error response from daemon: Cannot restart container nginx-test: driver failed programming external connectivity on endpoint nginx-test (9a6f7a503369a88aa1d200050e4a82ba09e755e48588f4000086c5abae6c3710): Bind for fc00:1337:1337:1337:1337:1337:1337:1337:80 failed: port is already allocatedUpon checking the status of the docker daemon via systemctl I can see there's an instance of
docker-proxyholding on to the port:The logs for the Docker daemon also provide the following output:
Describe the results you expected:
I expect that the container would have restarted successfully with the appropriate port being published on the specified IPv6 address.
Additional information you deem important (e.g. issue happens only occasionally):
I cannot reproduce the issue if publishing a port to a hosts IPv4 address, for example:
docker run -d --name nginx-test -p 10.10.10.4:80:80 nginxdocker restart nginx-testThe above commands will run successfully.
Downgrading to
19.03.14provides a temporary workaround as this issue does not exist in the older version.While my default runtime is set to
crunI also went back to the Docker defaults withruncand I was able to reproduce the issue.Output of
docker version:Output of
docker info:Additional environment details (AWS, VirtualBox, physical, etc.):
Not applicable.