docker server panic , when network plugin return nil StaticRoutes #28630
Description
Description
An new network plguin return nil in StaticRoutes of NetwrokDriver.join request! when attach new container to network, if that network has gateway (default route point to it , not default_bridge), dockerdocker server panic ! if no gateway , that request http goroutine panic
Steps to reproduce the issue:
- new network plugin return nil in StaticRoutes of NetwrokDriver.join request
- create docker network , one has gateway, other has no gateway
- create new container attach that network or restart container have been in that network
Describe the results you received:
- attach gateway network ,, docker panic log:
-- Logs begin at Tue 2016-11-08 05:11:32 EST, end at Fri 2016-11-11 05:56:08 EST. --
Nov 11 05:55:16 kvm-yz-dev-073-046-005 systemd[1]: docker.service failed.
Nov 11 05:55:16 kvm-yz-dev-073-046-005 systemd[1]: Unit docker.service entered failed state.
Nov 11 05:55:16 kvm-yz-dev-073-046-005 systemd[1]: docker.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/sandbox_externalkey
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: created by github.com/docker/libnetwork.(*controller).acceptClientConnections
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/sandbox_externalkey
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: github.com/docker/libnetwork.(*controller).acceptClientConnections.func1(0x7fd74c97e758, 0xc82
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/sandbox_externalkey
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: github.com/docker/libnetwork.(*controller).processExternalKey(0xc82042e000, 0x7fd74c97e758, 0x
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/sandbox.go:720 +0x6
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: github.com/docker/libnetwork.(*sandbox).SetKey(0xc8204a8960, 0xc821994e00, 0x12, 0x0, 0x0)
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: /root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/sandbox.go:871 +0xe
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: github.com/docker/libnetwork.(*sandbox).populateNetworkResources(0xc8204a8960, 0xc821305ce0, 0
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: /usr/local/go/src/net/ip.go:483 +0x42
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: net.(*IPNet).String(0x0, 0x0, 0x0)
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: /usr/local/go/src/net/ip.go:433 +0x42
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: net.networkNumberAndMask(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: /usr/local/go/src/runtime/panic.go:481 +0x3e6
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: panic(0x1a90b60, 0xc82000e070)
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: goroutine 31125 [running]:
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: [signal 0xb code=0x1 addr=0x0 pc=0x66de42]
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: panic: runtime error: invalid memory address or nil pointer dereference
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: time="2016-11-11T05:55:16.862258173-05:00" level=debug msg="sandbox set key processing took 11
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: time="2016-11-11T05:55:16-05:00" level=info msg="Firewalld running: false"
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: time="2016-11-11T05:55:16.688849459-05:00" level=debug msg="Calling GET /v1.24/networks/236b04
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: time="2016-11-11T05:55:16.688200313-05:00" level=debug msg="Calling GET /v1.24/networks/236b04
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: time="2016-11-11T05:55:16.653260421-05:00" level=debug msg="Programming external connectivity
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: time="2016-11-11T05:55:16.529823750-05:00" level=debug msg="Assigning addresses for endpoint z
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: time="2016-11-11T05:55:16.365286571-05:00" level=debug msg="RequestAddress(GlobalDefault/172.1
Nov 11 05:55:16 kvm-yz-dev-073-046-005 dockerd[5532]: time="2016-11-11T05:55:16.365250771-05:00" level=debug msg="Assigning addresses for endpoint
- attach no gateway network ,, goroutine panic log:
ov 11 05:59:55 kvm-yz-dev-073-046-005 dockerd[14595]: time="2016-11-11T05:59:55.003613671-05:00" level=debug msg="Closing buffered stdin pipe"
Nov 11 05:59:55 kvm-yz-dev-073-046-005 dockerd[14595]: http: panic serving @: runtime error: invalid memory address or nil pointer dereference
goroutine 287 [running]:
net/http.(*conn).serve.func1(0xc82154fe80)
/usr/local/go/src/net/http/server.go:1389 +0xc1
panic(0x1a90b60, 0xc82000e070)
/usr/local/go/src/runtime/panic.go:443 +0x4e9
net.networkNumberAndMask(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/ip.go:433 +0x42
net.(*IPNet).String(0x0, 0x0, 0x0)
/usr/local/go/src/net/ip.go:483 +0x42
github.com/docker/libnetwork.(*sandbox).needDefaultGW(0xc82126e1e0, 0x0)
/root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/default_ga
github.com/docker/libnetwork.(*endpoint).sbJoin(0xc82008d080, 0xc82126e1e0, 0x0, 0x0, 0x0, 0x
/root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/endpoint.g
github.com/docker/libnetwork.(*endpoint).Join(0xc82008cf20, 0x7fc9ba352da8, 0xc82126e1e0, 0x0
/root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/docker/libnetwork/endpoint.g
github.com/docker/docker/daemon.(*Daemon).connectToNetwork(0xc8204b0000, 0xc8204a2f00, 0xc821
/root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/contai
github.com/docker/docker/daemon.(*Daemon).allocateNetwork(0xc8204b0000, 0xc8204a2f00, 0x0, 0x
/root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/contai
github.com/docker/docker/daemon.(*Daemon).initializeNetworking(0xc8204b0000, 0xc8204a2f00, 0x
/root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/contai
github.com/docker/docker/daemon.(*Daemon).containerStart(0xc8204b0000, 0xc8204a2f00, 0x0, 0x0
/root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/start.
github.com/docker/docker/daemon.(*Daemon).ContainerStart(0xc8204b0000, 0xc8212fc017, 0x40, 0x
/root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/daemon/start.
github.com/docker/docker/api/server/router/container.(*containerRouter).postContainersStart(0
/root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/api/server/ro
github.com/docker/docker/api/server/router/container.(*containerRouter).(github.com/docker/do
/root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/api/server/ro
github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1(0x7fc9ba38
/root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/api/server/mi
github.com/docker/docker/api/server/middleware.UserAgentMiddleware.WrapHandler.func1(0x7fc9ba
/root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/api/server/mi
github.com/docker/docker/api/server/middleware.DebugRequestMiddleware.func1(0x7fc9ba4081b8, 0
/root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/api/server/mi
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x7fc9ba35cfb0, 0xc82015a
/root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/api/server/se
net/http.HandlerFunc.ServeHTTP(0xc82039a0c0, 0x7fc9ba35cfb0, 0xc82015a340, 0xc821af20e0)
/usr/local/go/src/net/http/server.go:1618 +0x3a
github.com/gorilla/mux.(*Router).ServeHTTP(0xc8214670e0, 0x7fc9ba35cfb0, 0xc82015a340, 0xc821
/root/rpmbuild/BUILD/docker-engine/vendor/src/github.com/gorilla/mux/mux.go:98 +0x29e
github.com/docker/docker/api/server.(*routerSwapper).ServeHTTP(0xc821664aa0, 0x7fc9ba35cfb0,
/root/rpmbuild/BUILD/docker-engine/.gopath/src/github.com/docker/docker/api/server/ro
net/http.serverHandler.ServeHTTP(0xc820412e00, 0x7fc9ba35cfb0, 0xc82015a340, 0xc821af20e0)
/usr/local/go/src/net/http/server.go:2081 +0x19e
net/http.(*conn).serve(0xc82154fe80)
/usr/local/go/src/net/http/server.go:1472 +0xf2e
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2137 +0x44e
Nov 11 05:59:54 kvm-yz-dev-073-046-005 dockerd[14595]: time="2016-11-11T05:59:54.910964983-05:00" level=debug msg="Assigning addresses for endpoint
Describe the results you expected:
return error without panic end, save some record info correctly, don't affect docker server restart!
Additional information you deem important (e.g. issue happens only occasionally):
follow step, it will panic every time!
Output of docker version:
[root@kvm-yz-dev-073-046-005 ~]# docker version
Client:
Version: 1.12.3
API version: 1.24
Go version: go1.6.3
Git commit: 6b644ec
Built:
OS/Arch: linux/amd64
Server:
Version: 1.12.3
API version: 1.24
Go version: go1.6.3
Git commit: 6b644ec
Built:
OS/Arch: linux/amd64
Output of docker info:
[root@kvm-yz-dev-073-046-005 ~]# docker info
Containers: 6
Running: 4
Paused: 0
Stopped: 2
Images: 3
Server Version: 1.12.3
Storage Driver: devicemapper
Pool Name: docker-thinpool
Pool Blocksize: 524.3 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file:
Metadata file:
Data Space Used: 714.6 MB
Data Space Total: 53.69 GB
Data Space Available: 52.97 GB
Metadata Space Used: 1.057 MB
Metadata Space Total: 10.74 GB
Metadata Space Available: 10.74 GB
Thin Pool Minimum Free Space: 5.369 GB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Library Version: 1.02.107-RHEL7 (2015-10-14)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: host null overlay vlcp bridge
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.642 GiB
Name: kvm-yz-dev-073-046-005
ID: 4JQB:RY3U:VX5B:NAHZ:M6LT:5UBR:GUFX:7RFM:EGED:MSZF:FO7N:NTO2
Docker Root Dir: /data/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 49
Goroutines: 93
System Time: 2016-11-19T21:22:00.699427473-05:00
EventsListeners: 3
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Cluster Store: zk://100.73.46.3:2181,100.73.46.4:2181,100.73.46.5:2181
Cluster Advertise: 100.73.46.5:2375
Insecure Registries:
127.0.0.0/8
Additional environment details (AWS, VirtualBox, physical, etc.):
We are using docker swarm (swarm standalone, not swarmkit) to manage the cluster.
when docker server panic , all node record container end-point info of an failed container!
[root@kvm-yz-dev-073-046-003 ~]# docker network inspect vxlan22
[
{
"Name": "vxlan22",
"Id": "236b04d394c6bbe64f4f7417bf5ac9b94457ffaa3b3c93075cf27c8443889c7f",
"Scope": "global",
"Driver": "vlcp",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "172.100.22.0/24",
"Gateway": "172.100.22.1"
}
]
},
"Internal": false,
"Containers": {
"ep-623121cf53bfdc2c546d53b8d4e6ccffa6df90ac848d433fd4a03bfde5a089b7": {
"Name": "zen_hawking",
"EndpointID": "623121cf53bfdc2c546d53b8d4e6ccffa6df90ac848d433fd4a03bfde5a089b7",
"MacAddress": "02:a4:b2:a0:5e:02",
"IPv4Address": "172.100.22.2/24",
"IPv6Address": ""
}
},
"Options": {
"physicalnetwork": "vxlan"
},
"Labels": {}
}
]
recover nil StaticRoutes, restart container will be success! but
Error response from daemon: service endpoint with name gpl already exists
I think some info record incorrectly!