Skip to content

docker daemon crashed after launched 9k+ containers #22978

Closed
Closed
docker daemon crashed after launched 9k+ containers#22978
Labels
version/master
@ligc

Description

@ligc
ligc
opened on May 25, 2016

Output of docker version:

Client:
 Version:      1.11.0-dev
 API version:  1.24
 Go version:   go1.6.1
 Git commit:   9e4b5e0
 Built:        Thu Apr 14 19:21:34 2016
 OS/Arch:      linux/ppc64le

Server:
 Version:      1.11.0-dev
 API version:  1.24
 Go version:   go1.6.1
 Git commit:   9e4b5e0
 Built:        Thu Apr 14 19:21:34 2016
 OS/Arch:      linux/ppc64le

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.11.0-dev
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 0
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: null host bridge
Kernel Version: 4.4.0-22-generic
Operating System: Ubuntu 16.04 LTS
OSType: linux
Architecture: ppc64le
CPUs: 160
Total Memory: 255.4 GiB
Name: redpower6
ID: DYHZ:MCJC:ADPZ:KPEM:JNAJ:CIKK:V345:IKHN:FS6J:BR77:635V:NZRZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):

Running docker on a physical machine.

Steps to reproduce the issue:

  1. Set ulimit
    root@redpower6:/test# cat /etc/security/limits.conf
    root soft nofile 1048576
    root hard nofile 1048576
    root soft nproc 10485760
    root hard nproc 10485760
    root@redpower6:    /test#
  2. Set systemd drop-in for docker

root@redpower6:/test# cat /etc/systemd/system/docker.service.d/tasksmax.conf
[Service]
TasksMax=infinity
root@redpower6:/test#
systemctl daemon-reload
systemctl restart docker

  1. Continuously launch new docker containers until 9k+

Describe the results you received:
When launched about 9240 docker containers, the docker daemon crashed with the following errors with "journalctl -u docker.service"

May 25 21:17:40 redpower6 docker[3386]: runtime/cgo: pthread_create failed: Resource temporarily unavailable
May 25 21:17:40 redpower6 docker[3386]: SIGABRT: abort
May 25 21:17:40 redpower6 docker[3386]: PC=0x10dd9798 m=18660
May 25 21:17:40 redpower6 docker[3386]: goroutine 0 [idle]:
May 25 21:17:40 redpower6 docker[3386]: goroutine 1 [chan receive, 649 minutes]:
May 25 21:17:40 redpower6 docker[3386]: main.(*DaemonCli).CmdDaemon(0xc8203fc220, 0xc82000a1a0, 0x2, 0x2, 0x0, 
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/docker/daemon.go:354 +0x202c
May 25 21:17:40 redpower6 docker[3386]: reflect.callMethod(0xc820378330, 0xc820861c30)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/reflect/value.go:631 +0x1e8
May 25 21:17:40 redpower6 docker[3386]: reflect.methodValueCall(0xc82000a1a0, 0x2, 0x2, 0x1, 0xc820378330, 0x0,
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/reflect/asm_ppc64x.s:32 +0x3c
May 25 21:17:40 redpower6 docker[3386]: github.com/docker/docker/cli.(*Cli).Run(0xc8203782a0, 0xc82000a190, 0x3
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/cli/cli.go:89 +0x318
May 25 21:17:40 redpower6 docker[3386]: main.main()
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/docker/docker.go:63 +0x45c
May 25 21:17:40 redpower6 docker[3386]: goroutine 17 [syscall, 649 minutes, locked to thread]:
May 25 21:17:40 redpower6 docker[3386]: runtime.goexit()
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/runtime/asm_ppc64x.s:1049 +0x4
May 25 21:17:40 redpower6 docker[3386]: goroutine 6 [syscall, 649 minutes]:
May 25 21:17:40 redpower6 docker[3386]: os/signal.signal_recv(0x0)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/runtime/sigqueue.go:116 +0x270
May 25 21:17:40 redpower6 docker[3386]: os/signal.loop()
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/os/signal/signal_unix.go:22 +0x24
May 25 21:17:40 redpower6 docker[3386]: created by os/signal.init.1
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/os/signal/signal_unix.go:28 +0x44
May 25 21:17:40 redpower6 docker[3386]: goroutine 205 [syscall, 649 minutes]:
May 25 21:17:40 redpower6 docker[3386]: syscall.Syscall6(0x72, 0xd62, 0xc8204f7d90, 0x0, 0xc820166000, 0x0, 0x0
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/syscall/asm_linux_ppc64x.s:41 +0xc
May 25 21:17:40 redpower6 docker[3386]: syscall.wait4(0xd62, 0xc8204f7d90, 0x0, 0xc820166000, 0x90, 0x0, 0x0)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/syscall/zsyscall_linux_ppc64le.go:172 +0x68
May 25 21:17:40 redpower6 docker[3386]: syscall.Wait4(0xd62, 0xc8204f7df0, 0x0, 0xc820166000, 0x0, 0x0, 0x0)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/syscall/syscall_linux.go:256 +0x54
May 25 21:17:40 redpower6 docker[3386]: os.(*Process).wait(0xc82036ca60, 0x0, 0x0, 0x0)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/os/exec_unix.go:22 +0xe8
May 25 21:17:40 redpower6 docker[3386]: os.(*Process).Wait(0xc82036ca60, 0x0, 0x0, 0x0)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/os/doc.go:49 +0x34
May 25 21:17:40 redpower6 docker[3386]: os/exec.(*Cmd).Wait(0xc8203b2000, 0x0, 0x0)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/os/exec/exec.go:396 +0x208
May 25 21:17:40 redpower6 docker[3386]: created by github.com/docker/docker/libcontainerd.(*remote).runContaine
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/libcontainerd/remote_linux.go:
May 25 21:17:40 redpower6 docker[3386]: goroutine 206 [select, 649 minutes]:
May 25 21:17:40 redpower6 docker[3386]: google.golang.org/grpc.(*Conn).transportMonitor(0xc82019c000)
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/vendor/src/google.golang.org/g
May 25 21:17:40 redpower6 docker[3386]: google.golang.org/grpc.NewConn.func1(0xc82019c000)
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/vendor/src/google.golang.org/g
May 25 21:17:40 redpower6 docker[3386]: created by google.golang.org/grpc.NewConn
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/vendor/src/google.golang.org/g
May 25 21:17:40 redpower6 docker[3386]: goroutine 207 [semacquire, 649 minutes]:
May 25 21:17:40 redpower6 docker[3386]: sync.runtime_Syncsemacquire(0xc82033e590)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/runtime/sema.go:241 +0x25c
May 25 21:17:40 redpower6 docker[3386]: sync.(*Cond).Wait(0xc82033e580)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/sync/cond.go:63 +0xe4
May 25 21:17:40 redpower6 docker[3386]: google.golang.org/grpc.(*Conn).WaitForStateChange(0xc82019c000, 0x3fff8
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/vendor/src/google.golang.org/g
May 25 21:17:40 redpower6 docker[3386]: google.golang.org/grpc.(*unicastPicker).WaitForStateChange(0xc82036caa0
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/vendor/src/google.golang.org/g
May 25 21:17:40 redpower6 docker[3386]: google.golang.org/grpc.(*ClientConn).WaitForStateChange(0xc820160580, 0
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/vendor/src/google.golang.org/g
May 25 21:17:40 redpower6 docker[3386]: github.com/docker/docker/libcontainerd.(*remote).handleConnectionChange
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/libcontainerd/remote_linux.go:
May 25 21:17:40 redpower6 docker[3386]: created by github.com/docker/docker/libcontainerd.New
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/libcontainerd/remote_linux.go:
May 25 21:17:40 redpower6 docker[3386]: goroutine 6601 [semacquire, 637 minutes]:
May 25 21:17:40 redpower6 docker[3386]: sync.runtime_Syncsemacquire(0xc8257f8b80)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/runtime/sema.go:241 +0x25c
May 25 21:17:40 redpower6 docker[3386]: sync.(*Cond).Wait(0xc8257f8b70)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/sync/cond.go:63 +0xe4
May 25 21:17:40 redpower6 docker[3386]: io.(*pipe).read(0xc8257f8b40, 0xc827b8c000, 0x8000, 0x8000, 0x0, 0x0, 0
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/io/pipe.go:52 +0x354
May 25 21:17:40 redpower6 docker[3386]: io.(*PipeReader).Read(0xc8212682a0, 0xc827b8c000, 0x8000, 0x8000, 0x800
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/io/pipe.go:134 +0x50
May 25 21:17:40 redpower6 docker[3386]: io.copyBuffer(0x3fff8c487420, 0xc82852c7c0, 0x3fff8c483890, 0xc8212682a
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/io/io.go:380 +0x1dc
May 25 21:17:40 redpower6 docker[3386]: io.Copy(0x3fff8c487420, 0xc82852c7c0, 0x3fff8c483890, 0xc8212682a0, 0xc
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/io/io.go:350 +0x5c
May 25 21:17:40 redpower6 docker[3386]: github.com/docker/docker/daemon.(*Daemon).AttachStreams.func2.1(0x3fff8
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/daemon/monitor.go:135 +0x50
May 25 21:17:40 redpower6 docker[3386]: created by github.com/docker/docker/daemon.(*Daemon).AttachStreams.func
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/daemon/monitor.go:139 +0xac
May 25 21:17:40 redpower6 docker[3386]: goroutine 1585 [syscall, 643 minutes]:
May 25 21:17:40 redpower6 docker[3386]: syscall.Syscall(0x3, 0xc2, 0xc8223f0000, 0x8000, 0x1032ed4c, 0x1032ed20
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/syscall/asm_linux_ppc64x.s:17 +0xc
May 25 21:17:40 redpower6 docker[3386]: syscall.read(0xc2, 0xc8223f0000, 0x8000, 0x8000, 0x117084c0, 0x0, 0x0)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/syscall/zsyscall_linux_ppc64le.go:783 +0x70
May 25 21:17:40 redpower6 docker[3386]: syscall.Read(0xc2, 0xc8223f0000, 0x8000, 0x8000, 0x9b, 0x0, 0x0)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/syscall/syscall_unix.go:161 +0x4c
May 25 21:17:40 redpower6 docker[3386]: os.(*File).read(0xc8200f6278, 0xc8223f0000, 0x8000, 0x8000, 0xc8223f000
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/os/file_unix.go:228 +0x54
May 25 21:17:40 redpower6 docker[3386]: os.(*File).Read(0xc8200f6278, 0xc8223f0000, 0x8000, 0x8000, 0x9b, 0x0, 
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/os/file.go:95 +0x84
May 25 21:17:40 redpower6 docker[3386]: io.copyBuffer(0x3fff8c483840, 0xc8200f6268, 0x3fff8d504270, 0xc8200f627
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/io/io.go:380 +0x1dc
May 25 21:17:40 redpower6 docker[3386]: io.Copy(0x3fff8c483840, 0xc8200f6268, 0x3fff8d504270, 0xc8200f6278, 0xc
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/io/io.go:350 +0x5c
May 25 21:17:40 redpower6 docker[3386]: github.com/docker/docker/libcontainerd.openReaderFromFifo.func1(0xc8214
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/libcontainerd/process_linux.go
May 25 21:17:40 redpower6 docker[3386]: created by github.com/docker/docker/libcontainerd.openReaderFromFifo
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/libcontainerd/process_linux.go
May 25 21:17:40 redpower6 docker[3386]: goroutine 210 [select, 2 minutes]:
May 25 21:17:40 redpower6 docker[3386]: google.golang.org/grpc/transport.(*http2Client).controller(0xc8203520f0
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/vendor/src/google.golang.org/g
May 25 21:17:40 redpower6 docker[3386]: created by google.golang.org/grpc/transport.newHTTP2Client
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/vendor/src/google.golang.org/g
May 25 21:17:40 redpower6 docker[3386]: goroutine 211 [IO wait, 2 minutes]:
May 25 21:17:40 redpower6 docker[3386]: net.runtime_pollWait(0x3fff8d50ebd0, 0x72, 0x0)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/runtime/netpoll.go:160 +0x8c
May 25 21:17:40 redpower6 docker[3386]: net.(*pollDesc).Wait(0xc8204020d0, 0x72, 0x0, 0x0)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/net/fd_poll_runtime.go:73 +0x40
May 25 21:17:40 redpower6 docker[3386]: net.(*pollDesc).WaitRead(0xc8204020d0, 0x0, 0x0)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/net/fd_poll_runtime.go:78 +0x3c
May 25 21:17:40 redpower6 docker[3386]: net.(*netFD).Read(0xc820402070, 0xc8201662f4, 0x9, 0x9, 0x0, 0x3fff8d50
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/net/fd_unix.go:250 +0x20c
May 25 21:17:40 redpower6 docker[3386]: net.(*conn).Read(0xc8200f6000, 0xc8201662f4, 0x9, 0x9, 0x0, 0x0, 0x0)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/net/net.go:172 +0xec
May 25 21:17:40 redpower6 docker[3386]: io.ReadAtLeast(0x3fff8c240028, 0xc8200f6000, 0xc8201662f4, 0x9, 0x9, 0x
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/io/io.go:297 +0xec
May 25 21:17:40 redpower6 docker[3386]: io.ReadFull(0x3fff8c240028, 0xc8200f6000, 0xc8201662f4, 0x9, 0x9, 0xc89
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/io/io.go:315 +0x60
May 25 21:17:40 redpower6 docker[3386]: golang.org/x/net/http2.readFrameHeader(0xc8201662f4, 0x9, 0x9, 0x3fff8c
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/vendor/src/golang.org/x/net/ht
May 25 21:17:40 redpower6 docker[3386]: golang.org/x/net/http2.(*Framer).ReadFrame(0xc8201662d0, 0x0, 0x0, 0x0,
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/vendor/src/golang.org/x/net/ht
May 25 21:17:40 redpower6 docker[3386]: google.golang.org/grpc/transport.(*framer).readFrame(0xc82027c000, 0x0,
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/vendor/src/google.golang.org/g
May 25 21:17:40 redpower6 docker[3386]: google.golang.org/grpc/transport.(*http2Client).reader(0xc8203520f0)
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/vendor/src/google.golang.org/g
May 25 21:17:40 redpower6 docker[3386]: created by google.golang.org/grpc/transport.newHTTP2Client
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/vendor/src/google.golang.org/g
May 25 21:17:40 redpower6 docker[3386]: goroutine 101 [IO wait, 2 minutes]:
May 25 21:17:40 redpower6 docker[3386]: net.runtime_pollWait(0x3fff8d50ec90, 0x72, 0xc82000e130)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/runtime/netpoll.go:160 +0x8c
May 25 21:17:40 redpower6 docker[3386]: net.(*pollDesc).Wait(0xc820478060, 0x72, 0x0, 0x0)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/net/fd_poll_runtime.go:73 +0x40
May 25 21:17:40 redpower6 docker[3386]: net.(*pollDesc).WaitRead(0xc820478060, 0x0, 0x0)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/net/fd_poll_runtime.go:78 +0x3c
May 25 21:17:40 redpower6 docker[3386]: net.(*netFD).accept(0xc820478000, 0x0, 0x3fff8c4c2800, 0xc8c6b6f560)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/net/fd_unix.go:426 +0x278
May 25 21:17:40 redpower6 docker[3386]: net.(*UnixListener).AcceptUnix(0xc82036c720, 0xc8202d5d80, 0x0, 0x0)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/net/unixsock_posix.go:305 +0x5c
May 25 21:17:40 redpower6 docker[3386]: net.(*UnixListener).Accept(0xc82036c720, 0x0, 0x0, 0x0, 0x0)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/net/unixsock_posix.go:315 +0x3c
May 25 21:17:40 redpower6 docker[3386]: net/http.(*Server).Serve(0xc820160400, 0x3fff8d514868, 0xc82036c720, 0x
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/net/http/server.go:2117 +0x134
May 25 21:17:40 redpower6 docker[3386]: github.com/docker/docker/api/server.(*HTTPServer).Serve(0xc82036c740, 0
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/api/server/server.go:112 +0x58
May 25 21:17:40 redpower6 docker[3386]: github.com/docker/docker/api/server.(*Server).serveAPI.func1(0xc8204320
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/api/server/server.go:85 +0x114
May 25 21:17:40 redpower6 docker[3386]: created by github.com/docker/docker/api/server.(*Server).serveAPI
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/api/server/server.go:89 +0x150
May 25 21:17:40 redpower6 docker[3386]: goroutine 213 [chan receive, 649 minutes]:
May 25 21:17:40 redpower6 docker[3386]: google.golang.org/grpc.NewClientStream.func1(0xc82036a0c0, 0xc8203fa180
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/vendor/src/google.golang.org/g
May 25 21:17:40 redpower6 docker[3386]: created by google.golang.org/grpc.NewClientStream
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/vendor/src/google.golang.org/g
May 25 21:17:40 redpower6 docker[3386]: goroutine 214 [select, 2 minutes]:
May 25 21:17:40 redpower6 docker[3386]: google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc82027c1e0,
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/vendor/src/google.golang.org/g
May 25 21:17:40 redpower6 docker[3386]: google.golang.org/grpc/transport.(*Stream).Read(0xc82036a0c0, 0xc8b0bde
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/vendor/src/google.golang.org/g
May 25 21:17:40 redpower6 docker[3386]: io.ReadAtLeast(0x3fff8c240218, 0xc82036a0c0, 0xc8b0bde9d5, 0x5, 0x5, 0x
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/io/io.go:297 +0xec
May 25 21:17:40 redpower6 docker[3386]: io.ReadFull(0x3fff8c240218, 0xc82036a0c0, 0xc8b0bde9d5, 0x5, 0x5, 0xc82
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/io/io.go:315 +0x60
May 25 21:17:40 redpower6 docker[3386]: google.golang.org/grpc.(*parser).recvMsg(0xc8203ba110, 0x4, 0x0, 0x0, 0
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/vendor/src/google.golang.org/g
May 25 21:17:40 redpower6 docker[3386]: google.golang.org/grpc.recv(0xc8203ba110, 0x3fff8d5149e8, 0x11cb11a8, 0
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/vendor/src/google.golang.org/g
May 25 21:17:40 redpower6 docker[3386]: google.golang.org/grpc.(*clientStream).RecvMsg(0xc8203fa180, 0x11236020
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/vendor/src/google.golang.org/g
May 25 21:17:40 redpower6 docker[3386]: github.com/docker/containerd/api/grpc/types.(*aPIEventsClient).Recv(0xc
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/vendor/src/github.com/docker/c
May 25 21:17:40 redpower6 docker[3386]: github.com/docker/docker/libcontainerd.(*remote).handleEventStream(0xc8
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/libcontainerd/remote_linux.go:
May 25 21:17:40 redpower6 docker[3386]: created by github.com/docker/docker/libcontainerd.(*remote).startEvents
May 25 21:17:40 redpower6 docker[3386]:         /go/src/github.com/docker/docker/libcontainerd/remote_linux.go:
May 25 21:17:40 redpower6 docker[3386]: goroutine 215 [select, 649 minutes, locked to thread]:
May 25 21:17:40 redpower6 docker[3386]: runtime.gopark(0x11708388, 0xc820116f00, 0x113cf020, 0x6, 0x18, 0x2)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/runtime/proc.go:262 +0x19c
May 25 21:17:40 redpower6 docker[3386]: runtime.selectgoImpl(0xc820116f00, 0x0, 0x18)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/runtime/select.go:392 +0xb2c
May 25 21:17:40 redpower6 docker[3386]: runtime.selectgo(0xc820116f00)
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/runtime/select.go:215 +0x14
May 25 21:17:40 redpower6 docker[3386]: runtime.ensureSigM.func1()
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/runtime/signal1_unix.go:279 +0x530
May 25 21:17:40 redpower6 docker[3386]: runtime.goexit()
May 25 21:17:40 redpower6 docker[3386]:         /usr/src/go/src/runtime/asm_ppc64x.s:1049 +0x4

Describe the results you expected:
I am trying to launch 10k containers on one docker host, I was able to run 10k busybox containers, but with production image, the docker daemon crashed when launched 9k+ containers.

Additional information you deem important (e.g. issue happens only occasionally):

may be related to #22690

Metadata

Metadata

Assignees

No one assigned

    Labels

    version/master

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions