Skip to content

docker 1.12 daemon crashed after launched 40+ busybox container #22690

Closed
Closed
docker 1.12 daemon crashed after launched 40+ busybox container#22690
@ligc

Description

@ligc
ligc
opened on May 12, 2016

Output of docker version:

Client:
 Version:      1.12.0-dev
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   c273163
 Built:        Thu May 12 04:16:04 2016
 OS/Arch:      linux/ppc64le

Server:
 Version:      1.12.0-dev
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   c273163
 Built:        Thu May 12 04:16:04 2016
 OS/Arch:      linux/ppc64le

Output of docker info:

Containers: 125
 Running: 42
 Paused: 0
 Stopped: 83
Images: 1
Server Version: 1.12.0-dev
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 251
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 4.4.0-22-generic
Operating System: Ubuntu 16.04 LTS
OSType: linux
Architecture: ppc64le
CPUs: 160
Total Memory: 255.4 GiB
Name: redpower6
ID: DYHZ:MCJC:ADPZ:KPEM:JNAJ:CIKK:V345:IKHN:FS6J:BR77:635V:NZRZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):

Steps to reproduce the issue:

  1. Get the ppc64le master binary from https://master.dockerproject.org/
  2. Test script

The purpose of this script is to run 10000 busybox docker instances on a baremetal machine.

#!/bin/bash

for i in `seq 1 10`; do
    for j in `seq 1 1000`; do
        let "z=($i-1)*1000+$j"
        echo $z
        docker run -d --net=none ppc64le/busybox sleep 100d > /dev/null
    done
done
  1. Run the test script

Describe the results you received:

After launched 40+ containers, the docker daemon crashed and could not launch any new containers.

root@redpower6:~# ./test
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
docker: Error response from daemon: rpc error: code = 2 desc = "oci runtime error: process_linux.go:329: running prestart hook 0 caused \"exit status 2: runtime/cgo: pthread_create failed: Resource temporarily unavailable\\nSIGABRT: abort\\nPC=0x10cca678 m=11\\n\\ngoroutine 0 [idle]:\\n\\ngoroutine 1 [runnable]:\\nsort.Sort(0x3fff80075800, 0xc820558000)\\n\\t/usr/src/go/src/sort/sort.go:221\\nencoding/json.typeFields(0x3fff820b0050, 0x110fc2a0, 0x0, 0x0, 0x0)\\n\\t/usr/src/go/src/encoding/json/encode.go:1097 +0xce8\\nencoding/json.cachedTypeFields(0x3fff820b0050, 0x110fc2a0, 0x0, 0x0, 0x0)\\n\\t/usr/src/go/src/encoding/json/encode.go:1187 +0xf4\\nencoding/json.(*decodeState).object(0xc820524000, 0x110fc2a0, 0xc820522000, 0x199)\\n\\t/usr/src/go/src/encoding/json/decode.go:639 +0xeec\\nencoding/json.(*decodeState).value(0xc820524000, 0x10d3a960, 0xc820522000, 0x16)\\n\\t/usr/src/go/src/encoding/json/decode.go:367 +0x430\\nencoding/json.(*decodeState).unmarshal(0xc820524000, 0x10d3a960, 0xc820522000, 0x0, 0x0)\\n\\t/usr/src/go/src/encoding/json/decode.go:168 +0x17c\\nencoding/json.Unmarshal(0xc82048e000, 0x13a, 0x600, 0x10d3a960, 0xc820522000, 0x0, 0x0)\\n\\t/usr/src/go/src/encoding/json/decode.go:96 +0x120\\ngithub.com/docker/libnetwork.processSetKeyReexec()\\n\\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/sandbox_externalkey_unix.go:48 +0x2b4\\ngithub.com/docker/docker/pkg/reexec.Init(0xc820352e80)\\n\\t/go/src/github.com/docker/docker/pkg/reexec/reexec.go:26 +0xb0\\nmain.main()\\n\\t/go/src/github.com/docker/docker/cmd/dockerd/docker.go:22 +0x30\\n\\ngoroutine 17 [syscall, locked to thread]:\\nruntime.goexit()\\n\\t/usr/src/go/src/runtime/asm_ppc64x.s:1049 +0x4\\n\\ngoroutine 6 [syscall]:\\nos/signal.signal_recv(0x0)\\n\\t/usr/src/go/src/runtime/sigqueue.go:116 +0x270\\nos/signal.loop()\\n\\t/usr/src/go/src/os/signal/signal_unix.go:22 +0x24\\ncreated by os/signal.init.1\\n\\t/usr/src/go/src/os/signal/signal_unix.go:28 +0x44\\n\\nr0   0xfa\\tr1   0x3fff79f3df60\\nr2   0x11a7cbb0\\tr3   0x0\\nr4   0x16f4\\tr5   0x6\\nr6   0x4000\\tr7   0x11a9cbb0\\nr8   0x0\\tr9   0x0\\nr10  0x0\\tr11  0x0\\nr12  0x0\\tr13  0x3fff79f46790\\nr14  0x0\\tr15  0x8\\nr16  0x0\\tr17  0x0\\nr18  0x180\\tr19  0x3fff7a77e280\\nr20  0x800000\\tr21  0x3fff79f3f100\\nr22  0x3fff79f3f790\\tr23  0x11a98b78\\nr24  0x0\\tr25  0x0\\nr26  0x0\\tr27  0x3fff7a77e240\\nr28  0x3fff79f3e950\\tr29  0x3fff700008c0\\nr30  0x3fff79f3f040\\tr31  0x6\\npc   0x10cca678\\tctr  0x0\\nlink 0x10c769bc\\txer  0x0\\nccr  0x44400424\\ttrap 0xc00\\n\"".
44
docker: Error response from daemon: rpc error: code = 2 desc = "oci runtime error: process_linux.go:329: running prestart hook 0 caused \"exit status 2: runtime/cgo: pthread_create failed: Resource temporarily unavailable\\nSIGABRT: abort\\nPC=0x10cca678 m=11\\n\\ngoroutine 0 [idle]:\\n\\ngoroutine 1 [runnable, locked to thread]:\\nregexp.onePassCopy(0xc82048a450, 0x0)\\n\\t/usr/src/go/src/regexp/onepass.go:227 +0x188\\nregexp.compileOnePass(0xc82048a450, 0xc82048a450)\\n\\t/usr/src/go/src/regexp/onepass.go:504 +0x2b4\\nregexp.compile(0x1135e2a0, 0x10, 0x3fff760000d4, 0x114186a0, 0x0, 0x0)\\n\\t/usr/src/go/src/regexp/regexp.go:179 +0x120\\nregexp.Compile(0x1135e2a0, 0x10, 0x10284fd0, 0x0, 0x0)\\n\\t/usr/src/go/src/regexp/regexp.go:129 +0x50\\nregexp.MustCompile(0x1135e2a0, 0x10, 0x3fff76d00000)\\n\\t/usr/src/go/src/regexp/regexp.go:230 +0x38\\ngoogle.golang.org/api/googleapi.init()\\n\\t/go/src/github.com/docker/docker/vendor/src/google.golang.org/api/googleapi/googleapi.go:358 +0x110\\ngoogle.golang.org/api/logging/v1beta3.init()\\n\\t/go/src/github.com/docker/docker/vendor/src/google.golang.org/api/logging/v1beta3/logging-gen.go:4787 +0x90\\ngoogle.golang.org/cloud/logging.init()\\n\\t/go/src/github.com/docker/docker/vendor/src/google.golang.org/cloud/logging/logging.go:468 +0x8c\\ngithub.com/docker/docker/daemon/logger/gcplogs.init()\\n\\t/go/src/github.com/docker/docker/daemon/logger/gcplogs/gcplogging.go:191 +0x94\\ngithub.com/docker/docker/daemon.init()\\n\\t/go/src/github.com/docker/docker/daemon/wait.go:17 +0x244\\nmain.init()\\n\\t/go/src/github.com/docker/docker/cmd/dockerd/service_unsupported.go:7 +0xd4\\n\\ngoroutine 17 [syscall, locked to thread]:\\nruntime.goexit()\\n\\t/usr/src/go/src/runtime/asm_ppc64x.s:1049 +0x4\\n\\ngoroutine 21 [syscall]:\\nos/signal.signal_recv(0x0)\\n\\t/usr/src/go/src/runtime/sigqueue.go:116 +0x270\\nos/signal.loop()\\n\\t/usr/src/go/src/os/signal/signal_unix.go:22 +0x24\\ncreated by os/signal.init.1\\n\\t/usr/src/go/src/os/signal/signal_unix.go:28 +0x44\\n\\nr0   0xfa\\tr1   0x3fff5fffdf60\\nr2   0x11a7cbb0\\tr3   0x0\\nr4   0x172e\\tr5   0x6\\nr6   0x4000\\tr7   0x11a9cbb0\\nr8   0x0\\tr9   0x0\\nr10  0x0\\tr11  0x0\\nr12  0x0\\tr13  0x3fff60006790\\nr14  0x0\\tr15  0x8\\nr16  0x0\\tr17  0x0\\nr18  0x180\\tr19  0x3fff74ade280\\nr20  0x800000\\tr21  0x3fff5ffff100\\nr22  0x3fff5ffff790\\tr23  0x11a98b78\\nr24  0x0\\tr25  0x0\\nr26  0x0\\tr27  0x3fff74ade240\\nr28  0x3fff5fffe950\\tr29  0x3fff580008c0\\nr30  0x3fff5ffff040\\tr31  0x6\\npc   0x10cca678\\tctr  0x0\\nlink 0x10c769bc\\txer  0x0\\nccr  0x44400424\\ttrap 0xc00\\n\"".
45
docker: Error response from daemon: rpc error: code = 2 desc = "oci runtime error: process_linux.go:329: running prestart hook 0 caused \"exit status 2: runtime/cgo: pthread_create failed: Resource temporarily unavailable\\nSIGABRT: abort\\nPC=0x10cca678 m=12\\n\\ngoroutine 0 [idle]:\\n\\ngoroutine 1 [runnable]:\\nreflect.StructTag.Get(0x1128b1f0, 0xb, 0x11263750, 0x4, 0x0, 0x0)\\n\\t/usr/src/go/src/reflect/type.go:765\\nencoding/json.typeFields(0x3fffaa3d0050, 0x110fc2a0, 0x0, 0x0, 0x0)\\n\\t/usr/src/go/src/encoding/json/encode.go:1033 +0x3a8\\nencoding/json.cachedTypeFields(0x3fffaa3d0050, 0x110fc2a0, 0x0, 0x0, 0x0)\\n\\t/usr/src/go/src/encoding/json/encode.go:1187 +0xf4\\nencoding/json.(*decodeState).object(0xc82052a000, 0x110fc2a0, 0xc820528000, 0x199)\\n\\t/usr/src/go/src/encoding/json/decode.go:639 +0xeec\\nencoding/json.(*decodeState).value(0xc82052a000, 0x10d3a960, 0xc820528000, 0x16)\\n\\t/usr/src/go/src/encoding/json/decode.go:367 +0x430\\nencoding/json.(*decodeState).unmarshal(0xc82052a000, 0x10d3a960, 0xc820528000, 0x0, 0x0)\\n\\t/usr/src/go/src/encoding/json/decode.go:168 +0x17c\\nencoding/json.Unmarshal(0xc82048e000, 0x13a, 0x600, 0x10d3a960, 0xc820528000, 0x0, 0x0)\\n\\t/usr/src/go/src/encoding/json/decode.go:96 +0x120\\ngithub.com/docker/libnetwork.processSetKeyReexec()\\n\\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/sandbox_externalkey_unix.go:48 +0x2b4\\ngithub.com/docker/docker/pkg/reexec.Init(0xc820352e80)\\n\\t/go/src/github.com/docker/docker/pkg/reexec/reexec.go:26 +0xb0\\nmain.main()\\n\\t/go/src/github.com/docker/docker/cmd/dockerd/docker.go:22 +0x30\\n\\ngoroutine 17 [syscall, locked to thread]:\\nruntime.goexit()\\n\\t/usr/src/go/src/runtime/asm_ppc64x.s:1049 +0x4\\n\\ngoroutine 6 [syscall]:\\nos/signal.signal_recv(0x0)\\n\\t/usr/src/go/src/runtime/sigqueue.go:116 +0x270\\nos/signal.loop()\\n\\t/usr/src/go/src/os/signal/signal_unix.go:22 +0x24\\ncreated by os/signal.init.1\\n\\t/usr/src/go/src/os/signal/signal_unix.go:28 +0x44\\n\\nr0   0xfa\\tr1   0x3fffa5a8df60\\nr2   0x11a7cbb0\\tr3   0x0\\nr4   0x176a\\tr5   0x6\\nr6   0x4000\\tr7   0x11a9cbb0\\nr8   0x0\\tr9   0x0\\nr10  0x0\\tr11  0x0\\nr12  0x0\\tr13  0x3fffa5a96790\\nr14  0x0\\tr15  0x8\\nr16  0x0\\tr17  0x0\\nr18  0x180\\tr19  0x3fffa62ce280\\nr20  0x800000\\tr21  0x3fffa5a8f100\\nr22  0x3fffa5a8f790\\tr23  0x11a98b78\\nr24  0x0\\tr25  0x0\\nr26  0x0\\tr27  0x3fffa62ce240\\nr28  0x3fffa5a8e950\\tr29  0x3fff900008c0\\nr30  0x3fffa5a8f040\\tr31  0x6\\npc   0x10cca678\\tctr  0x0\\nlink 0x10c769bc\\txer  0x0\\nccr  0x44800424\\ttrap 0xc00\\n\"".
46
docker: Error response from daemon: rpc error: code = 2 desc = "oci runtime error: process_linux.go:329: running prestart hook 0 caused \"exit status 2: runtime/cgo: pthread_create failed: Resource temporarily unavailable\\nSIGABRT: abort\\nPC=0x10cca678 m=10\\n\\ngoroutine 0 [idle]:\\n\\ngoroutine 1 [runnable, locked to thread]:\\nunicode.SimpleFold(0xc80000002f, 0xc82048e5f0)\\n\\t/usr/src/go/src/unicode/letter.go:334\\nregexp/syntax.(*compiler).rune(0xc820367a60, 0xc82048e5f0, 0x1, 0x2, 0x11550000, 0x0)\\n\\t/usr/src/go/src/regexp/syntax/compile.go:271 +0x268\\nregexp/syntax.(*compiler).compile(0xc820367a60, 0xc82048e5b0, 0x0)\\n\\t/usr/src/go/src/regexp/syntax/compile.go:110 +0x598\\nregexp/syntax.(*compiler).compile(0xc820367a60, 0xc82048e930, 0x0)\\n\\t/usr/src/go/src/regexp/syntax/compile.go:156 +0x1cdc\\nregexp/syntax.Compile(0xc82048e930, 0xc82048e930, 0x0, 0x0)\\n\\t/usr/src/go/src/regexp/syntax/compile.go:83 +0x1c8\\nregexp.compile(0x113ce860, 0x24, 0xd4, 0x3f9, 0x0, 0x0)\\n\\t/usr/src/go/src/regexp/regexp.go:172 +0xd8\\nregexp.Compile(0x113ce860, 0x24, 0xc820367c58, 0x0, 0x0)\\n\\t/usr/src/go/src/regexp/regexp.go:129 +0x50\\nregexp.MustCompile(0x113ce860, 0x24, 0x11268468)\\n\\t/usr/src/go/src/regexp/regexp.go:230 +0x38\\ngithub.com/docker/distribution/registry/client/transport.init()\\n\\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/distribution/registry/client/transport/http_reader.go:14 +0xa8\\ngithub.com/docker/distribution/registry/client.init()\\n\\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/distribution/registry/client/repository.go:832 +0xb4\\ngithub.com/docker/distribution/registry/client/auth.init()\\n\\t/go/src/github.com/docker/docker/vendor/src/github.com/docker/distribution/registry/client/auth/session.go:480 +0x98\\ngithub.com/docker/docker/registry.init()\\n\\t/go/src/github.com/docker/docker/registry/types.go:70 +0x98\\ngithub.com/docker/docker/daemon.init()\\n\\t/go/src/github.com/docker/docker/daemon/wait.go:17 +0x10c\\nmain.init()\\n\\t/go/src/github.com/docker/docker/cmd/dockerd/service_unsupported.go:7 +0xd4\\n\\ngoroutine 17 [syscall, locked to thread]:\\nruntime.goexit()\\n\\t/usr/src/go/src/runtime/asm_ppc64x.s:1049 +0x4\\n\\ngoroutine 6 [syscall]:\\nos/signal.signal_recv(0x0)\\n\\t/usr/src/go/src/runtime/sigqueue.go:116 +0x270\\nos/signal.loop()\\n\\t/usr/src/go/src/os/signal/signal_unix.go:22 +0x24\\ncreated by os/signal.init.1\\n\\t/usr/src/go/src/os/signal/signal_unix.go:28 +0x44\\n\\nr0   0xfa\\tr1   0x3fff92f7df60\\nr2   0x11a7cbb0\\tr3   0x0\\nr4   0x17a5\\tr5   0x6\\nr6   0x4000\\tr7   0x11a9cbb0\\nr8   0x0\\tr9   0x0\\nr10  0x0\\tr11  0x0\\nr12  0x0\\tr13  0x3fff92f86790\\nr14  0x0\\tr15  0x8\\nr16  0x0\\tr17  0x0\\nr18  0x180\\tr19  0x3fff9921e210\\nr20  0x800000\\tr21  0x3fff92f7f100\\nr22  0x3fff92f7f790\\tr23  0x11a98b78\\nr24  0x0\\tr25  0x0\\nr26  0x0\\tr27  0x3fff9921e1d0\\nr28  0x3fff92f7e950\\tr29  0x3fff880008c0\\nr30  0x3fff92f7f040\\tr31  0x6\\npc   0x10cca678\\tctr  0x0\\nlink 0x10c769bc\\txer  0x0\\nccr  0x44800424\\ttrap 0xc00\\n\"".
47
docker: Error response from daemon: rpc error: code = 2 desc = "oci runtime error: process_linux.go:329: running prestart hook 0 caused \"exit status 2: runtime/cgo: pthread_create failed: Resource temporarily unavailable\\nSIGABRT: abort\\nPC=0x10cca678 m=9\\n\\ngoroutine 0 [idle]:\\n\\ngoroutine 1 [runnable, locked to thread]:\\nregexp.compile(0x114186a0, 0x2b, 0x100000d4, 0xc820447360, 0x0, 0x0)\\n\\t/usr/src/go/src/regexp/regexp.go:183 +0x14c\\nregexp.Compile(0x114186a0, 0x2b, 0xc820447360, 0x0, 0x0)\\n\\t/usr/src/go/src/regexp/regexp.go:129 +0x50\\nregexp.MustCompile(0x114186a0, 0x2b, 0xc820447360)\\n\\t/usr/src/go/src/regexp/regexp.go:230 +0x38\\ngoogle.golang.org/api/googleapi/internal/uritemplates.init()\\n\\t/go/src/github.com/docker/docker/vendor/src/google.golang.org/api/googleapi/internal/uritemplates/uritemplates.go:33 +0x10c\\ngoogle.golang.org/api/googleapi.init()\\n\\t/go/src/github.com/docker/docker/vendor/src/google.golang.org/api/googleapi/types.go:182 +0xb8\\ngoogle.golang.org/api/logging/v1beta3.init()\\n\\t/go/src/github.com/docker/docker/vendor/src/google.golang.org/api/logging/v1beta3/logging-gen.go:4787 +0x90\\ngoogle.golang.org/cloud/logging.init()\\n\\t/go/src/github.com/docker/docker/vendor/src/google.golang.org/cloud/logging/logging.go:468 +0x8c\\ngithub.com/docker/docker/daemon/logger/gcplogs.init()\\n\\t/go/src/github.com/docker/docker/daemon/logger/gcplogs/gcplogging.go:191 +0x94\\ngithub.com/docker/docker/daemon.init()\\n\\t/go/src/github.com/docker/docker/daemon/wait.go:17 +0x244\\nmain.init()\\n\\t/go/src/github.com/docker/docker/cmd/dockerd/service_unsupported.go:7 +0xd4\\n\\ngoroutine 17 [syscall, locked to thread]:\\nruntime.goexit()\\n\\t/usr/src/go/src/runtime/asm_ppc64x.s:1049 +0x4\\n\\ngoroutine 21 [syscall]:\\nos/signal.signal_recv(0x0)\\n\\t/usr/src/go/src/runtime/sigqueue.go:116 +0x270\\nos/signal.loop()\\n\\t/usr/src/go/src/os/signal/signal_unix.go:22 +0x24\\ncreated by os/signal.init.1\\n\\t/usr/src/go/src/os/signal/signal_unix.go:28 +0x44\\n\\nr0   0xfa\\tr1   0x3fff78b3df60\\nr2   0x11a7cbb0\\tr3   0x0\\nr4   0x17de\\tr5   0x6\\nr6   0x4000\\tr7   0x11a9cbb0\\nr8   0x0\\tr9   0x0\\nr10  0x0\\tr11  0x0\\nr12  0x0\\tr13  0x3fff78b46790\\nr14  0x0\\tr15  0x8\\nr16  0x0\\tr17  0x0\\nr18  0x180\\tr19  0x3fff7933e280\\nr20  0x800000\\tr21  0x3fff78b3f100\\nr22  0x3fff78b3f790\\tr23  0x11a98b78\\nr24  0x0\\tr25  0x0\\nr26  0x0\\tr27  0x3fff7933e240\\nr28  0x3fff78b3e950\\tr29  0x3fff640008c0\\nr30  0x3fff78b3f040\\tr31  0x6\\npc   0x10cca678\\tctr  0x0\\nlink 0x10c769bc\\txer  0x0\\nccr  0x44400424\\ttrap 0xc00\\n\"".
48
docker: Error response from daemon: rpc error: code = 2 desc = "containerd: container not started".
49
docker: Error response from daemon: rpc error: code = 2 desc = "containerd: container not started".
50
docker: Error response from daemon: rpc error: code = 2 desc = "containerd: container not started".
51
docker: Error response from daemon: rpc error: code = 2 desc = "oci runtime error: process_linux.go:234: starting init process command caused \"fork/exec /proc/self/exe: resource temporarily unavailable\"".
52
docker: Error response from daemon: rpc error: code = 2 desc = "containerd: container not started".
53
docker: Error response from daemon: rpc error: code = 2 desc = "containerd: container not started".
54
docker: Error response from daemon: rpc error: code = 2 desc = "containerd: container not started".
55
docker: Error response from daemon: rpc error: code = 2 desc = "containerd: container not started".
56
docker: Error response from daemon: rpc error: code = 2 desc = "containerd: container not started".
57
^Cdocker: Error response from daemon: rpc error: code = 2 desc = "oci runtime error: process_linux.go:234: starting init process command caused \"fork/exec /proc/self/exe: resource temporarily unavailable\"".
58
docker: Error response from daemon: rpc error: code = 2 desc = "containerd: container not started".
59
docker: Error response from daemon: rpc error: code = 2 desc = "containerd: container not started".
60
docker: Error response from daemon: rpc error: code = 2 desc = "containerd: container not started".
61
docker: Error response from daemon: rpc error: code = 2 desc = "containerd: container not started".
62
docker: Error response from daemon: rpc error: code = 2 desc = "containerd: container not started".
^C
root@redpower6:~# ^C

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions