Output of docker version:
Client:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 22:00:43 2016
OS/Arch: linux/amd64
Server:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 22:00:43 2016
OS/Arch: linux/amd64
Output of docker info:
Containers: 1918
Running: 1853
Paused: 0
Stopped: 65
Images: 2
Server Version: 1.11.2
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 3858
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge null host
Kernel Version: 4.4.0-21-generic
Operating System: Ubuntu 16.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 188.7 GiB
Name: x3650m4n08
ID: WL45:4OZ5:5ANE:LX2U:HRYA:DEKQ:DNE6:2HY6:CK3E:HJ7I:MAM5:JY6P
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Additional environment details (AWS, VirtualBox, physical, etc.):
Steps to reproduce the issue:
- docker pull yeasy/hyperledger-peer:noops
- Run the test script as shown below, the script sets the related limits to be bigger numbers and tries to launch 10000 docker containers.
#!/bin/bash
# Prerequisites setup
echo "root soft nofile 1048576" > /etc/security/limits.conf
echo "root hard nofile 1048576" >> /etc/security/limits.conf
echo "root soft nproc 10485760" >> /etc/security/limits.conf
echo "root hard nproc 10485760" >> /etc/security/limits.conf
echo "root soft stack 32768" >> /etc/security/limits.conf
echo "root hard stack 32768" >> /etc/security/limits.conf
echo "kernel.threads-max = 2091845" >> /etc/sysctl.conf
sysctl -p
echo 210000 > /proc/sys/kernel/pty/max
mkdir -p /etc/systemd/system/docker.service.d/
tee /etc/systemd/system/docker.service.d/tasksmax.conf <<-'EOF'
[Service]
TasksMax=infinity
EOF
systemctl daemon-reload
systemctl restart docker
for i in `seq 1 10`;
do
brctl addbr docker$i
ifconfig docker$i 172.$i.254.254/16 up
docker network create --driver=bridge --ip-range=172.$i.0.0/16 --subnet=172.$i.0.0/16 -o "com.docker.network.bridge.name=docker$i" docker$i
done
for i in `seq 1 10`; do
for j in `seq 1 1000`; do
let "z=($i-1)*1000+$j"
echo $z
docker run --restart=unless-stopped --net=docker$i -d -e CORE_PEER_ID=vp0 -e CORE_VM_ENDPOINT=http://172.$i.254.254:2375 -e CORE_PEER_ADDRESSAUTODETECT=true -e CORE_NOOPS_BLOCK_TIMEOUT=10 yeasy/hyperledger-peer:noops > /dev/null
done
done
- When launches about 1850 containers, will get "pthread_create failed" error and could not launch any new containers.
Describe the results you received:
The output from the script:
1844
1845
1846
1847
1848
1849
1850
1851
1852
1853
1854
docker: Error response from daemon: rpc error: code = 2 desc = "oci runtime error: exit status 2: runtime/cgo: pthread_create failed: Resource temporarily unavailable\nruntime/cgo: pthread_create failed: Resource temporarily unavailable\nSIGABRT: abort\nPC=0x7f9834314418 m=0\n\ngoroutine 0 [idle]:\n\ngoroutine 1 [runnable, locked to thread]:\nruntime.lock(0xc82006c058)\n\t/usr/local/go/src/runtime/lock_futex.go:42 fp=0xc82005fe38 sp=0xc82005fe30\nruntime.chanrecv(0x120dee0, 0xc82006c000, 0x0, 0xc82005ff01, 0x420000)\n\t/usr/local/go/src/runtime/chan.go:401 +0x148 fp=0xc82005fef8 sp=0xc82005fe38\nruntime.chanrecv1(0x120dee0, 0xc82006c000, 0x0)\n\t/usr/local/go/src/runtime/chan.go:349 +0x2b fp=0xc82005ff28 sp=0xc82005fef8\nruntime.gcenable()\n\t/usr/local/go/src/runtime/mgc.go:207 +0x76 fp=0xc82005ff50 sp=0xc82005ff28\nruntime.main()\n\t/usr/local/go/src/runtime/proc.go:73 +0x138 fp=0xc82005ffa0 sp=0xc82005ff50\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1721 +0x1 fp=0xc82005ffa8 sp=0xc82005ffa0\n\ngoroutine 17 [syscall, locked to thread]:\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1721 +0x1\n\nrax 0x0\nrbx 0x7f98346a3700\nrcx 0x7f9834314418\nrdx 0x6\nrdi 0xa20\nrsi 0xa20\nrbp 0x1ad238a\nrsp 0x7fff3bd65218\nr8 0x7f98346a4770\nr9 0x7f98352e5880\nr10 0x8\nr11 0x202\nr12 0x2897320\nr13 0x1a79070\nr14 0x0\nr15 0x8\nrip 0x7f9834314418\nrflags 0x202\ncs 0x33\nfs 0x0\ngs 0x0\n".
1855
docker: Error response from daemon: rpc error: code = 2 desc = "containerd: container not started".
1856
Describe the results you expected:
As indicated in #22978, I could launch more than 9000 containers with the same docker image(different arch of course) and similar hardware configuration on ppc64le platform, I expect to be able to launch similar number of docker containers on x86 also.
Additional information you deem important (e.g. issue happens only occasionally):
Output of
docker version:Output of
docker info:Additional environment details (AWS, VirtualBox, physical, etc.):
Steps to reproduce the issue:
Describe the results you received:
The output from the script:
Describe the results you expected:
As indicated in #22978, I could launch more than 9000 containers with the same docker image(different arch of course) and similar hardware configuration on ppc64le platform, I expect to be able to launch similar number of docker containers on x86 also.
Additional information you deem important (e.g. issue happens only occasionally):