Skip to content

could not launch any new docker container after 1850 containers launched #23332

@ligc

Description

@ligc

Output of docker version:

Client:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 22:00:43 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 22:00:43 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 1918
 Running: 1853
 Paused: 0
 Stopped: 65
Images: 2
Server Version: 1.11.2
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 3858
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 4.4.0-21-generic
Operating System: Ubuntu 16.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 188.7 GiB
Name: x3650m4n08
ID: WL45:4OZ5:5ANE:LX2U:HRYA:DEKQ:DNE6:2HY6:CK3E:HJ7I:MAM5:JY6P
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):

Steps to reproduce the issue:

  1. docker pull yeasy/hyperledger-peer:noops
  2. Run the test script as shown below, the script sets the related limits to be bigger numbers and tries to launch 10000 docker containers.
#!/bin/bash

# Prerequisites setup

echo "root soft nofile 1048576" > /etc/security/limits.conf
echo "root hard nofile 1048576" >> /etc/security/limits.conf
echo "root soft nproc 10485760" >> /etc/security/limits.conf
echo "root hard nproc 10485760" >> /etc/security/limits.conf
echo "root soft stack 32768" >> /etc/security/limits.conf
echo "root hard stack 32768" >> /etc/security/limits.conf
echo "kernel.threads-max = 2091845" >> /etc/sysctl.conf
sysctl -p
echo 210000 > /proc/sys/kernel/pty/max
mkdir -p /etc/systemd/system/docker.service.d/

tee /etc/systemd/system/docker.service.d/tasksmax.conf <<-'EOF'
[Service]
TasksMax=infinity
EOF
systemctl daemon-reload
systemctl restart docker

for i in `seq 1 10`;
do
    brctl addbr docker$i
    ifconfig docker$i 172.$i.254.254/16 up
    docker network create --driver=bridge --ip-range=172.$i.0.0/16 --subnet=172.$i.0.0/16 -o "com.docker.network.bridge.name=docker$i" docker$i
done

for i in `seq 1 10`; do
    for j in `seq 1 1000`; do
        let "z=($i-1)*1000+$j"
        echo $z
        docker run --restart=unless-stopped --net=docker$i -d -e CORE_PEER_ID=vp0 -e CORE_VM_ENDPOINT=http://172.$i.254.254:2375 -e CORE_PEER_ADDRESSAUTODETECT=true -e CORE_NOOPS_BLOCK_TIMEOUT=10 yeasy/hyperledger-peer:noops > /dev/null
    done
done
  1. When launches about 1850 containers, will get "pthread_create failed" error and could not launch any new containers.

Describe the results you received:

The output from the script:

1844
1845
1846
1847
1848
1849
1850
1851
1852
1853
1854
docker: Error response from daemon: rpc error: code = 2 desc = "oci runtime error: exit status 2: runtime/cgo: pthread_create failed: Resource temporarily unavailable\nruntime/cgo: pthread_create failed: Resource temporarily unavailable\nSIGABRT: abort\nPC=0x7f9834314418 m=0\n\ngoroutine 0 [idle]:\n\ngoroutine 1 [runnable, locked to thread]:\nruntime.lock(0xc82006c058)\n\t/usr/local/go/src/runtime/lock_futex.go:42 fp=0xc82005fe38 sp=0xc82005fe30\nruntime.chanrecv(0x120dee0, 0xc82006c000, 0x0, 0xc82005ff01, 0x420000)\n\t/usr/local/go/src/runtime/chan.go:401 +0x148 fp=0xc82005fef8 sp=0xc82005fe38\nruntime.chanrecv1(0x120dee0, 0xc82006c000, 0x0)\n\t/usr/local/go/src/runtime/chan.go:349 +0x2b fp=0xc82005ff28 sp=0xc82005fef8\nruntime.gcenable()\n\t/usr/local/go/src/runtime/mgc.go:207 +0x76 fp=0xc82005ff50 sp=0xc82005ff28\nruntime.main()\n\t/usr/local/go/src/runtime/proc.go:73 +0x138 fp=0xc82005ffa0 sp=0xc82005ff50\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1721 +0x1 fp=0xc82005ffa8 sp=0xc82005ffa0\n\ngoroutine 17 [syscall, locked to thread]:\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1721 +0x1\n\nrax    0x0\nrbx    0x7f98346a3700\nrcx    0x7f9834314418\nrdx    0x6\nrdi    0xa20\nrsi    0xa20\nrbp    0x1ad238a\nrsp    0x7fff3bd65218\nr8     0x7f98346a4770\nr9     0x7f98352e5880\nr10    0x8\nr11    0x202\nr12    0x2897320\nr13    0x1a79070\nr14    0x0\nr15    0x8\nrip    0x7f9834314418\nrflags 0x202\ncs     0x33\nfs     0x0\ngs     0x0\n".
1855
docker: Error response from daemon: rpc error: code = 2 desc = "containerd: container not started".
1856

Describe the results you expected:
As indicated in #22978, I could launch more than 9000 containers with the same docker image(different arch of course) and similar hardware configuration on ppc64le platform, I expect to be able to launch similar number of docker containers on x86 also.

Additional information you deem important (e.g. issue happens only occasionally):

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions