Docker has been touted as the holy grail of on-premises software container solutions. Docker Swarm is the orchestration upgrade that allows you to scale your containers from a single host to many hosts, and from tens of containers into thousands of them.<\/p>\n
But does it deliver on that promise? Let\u2019s run 1,000 containers to find out.<\/p>\n
I\u2019m starting with a three-node cluster, running the latest version of Docker.<\/p>\n
root@swarm1:~# docker node ls\r\nID HOSTNAME STATUS AVAILABILITY MANAGER STATUS\r\n2jrjw0f0t6k2hbf2w0d41db2x * swarm1 Ready Active Reachable\r\naet822iruebj35gu4vlkn70bb swarm3 Ready Active Leader\r\nl4bdxu989ktteb61cte66dqbc swarm2 Ready Active Reachable<\/pre>\nWe will need to set up a multihost network. This network is the common communication channel between the containers, or any other containers. For example, if you would run a webserver and a database, you would put them on the same network, so they can talk to each other.<\/p>\n
Using Docker Engine running in Swarm mode, you can create an overlay network on a manager node. The swarm makes the overlay network available only to nodes in the swarm that require it for a service. When you create a service that uses an overlay network, the manager node automatically extends the overlay network to nodes that run service tasks.<\/p><\/blockquote>\n
In order for containers to be discoverable between hosts in a Docker swarm, we will need to create an
overlay<\/code> network on one of the manager nodes.<\/p>\n# docker network create \\\r\n --driver overlay \\\r\n --subnet 10.0.0.0\/20 \\\r\n --attachable \\\r\n party-swarm<\/pre>\nSince we can use a higher number of containers, we can declare a larger number of IPs available, using the CIDR notation<\/a>. A bit mask with 20 bits leaves 12 bits available for IPs \u2014 4,096 of them can be run on this network.<\/p>\n
The switch
--attachable<\/code> creates this network in a way that may be used by non-swarm containers. This provides access to the network from the classicdocker run<\/code> way of running your containers.<\/p>\nPrepare the Host<\/h2>\n
Due to the very large number of containers, which means an even larger number of system processes on your Docker host, we will need to tune some settings in order to avoid instability when running our service at such a scale.<\/p>\n
Various Linux subsystems, especially in regards to networking, have many exposed configuration flags that are managed by sysctl. The defaults of these flags are more suited for laptops than servers that would run thousands of processes.<\/p>\n
ARP cache<\/h3>\n
When I first tried to run 1,000 containers in a service, I hit a limitation that resulted in the Docker swarm becoming unresponsive, with occasional connectivity interruptions. When inspecting this, I found many messages like the following one in output of
dmesg<\/code>.<\/p>\n
neighbour: arp_cache: neighbor table overflow!<\/code><\/p>\nThe ARP cache is a look-up table that connects IP addresses (OSI Layer 3) to MAC addresses (OSI Layer 2). When we\u2019re running 1,000 containers and assigning them IPs, we will inevitably declare a large number of IP\/MAC address pairs. The issue with connectivity interruptions was caused by hitting the ARP cache limits, where garbage collection is performed.<\/p>\n
Keep in mind, when we start a service, the service will attach to three networks:<\/p>\n
\n
party-swarm<\/code> \u2013 the shared network we set up<\/li>\ningress<\/code> \u2013 where traffic comes from; we will forward a port to the service<\/li>\ndocker_gwbridge<\/code> \u2013 the public internet access network<\/li>\n<\/ul>\nSome of these networks will contain multiple IP addresses in each container. On my count, it means about 5,000 network addresses allocated for a 1,000 containers.<\/p>\n
A bit of digging around led me to some sysctl tunable settings which can be used to work around the problem, basically by increasing the thresholds where ARP cache collection occurs.<\/p>\n
Issue this on every Docker swarm host:<\/p>\n
sysctl -w net.ipv4.neigh.default.gc_thresh1=8096\r\nsysctl -w net.ipv4.neigh.default.gc_thresh2=12288\r\nsysctl -w net.ipv4.neigh.default.gc_thresh3=16384<\/pre>\nAnd this is what the values do:<\/p>\n
\n
gc_thresh1<\/code> \u2013 the minimum number of entries to keep in the ARP cache<\/li>\ngc_thresh2<\/code> \u2013 the soft maximum number of entries to keep in the ARP cache<\/li>\ngc_thresh3<\/code> \u2013 the hard maximum number of entries to keep in the ARP cache<\/li>\n<\/ul>\nThe operative value is
gc_thresh2<\/code>, as garbage collection there happens five seconds after the ARP cache crosses this many entries. The default value was 512, which caused very frequent garbage collection runs and related system instability.<\/p>\nThe reason why I chose this value is that I needed a sane default that would not be hit. Generally the recommendation is that you should multiply all the
gc_thresh*<\/code> values by 2 until your system will behave nicely. As is the case most of the time, the sysctl tunable settings are very poorly documented, so figuring out what kind of impact they have in terms of memory use or other pitfalls is possible only in the rarest occasions.<\/p>\nSysctl tunables<\/h3>\n
There are other sysctl tunable settings which might be useful for you:<\/p>\n
# Have a larger connection range available\r\nnet.ipv4.ip_local_port_range=1024 65000\r\n\r\n# Reuse closed sockets faster\r\nnet.ipv4.tcp_tw_reuse=1\r\nnet.ipv4.tcp_fin_timeout=15\r\n\r\n# The maximum number of \"backlogged sockets\". Default is 128.\r\nnet.core.somaxconn=4096\r\nnet.core.netdev_max_backlog=4096\r\n\r\n# 16MB per socket - which sounds like a lot,\r\n# but will virtually never consume that much.\r\nnet.core.rmem_max=16777216\r\nnet.core.wmem_max=16777216\r\n\r\n# Various network tunables\r\nnet.ipv4.tcp_max_syn_backlog=20480\r\nnet.ipv4.tcp_max_tw_buckets=400000\r\nnet.ipv4.tcp_no_metrics_save=1\r\nnet.ipv4.tcp_rmem=4096 87380 16777216\r\nnet.ipv4.tcp_syn_retries=2\r\nnet.ipv4.tcp_synack_retries=2\r\nnet.ipv4.tcp_wmem=4096 65536 16777216\r\n#vm.min_free_kbytes=65536\r\n\r\n# Connection tracking to prevent dropped connections (usually issue on LBs)\r\nnet.netfilter.nf_conntrack_max=262144\r\nnet.ipv4.netfilter.ip_conntrack_generic_timeout=120\r\nnet.netfilter.nf_conntrack_tcp_timeout_established=86400\r\n\r\n# ARP cache settings for a highly loaded docker swarm\r\nnet.ipv4.neigh.default.gc_thresh1=8096\r\nnet.ipv4.neigh.default.gc_thresh2=12288\r\nnet.ipv4.neigh.default.gc_thresh3=16384<\/pre>\nThese sysctl settings deal mainly with how many connections can be open at a given time, which ports are used for the connections, how fast connections are recycled\u2026 I can\u2019t imagine a high-traffic, load-balancer host running without at least some of these.<\/p>\n
!New Call-to-action<\/p>\n
Check Resource Usage<\/h2>\n
For our purposes, my Swarm cluster has three nodes, which each have six CPU cores and 8GB of RAM. I have a bit more than 100GB of disk space available, but if you\u2019re following along, 10GB of available disk space will be enough to run 1,000 containers.<\/p>\n
Calculating needed disk space<\/h3>\n
In order to calculate the theoretical capacity of your system for a single Docker container, we can look at the image size. A Docker image is created as a composite of many layers.<\/p>\n
# docker history titpetric\/sonyflake\r\nIMAGE INFO SIZE\r\naf387743bcd5 ENTRYPOINT [\"\/sonyflake\"] 0 B\r\nfdf6859f28f4 ADD file:62a837b350878b1d75 6.107 MB\r\n2a13f6fd8e6c MAINTAINER Tit Petric <blac 0 B\r\n88e169ea8f46 ADD file:92ab746eb22dd3ed2b 3.98 MB<\/pre>\nFor example, the image I\u2019m using uses
alpine:3.5<\/code> as a base (4MB) and adds a compiled application which is about 6MB in size. The application is an ID generator written in Go (source code available on titpetric\/sonyflake GitHub<\/a>).<\/p>\ncapacity = available resources (10GB) \/ size of a container (10MB)<\/pre>\nUsing this simple calculation, we can estimate that we can run about 1,000 containers on a single host with 10GB of available disk space.<\/p>\n
Calculating memory use<\/h3>\n
Memory use is a bit harder to calculate, because it can vary from the state when your application just starts (cold) or after the state when your application is serving real traffic for a while. The basic calculation is pretty much the same, but I do caution that you keep some buffer when doing calculations with more fluid resources.<\/p>\n
# docker stats sonyflake\r\nCONTAINER MEM USAGE \/ LIMIT\r\nsonyflake 5.273 MiB \/ 7.792 GiB<\/pre>\nI want to reserve about 20 percent of memory for other uses, so I assume that I only have 6GB of available resources. I round up the memory usage of the container as well.<\/p>\n
capacity = available resources (6GB) \/ size of a container (6MB)<\/pre>\nIt just so happens that we can comfortably run 1,000 containers on a single host because of the low memory footprint of the sonyflake app.<\/p>\n
Calculating CPU use<\/h3>\n
You don\u2019t. You should use a monitoring system that will monitor CPU usage of your hosts and containers. There are some free and cool tools you can use. For example, ctop<\/a> gives you a real-time view of your Docker containers on a single host. Netdata<\/a> provides an overview of real-time data as well as limited historical data and alerting.<\/p>\n
You can also resort to benchmarks, with software like wrk<\/a> or siege<\/a>, to estimate what kind of capacity you can expect with your setup.<\/p>\n
Running 1,000 Containers<\/h2>\n
After we get all the configuration and tuning out of the way, we\u2019re at a point where we can run our service and create 1,000 instances. For this example, I\u2019ve chosen the sony\/sonyflake<\/a> ID generator and prepared an HTTP server following 12-factor app guidelines<\/a>. The source code for the HTTP server is on titpetric\/sonyflake GitHub<\/a>.<\/p>\n
# docker service create \\\r\n> --replicas 1000 \\\r\n> --network party-swarm \\\r\n> --update-parallelism 5 \\\r\n> --name sonyflake \\\r\n> -p 80:80 titpetric\/sonyflake\r\nje2np5ab1s6ztf55qj62im72g<\/pre>\nIt takes about ten minutes to start up all the containers. You can inspect the service as it\u2019s starting up containers:<\/p>\n
# docker service ls\r\nID NAME MODE REPLICAS IMAGE\r\nje2np5ab1s6z sonyflake replicated 371\/1000 titpetric\/sonyflake:latest<\/pre>\nThe service is already usable as soon as one container is online.<\/p>\n
# curl -s http:\/\/127.0.0.1\r\n{\"id\":134286618554008122,\"machine-id\":570,\"msb\":0,\"sequence\":0,\"time\":8004106197}<\/pre>\nAfter all the containers have started, there will be a nice 1000\/1000 replicas column in the output of
docker service ls<\/code>.<\/p>\n# docker service ls\r\nID NAME MODE REPLICAS IMAGE\r\nje2np5ab1s6z sonyflake replicated 1000\/1000 titpetric\/sonyflake:latest<\/pre>\nAchievement unlocked!<\/p>\n
Let\u2019s give it a kick with a benchmark tool just to see that it works. As wrk<\/a> is also available as a Docker container, I\u2019ll be running it to attach to the custom network which we created and to benchmark the
sonyflake<\/code> service.<\/p>\n# docker run --net=party-swarm --rm williamyeh\/wrk -t 6 -c 30 http:\/\/sonyflake\r\nRunning 10s test @ http:\/\/sonyflake\r\n 6 threads and 30 connections\r\n Thread Stats Avg Stdev Max +\/- Stdev\r\n Latency 2.82ms 5.26ms 104.23ms 89.64%\r\n Req\/Sec 3.48k 1.31k 7.92k 67.57%\r\n 207348 requests in 10.07s, 40.36MB read\r\n Socket errors: connect 0, read 0, write 0, timeout 37\r\nRequests\/sec: 20598.27\r\nTransfer\/sec: 4.01MB<\/pre>\nI\u2019m feeling pretty nervous, so I\u2019ll also scale down the service to 30 containers, wait about ten minutes until it completes that, and rerun the benchmark.<\/p>\n
# docker service scale sonyflake=30\r\nsonyflake scaled to 30<\/pre>\nYou can inspect the state of your containers by issuing something like this:<\/p>\n
# docker ps -a --format \"{{.Status}}\" | sort | uniq -c\r\n 1 Created\r\n 164 Removal In Progress\r\n 2 Up 11 minutes\r\n 1 Up 12 minutes\r\n 2 Up 13 minutes\r\n 2 Up 7 minutes\r\n 3 Up 9 minutes<\/pre>\nWhen \u201cRemoval in Progress\u201d drops to 0 on all hosts, it should be okay to rerun the benchmark.<\/p>\n
# docker run --net=party-swarm --rm williamyeh\/wrk -t 6 -c 30 http:\/\/sonyflake\r\nRunning 10s test @ http:\/\/sonyflake\r\n 6 threads and 30 connections\r\n Thread Stats Avg Stdev Max +\/- Stdev\r\n Latency 0.89ms 1.37ms 37.99ms 93.22%\r\n Req\/Sec 5.69k 1.83k 9.61k 60.83%\r\n 340455 requests in 10.04s, 66.30MB read\r\n Socket errors: connect 0, read 0, write 0, timeout 46\r\nRequests\/sec: 33904.03\r\nTransfer\/sec: 6.60MB<\/pre>\nHere we can see the effect that process scheduling on Linux has on containers. When we run 1,000 containers, it means that the operating system has to divide all available time between them. Figuring out which process needs CPU time is a completely different problem when you\u2019re running 1,000 containers or when you\u2019re running only 30.<\/p>\n
This is why the benchmark for fewer containers is faster. But consider that increasing containers from 30 to 1,000 containers is a 30x increase, and the penalty is still just 30 percent of your total capacity. Amazing.<\/p>\n
Conclusion<\/h2>\n
Running a service in Docker Swarm feels very natural and simple. Scaling to 1,000 containers was a fun challenge, with honestly not very many obstacles.<\/p>\n
Of course, it very much depends on what kind of application you are scaling. If you follow the 12-factor application guidelines, it makes it very easy to go from 1 to 1,000 containers, but as soon as you\u2019re thinking of adding on another thousand containers, some systems settings should be revisited.<\/p>\n
If you need some
sysctl<\/code> settings inside your container \u2014 for example, if you have a high-traffic reverse proxy or Redis recommends some settings for you \u2014 it seems that you need to stick withdocker run<\/code> for a while longer.<\/p>\n\n