Concurrent POST networks/create results in "Another app is currently holding the xtables lock" #26111
Description
Output of docker version:
Client:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Thu Aug 18 05:33:38 2016
OS/Arch: linux/amd64
Server:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Thu Aug 18 05:33:38 2016
OS/Arch: linux/amd64
Output of docker info:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 147
Server Version: 1.12.1
Storage Driver: overlay2
Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: host bridge null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-34-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 28.01 GiB
Name: ubuntu-vm-main
ID: QLXW:RPE6:XA5D:VFSY:OF6O:2TRE:WL22:DP7Y:37PS:ANX3:HMFU:ZKUZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: indygreg
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
127.0.0.0/8
Additional environment details (AWS, VirtualBox, physical, etc.):
Ubuntu 16.04 running under Hyper-V (but Hyper-V shouldn't matter for this bug report).
Steps to reproduce the issue:
- Make concurrent calls to
POST networks/createto create multiple networks in rapid succession - Docker API fails reliably with complaints of xtables lock being held.
Here is short Python script to reproduce the issue:
import docker
import concurrent.futures as futures
import uuid
client = docker.Client()
fs = []
with futures.ThreadPoolExecutor(8) as pool:
for i in range(100):
network_name = str(uuid.uuid1())
fs.append(pool.submit(client.create_network, network_name, driver='bridge'))
# check results of futures here if you want
Describe the results you received:
The server log is littered with the following:
Handler for POST /v1.24/networks/create returned error: Failed to Setup IP tables: Unable to enable SKIP DNAT rule: Another app is currently holding the xtables lock; waiting (1s) for it to exit...\n (<nil>)
Handler for POST /v1.24/networks/create returned error: Failed to Setup IP tables: Unable to allow intercontainer communication: Another app is currently holding the xtables lock; waiting (1s) for it to exit...\n (<nil>)
Describe the results you expected:
I would expect network API calls to work under load. If it could take several seconds for the request to complete, I'd appreciate if the client could specify a timeout so it doesn't have to retry requests until they succeed.
Additional information you deem important (e.g. issue happens only occasionally):
I'm creating many short-lived containers and user-defined networks as part of a testing environment. Each test essentially creates its own isolated "cluster" of containers. This is all using the Remote API directly (via the Python client). Running multiple tests concurrently was generally working fine until we started using user-defined networks. It appears these APIs can't handle load as well as e.g. the container create/start/stop/delete APIs.