Global mode target replicas keep increasing #30854
Description
Description
Global mode target replicas keep increasing until a certain number, and current replicas is 0. Similar issue as http://stackoverflow.com/questions/41934123/docker-1-13-stack-deploy-with-global-service-malfunction-target-replicas-increa.
Steps to reproduce the issue:
- Create 2 manager node swarm cluster.
- Edit docker-compose.yml on master node with service deploy set to global mode.
docker stack deploy -c docker-compose.yml stackdocker service lsdocker service lsagain after some time.docker service lsagain after a few minutes.
Describe the results you received:
$ docker service ls
ID NAME MODE REPLICAS IMAGE
y9vgl0f9b15c project_caddy global 0/2 chrishawes/caddy:latest
$ docker service ls
y9vgl0f9b15c project_caddy global 0/17 chrishawes/caddy:latest
$ docker service ls
ID NAME MODE REPLICAS IMAGE
y9vgl0f9b15c project_caddy global 0/29 chrishawes/caddy:latest
Describe the results you expected:
$ docker service ls
ID NAME MODE REPLICAS IMAGE
y9vgl0f9b15c project_caddy global 2/2 chrishawes/caddy:latest
Additional information you deem important (e.g. issue happens only occasionally):
If redis is used instead of caddy, target replica is correct (in this case swarm has 3 managers) but current replica is always 1:
$ docker service ls
ID NAME MODE REPLICAS IMAGE
vjqgk8qxoeai project_redis global 1/3 redis:latest
Listing the containers in one of the manager nodes, there is a bunch of redis containers created but not running:
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c104fc03cc52 redis@sha256:afa4b429ef3ee08c8b198e50d684c5da0ffa43ae58631f61b08829bd6df3c500 "docker-entrypoint..." Less than a second ago Created project_redis.slk8qedylxazh54ymvfm3nko1.y3yohlpp9hkb9wmdix62wsv08
27e5331d4dbe redis@sha256:afa4b429ef3ee08c8b198e50d684c5da0ffa43ae58631f61b08829bd6df3c500 "docker-entrypoint..." 6 seconds ago Created project_redis.slk8qedylxazh54ymvfm3nko1.r599un5n1kifrgpoy2t41z57x
0c1ee32c1ab7 redis@sha256:afa4b429ef3ee08c8b198e50d684c5da0ffa43ae58631f61b08829bd6df3c500 "docker-entrypoint..." 11 seconds ago Created project_redis.slk8qedylxazh54ymvfm3nko1.rz6x512dc4sbojrvwnyrltd2o
4d1c85c2b582 redis@sha256:afa4b429ef3ee08c8b198e50d684c5da0ffa43ae58631f61b08829bd6df3c500 "docker-entrypoint..." 16 seconds ago Created project_redis.slk8qedylxazh54ymvfm3nko1.o9r9dcqmolxso4bj4hdqbfvi7
d14cf376d8e3 redis@sha256:afa4b429ef3ee08c8b198e50d684c5da0ffa43ae58631f61b08829bd6df3c500 "docker-entrypoint..." 21 seconds ago Created project_redis.slk8qedylxazh54ymvfm3nko1.z5s62mfnx2v00ac6aiclq3vwz
7c2697d3d1ea redis@sha256:afa4b429ef3ee08c8b198e50d684c5da0ffa43ae58631f61b08829bd6df3c500 "docker-entrypoint..." 27 seconds ago Created project_redis.slk8qedylxazh54ymvfm3nko1.h50xvf1fa26y65t094e3v0syl
In the manager node that has one redis container running, there is another redis created but not running:
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
redis@sha256:afa4b429ef3ee08c8b198e50d684c5da0ffa43ae58631f61b08829bd6df3c500 "docker-entrypoint..." 6 minutes ago Up 6 minutes 6379/tcp project_redis.n0iu4zdto5ezdiscnu36urnsq.wzn63rk17f7yhmf8ouwvlywzk
6a93743b60ec redis@sha256:afa4b429ef3ee08c8b198e50d684c5da0ffa43ae58631f61b08829bd6df3c500 "docker-entrypoint..." 6 minutes ago Created project_redis.n0iu4zdto5ezdiscnu36urnsq.tqzv5yech4b39k0n38xa2co3z
Output of docker version:
Client:
Version: 1.13.1
API version: 1.26
Go version: go1.7.5
Git commit: 092cba3
Built: Wed Feb 8 06:50:14 2017
OS/Arch: linux/amd64
Server:
Version: 1.13.1
API version: 1.26 (minimum version 1.12)
Go version: go1.7.5
Git commit: 092cba3
Built: Wed Feb 8 06:50:14 2017
OS/Arch: linux/amd64
Experimental: false
Output of docker info:
Containers: 35
Running: 0
Paused: 0
Stopped: 35
Images: 12
Server Version: 1.13.1
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 121
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: active
NodeID: gvqvt20ff1cpvq2tonlb7l92q
Is Manager: true
ClusterID: xd92cbileld0mieqx8qb8pf05
Managers: 2
Nodes: 2
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Node Address: 172.31.16.112
Manager Addresses:
172.31.16.112:2377
172.31.7.169:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-62-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.952 GiB
Name: ip-172-31-16-112
ID: SWAH:VWD3:HGAX:NKEB:MKVC:LK7B:BPQ2:S6X5:DA6J:QPXD:Y3AN:XAQV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: khamoud
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.):
Nodes are AWS t2.small instances.