Unhealthy cluster: grpc: received message larger than max (4246471 vs. XXXX) #37941
Description
Description
Steps to reproduce the issue:
- I don't know, the problem happened randomly.
Describe the results you received:
I received this error on /var/log/messages
Sep 24 10:05:06 manager1 dockerd: time="2018-09-24T10:05:06.149068150-03:00" level=error msg="Handler for GET /tasks returned error: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4246471 vs. 4194304)"
Sep 24 10:05:08 manager1 dockerd: time="2018-09-24T10:05:08.164933960-03:00" level=error msg="Error getting tasks: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4246471 vs. 4194304)"
Describe the results you expected:
Any error and health cluster.
Additional information you deem important (e.g. issue happens only occasionally):
The issue happens only occasionally, when the issue happen i remove some stacks (until grpc size message is lower than 4246471 and start again).
Output of docker version:
Client:
Version: 18.06.1-ce
API version: 1.38
Go version: go1.10.3
Git commit: e68fc7a
Built: Tue Aug 21 17:23:03 2018
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 18.06.1-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: e68fc7a
Built: Tue Aug 21 17:25:29 2018
OS/Arch: linux/amd64
Experimental: false
Output of docker info:
Containers: 8
Running: 8
Paused: 0
Stopped: 0
Images: 45
Server Version: 18.06.1-ce
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: djwoo6fnyoioqjcjbrn9vtwux
Is Manager: true
ClusterID: 9ftd1qnqfm2kny7gk8bjsxss0
Managers: 3
Nodes: 30
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 10.0.0.16
Manager Addresses:
10.0.0.16:2377
10.0.1.8:2377
10.0.2.57:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-862.11.6.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.637GiB
Name: manager1.domain.local
ID: JPSI:X5AM:AAFW:SP5P:UK2Z:F6S2:MA23:HVMJ:TC2U:3WT2:QB7I:BUHV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: opentecnologia
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.):
Docker swarm cluster on AWS.