Kind

Enhancement

Description

When a huge number of Swarmkit objects (services, configs and secrets) are created in the same time, this error appears:

level=error msg="agent: session failed" backoff=100ms error="rpc error: code = ResourceExhausted desc = grpc: received message larger than max (7263689 vs. 4194304)"

This error seems to happen on all the cluster nodes in two differents use cases:

• When a big swarmkit object is created/updated or when a lot of swamkit objects are created/updated in the same, UC1 below.
• When the cluster leader comes down, UC2 below.

UC1

The error seems to happen because the gRPC client connection used for the Assignments method can not received messages which have a size greater than 4194304.
This method is called thanks to the ConnectionBroker Client Connection which is defined in two functions:

• Here to define the local connection to use on the leader
• Here to define the remote connection to use by the others nodes

UC2

The error seems to happen because the grpc client connection used in the Raft Transport (the structure which manages remote raft peers and sends messages to them) can not received messages which have a size greater than 4194304.

The Transport ClientConnection is initialized here.

Proposition

I propose to add an option on the Node Config to allow users to add a slice of grpc.DialOption which can be provided to the internal ClientConnections.