Skip to content

SignalR backplane with disabled CLUSTER command generates hundreds of connection errors #2012

@tylerohlsen

Description

@tylerohlsen

Background

I'm using a Redis cluster as a SignalR backplane and only using it for PubSub. Each node of the cluster is physically separated by a large distance. I have the networking set up so all nodes of the Redis cluster can talk to each other on a private subnet, but the applications that connect to the Redis nodes can only communicate with the Redis node in the same physical location.

This setup works very well so users that connect to the application in same physical location have very low latency with each other, and users that connect to different physical locations have higher latency, but they can still receive messages across the backplane.

Problem

The problem I'm running into is the StackExchange.Redis auto-discovery of the cluster nodes is causing the applications to silently attempt to connect to the Redis nodes in the distant physical locations. I know they will never be able to connect to those other Redis nodes, so I attempted to disable the auto-discovery of the other nodes in the cluster by adding $CLUSTER= on the connection string. This does achieve my goal, but it has the side effect of a one-time burst of hundreds of the following error:

StackExchange.Redis.RedisConnectionException: InternalFailure (None, last-recv: 364) on TEST-REDIS-0:6379/Interactive, Writing/ReadAsync, last: CLUSTER, origin: WriteMessageToServerInsideWriteLock, outstanding: 2, last-read: 0s ago, last-write: 0s ago, unanswered-write: 0s ago, keep-alive: 60s, state: ConnectedEstablished, mgr: 10 of 10 available, in: 0, in-pipe: 0, out-pipe: 41, last-heartbeat: never, last-mbeat: 1s ago, global: 0s ago, v: 2.2.88.56325
 ---> StackExchange.Redis.RedisCommandException: This operation has been disabled in the command-map and cannot be used: CLUSTER
   at StackExchange.Redis.PhysicalConnection.WriteHeader(RedisCommand command, Int32 arguments, CommandBytes commandBytes) in /_/src/StackExchange.Redis/PhysicalConnection.cs:line 734
   at StackExchange.Redis.Message.CommandValueMessage.WriteImpl(PhysicalConnection physical) in /_/src/StackExchange.Redis/Message.cs:line 1250
   at StackExchange.Redis.Message.WriteTo(PhysicalConnection physical) in /_/src/StackExchange.Redis/Message.cs:line 768
   at StackExchange.Redis.PhysicalBridge.WriteMessageToServerInsideWriteLock(PhysicalConnection connection, Message message) in /_/src/StackExchange.Redis/PhysicalBridge.cs:line 1312
   --- End of inner exception stack trace ---

Unfortunately, that stack trace does not identify where the CLUSTER NODES command was executed, but I suspect it's this block:

if (clusterCount > 0 && !encounteredConnectedClusterServer)
{
// We have encountered a connected server with a cluster type for the first time.
// so we will get list of other nodes from this server using "CLUSTER NODES" command
// and try to connect to these other nodes in the next iteration
encounteredConnectedClusterServer = true;
updatedClusterEndpointCollection = await GetEndpointsFromClusterNodes(server, log).ForAwait();
}

Proposed Solution

Could this check be added to be added to that if statement?

CommandMap.IsAvailable(RedisCommand.CLUSTER)

I'd be willing to submit a PR for that change if you agree.
AtDhVaAnNkCsE!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions