-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Description
Background
I'm using a Redis cluster as a SignalR backplane and only using it for PubSub. Each node of the cluster is physically separated by a large distance. I have the networking set up so all nodes of the Redis cluster can talk to each other on a private subnet, but the applications that connect to the Redis nodes can only communicate with the Redis node in the same physical location.
This setup works very well so users that connect to the application in same physical location have very low latency with each other, and users that connect to different physical locations have higher latency, but they can still receive messages across the backplane.
Problem
The problem I'm running into is the StackExchange.Redis auto-discovery of the cluster nodes is causing the applications to silently attempt to connect to the Redis nodes in the distant physical locations. I know they will never be able to connect to those other Redis nodes, so I attempted to disable the auto-discovery of the other nodes in the cluster by adding $CLUSTER= on the connection string. This does achieve my goal, but it has the side effect of a one-time burst of hundreds of the following error:
StackExchange.Redis.RedisConnectionException: InternalFailure (None, last-recv: 364) on TEST-REDIS-0:6379/Interactive, Writing/ReadAsync, last: CLUSTER, origin: WriteMessageToServerInsideWriteLock, outstanding: 2, last-read: 0s ago, last-write: 0s ago, unanswered-write: 0s ago, keep-alive: 60s, state: ConnectedEstablished, mgr: 10 of 10 available, in: 0, in-pipe: 0, out-pipe: 41, last-heartbeat: never, last-mbeat: 1s ago, global: 0s ago, v: 2.2.88.56325
---> StackExchange.Redis.RedisCommandException: This operation has been disabled in the command-map and cannot be used: CLUSTER
at StackExchange.Redis.PhysicalConnection.WriteHeader(RedisCommand command, Int32 arguments, CommandBytes commandBytes) in /_/src/StackExchange.Redis/PhysicalConnection.cs:line 734
at StackExchange.Redis.Message.CommandValueMessage.WriteImpl(PhysicalConnection physical) in /_/src/StackExchange.Redis/Message.cs:line 1250
at StackExchange.Redis.Message.WriteTo(PhysicalConnection physical) in /_/src/StackExchange.Redis/Message.cs:line 768
at StackExchange.Redis.PhysicalBridge.WriteMessageToServerInsideWriteLock(PhysicalConnection connection, Message message) in /_/src/StackExchange.Redis/PhysicalBridge.cs:line 1312
--- End of inner exception stack trace ---
Unfortunately, that stack trace does not identify where the CLUSTER NODES command was executed, but I suspect it's this block:
StackExchange.Redis/src/StackExchange.Redis/ConnectionMultiplexer.cs
Lines 1825 to 1832 in 6718fea
| if (clusterCount > 0 && !encounteredConnectedClusterServer) | |
| { | |
| // We have encountered a connected server with a cluster type for the first time. | |
| // so we will get list of other nodes from this server using "CLUSTER NODES" command | |
| // and try to connect to these other nodes in the next iteration | |
| encounteredConnectedClusterServer = true; | |
| updatedClusterEndpointCollection = await GetEndpointsFromClusterNodes(server, log).ForAwait(); | |
| } |
Proposed Solution
Could this check be added to be added to that if statement?
CommandMap.IsAvailable(RedisCommand.CLUSTER)I'd be willing to submit a PR for that change if you agree.
AtDhVaAnNkCsE!