Reading from streams with command flag "PreferReplica", does not recover after rebooting replica node #2020
Description
We have an Azure Redis cluster (2 masters, 2 replicas), and are using SE version 2.2.88 in a .NET Core 3.1 service. We are reading streams from the replica nodes (using PreferReplica command flag). After we reboot the replica node where the stream is located, we get continuous errors when trying to read from the stream, it never seems to recover. The error that we get is Key has MOVED to Endpoint xx.xx.xx.xx:port and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD. We are able to consistently reproduce this issue with the code snippet below. We are assuming this issue is related to streams, because if we replace StreamReadAsync with StringGetAsync we can't reproduce the issue. Also, we can't reproduce the issue if we read the stream without specifying the command flag, and reboot the master node.
var options = ConfigurationOptions.Parse(
"host123:1234,password=123,ssl=True,abortConnect=False");
options.CommandMap = CommandMap.Create(new HashSet<string> { "SUBSCRIBE" }, false);
options.CertificateValidation += (sender, certificate, chain, sslPolicyErrors) => true;
var connectionMultiplexer = await ConnectionMultiplexer.ConnectAsync(options);
const string key = "{06S}TestPreferReplicaStream";
await connectionMultiplexer.GetDatabase().StreamAddAsync(key, "Field", "hello world from shard 0");
await connectionMultiplexer.GetDatabase().KeyExpireAsync(key, TimeSpan.FromHours(1));
while (true)
{
try
{
await connectionMultiplexer.GetDatabase().StreamReadAsync(key, "0-0",
1000, CommandFlags.PreferReplica);
}
catch (Exception e)
{
await Task.Delay(100);
Console.WriteLine("------------------------------------------------------------------------------------------------------------");
Console.WriteLine($"{DateTime.Now}: " + e);
}
}Below is the output from the code snippet above.
------------------------------------------------------------------------------------------------------------
3/3/2022 1:39:18 PM: StackExchange.Redis.RedisConnectionException: SocketFailure on xx.xxx.xx.xx:15000/Interactive, Idle/Faulted, last: XREAD, origin: ReadFromPipe, outstanding: 1, last-read: 0s ago, last-write: 0s ago, unanswered-write: 0s ago, keep-alive: 60s, state: ConnectedEstablished, mgr: 9 of 10 available, in: 0, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.2.88.56325
---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
at Pipelines.Sockets.Unofficial.StreamConnection.AsyncStreamPipe.CopyFromStreamToReadPipe() in /_/src/Pipelines.Sockets.Unofficial/StreamConnection.AsyncStreamPipe.cs:line 94
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
at System.IO.Pipelines.Pipe.GetReadAsyncResult()
at StackExchange.Redis.PhysicalConnection.ReadFromPipe()
--- End of inner exception stack trace ---
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:39:40 PM: StackExchange.Redis.RedisConnectionException: UnableToConnect on xx.xxx.xx.xx:15000/Interactive, Initializing/NotStarted, last: XREAD, origin: BeginConnectAsync, outstanding: 0, last-read: 1s ago, last-write: 1s ago, keep-alive: 60s, state: Connecting, mgr: 10 of 10 available, last-heartbeat: never, last-mbeat: 0s ago, global: 0s ago, v: 2.2.88.56325
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:39:42 PM: StackExchange.Redis.RedisConnectionException: UnableToConnect on xx.xxx.xx.xx:15000/Interactive, Initializing/NotStarted, last: XREAD, origin: BeginConnectAsync, outstanding: 0, last-read: 1s ago, last-write: 1s ago, keep-alive: 60s, state: Connecting, mgr: 10 of 10 available, last-heartbeat: never, last-mbeat: 0s ago, global: 0s ago, v: 2.2.88.56325
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:40:29 PM: StackExchange.Redis.RedisConnectionException: UnableToConnect on xx.xxx.xx.xx:15000/Interactive, Initializing/NotStarted, last: XREAD, origin: BeginConnectAsync, outstanding: 0, last-read: 5s ago, last-write: 5s ago, keep-alive: 60s, state: Connecting, mgr: 10 of 10 available, last-heartbeat: never, last-mbeat: 0s ago, global: 0s ago, v: 2.2.88.56325
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:40:54 PM: StackExchange.Redis.RedisTimeoutException: The timeout was reached before the message could be written to the output buffer, and it was not sent, command=XREAD, timeout: 5000, inst: 0, qu: 10, qs: 0, aw: True, bw: RecordingTimeout, rs: ReadAsync, ws: Flushed, in: 0, serverEndpoint: xx.xxx.xx.xx:15000, mc: 1/1/0, mgr: 10 of 10 available, clientName: S-TB-W9, IOCP: (Busy=0,Free=1000,Min=8,Max=1000), WORKER: (Busy=0,Free=32767,Min=8,Max=32767), v: 2.2.88.56325 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:40:55 PM: StackExchange.Redis.RedisConnectionException: Loading on xx.xxx.xx.xx:15000/Interactive, Idle/ComputeResult, last: ECHO, origin: SetResult, outstanding: 0, last-read: 0s ago, last-write: 0s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 9 of 10 available, last-heartbeat: never, last-mbeat: 0s ago, global: 0s ago, v: 2.2.88.56325
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:40:56 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:40:57 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:40:58 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:40:59 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:00 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:01 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:02 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:03 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:04 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:05 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:06 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:07 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:08 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:08 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:08 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:08 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:08 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:08 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:08 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:09 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:09 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:09 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:09 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:09 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:09 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:09 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
Keeps on going with same error