-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Description
We have been discussing retrycommand with Nick and Marc and I am capturing the proposal here.
Today in case of a transient network error during planned or unplanned event SE.Redis
1. Cancels all in flight and requests awaiting to be sent and throws RedisConnection exception.
2. While SE.Redis is reconnecting any new requests coming are failed fast by throwing RedisConnection exception
This proposal is to improve reliability of SE.Redis commands by transparently retrying in case of a transient network failure. SE.Redis will hold the requests in a retry queue while it's reconnecting during a transient network blip. Commands can get timedout while waiting in the retry queue.
Following is the sequence for retry of a request during a transient network failure:
1. Request is sent to Se.Redis
2. Se.Redis while sending to Redis-server experiences a network disconnect.
3. Instead of throwing an exception Se.Redis enqueue the request to a retriable queue
4. A background thread starts processing the retriable queue
a) Request is sent if a server endpoint is available for it
b) Connection exception is thrown for a particular request if it timed out while waiting in the retry queue
5. You can control the behavior of retriability using configuration options
API:
You can specify retrypolicy on configurationoption, default is no retry.
options.CommandRetryPolicy = new CommandRetryPolicy().AlwaysRetryOnConnectionException();
options.CommandRetryPolicy = new CommandRetryPolicy().RetryIfNotYetSentOnConnectionException();
AlwaysRetryOnConnectionException implements ICommandRetry
/// <summary>
/// interface to implement command retry policy
/// </summary>
public interface ICommandRetryPolicy
{
/// <summary>
/// Called when a message failed due to connection error
/// </summary>
/// <param name="commandStatus">current state of the command</param>
/// <returns></returns>
public bool ShouldRetryOnConnectionException(CommandStatus commandStatus);
}
Note: naming here is still TDB and I am taking some help from Nick to have better naming here :)
You can also override the retrypolicy for a particular command with commandflags.
for example:
mux.GetDatabase().StringIncrement("test", 1, CommandFlags.NoRetry);
Out of scope:
The proposal is to retry connection exceptions during transient networking blips other exceptions are currently not in scope
