Skip to content

CommandRetry onConnection Failure #1818

@deepakverma

Description

@deepakverma

We have been discussing retrycommand with Nick and Marc and I am capturing the proposal here.

Today in case of a transient network error during planned or unplanned event SE.Redis
1. Cancels all in flight and requests awaiting to be sent and throws RedisConnection exception.
2. While SE.Redis is reconnecting any new requests coming are failed fast by throwing RedisConnection exception

This proposal is to improve reliability of SE.Redis commands by transparently retrying in case of a transient network failure. SE.Redis will hold the requests in a retry queue while it's reconnecting during a transient network blip. Commands can get timedout while waiting in the retry queue.

Following is the sequence for retry of a request during a transient network failure:
1. Request is sent to Se.Redis
2. Se.Redis while sending to Redis-server experiences a network disconnect.
3. Instead of throwing an exception Se.Redis enqueue the request to a retriable queue
4. A background thread starts processing the retriable queue
a) Request is sent if a server endpoint is available for it
b) Connection exception is thrown for a particular request if it timed out while waiting in the retry queue
5. You can control the behavior of retriability using configuration options

API:
You can specify retrypolicy on configurationoption, default is no retry.

options.CommandRetryPolicy = new CommandRetryPolicy().AlwaysRetryOnConnectionException();
options.CommandRetryPolicy = new CommandRetryPolicy().RetryIfNotYetSentOnConnectionException();

AlwaysRetryOnConnectionException implements ICommandRetry

 /// <summary>
    /// interface to implement command retry policy
    /// </summary>
    public interface ICommandRetryPolicy
    {
        /// <summary>
        /// Called when a message failed due to connection error
        /// </summary>
        /// <param name="commandStatus">current state of the command</param>
        /// <returns></returns>
        public bool ShouldRetryOnConnectionException(CommandStatus commandStatus);
        
    }

Note: naming here is still TDB and I am taking some help from Nick to have better naming here :)

You can also override the retrypolicy for a particular command with commandflags.
for example:
mux.GetDatabase().StringIncrement("test", 1, CommandFlags.NoRetry);

Out of scope:
The proposal is to retry connection exceptions during transient networking blips other exceptions are currently not in scope

High level flow diagram
image

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions