Skip to content

SADD returning illegal result under concurrent load (for example, 2 when adding 1 item) #1833

Description

@mgravell

Describe the bug

Server: 2.0.0-beta.7 as garnet-server from NuGet

Under concurrent heavy load, I'm seeing client-library failures from SADD {key} {value} (i.e. single add). The expected results of this are :0 (value already exists in the set), :1 (new item added to the set), or -ERR bad things. However, I'm seeing failures because the client is getting the result :2. This happens specifically when there is SPOP competing with SADD.

Steps to reproduce the bug

The easiest way to repro this is with something comparable to redis-benchmark. For your convenience, I have such a tool in the SE.Redis v3 branch, specifically if you pull at commit f6d6503898483ec5cab45cbeae4633012c91d1af.

Example usage in the src/RESPite.Benchmark sub-directory:

YOU NEED TWO RUNNING INSTANCES OF THIS!!!

dotnet run -p:TargetVer=3 -f net10.0 -c Debug -- -q +m --queue -n 500000 -P 50 -w 1 -l -t SPOP

and to emphasise the TWO RUNNING INSTANCES, in another terminal, repeat yourself:

dotnet run -p:TargetVer=3 -f net10.0 -c Debug -- -q +m --queue -n 500000 -P 50 -w 1 -l -t SPOP

(the -c Debug here is useful for including the payload in the exception; with -c Release it is much faster, but we don't expose payload data; for the meaning of the other options: see the readme.md in that folder)

It is perfectly stable against redis-server; but garnet-server blows up with errors like:

FormatException: Invalid format parsing Integer as Boolean: '2'

Expected behavior

It doesn't do that, reporting :0 or :1 reliably.

Screenshots

SAdd failed after 10000 operations

### EXCEPTION: PipelineTyped

FormatException: Invalid format parsing Integer as Boolean: '2'
SAdd failed after 10000 operations

### EXCEPTION: PipelineTyped

FormatException: Invalid format parsing Integer as Boolean: '2'
           at RESPite.Messages.RespReader.ThrowFormatException(String type) in /home/marc/ser/v3/src/RESPite/Messages/RespReader.Utils.cs:line 119
   at RESPite.Messages.RespReader.ReadBoolean() in /home/marc/ser/v3/src/RESPite/Messages/RespReader.cs:line 1943
   at StackExchange.Redis.ResultProcessor.BooleanProcessor.SetResultCore(PhysicalConnection connection, Message message, RespReader& reader) in /home/marc/ser/v3/src/StackExchange.Redis/ResultProcessor.cs:line 1142
   at StackExchange.Redis.ResultProcessor.SetResult(PhysicalConnection connection, Message message, RespReader& reader) in /home/marc/ser/v3/src/StackExchange.Redis/ResultProcessor.cs:line 251
   at StackExchange.Redis.Message.ComputeResult(PhysicalConnection connection, RespReader& reader) in /home/marc/ser/v3/src/StackExchange.Redis/Message.cs:line 690
--- End of stack trace from previous location ---
   at RESPite.Benchmark.BenchmarkBase`1.PipelineTyped[T](TClient client, Func`2 operation, Int32 divisor) in /home/marc/ser/v3/src/RESPite.Benchmark/BenchmarkBase.cs:line 295
        got: Integer

           at RESPite.Messages.RespReader.ThrowFormatException(String type) in /home/marc/ser/v3/src/RESPite/Messages/RespReader.Utils.cs:line 119
   at RESPite.Messages.RespReader.ReadBoolean() in /home/marc/ser/v3/src/RESPite/Messages/RespReader.cs:line 1943
   at StackExchange.Redis.ResultProcessor.BooleanProcessor.SetResultCore(PhysicalConnection connection, Message message, RespReader& reader) in /home/marc/ser/v3/src/StackExchange.Redis/ResultProcessor.cs:line 1142
   at StackExchange.Redis.ResultProcessor.SetResult(PhysicalConnection connection, Message message, RespReader& reader) in /home/marc/ser/v3/src/StackExchange.Redis/ResultProcessor.cs:line 251
   at StackExchange.Redis.Message.ComputeResult(PhysicalConnection connection, RespReader& reader) in /home/marc/ser/v3/src/StackExchange.Redis/Message.cs:line 690
--- End of stack trace from previous location ---
   at RESPite.Benchmark.BenchmarkBase`1.PipelineTyped[T](TClient client, Func`2 operation, Int32 divisor) in /home/marc/ser/v3/src/RESPite.Benchmark/BenchmarkBase.cs:line 295
        got: Integer

SAdd failed after 10000 operations

### EXCEPTION: PipelineTyped

FormatException: Invalid format parsing Integer as Boolean: '2'
           at RESPite.Messages.RespReader.ThrowFormatException(String type) in /home/marc/ser/v3/src/RESPite/Messages/RespReader.Utils.cs:line 119
   at RESPite.Messages.RespReader.ReadBoolean() in /home/marc/ser/v3/src/RESPite/Messages/RespReader.cs:line 1943
   at StackExchange.Redis.ResultProcessor.BooleanProcessor.SetResultCore(PhysicalConnection connection, Message message, RespReader& reader) in /home/marc/ser/v3/src/StackExchange.Redis/ResultProcessor.cs:line 1142
   at StackExchange.Redis.ResultProcessor.SetResult(PhysicalConnection connection, Message message, RespReader& reader) in /home/marc/ser/v3/src/StackExchange.Redis/ResultProcessor.cs:line 251
   at StackExchange.Redis.Message.ComputeResult(PhysicalConnection connection, RespReader& reader) in /home/marc/ser/v3/src/StackExchange.Redis/Message.cs:line 690
--- End of stack trace from previous location ---
   at RESPite.Benchmark.BenchmarkBase`1.PipelineTyped[T](TClient client, Func`2 operation, Int32 divisor) in /home/marc/ser/v3/src/RESPite.Benchmark/BenchmarkBase.cs:line 295
        got: Integer

SAdd failed after 10000 operations

Release version

2.0.0-beta.7

IDE

N/A

OS version

tested on Kubuntu 26.04

Additional context

No response

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions