[Bugfix] SortedSet Race Condition - Mutating under Shared Lock#1642
Merged
Conversation
hamdaankhalid
commented
Mar 19, 2026
hamdaankhalid
force-pushed
the
hkhalid/race-condition-RCU-object-store
branch
from
a9a1fc1 to
b7c4bf8
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes a SortedSet thread-safety issue where ZTTL/ZPTTL (read-path operations) mutated SortedSet state by calling DeleteExpiredItems() under Tsavorite’s shared/read lock, breaking the object-store locking invariant. Adds a new test intended to reproduce a related race involving shallow cloning during flush serialization.
Changes:
- Remove
DeleteExpiredItems()fromSortedSetTimeToLiveto prevent mutations on the read path. - Add
SortedSetShallowCloneRaceTeststo exercise a flush + CopyUpdater shallow-clone concurrency scenario.
Reviewed changes
Copilot reviewed 1 out of 1 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
libs/server/Objects/SortedSet/SortedSetObjectImpl.cs |
Stops ZTTL from mutating SortedSet internals during read operations by removing DeleteExpiredItems(). |
test/Garnet.test/SortedSetShallowCloneRaceTests.cs |
Introduces a new race-repro test intended to demonstrate shallow-clone mutation racing with serialization during flush. |
Comments suppressed due to low confidence (1)
libs/server/Objects/SortedSet/SortedSetObjectImpl.cs:868
- Removing DeleteExpiredItems() avoids mutating under a shared/read lock, but SortedSetTimeToLive now treats an expired member like a non-expiring member: GetExpiration returns the stored ticks, and ConvertUtils.*FromDiffUtcNowTicks returns -1 once the timestamp is in the past. That conflates “expired” with “no expiration”, and can change ZTTL/ZPTTL semantics. Consider returning -2 for expired members without mutating state (e.g., check IsExpired(member) or compare the returned expiration ticks to UtcNow and treat as not found).
private void SortedSetTimeToLive(ref ObjectInput input, ref GarnetObjectStoreOutput output, byte respProtocolVersion)
{
var isMilliseconds = input.arg1 == 1;
var isTimestamp = input.arg2 == 1;
var numFields = input.parseState.Count;
using var writer = new RespMemoryWriter(respProtocolVersion, ref output.SpanByteAndMemory);
writer.WriteArrayLength(numFields);
foreach (var item in input.parseState.Parameters)
{
var result = GetExpiration(item.ToArray());
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
hamdaankhalid
force-pushed
the
hkhalid/race-condition-RCU-object-store
branch
from
b7c4bf8 to
c4040cf
Compare
badrishc
approved these changes
Mar 27, 2026
vazois
added a commit
that referenced
this pull request
Mar 27, 2026
* fixed * SortedSet Race Condition - Mutation under Shared Lock --------- Co-authored-by: Hamdaan Khalid <[email protected]> Co-authored-by: Vasileios Zois <[email protected]>
vazois
added a commit
that referenced
this pull request
Apr 2, 2026
* Add es-metadata.yml (#1641) * Fix INFO all/default/everything returning empty response (#1645) * Fix INFO all/default/everything returning empty response Implement support for INFO all, default, and everything options: - all: returns all DefaultInfo sections excluding module-generated ones - default: returns the default set of sections (same as no-arg INFO) - everything: returns all DefaultInfo sections including modules Added pre-declared HashSet collections (AllInfoSet, EverythingInfoSet) in GarnetInfoMetrics.cs derived from DefaultInfo to support these options. Updated InfoCommand.cs to use UnionWith with the new HashSets instead of silently skipping the ALL keyword. Added tests for all three options, verifying correct section inclusion/exclusion. Fixes #1643 Co-authored-by: Copilot <[email protected]> * make everything option use DefaultInfo * Add null/empty guard in GetSectionHeaders test helper Adds explicit asserts before splitting INFO output so test failures surface a clear message instead of a NullReferenceException. Co-authored-by: Copilot <[email protected]> * wip; restructuring cluster tests to reduce CI duration * separate dispose from close and configure socket to allow rapid connect * ensure socket is disposed succesfully * fix failing test * update global.json --------- Co-authored-by: Copilot <[email protected]> * Fix revivification CLI flag ordering dependency (#1644) The validation in Options.GetServerOptions() checked the raw enableRevivification flag (from --reviv) instead of the computed useRevivBinsPowerOf2 state. This caused specifying --reviv alongside explicit --reviv-bin-record-sizes and --reviv-bin-record-counts to always fail, despite the --reviv help text documenting that it can be overridden by the combination of these flags. Changed the check from enableRevivification to useRevivBinsPowerOf2, which correctly reflects that --reviv-bin-record-sizes already overrides the power-of-2 default when present. Co-authored-by: Hamdaan Khalid <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Vasileios Zois <[email protected]> * [Bugfix] SortedSet Race Condition - Mutating under Shared Lock (#1642) * fixed * SortedSet Race Condition - Mutation under Shared Lock --------- Co-authored-by: Hamdaan Khalid <[email protected]> Co-authored-by: Vasileios Zois <[email protected]> * Bugfix: Accept Loop socket error handling (#1646) * Bugfix Accept Loop * race condition and tiny loggin fix * Dispose potentially partial sockets. * Fix accept loop zombie state: tiered error handling with backoff retry The accept loop in GarnetServerTcp.HandleNewConnection previously treated all SocketError != Success as fatal, disposing the SocketAsyncEventArgs and permanently killing the accept loop. This left the server in a zombie state where existing connections worked but no new connections were accepted. Changes: - Extract HandleAcceptError with three-tier error categorization: - Tier 1 (fatal): OperationAborted, NotSocket, etc. — stop loop - Tier 2 (resource pressure): TooManyOpenSockets, NoBufferSpaceAvailable, etc. — backoff via Timer and retry (100ms initial, 5s cap) - Tier 3 (client-caused transient): ConnectionReset, etc. — log and continue - Add ScheduleAcceptRetry using Timer to avoid blocking IOCP threads - Dispose retry timer on server shutdown - Add AcceptLoopTests with RST flood and extended attack simulation Co-authored-by: Copilot <[email protected]> * PR comments * Nit * Nit * simplify greatly * easy win * small updates --------- Co-authored-by: Hamdaan Khalid <[email protected]> Co-authored-by: Badrish Chandramouli <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Vasileios Zois <[email protected]> * Skip Value Expiration Check When Scanning a Tombstoned Record (#1612) * fix migrate write test * prevent expiration check on tombstoned key while scanning * fix formatting * ensure reviv pause signal is observed through epoch protection * make revivPauseEvent readonly * Update test/Garnet.test.cluster/ClusterMigrateTests.cs Co-authored-by: Copilot <[email protected]> * addressing first comments * update comment in MigrateScanFunctions * release epoch when acquiring exlucisve SuspendConfigMerge lock * add ReaderWriterLock custom implementation * fix formatting * fixing simple tests * make pause reviv thread safe --------- Co-authored-by: Copilot <[email protected]> * Bump picomatch from 2.3.1 to 2.3.2 in /website (#1649) Bumps [picomatch](https://github.com/micromatch/picomatch) from 2.3.1 to 2.3.2. - [Release notes](https://github.com/micromatch/picomatch/releases) - [Changelog](https://github.com/micromatch/picomatch/blob/master/CHANGELOG.md) - [Commits](micromatch/picomatch@2.3.1...2.3.2) --- updated-dependencies: - dependency-name: picomatch dependency-version: 2.3.2 dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump brace-expansion from 1.1.12 to 1.1.13 in /website (#1651) Bumps [brace-expansion](https://github.com/juliangruber/brace-expansion) from 1.1.12 to 1.1.13. - [Release notes](https://github.com/juliangruber/brace-expansion/releases) - [Commits](juliangruber/brace-expansion@v1.1.12...v1.1.13) --- updated-dependencies: - dependency-name: brace-expansion dependency-version: 1.1.13 dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Handle SocketException on accepted socket configuration (#1648) When a client RSTs between accept completing and socket setup (e.g. NoDelay), the SocketException was unhandled and crashed the process. Wrap the post-accept socket configuration in a try/catch, dispose the dead socket, and continue the accept loop. Co-authored-by: Copilot <[email protected]> * Fix CLUSTER NODES and CLUSTER SHARDS metadata output (#1657) * Fix CLUSTER NODES and CLUSTER SHARDS metadata output (#1650) - CLUSTER NODES: Only append ,hostname when hostname is non-empty - CLUSTER SHARDS: Replace 'address' field with Redis-compatible 'ip', 'endpoint', and optional 'hostname' fields - CLUSTER SHARDS: Honor ClusterPreferredEndpointType for endpoint field (ip address when Ip, hostname when Hostname) - CLUSTER SHARDS: Fix role output to use 'master'/'slave' instead of enum names 'PRIMARY'/'REPLICA' - Update NodeInfo struct and ClusterShards parser in test utilities for dynamic key-value field parsing - Add ClusterShardsTest and ClusterNodesHostnameTest unit tests Co-authored-by: Copilot <[email protected]> * Refactor CLUSTER SLOTS/SHARDS to use StringBuilder Replace string concatenation with StringBuilder in GetShardsInfo, GetSlotsInfo, and their helper methods (AppendFormattedSlotInfo, AppendNodeNetworkingInfo) to reduce intermediate string allocations. Co-authored-by: Copilot <[email protected]> * Address PR review comments and fix endpoint validation - Fix endpoint validation (Options.cs): allow cluster-announce-ip when bind address is 0.0.0.0/:: (wildcard), since the server listens on all interfaces. Only require port match in that case. - Handle ClusterPreferredEndpointType.Unknown in CLUSTER SHARDS: set endpoint to '?' consistent with CLUSTER SLOTS and redirects. - Add ClusterShardsTest cases for Unknown endpoint type. - Improve ClusterNodesHostnameTest to handle both hostname-present and hostname-absent branches. - Add even-length guard and explicit role validation in ClusterShards test parser. Co-authored-by: Copilot <[email protected]> --------- Co-authored-by: Copilot <[email protected]> * Apply suggestions from code review Co-authored-by: Copilot <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Badrish Chandramouli <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Hamdaan Khalid <[email protected]> Co-authored-by: Hamdaan Khalid <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR:
SortedSetTimeToLive/ZTTL uses the Read Tsavorite method which uses Shared Lock, and then in SortedSetObjectImpl layer it was mutating shared data structure.
Reasoning:
As I understand, we handle thread safety for SortedSet data structures by depending on the state mutating methods like DeleteExpiredItem to be called for commands under the ExclusiveLock commands using Tsavorite Upsert or RMW methods.
If a SortedSet command is using the Read methods and then it mutates state, that breaks the invariant.