Skip to content

Skip Value Expiration Check When Scanning a Tombstoned Record#1612

Merged
vazois merged 19 commits into
mainfrom
vazois/migrate-fix
Mar 18, 2026
Merged

Skip Value Expiration Check When Scanning a Tombstoned Record#1612
vazois merged 19 commits into
mainfrom
vazois/migrate-fix

Conversation

@vazois

@vazois vazois commented Mar 6, 2026

Copy link
Copy Markdown
Contributor

This PR attempts to fix #1560. It also includes the following:

  • Skip expiration check when SingleReader pushes a tombstoned record.
    This is safe since expiration validation will occur when reading the actual value during TransmitKeys. If the key has expired at that point, it will be skipped.
  • Ensure pause reviv signal is observed under epoch protection.
  • Fix ClusterMigrateTest.
    Use GarnetClientSession to ensure the correct redirection messages are generated.
  • Release epoch when acquiring SuspendConfigMerge lock to prevent deadlock/livelock.

@vazois vazois force-pushed the vazois/migrate-fix branch from 423b177 to d6a7896 Compare March 6, 2026 05:28
@vazois vazois marked this pull request as ready for review March 6, 2026 05:28
Copilot AI review requested due to automatic review settings March 6, 2026 05:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes cluster slot migration failures when scans include tombstoned records (issue #1560) by avoiding expiration checks on tombstones, strengthening revivification pausing semantics, and updating/adding cluster migration tests to validate behavior.

Changes:

  • Skip expiration validation in migration scan readers when the scanned record is a tombstone.
  • Update TsavoriteKV.PauseRevivification() to coordinate pause visibility via epoch bump + signaling.
  • Fix and extend cluster migration tests, including a new tombstone-focused migration test.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

File Description
test/Garnet.test.cluster/ClusterMigrateTests.cs Adjusts migrate test flow/expectations and adds a new tombstone migration test.
libs/storage/Tsavorite/cs/src/core/Index/Tsavorite/Tsavorite.cs Implements epoch-coordinated revivification pause signaling and disposes the new event.
libs/cluster/Server/Migration/MigrateScanFunctions.cs Skips expiration checks for tombstone records during scan to avoid null/value issues.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread test/Garnet.test.cluster/ClusterMigrateTests.cs Outdated
Comment thread test/Garnet.test.cluster/ClusterMigrateTests.cs
Comment thread libs/cluster/Server/Migration/MigrateScanFunctions.cs Outdated
Comment thread libs/storage/Tsavorite/cs/src/core/Index/Tsavorite/Tsavorite.cs Outdated
Comment thread libs/storage/Tsavorite/cs/src/core/Index/Tsavorite/Tsavorite.cs Outdated
Comment thread test/Garnet.test.cluster/ClusterMigrateTests.cs Outdated
@vazois vazois requested a review from badrishc March 10, 2026 00:05
@vazois vazois force-pushed the vazois/migrate-fix branch from 5d4eb4d to cf8f26f Compare March 12, 2026 22:00
@vazois vazois merged commit 52eba4d into main Mar 18, 2026
62 of 63 checks passed
@vazois vazois deleted the vazois/migrate-fix branch March 18, 2026 01:11
vazois added a commit that referenced this pull request Mar 19, 2026
* fix migrate write test

* prevent expiration check on tombstoned key while scanning

* fix formatting

* ensure reviv pause signal is observed through epoch protection

* make revivPauseEvent readonly

* Update test/Garnet.test.cluster/ClusterMigrateTests.cs

Co-authored-by: Copilot <[email protected]>

* addressing first comments

* update comment in MigrateScanFunctions

* release epoch when acquiring exlucisve SuspendConfigMerge lock

* add ReaderWriterLock custom implementation

* fix formatting

* fixing simple tests

* make pause reviv thread safe

---------

Co-authored-by: Copilot <[email protected]>
vazois added a commit that referenced this pull request Apr 2, 2026
* fix migrate write test

* prevent expiration check on tombstoned key while scanning

* fix formatting

* ensure reviv pause signal is observed through epoch protection

* make revivPauseEvent readonly

* Update test/Garnet.test.cluster/ClusterMigrateTests.cs

Co-authored-by: Copilot <[email protected]>

* addressing first comments

* update comment in MigrateScanFunctions

* release epoch when acquiring exlucisve SuspendConfigMerge lock

* add ReaderWriterLock custom implementation

* fix formatting

* fixing simple tests

* make pause reviv thread safe

---------

Co-authored-by: Copilot <[email protected]>
vazois added a commit that referenced this pull request Apr 2, 2026
* Add es-metadata.yml (#1641)

* Fix INFO all/default/everything returning empty response (#1645)

* Fix INFO all/default/everything returning empty response

Implement support for INFO all, default, and everything options:
- all: returns all DefaultInfo sections excluding module-generated ones
- default: returns the default set of sections (same as no-arg INFO)
- everything: returns all DefaultInfo sections including modules

Added pre-declared HashSet collections (AllInfoSet, EverythingInfoSet)
in GarnetInfoMetrics.cs derived from DefaultInfo to support these options.

Updated InfoCommand.cs to use UnionWith with the new HashSets instead of
silently skipping the ALL keyword.

Added tests for all three options, verifying correct section inclusion/exclusion.

Fixes #1643

Co-authored-by: Copilot <[email protected]>

* make everything option use DefaultInfo

* Add null/empty guard in GetSectionHeaders test helper

Adds explicit asserts before splitting INFO output so test failures
surface a clear message instead of a NullReferenceException.

Co-authored-by: Copilot <[email protected]>

* wip; restructuring cluster tests to reduce CI duration

* separate dispose from close and configure socket to allow rapid connect

* ensure socket is disposed succesfully

* fix failing test

* update global.json

---------

Co-authored-by: Copilot <[email protected]>

* Fix revivification CLI flag ordering dependency (#1644)

The validation in Options.GetServerOptions() checked the raw
enableRevivification flag (from --reviv) instead of the computed
useRevivBinsPowerOf2 state. This caused specifying --reviv alongside
explicit --reviv-bin-record-sizes and --reviv-bin-record-counts to
always fail, despite the --reviv help text documenting that it can be
overridden by the combination of these flags.

Changed the check from enableRevivification to useRevivBinsPowerOf2,
which correctly reflects that --reviv-bin-record-sizes already
overrides the power-of-2 default when present.

Co-authored-by: Hamdaan Khalid <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Vasileios Zois <[email protected]>

* [Bugfix] SortedSet Race Condition - Mutating under Shared Lock (#1642)

* fixed

* SortedSet Race Condition - Mutation under Shared Lock

---------

Co-authored-by: Hamdaan Khalid <[email protected]>
Co-authored-by: Vasileios Zois <[email protected]>

* Bugfix: Accept Loop socket error handling (#1646)

* Bugfix Accept Loop

* race condition and tiny loggin fix

* Dispose potentially partial sockets.

* Fix accept loop zombie state: tiered error handling with backoff retry

The accept loop in GarnetServerTcp.HandleNewConnection previously treated
all SocketError != Success as fatal, disposing the SocketAsyncEventArgs and
permanently killing the accept loop. This left the server in a zombie state
where existing connections worked but no new connections were accepted.

Changes:
- Extract HandleAcceptError with three-tier error categorization:
  - Tier 1 (fatal): OperationAborted, NotSocket, etc. — stop loop
  - Tier 2 (resource pressure): TooManyOpenSockets, NoBufferSpaceAvailable,
    etc. — backoff via Timer and retry (100ms initial, 5s cap)
  - Tier 3 (client-caused transient): ConnectionReset, etc. — log and continue
- Add ScheduleAcceptRetry using Timer to avoid blocking IOCP threads
- Dispose retry timer on server shutdown
- Add AcceptLoopTests with RST flood and extended attack simulation

Co-authored-by: Copilot <[email protected]>

* PR comments

* Nit

* Nit

* simplify greatly

* easy win

* small updates

---------

Co-authored-by: Hamdaan Khalid <[email protected]>
Co-authored-by: Badrish Chandramouli <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Vasileios Zois <[email protected]>

* Skip Value Expiration Check When Scanning a Tombstoned Record (#1612)

* fix migrate write test

* prevent expiration check on tombstoned key while scanning

* fix formatting

* ensure reviv pause signal is observed through epoch protection

* make revivPauseEvent readonly

* Update test/Garnet.test.cluster/ClusterMigrateTests.cs

Co-authored-by: Copilot <[email protected]>

* addressing first comments

* update comment in MigrateScanFunctions

* release epoch when acquiring exlucisve SuspendConfigMerge lock

* add ReaderWriterLock custom implementation

* fix formatting

* fixing simple tests

* make pause reviv thread safe

---------

Co-authored-by: Copilot <[email protected]>

* Bump picomatch from 2.3.1 to 2.3.2 in /website (#1649)

Bumps [picomatch](https://github.com/micromatch/picomatch) from 2.3.1 to 2.3.2.
- [Release notes](https://github.com/micromatch/picomatch/releases)
- [Changelog](https://github.com/micromatch/picomatch/blob/master/CHANGELOG.md)
- [Commits](micromatch/picomatch@2.3.1...2.3.2)

---
updated-dependencies:
- dependency-name: picomatch
  dependency-version: 2.3.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump brace-expansion from 1.1.12 to 1.1.13 in /website (#1651)

Bumps [brace-expansion](https://github.com/juliangruber/brace-expansion) from 1.1.12 to 1.1.13.
- [Release notes](https://github.com/juliangruber/brace-expansion/releases)
- [Commits](juliangruber/brace-expansion@v1.1.12...v1.1.13)

---
updated-dependencies:
- dependency-name: brace-expansion
  dependency-version: 1.1.13
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Handle SocketException on accepted socket configuration (#1648)

When a client RSTs between accept completing and socket setup (e.g. NoDelay),
the SocketException was unhandled and crashed the process. Wrap the post-accept
socket configuration in a try/catch, dispose the dead socket, and continue
the accept loop.

Co-authored-by: Copilot <[email protected]>

* Fix CLUSTER NODES and CLUSTER SHARDS metadata output (#1657)

* Fix CLUSTER NODES and CLUSTER SHARDS metadata output (#1650)

- CLUSTER NODES: Only append ,hostname when hostname is non-empty
- CLUSTER SHARDS: Replace 'address' field with Redis-compatible 'ip',
  'endpoint', and optional 'hostname' fields
- CLUSTER SHARDS: Honor ClusterPreferredEndpointType for endpoint field
  (ip address when Ip, hostname when Hostname)
- CLUSTER SHARDS: Fix role output to use 'master'/'slave' instead of
  enum names 'PRIMARY'/'REPLICA'
- Update NodeInfo struct and ClusterShards parser in test utilities
  for dynamic key-value field parsing
- Add ClusterShardsTest and ClusterNodesHostnameTest unit tests

Co-authored-by: Copilot <[email protected]>

* Refactor CLUSTER SLOTS/SHARDS to use StringBuilder

Replace string concatenation with StringBuilder in GetShardsInfo,
GetSlotsInfo, and their helper methods (AppendFormattedSlotInfo,
AppendNodeNetworkingInfo) to reduce intermediate string allocations.

Co-authored-by: Copilot <[email protected]>

* Address PR review comments and fix endpoint validation

- Fix endpoint validation (Options.cs): allow cluster-announce-ip when
  bind address is 0.0.0.0/:: (wildcard), since the server listens on
  all interfaces. Only require port match in that case.
- Handle ClusterPreferredEndpointType.Unknown in CLUSTER SHARDS: set
  endpoint to '?' consistent with CLUSTER SLOTS and redirects.
- Add ClusterShardsTest cases for Unknown endpoint type.
- Improve ClusterNodesHostnameTest to handle both hostname-present
  and hostname-absent branches.
- Add even-length guard and explicit role validation in ClusterShards
  test parser.

Co-authored-by: Copilot <[email protected]>

---------

Co-authored-by: Copilot <[email protected]>

* Apply suggestions from code review

Co-authored-by: Copilot <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Badrish Chandramouli <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Hamdaan Khalid <[email protected]>
Co-authored-by: Hamdaan Khalid <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@github-actions github-actions Bot locked and limited conversation to collaborators May 17, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

3 participants