Summary

Eliminates tempKv-driven temporary memory spikes in Garnet's scan-based key iteration by routing all four Garnet/Cluster callsites of Session.Iterate(...) through a new lookup-based Session.IterateLookupSnapshot(...) API on Tsavorite's ClientSession. The snapshot variant captures Log.TailAddress once and pins both untilAddress and maxAddress to it — preserving the prior snapshot semantics (each unique live key emitted exactly once based on its latest in-range version, with the in-flight RCU race window correctly handled) without the O(N) tempKv allocation cost.

Why

Session.Iterate(...) (pull and push variants) builds a parallel TsavoriteKV<...> index keyed by every distinct key encountered while scanning the main log (libs/storage/Tsavorite/cs/src/core/Index/Tsavorite/TsavoriteIterator.cs:90) and copies in-range non-tail records into it. The hash index is sized like the main store; the in-memory log buffer is in addition to the main log. So each pull/push Session.Iterate(...) call causes a temporary memory spike proportional to the keyspace — exactly the problem this PR fixes.

The lookup-based equivalent uses the main store's hash chain for liveness checking and never builds a parallel KV. The snapshot variant additionally pins maxAddress = untilAddress so concurrent RCUs above the snapshot don't silently drop in-range records (rule S2).

Audit results

Garnet/Cluster callsites that built tempKv (now converted):

Already lookup-based (no change): SCAN cmd, DBSIZE, expired-key deletion, cluster KEYSINSLOT/COUNTKEYSINSLOT, MigrateOperation.Scan, StreamingSnapshot replication.

Raw Log.Scan(...) callsites (verified tempKv-free, out of scope): post-checkpoint serialised-blob clear, hybrid log stats diagnostics, AOF replay.

API changes

Added: ClientSession.IterateLookupSnapshot<TScanFunctions>(ref scanFns, bool includeTombstones = false) (Tsavorite). Wraps ScanCursor with the snapshot pattern untilAddress = maxAddress = capturedTail. Drops boilerplate at four callsites.

Removed (breaking):

• Parameterless pull IGarnetApi.IterateStore() overload (returned ITsavoriteScanIterator). Sole tempKv-instantiating Garnet API. The push overload IterateStore<TScanFunctions>(ref scanFns, ref cursor, ...) is unchanged. Out-of-tree extensions binding to IGarnetApi will need to migrate to the push overload.
• LogCompactionType.ShiftForced enum value (deprecated 2024-06-23 in PR Add compaction docs, update website, add compaction switch #482, ~2 years ago) + back-compat shim. Migration: --compaction-type Shift --compaction-force-delete true. CommandLineParser now produces a clear startup error listing the now-valid values.

Reordered (no behavior change): LogCompactionType enum members + help text + defaults.conf comment + website docs to None | Shift | Lookup | Scan so users see the recommended Lookup first. Added a startup LogWarning if --compaction-type Scan is selected, flagging the temporary-memory-spike cost. Scan itself still works as before.

Implementation conventions

All new callbacks follow the existing cached-field pattern (xxxFuncs ??= new(); xxxFuncs.Initialize(...)) used by unifiedStoreDbKeysFuncs, unifiedStoreDbScanFuncs, unifiedStoreDbSizeFuncs, expiredKeyDeletionScanFuncs. Steady-state per-call callback allocations are zero.

Per-call cost on the four hot paths drops from {1–2 sessions + 1–2 TsavoriteKV instances + 2 iterators + O(N) record copies} to {1 session, 0 callback alloc}.

Tests

New (this PR):

• SpanByteIterateLookupSnapshotBasicCorrectness (libs/storage/Tsavorite/cs/test/SpanByteIterationTests.cs): inserts 200 keys, RCUs all of them, asserts IterateLookupSnapshot emits each unique live key exactly once with the post-RCU value; asserts the snapshot is stable across repeated calls.
• ClusterDelKeysInSlotRemovesStringAndObjectKeys (test/Garnet.test.cluster/ClusterMigrateTests.cs): exercises CLUSTER DELKEYSINSLOT over both raw-string and collection-object keys; asserts both are deleted via the unified scan.
• SeKeysNoDuplicatesUnderRcu (test/Garnet.test/RespScanCommandsTests.cs): runs concurrent SET RCUs while issuing KEYS in a loop; asserts each key is returned exactly once across multiple rounds.

Pre-existing tests touching the changed paths (all pass):

• SpanByteIteration* suite (libs/storage/Tsavorite/cs/test/): 16/16 (+6 device-skips).
• RespScanCommandsTests (KEYS, SCAN, DBSIZE, expired-key deletion): 22/22.
• ClusterResetFailsForMasterWithKeysInSlots* (string + object), ClusterResetAfterFLushAll, ClusterSimpleSlotInfo, ClusterSimpleMigrateSlots: 9/9.
• RespVectorSetTests.DeleteVectorSet: passes.

dotnet build clean; dotnet format Garnet.slnx --verify-no-changes clean on changed files.

Migration notes for downstream users

• --compaction-type ShiftForced → use --compaction-type Shift --compaction-force-delete true.
• IGarnetApi.IterateStore() (pull, no args) → use IGarnetApi.IterateStore<TScanFunctions>(ref scanFns, ref cursor, ...) (push, lookup-based) instead. The push overload is the recommended pattern and is already used by all in-tree cluster code.
• Users explicitly opting into --compaction-type Scan will see a one-time startup LogWarning. No behavior change — Scan compaction still works exactly as before.

Verified facts cited in design

• tempKv instantiation site: libs/storage/Tsavorite/cs/src/core/Index/Tsavorite/TsavoriteIterator.cs:90.
• Snapshot semantics rationale (maxAddress strict check at < maxAddress): libs/storage/Tsavorite/cs/src/core/Index/Tsavorite/Implementation/FindRecord.cs:140.
• Reader runs with epoch suspended (DELETE-from-Reader is safe; precedent: ExpiredKeyDeletionScan): libs/storage/Tsavorite/cs/src/core/Allocator/AllocatorScan.cs:312-331.
• ScanCursorState.functions is interface-typed → struct callbacks are boxed, so all callbacks are classes: libs/storage/Tsavorite/cs/src/core/Allocator/ScanCursorState.cs:8.