Remove Vector Set Limits#1900
Merged
Merged
Conversation
…savorite tests passing
…ber of Vector Sets
…xes for cluster work (tests are still failing); test run is pretty long now... need to clean that up
18 tasks
Contributor
There was a problem hiding this comment.
Pull request overview
This PR removes the previous Vector Set context-allocation ceiling by introducing scalable context metadata (multiple ContextMetadata blocks) and enabling variable-length namespace storage in Tsavorite records. This allows Vector Sets to use namespaces larger than a single byte (e.g., 4-byte contexts) while keeping legacy 1-byte namespaces for small contexts.
Changes:
- Added extended-namespace support in Tsavorite record layout (
RecordDataHeader/LogRecord.Namespace) and updated Tsavorite tests to cover namespaced keys + recovery. - Reworked Vector Set context tracking to scale across multiple
ContextMetadatablocks and updated migration/replication serialization to handle 4-byte namespaces. - Expanded Vector Set and cluster test coverage (including pre-allocation scenarios and a “many vector sets” stress test).
Reviewed changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| test/standalone/Garnet.test.vectorset/VectorElementKeyTests.cs | New tests validating VectorElementKey equality/hash behavior across namespace sizes/contexts. |
| test/standalone/Garnet.test.vectorset/RespVectorSetTests.cs | Parameterized fixtures for pre-allocated contexts; adds “many vector sets” stress test; adjusts quantization backfill test parameters. |
| test/cluster/Garnet.test.cluster.vectorsets/VectorSets/ClusterVectorSetTests.cs | Adds pre-allocation fixture support; ensures contexts are preallocated across nodes; minor parsing change. |
| libs/storage/Tsavorite/cs/test/TestTypes.cs | Adds a namespaced key test type and threads extended-namespace sizing into RecordFieldInfo. |
| libs/storage/Tsavorite/cs/test/test.hlog/LogRecordTests.cs | Updates assertions for new namespace address layout when extended namespace storage is used. |
| libs/storage/Tsavorite/cs/test/NamespaceTests.cs | New tests covering basic ops + recovery using keys with namespaces across device types. |
| libs/storage/Tsavorite/cs/src/core/Allocator/ObjectSerialization/RecordDataHeader.cs | Adds max single-byte namespace constant; supports extended namespace address selection; exposes extended namespace info. |
| libs/storage/Tsavorite/cs/src/core/Allocator/LogRecord.cs | Implements returning extended namespaces instead of throwing; removes outdated single-byte namespace assertions. |
| libs/server/Storage/Functions/VectorStore/VectorSessionFunctions.cs | Threads ExtendedNamespaceSize into allocation sizing; centralizes migrated record serialization via VectorManager. |
| libs/server/Storage/Functions/GarnetRecordTriggers.cs | Updates recovery trigger logic to recognize and restore persisted ContextMetadata records. |
| libs/server/Resp/Vector/VectorManager.cs | Tracks recovered context metadata; rebuilds context blocks post-recovery; marks missing-but-inuse contexts for cleanup. |
| libs/server/Resp/Vector/VectorManager.ContextMetadata.cs | Replaces single-metadata model with block array; adds helpers to compose/decompose contexts; persists metadata per-block; adds test-only preallocation hook. |
| libs/server/Resp/Vector/VectorManager.Replication.cs | Updates replication to handle 4-byte serialized namespaces for migrated element keys; persists context-metadata updates per block. |
| libs/server/Resp/Vector/VectorManager.Migration.cs | Adds centralized (de)serialization helpers for migrated keys and namespace rewriting; updates migration logic to handle expanded namespaces. |
| libs/server/Resp/Vector/VectorManager.Cleanup.cs | Updates cleanup scanning/deletion to accept 1- or 4-byte namespaces and to operate across metadata blocks. |
| libs/server/Resp/Vector/VectorManager.Callbacks.cs | Updates DiskANN callbacks to write contexts into 1- or 4-byte namespaces safely. |
| libs/common/VectorElementKey.cs | Allows 1-byte or 4-byte namespace encodings and validates the encoding rules in Debug. |
| libs/common/GarnetKeyComparer.cs | Uses SpanByteComparer.StaticEquals for key comparison in the mixed generic comparer path. |
| libs/cluster/Session/RespClusterReplicationCommands.cs | Updates reservation path to use the new ReserveContextsForMigration return type. |
| libs/cluster/Session/RespClusterMigrateCommands.cs | Switches migrated payload parsing to new VectorManager serialization helpers. |
| libs/cluster/Server/Migration/Sketch.cs | Hash seed now uses decoded namespace/context (supports 1- or 4-byte namespaces). |
| libs/cluster/Server/Migration/MigrateSessionSlots.cs | Uses VectorManager index-key serialization helper. |
| libs/cluster/Server/Migration/MigrateSessionCommonUtils.cs | Uses VectorManager.UpdateMigratedElementNamespaces; reads element keys with variable namespace sizes. |
| libs/cluster/Server/Migration/MigrateOperation.cs | Updates namespace matching to support 1- or 4-byte namespaces and ignores other sizes. |
…ure storage can sometimes produce supersets of expected status, fix tests accordingly
badrishc
approved these changes
Jun 27, 2026
badrishc
approved these changes
Jun 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Update
ContextMetadataand various namespace handling code to allow up to2^32-1allocated contexts, greatly increasing the number of simultanously allocated Vector Sets.Concretely, the number of assignable contexts is now 4,294,967,287 (eight less than
uint.MaxValueas they are reserved) and each Vector Set uses 8 contexts for a total of 536,870,910 simultaneous Vector Sets. We could push this even further (to ~17B) if DiskANN reserved 8-bytes before keys in the FFI - then the limit would be imposed by .NETs maximum array sizes.This expands tests in Tsavorite and Garnet to cover larger namespaces. It also removes a few permutations of Vector Set tests to keep CI runtimes from exploding.
TODO: