[MOD-8474] Scalable vector search library intergration support by markovamaria · Pull Request #6329 · RediSearch/RediSearch

markovamaria · 2025-06-16T14:43:10Z

This PR reflects changes in Vector Similarity to support Scalable Vector Search (SVS) library integration.

Changes include

Add 'SVS' algorithm aupport to the 'FT.CREATE' parser
Parse and expose SVS specific index parameters.
Add functionality for showing SVS algorithm parameters in redis-client 'FT.INFO'

Mark if applicable

This PR introduces API changes
This PR introduces serialization changes

markovamaria · 2025-06-16T15:40:01Z

$ ./build.sh RUN_PYTEST TEST=test_vecsim
Total Tests Run: 47, Total Tests Failed: 2, Total Tests Passed: 45
Failed Tests Summary:
test_vecsim:test_create_multi_value_json
❌  (FAIL):     0 == 1  test_vecsim.py:1778 [SVS, $..vec]
❌  (FAIL):     0 == 1  test_vecsim.py:1778 [SVS, $.vecs[*]]
❌  (FAIL):     0 == 1  test_vecsim.py:1778 [SVS, $.*.vec]
test_vecsim:test_default_block_size_and_initial_capacity
Exception raised during test execution. See logs

TODO: verify support for SVS in tests except tests based on VECSIM_ALGOS

alonre24

Looks good, let's discuss the required changes.

tests/pytests/test_vecsim.py

src/vector_index.h

src/spec.c

src/info/info_command.c

alonre24

Nice job! few more comments

src/spec.c

src/vector_index.h

tests/pytests/test_vecsim.py

src/info/info_command.c

alonre24 · 2025-06-18T19:41:04Z

src/spec.c

+    memset(params->primaryIndexParams, 0, sizeof(VecSimParams));
+


Let's set the default svs-vamana parameters here as we do for HNSW, so we can have those values set to return them in info. This will require defining them in vecsim

Added w/o usage of default defines yet. TODO: update when RedisAI/VectorSimilarity#702 will be merged

alonre24

few more comments

src/spec.c

tests/pytests/test_vecsim.py

src/info/info_command.c

alonre24

Added few more comments

src/spec.c

src/vector_index.c

src/vector_index.h

tests/pytests/common.py

tests/pytests/test_vecsim.py

codecov · 2025-06-22T06:49:40Z

Codecov Report

Attention: Patch coverage is 86.13861% with 28 lines in your changes missing coverage. Please review.

Project coverage is 88.94%. Comparing base (2dfefe9) to head (226ca39).
Report is 6 commits behind head on master.

Files with missing lines	Patch %	Lines
src/spec.c	91.01%	15 Missing ⚠️
src/vector_index.c	35.00%	13 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6329      +/-   ##
==========================================
- Coverage   88.98%   88.94%   -0.05%     
==========================================
  Files         250      250              
  Lines       41415    41617     +202     
  Branches     3556     3556              
==========================================
+ Hits        36854    37016     +162     
- Misses       4512     4552      +40     
  Partials       49       49

Flag	Coverage Δ
flow	`81.40% <86.13%> (-0.13%)`	⬇️
unit	`46.82% <0.00%> (-0.24%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

alonre24

Great, we are getting there! whats left is:

We need to wait for clang18 fix to be merged into vecsim, and then update the vecsim submodule in this PR.
Spellcheck fix - https://github.com/RediSearch/RediSearch/actions/runs/15798445088/job/44545526121?pr=6329#step:6:14
Adding coverage for error handeling in two places (no need to add covergae for other places that are not covered)

src/spec.c

Verified & Passed tests: test_sanity_cosine, test_sanity_l2, test_sanity_zero_results, test_ft_aggregate_basic, test_range_query_basic

- Fixes for typos - Update deps/vecsim to latest - Changed naming for SVS (to SVS-VAMANA) and multiple SVS Params - Removed block_size from svs params processing - Check for appropriate date type for SVS-VAMANA - Added a validation to keep NUM_THREADS within MAX_WORKER_THREADS - Removed redundant VecSimAlgo_SVS blocks

- removed extra line - removed redundant define - fixed compresion values - fixed svs self.type in tests for better code - fixed typo in info_command

- Updated test test_vecsim:test_create_errors for SVS-VAMANA - Excluded SVS-VAMANA from test test_create_multi_value_json, test_default_block_size_and_initial_capacity - Default parameters for SVS-VAMANA are setup explicitly - Extra checks for TRAINING_THRESHOLD, related test - Processing for absent quantBits value is added - Fix for QueryError function call - Removed MAX_CANDIDATE_POOL_SIZE and PRUNE_TO from info

Fix for TRAINING_THRESHOLD related error message Added processing for VECSIM_LVQ_SCALAR compression case Added skip for SVS-VAMANA in test_multithread Updated test_sanity_cosine,test_sanity_l2 with case COMPRESSION,LVQ8 for SVS-VAMANA Typo fix

Typo fix

Introducing a new SVS-VAMANA vector type index which supports vector compression (optimized on Intel machines) - RediSearch/RediSearch#6329, RediSearch/RediSearch#6394

[#Q6329](RediSearch/RediSearch#6329), [#Q6329](RediSearch/RediSearch#6394) - Introducing the new SVS-VAMANA vector index type which supports vector compression (optimized on Intel machines)

markovamaria force-pushed the scalable-vector-search branch from 245d914 to 9cffa37 Compare June 16, 2025 14:45

markovamaria marked this pull request as ready for review June 16, 2025 15:50

markovamaria marked this pull request as draft June 16, 2025 15:55

alonre24 requested changes Jun 17, 2025

View reviewed changes

markovamaria force-pushed the scalable-vector-search branch 3 times, most recently from 33cd184 to 847a843 Compare June 18, 2025 07:52

alonre24 changed the title ~~Scalable vector search library intergration support~~ [MOD-7484] Scalable vector search library intergration support Jun 18, 2025

markovamaria marked this pull request as ready for review June 18, 2025 14:04

markovamaria requested a review from alonre24 June 18, 2025 14:10

alonre24 reviewed Jun 18, 2025

View reviewed changes

markovamaria force-pushed the scalable-vector-search branch from ca16842 to 0a95f76 Compare June 19, 2025 13:05

alonre24 reviewed Jun 19, 2025

View reviewed changes

src/spec.c Show resolved Hide resolved

tests/pytests/test_vecsim.py Show resolved Hide resolved

src/info/info_command.c Show resolved Hide resolved

markovamaria force-pushed the scalable-vector-search branch from 11fa2c0 to 58efadc Compare June 19, 2025 23:52

markovamaria requested a review from alonre24 June 19, 2025 23:54

markovamaria force-pushed the scalable-vector-search branch 2 times, most recently from c50cc1e to d1079d6 Compare June 20, 2025 00:23

alonre24 reviewed Jun 20, 2025

View reviewed changes

markovamaria force-pushed the scalable-vector-search branch from d1079d6 to c2996a8 Compare June 21, 2025 18:22

markovamaria requested a review from alonre24 June 21, 2025 18:23

alonre24 reviewed Jun 22, 2025

View reviewed changes

src/spec.c Show resolved Hide resolved

src/spec.c Show resolved Hide resolved

markovamaria added 7 commits June 23, 2025 15:52

Updated deps/VectorSimilarity to main:fa8c459 (Jun'15)

11284c2

Support for SVS library integration

5796ed7

Added support for SVS library VECSIM_ALGOS in pytests

a642ed6

Verified & Passed tests: test_sanity_cosine, test_sanity_l2, test_sanity_zero_results, test_ft_aggregate_basic, test_range_query_basic

Added basic support for SVS_TIRED, LEANVEC

ee19383

Added training_threshold parameter for SVS-VAMANA

c19c867

Fixes for review

10f4b5f

- removed extra line - removed redundant define - fixed compresion values - fixed svs self.type in tests for better code - fixed typo in info_command

markovamaria added 3 commits June 23, 2025 15:52

Naming fix: QuantBits -> Compression

50a685c

Fix for TRAINING_THRESHOLD related error message Added processing for VECSIM_LVQ_SCALAR compression case Added skip for SVS-VAMANA in test_multithread Updated test_sanity_cosine,test_sanity_l2 with case COMPRESSION,LVQ8 for SVS-VAMANA Typo fix

Added tests for SVS-VAMANA to test_create_errors

ca7cfb5

Typo fix

markovamaria force-pushed the scalable-vector-search branch from c2996a8 to ca7cfb5 Compare June 23, 2025 14:10

markovamaria requested a review from alonre24 June 23, 2025 14:11

Update deps to latest (f8bc695 Add Clang support to SVS)

226ca39

alonre24 changed the title ~~[MOD-7484] Scalable vector search library intergration support~~ [MOD-8474] Scalable vector search library intergration support Jun 24, 2025

alonre24 enabled auto-merge June 24, 2025 11:33

alonre24 approved these changes Jun 24, 2025

View reviewed changes

alonre24 added this pull request to the merge queue Jun 24, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 24, 2025

alonre24 added this pull request to the merge queue Jun 24, 2025

Merged via the queue into RediSearch:master with commit 53d4982 Jun 24, 2025
15 checks passed

rfsaliev mentioned this pull request Jun 25, 2025

Add Scalable Vector Search (SVS) library integration support #5640

Closed

2 tasks

alonre24 added a commit to alonre24/redis that referenced this pull request Jul 1, 2025

Update redisearch to 8.2 RC1

266d287

Introducing a new SVS-VAMANA vector type index which supports vector compression (optimized on Intel machines) - RediSearch/RediSearch#6329, RediSearch/RediSearch#6394

alonre24 mentioned this pull request Jul 1, 2025

Update redisearch to 8.2 RC1 redis/redis#14159

Merged

		memset(params->primaryIndexParams, 0, sizeof(VecSimParams));

Conversation

markovamaria commented Jun 16, 2025 • edited by alonre24 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes include

Mark if applicable

Uh oh!

markovamaria commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alonre24 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alonre24 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alonre24 Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

markovamaria Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

alonre24 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alonre24 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Jun 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

alonre24 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

markovamaria commented Jun 16, 2025 •

edited by alonre24

Loading

markovamaria commented Jun 16, 2025 •

edited

Loading

codecov bot commented Jun 22, 2025 •

edited

Loading