[MOD-12263] Enhance FT.PROFILE with vector search execution details by meiravgri · Pull Request #7408 · RediSearch/RediSearch

meiravgri · 2025-11-18T14:52:36Z

This PR enhances FT.PROFILE output for vector queries by adding execution mode information and batch statistics.

Vector Search Mode

All vector queries now report their execution mode in the profile output:

Modes:

STANDARD_KNN - Pure vector search without filters
HYBRID_ADHOC_BF - Ad-hoc brute force for filtered queries
HYBRID_BATCHES - Batch-based filtered search
HYBRID_BATCHES_TO_ADHOC_BF - Dynamic mode switching
RANGE_QUERY - Range-based vector search

Example:

FT.PROFILE idx SEARCH QUERY "(@t:hello world)=>[KNN 3 @v $vec]" ...

'Iterators profile', 
  'Type', 'VECTOR',
  'Vector search mode', 'HYBRID_BATCHES',
  ...

Batch Execution Statistics

For queries using batch modes (HYBRID_BATCHES or HYBRID_BATCHES_TO_ADHOC_BF), the profile now includes:

Batches number - Total iterations executed
Largest batch size - Maximum batch size used
Largest batch iteration (zero based) - When the largest batch occurred

Example:

'Vector search mode', 'HYBRID_BATCHES',
'Batches number', 13,
'Largest batch size', 20001,
'Largest batch iteration (zero based)', 12

Note

Expose vector search execution mode and largest-batch statistics in FT.PROFILE for vector queries, with supporting iterator/state changes and tests.

FT.PROFILE output:
- Add Vector search mode to iterator profiles (VECTOR, METRIC - VECTOR DISTANCE).
- For batch modes, include Batches number, Largest batch size, and Largest batch iteration (zero based).
Hybrid iterator (src/iterators/hybrid_reader.*):
- Track maxBatchSize and maxBatchIteration; initialize/reset on create/rewind; update during batched search.
Profile printing (src/profile.*):
- Print vector search mode and new batch stats when applicable.
Vector index (src/vector_index.*):
- Add VecSimSearchMode_ToString API for human-readable modes.
Tests (tests/pytests/test_profile.py):
- Update expectations to include search mode and batch stats; add RESP3 tests validating modes for SEARCH/AGGREGATE.

^{Written by Cursor Bugbot for commit 07574d7. This will update automatically on new commits. Configure here.}

add VecSimSearchMode_ToString

codecov · 2025-11-18T15:31:49Z

Codecov Report

❌ Patch coverage is 89.28571% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.90%. Comparing base (f81354f) to head (07574d7).
⚠️ Report is 11 commits behind head on master.

Files with missing lines	Patch %	Lines
src/vector_index.c	80.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7408      +/-   ##
==========================================
- Coverage   85.17%   84.90%   -0.28%     
==========================================
  Files         346      346              
  Lines       53049    53239     +190     
  Branches    13773    13935     +162     
==========================================
+ Hits        45184    45201      +17     
- Misses       7671     7844     +173     
  Partials      194      194

Flag	Coverage Δ
flow	`84.52% <89.28%> (-0.09%)`	⬇️
unit	`52.21% <28.57%> (-0.14%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

cursor

Bug: Iterator Rewind Corrupts Profile Statistics

The HR_Rewind function doesn't reset maxBatchSize and maxBatchIteration fields. When a hybrid iterator is rewound and executed again, these statistics retain values from the previous execution, causing incorrect profile output on subsequent runs. The fields should be reset to 0 along with numIterations.

src/iterators/hybrid_reader.c#L384-L405

RediSearch/src/iterators/hybrid_reader.c

Lines 384 to 405 in b4d094d

    
           static void HR_Rewind(QueryIterator *ctx) { 
        
             HybridIterator *hr = (HybridIterator *)ctx; 
        
             hr->resultsPrepared = false; 
        
             hr->numIterations = 0; 
        
             VecSimQueryReply_Free(hr->reply); 
        
             VecSimQueryReply_IteratorFree(hr->iter); 
        
             hr->reply = NULL; 
        
             hr->iter = NULL; 
        
             hr->base.lastDocId = 0; 
        
             hr->base.atEOF = false; 
        
             if (hr->base.current) { 
        
               IndexResult_Free(hr->base.current); 
        
               hr->base.current = NULL; 
        
             } 
        
             if (hr->searchMode == VECSIM_HYBRID_ADHOC_BF || hr->searchMode == VECSIM_HYBRID_BATCHES) { 
        
               // Clean the saved and returned results (in case of HYBRID mode). 
        
               mmh_clear(hr->topResults); 
        
               hr->child->Rewind(hr->child); 
        
             } 
        
           }

alonre24

Nice and clean :)
one comment

alonre24 · 2025-11-19T08:53:14Z

tests/pytests/test_profile.py

+    # Check at least one shard has the expected search mode
+    # res['Profile']['Shards'][0]['Iterators profile']['Vector search mode']


Why at least one shard and not all?
Also, can we validate the expected number of batches and max batch size (if relevant) here?

This one is runing both in SA and in cluster so

Since the heuristic depends on the number of documents (and their text content) it might be complicated to create it for a unified scenario for all shards

im afraid that testing it will complicate the test. We test it in testProfileVector, i don't think testing it here will add value

redisearch-backport-pull-request · 2025-11-19T16:23:38Z

Backport failed for 2.8, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 2.8
git worktree add -d .worktree/backport-7408-to-2.8 origin/2.8
cd .worktree/backport-7408-to-2.8
git switch --create backport-7408-to-2.8
git cherry-pick -x a43aacf782c0cd2ca57d9a797c4e8b690e7d7d72

redisearch-backport-pull-request · 2025-11-19T16:23:40Z

Backport failed for 2.10, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 2.10
git worktree add -d .worktree/backport-7408-to-2.10 origin/2.10
cd .worktree/backport-7408-to-2.10
git switch --create backport-7408-to-2.10
git cherry-pick -x a43aacf782c0cd2ca57d9a797c4e8b690e7d7d72

redisearch-backport-pull-request · 2025-11-19T16:23:42Z

Backport failed for 8.2, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 8.2
git worktree add -d .worktree/backport-7408-to-8.2 origin/8.2
cd .worktree/backport-7408-to-8.2
git switch --create backport-7408-to-8.2
git cherry-pick -x a43aacf782c0cd2ca57d9a797c4e8b690e7d7d72

…7408) * add vector search mode to profule add VecSimSearchMode_ToString * ADD 'Largest batch size',and 'Largest batch iteration (zero based) * reset maxBatchSize and maxBatchIteration (cherry picked from commit a43aacf)

redisearch-backport-pull-request · 2025-11-19T16:23:49Z

Successfully created backport PR for 8.4:

[8.4] [MOD-12263] Enhance FT.PROFILE with vector search execution details #7427

…ails (#7427) [MOD-12263] Enhance FT.PROFILE with vector search execution details (#7408) * add vector search mode to profule add VecSimSearchMode_ToString * ADD 'Largest batch size',and 'Largest batch iteration (zero based) * reset maxBatchSize and maxBatchIteration (cherry picked from commit a43aacf) Co-authored-by: meiravgri <[email protected]>

…7408) profile.c changes are in index.c

…ails (#7436) [MOD-12263] Enhance FT.PROFILE with vector search execution details (#7408) profile.c changes are in index.c

add vector search mode to profule

d81ca65

add VecSimSearchMode_ToString

github-actions bot added the size:M label Nov 18, 2025

ADD 'Largest batch size',and 'Largest batch iteration (zero based)

b4d094d

cursor bot reviewed Nov 18, 2025

View reviewed changes

meiravgri changed the title ~~add vector search mode to profule~~ [MOD-12263] Enhance FT.PROFILE with vector search execution details Nov 18, 2025

meiravgri added backport 2.8 backport 2.10 backport 8.2 backport 8.4 labels Nov 18, 2025

reset maxBatchSize and maxBatchIteration

07574d7

meiravgri requested a review from alonre24 November 19, 2025 08:50

alonre24 reviewed Nov 19, 2025

View reviewed changes

alonre24 approved these changes Nov 19, 2025

View reviewed changes

meiravgri added this pull request to the merge queue Nov 19, 2025

Merged via the queue into master with commit a43aacf Nov 19, 2025
19 checks passed

meiravgri deleted the meiravg_profile_vector_serach_mode branch November 19, 2025 16:23

redisearch-backport-pull-request bot mentioned this pull request Nov 19, 2025

[8.4] [MOD-12263] Enhance FT.PROFILE with vector search execution details #7427

Merged

meiravgri added a commit that referenced this pull request Nov 20, 2025

[MOD-12263] Enhance FT.PROFILE with vector search execution details (#…

db0a94a

…7408) profile.c changes are in index.c

meiravgri mentioned this pull request Nov 20, 2025

[8.2] [MOD-12263] Enhance FT.PROFILE with vector search execution details #7436

Merged

meiravgri removed backport 2.8 backport 2.10 labels Nov 20, 2025

github-merge-queue bot pushed a commit that referenced this pull request Nov 20, 2025

[8.2] [MOD-12263] Enhance FT.PROFILE with vector search execution det…

c219c51

…ails (#7436) [MOD-12263] Enhance FT.PROFILE with vector search execution details (#7408) profile.c changes are in index.c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MOD-12263] Enhance FT.PROFILE with vector search execution details#7408

[MOD-12263] Enhance FT.PROFILE with vector search execution details#7408
meiravgri merged 3 commits intomasterfrom
meiravg_profile_vector_serach_mode

meiravgri commented Nov 18, 2025 •

edited by cursor bot

Loading

Uh oh!

codecov bot commented Nov 18, 2025 •

edited

Loading

Uh oh!

cursor bot left a comment

Uh oh!

alonre24 left a comment

Uh oh!

alonre24 Nov 19, 2025

Uh oh!

meiravgri Nov 19, 2025

Uh oh!

Uh oh!

redisearch-backport-pull-request bot commented Nov 19, 2025

Uh oh!

redisearch-backport-pull-request bot commented Nov 19, 2025

Uh oh!

redisearch-backport-pull-request bot commented Nov 19, 2025

Uh oh!

redisearch-backport-pull-request bot commented Nov 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


	static void HR_Rewind(QueryIterator *ctx) {
	HybridIterator hr = (HybridIterator )ctx;
	hr->resultsPrepared = false;
	hr->numIterations = 0;
	VecSimQueryReply_Free(hr->reply);
	VecSimQueryReply_IteratorFree(hr->iter);
	hr->reply = NULL;
	hr->iter = NULL;
	hr->base.lastDocId = 0;
	hr->base.atEOF = false;
	if (hr->base.current) {
	IndexResult_Free(hr->base.current);
	hr->base.current = NULL;
	}

	if (hr->searchMode == VECSIM_HYBRID_ADHOC_BF \|\| hr->searchMode == VECSIM_HYBRID_BATCHES) {
	// Clean the saved and returned results (in case of HYBRID mode).
	mmh_clear(hr->topResults);
	hr->child->Rewind(hr->child);
	}
	}

		# Check at least one shard has the expected search mode
		# res['Profile']['Shards'][0]['Iterators profile']['Vector search mode']

Conversation

meiravgri commented Nov 18, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Vector Search Mode

Batch Execution Statistics

Uh oh!

codecov bot commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Bug: Iterator Rewind Corrupts Profile Statistics

Uh oh!

alonre24 left a comment

Choose a reason for hiding this comment

Uh oh!

alonre24 Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

meiravgri Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

redisearch-backport-pull-request bot commented Nov 19, 2025

Uh oh!

redisearch-backport-pull-request bot commented Nov 19, 2025

Uh oh!

redisearch-backport-pull-request bot commented Nov 19, 2025

Uh oh!

redisearch-backport-pull-request bot commented Nov 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

meiravgri commented Nov 18, 2025 •

edited by cursor bot

Loading

codecov bot commented Nov 18, 2025 •

edited

Loading