[MOD-13432] Fix FULLTEXT field metric count by lerman25 · Pull Request #8037 · RediSearch/RediSearch

lerman25 · 2026-01-13T18:16:23Z

Changed search_total_indexing_ops_text_fields metric to count unique terms indexed instead of documents.

Moved the metric update call from fulltextPreprocessor() to Indexer_Add(), after writeCurEntries() completes successfully. This ensures the metric accurately reflects the number of unique terms added to the inverted index, persisting even after document deletion.

Note

Adjusts field indexing metrics to be accurate and consistent across types.

FULLTEXT: move metric update from fulltextPreprocessor to writeCurEntries() and increment by new unique terms (spec->stats.numTerms delta)
Stats API: change FieldsGlobalStats_UpdateFieldDocsIndexed to accept FieldType (was FieldSpec*); update all call sites
INFO metrics: clarify semantics as total indexing operations; for TEXT count unique terms, others counted per-field per-doc; ensure JSON NULLs don’t increment; multi-value JSON counts once per doc
Tests: remove TEXT from doc-count tests, add test_total_terms_indexed_text_fields, update overlapping-index and same-type-field cases, and add JSON NULL field coverage

^{Written by Cursor Bugbot for commit 29d1782. This will update automatically on new commits. Configure here.}

src/indexer.c

codecov · 2026-01-13T19:05:19Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.90%. Comparing base (83ef6c2) to head (29d1782).
⚠️ Report is 7 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8037      +/-   ##
==========================================
- Coverage   84.01%   83.90%   -0.12%     
==========================================
  Files         365      365              
  Lines       55097    55167      +70     
  Branches    14296    14296              
==========================================
- Hits        46292    46290       -2     
- Misses       8644     8716      +72     
  Partials      161      161

Flag	Coverage Δ
flow	`84.82% <100.00%> (-0.07%)`	⬇️
unit	`50.75% <83.33%> (-0.07%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

src/document.c

raz-mon · 2026-01-14T07:30:40Z

src/indexer.c

+
+    // Update global statistics for fulltext fields after successful indexing
+    if (aCtx->numTextFields > 0) {
+      FieldsGlobalStats_UpdateTextFieldDocsIndexed(aCtx->numTextFields);


Any reason not to just put this in the while loop in writeCurEntries?
What is the exact metric this counter is counting? If it's the amount of entries we write for the text field, I think it can be there

IIUC, the loop is for terms rather than for each field? We want to count docs per field

I thought the meaning of this metric is how many times we indexed documents for text fields..
// Total number of documents indexed by each field type
Actually by this description I would think it's something else - one increment per indexed doc, no matter the field count. I know that's not the meaning since you increment in bulkAdd, but it's a bit confusing.
Anyways, we can go with what you defined it to be, if it's the number of fields then you're right. If you wish to change to the number of index entries that would probably make a bit more sense IMHO.

src/info/global_stats.c

src/document.c

…-metric-count

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

cursor · 2026-01-18T12:18:05Z

src/indexer.c

  }
+
+  // Update the number of terms added for metrics
+  FieldsGlobalStats_UpdateFieldDocsIndexed(INDEXFLD_T_FULLTEXT, spec->stats.numTerms - prevNumTerms);


FULLTEXT metric counts new terms instead of fields indexed

High Severity

The FULLTEXT field metric now tracks new unique terms added rather than text fields indexed. spec->stats.numTerms - prevNumTerms counts only terms that are new to the index; if a term like "hello" already exists from a previous document, it won't increment the counter. This differs from other field types which increment by 1 per field in IndexerBulkAdd. A document with two text fields containing "hello" and "world" would only add 1 to the metric if "hello" was previously indexed, breaking the test expectation of +2 (per field).

src/indexer.c

redisearch-backport-pull-request · 2026-01-18T16:25:30Z

Backport failed for 8.2, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 8.2
git worktree add -d .worktree/backport-8037-to-8.2 origin/8.2
cd .worktree/backport-8037-to-8.2
git switch --create backport-8037-to-8.2
git cherry-pick -x 4e09df11613f0369318038f8012b0b3190ab7285

redisearch-backport-pull-request · 2026-01-18T16:25:31Z

Backport failed for 8.4, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 8.4
git worktree add -d .worktree/backport-8037-to-8.4 origin/8.4
cd .worktree/backport-8037-to-8.4
git switch --create backport-8037-to-8.4
git cherry-pick -x 4e09df11613f0369318038f8012b0b3190ab7285

* Fix metric full text metric counting * initialize fieldspec * Add missing include * remove redundant lines * test null fields * Don't count null text fields * reset counter * typo fix * remove redudant wrapper * Count terms indexed * remove extra line * test text indexing * fix test (cherry picked from commit 4e09df1)

…com/RediSearch/RediSearch/issues/8037)](https://github.com/RediSearch/RediSearch/issues/8037)) * Fix metric full text metric counting * initialize fieldspec * Add missing include * remove redundant lines * test null fields * Don't count null text fields * reset counter * typo fix * remove redudant wrapper * Count terms indexed * remove extra line * test text indexing * fix test (cherry picked from commit 4e09df1)

* Fix metric full text metric counting * initialize fieldspec * Add missing include * remove redundant lines * test null fields * Don't count null text fields * reset counter * typo fix * remove redudant wrapper * Count terms indexed * remove extra line * test text indexing * fix test (cherry picked from commit 4e09df1)

…com/RediSearch/RediSearch/issues/8037)](https://github.com/RediSearch/RediSearch/issues/8037)) * Fix metric full text metric counting * initialize fieldspec * Add missing include * remove redundant lines * test null fields * Don't count null text fields * reset counter * typo fix * remove redudant wrapper * Count terms indexed * remove extra line * test text indexing * fix test (cherry picked from commit 4e09df1)

lerman25 · 2026-01-19T16:44:08Z

/backport

redisearch-backport-pull-request · 2026-01-19T16:44:25Z

Backport failed for 2.8, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 2.8
git worktree add -d .worktree/backport-8037-to-2.8 origin/2.8
cd .worktree/backport-8037-to-2.8
git switch --create backport-8037-to-2.8
git cherry-pick -x 4e09df11613f0369318038f8012b0b3190ab7285

redisearch-backport-pull-request · 2026-01-19T16:44:27Z

Backport failed for 2.10, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 2.10
git worktree add -d .worktree/backport-8037-to-2.10 origin/2.10
cd .worktree/backport-8037-to-2.10
git switch --create backport-8037-to-2.10
git cherry-pick -x 4e09df11613f0369318038f8012b0b3190ab7285

redisearch-backport-pull-request · 2026-01-19T16:44:29Z

Backport failed for 8.2, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 8.2
git worktree add -d .worktree/backport-8037-to-8.2 origin/8.2
cd .worktree/backport-8037-to-8.2
git switch --create backport-8037-to-8.2
git cherry-pick -x 4e09df11613f0369318038f8012b0b3190ab7285

redisearch-backport-pull-request · 2026-01-19T16:44:32Z

Backport failed for 8.4, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 8.4
git worktree add -d .worktree/backport-8037-to-8.4 origin/8.4
cd .worktree/backport-8037-to-8.4
git switch --create backport-8037-to-8.4
git cherry-pick -x 4e09df11613f0369318038f8012b0b3190ab7285

* Fix metric full text metric counting * initialize fieldspec * Add missing include * remove redundant lines * test null fields * Don't count null text fields * reset counter * typo fix * remove redudant wrapper * Count terms indexed * remove extra line * test text indexing * fix test (cherry picked from commit 4e09df1)

redisearch-backport-pull-request · 2026-01-19T16:44:39Z

Successfully created backport PR for 8.6:

[8.6] [MOD-13432] Fix FULLTEXT field metric count #8086

[MOD-13432] Fix FULLTEXT field metric count ([[#8037](https://github.com/RediSearch/RediSearch/issues/8037)](https://github.com/RediSearch/RediSearch/issues/8037)) * Fix metric full text metric counting * initialize fieldspec * Add missing include * remove redundant lines * test null fields * Don't count null text fields * reset counter * typo fix * remove redudant wrapper * Count terms indexed * remove extra line * test text indexing * fix test (cherry picked from commit 4e09df1)

[MOD-13432] Fix FULLTEXT field metric count ([#8037](#8037)) * Fix metric full text metric counting * initialize fieldspec * Add missing include * remove redundant lines * test null fields * Don't count null text fields * reset counter * typo fix * remove redudant wrapper * Count terms indexed * remove extra line * test text indexing * fix test (cherry picked from commit 4e09df1)

Fix metric full text metric counting

c441374

lerman25 requested a review from raz-mon January 13, 2026 18:16

github-actions bot added the size:S label Jan 13, 2026

lerman25 removed the request for review from raz-mon January 13, 2026 18:16

lerman25 added backport 2.8 backport 2.10 backport 8.2 backport 8.4 labels Jan 13, 2026

cursor bot reviewed Jan 13, 2026

View reviewed changes

src/indexer.c Outdated Show resolved Hide resolved

initialize fieldspec

3fa85e4

cursor bot reviewed Jan 14, 2026

View reviewed changes

src/document.c Outdated Show resolved Hide resolved

Add missing include

126885c

raz-mon reviewed Jan 14, 2026

View reviewed changes

src/info/global_stats.c Outdated Show resolved Hide resolved

lerman25 added 3 commits January 14, 2026 10:56

remove redundant lines

d88a0f9

test null fields

6eadac3

Don't count null text fields

570483b

cursor bot reviewed Jan 14, 2026

View reviewed changes

src/document.c Outdated Show resolved Hide resolved

reset counter

0295b0d

lerman25 enabled auto-merge January 14, 2026 15:41

lerman25 added 5 commits January 15, 2026 15:54

typo fix

7d43fb2

remove redudant wrapper

d3e8b71

\Merge remote-tracking branch 'origin/master' into omerl-fix-fulltext…

d01c2e5

…-metric-count

Count terms indexed

747c2bc

remove extra line

cf93c38

cursor bot reviewed Jan 18, 2026

View reviewed changes

test text indexing

e4aadf0

github-actions bot added the size:M label Jan 18, 2026

raz-mon reviewed Jan 18, 2026

View reviewed changes

src/indexer.c Show resolved Hide resolved

This was referenced Jan 19, 2026

[8.4] [MOD-13432] Fix FULLTEXT field metric count (#8037) #8082

Merged

[8.2] [MOD-13432] Fix FULLTEXT field metric count (#8037) #8083

Merged

lerman25 mentioned this pull request Jan 19, 2026

[2.10] [MOD-13432] Fix FULLTEXT field metric count (#8037) #8084

Merged

lerman25 mentioned this pull request Jan 19, 2026

[2.8] [MOD-13432] Fix FULLTEXT field metric count (#8037) #8085

Merged

lerman25 added the backport 8.6 label Jan 19, 2026

redisearch-backport-pull-request bot mentioned this pull request Jan 19, 2026

[8.6] [MOD-13432] Fix FULLTEXT field metric count #8086

Merged

Conversation

lerman25 commented Jan 13, 2026 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

raz-mon Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

lerman25 Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

raz-mon Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor bot Jan 18, 2026

Choose a reason for hiding this comment

FULLTEXT metric counts new terms instead of fields indexed

Uh oh!

lerman25 Jan 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

redisearch-backport-pull-request bot commented Jan 18, 2026

Uh oh!

redisearch-backport-pull-request bot commented Jan 18, 2026

Uh oh!

lerman25 commented Jan 19, 2026

Uh oh!

redisearch-backport-pull-request bot commented Jan 19, 2026

Uh oh!

redisearch-backport-pull-request bot commented Jan 19, 2026

Uh oh!

redisearch-backport-pull-request bot commented Jan 19, 2026

Uh oh!

redisearch-backport-pull-request bot commented Jan 19, 2026

Uh oh!

redisearch-backport-pull-request bot commented Jan 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lerman25 commented Jan 13, 2026 •

edited by cursor bot

Loading

codecov bot commented Jan 13, 2026 •

edited

Loading