[8.2] [MOD-11233] - fix: empty string token should be counted in byteOffset…#6973
Merged
[8.2] [MOD-11233] - fix: empty string token should be counted in byteOffset…#6973
Conversation
…s for highlighting (#6938) * initial commit * add newline at end of file * add debug prints * test: fix testing index being awaited * add test from the MOD-11233 * test: simplify tests to be easier to handle/debug * test: update test * test: improve test assertion * add test with some debug prints * fix: fix tokenization empty * remove useless field from tokenizer * fix: simpler fix * fix: improved behavior for forward index * fix spell check * change test name and var name * change test text * refactor: byteoffset handling independent * test: add more tests with highlighting and empty combination * apply changes as per code review --------- Co-authored-by: jonathan keinan <[email protected]> (cherry picked from commit 521847a)
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
| while (0 != aCtx->tokenizer->Next(aCtx->tokenizer, &tok)) { | ||
| // We always want to write the byte offset, even when string is empty since it is global across all fields and | ||
| // we need to know the start position of the next field. This is required for highlighting. | ||
| writeByteOffsets(&tokCtx, &tok); |
There was a problem hiding this comment.
Bug: Memory Leak from Skipped Empty Tokens
Skipping empty tokens can lead to a memory leak. When an empty token is encountered and the continue statement is hit, the cleanup for tok.allocatedTok is bypassed, leaving allocated memory unfreed.
Collaborator
Author
There was a problem hiding this comment.
in the tokenizer, if tokLen=0 it means that in the normalizer dstLen cannot be larger than the original, and then no more memory is allocated for this to be handled here.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## 8.2 #6973 +/- ##
==========================================
- Coverage 89.47% 89.44% -0.03%
==========================================
Files 253 253
Lines 41663 41665 +2
Branches 3725 3725
==========================================
- Hits 37278 37269 -9
- Misses 4336 4347 +11
Partials 49 49
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
kei-nan
approved these changes
Oct 6, 2025
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Ensures empty-string tokens are recorded in global byte offsets for accurate highlighting, updates forward-index token function signature, and adds highlight tests.
writeByteOffsetsinsrc/document.c, including empty tokens, to maintain correct cross-field positions for highlighting.indexesEmptycheck until after writing offsets; empty tokens are still skipped for indexing when appropriate.forwardIndexTokenFuncsignature toint forwardIndexTokenFunc(ForwardIndexTokenizerCtx *tokCtx, const Token *tokInfo)and document it; remove offset writing from this function.src/forward_index.haccordingly.tests/pytests/test_highlight.pycovering highlighting across empty strings, missing fields, array cases, field order, andINDEXEMPTYbehavior.Written by Cursor Bugbot for commit 7cdb3ea. This will update automatically on new commits. Configure here.