fix(fieldNorm): count word-starts instead of space transitions to avoid over-counting leading/trailing spaces#830
Merged
Conversation
Leading and trailing spaces were silently inflating numTokens by one boundary each, producing a lower norm value (0.707 instead of 1.0 for a single-word field) and therefore a higher (worse) final search score for any document whose field value contained stray whitespace. Root cause: the original loop started numTokens at 1 and incremented on every space→non-space transition. A leading space was treated as a transition from an implicit word, so ' hello' gave numTokens=2 instead of 1. Similarly, 'hello ' transitioned word→space near the end, giving numTokens=2 even though only one word is present. Fix: rewrite the counter to tally word-starts (non-space character that follows a space or the string start). This is equivalent for typical strings — 'hello world' still yields 2 — but correctly ignores leading and trailing whitespace. Empty and all-whitespace strings are clamped to 1 to preserve the existing fallback behaviour. Test: add a failing test that demonstrates the bug for ' hello', 'hello ', ' hello world ', and ' ', then verify it passes after fix.
krisk
approved these changes
Jun 22, 2026
krisk
left a comment
krisk
left a comment
Owner
There was a problem hiding this comment.
Nice catch, this one's real. " hello" was counting as 2 tokens since values reach norm.get() untrimmed and only all-whitespace gets filtered. Fix looks good and trimmed strings come out the same as before.
Tiny thing: the table says old count for " hello world " is 3, it's actually 4.
Approving, thanks!
krisk
added a commit
that referenced
this pull request
Jun 22, 2026
krisk
pushed a commit
to chatman-media/Fuse
that referenced
this pull request
Jul 3, 2026
The word counter added in krisk#830 only checked for charCode 32 (plain ASCII space), so any field that separated words with a tab or newline instead was scored as a single long word. That gives it an artificially better field-length norm than a same-content field using regular spaces - e.g. 'foo\tbar\tbaz' scores 20x better than 'foo bar baz' even though they're the same three words. Added a small isWordSeparator helper covering the usual whitespace range (tab/LF/VT/FF/CR, space, NBSP) instead of the single charCode check, plus a test that fails without the fix.
krisk
pushed a commit
that referenced
this pull request
Jul 3, 2026
The word counter added in #830 only checked for charCode 32 (plain ASCII space), so any field that separated words with a tab or newline instead was scored as a single long word. That gives it an artificially better field-length norm than a same-content field using regular spaces - e.g. 'foo\tbar\tbaz' scores 20x better than 'foo bar baz' even though they're the same three words. Added a small isWordSeparator helper covering the usual whitespace range (tab/LF/VT/FF/CR, space, NBSP) instead of the single charCode check, plus a test that fails without the fix.
OIRNOIR
pushed a commit
to OIRNOIR/YouTube-Helper-Server
that referenced
this pull request
Jul 21, 2026
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [fuse.js](http://fusejs.io) ([source](https://github.com/krisk/Fuse)) | imports | minor | [`7.4.2` -> `7.5.0`](https://renovatebot.com/diffs/npm/fuse.js/7.4.2/7.5.0) | --- ### Release Notes <details> <summary>krisk/Fuse (fuse.js)</summary> ### [`v7.5.0`](https://github.com/krisk/Fuse/blob/HEAD/CHANGELOG.md#750-2026-07-13) [Compare Source](krisk/Fuse@v7.4.2...v7.5.0) #####⚠️ Behavior changes Every change in this release is a bug fix, but each one corrects a **scoring or ranking** bug. Scores and result ordering will shift for some queries. That is why this ships as a minor rather than a patch: the public API is unchanged and upgrading is a drop-in, but the results you get back can differ, and that should not arrive silently in a patch bump. If you assert on exact `score` values or on a specific result order, expect those assertions to need updating. Re-baseline them against 7.5.0 rather than pinning to 7.4.x, since the 7.4.x behavior was wrong in the cases below. - **Field-length normalisation now counts words correctly.** Tabs and newlines were not treated as word separators, so a multi-line or tab-delimited field was scored as though it were one long word, making it look far shorter than it is. Fields containing `\t`, `\n`, or `\r` now score differently ([#​830](krisk/Fuse#830)). - **Key weights are now normalised in object and keyless-logical search.** Weights that did not sum to `1` were applied unnormalised, skewing the relative influence of each key. If your `keys` weights do not already sum to `1`, your relative ranking changes ([#​833](krisk/Fuse#833)). - **`limit` now returns the correct top-N when scores tie.** A tie at the cutoff boundary could evict a result that should have been kept, so `limit` could return the *wrong* items, not merely the right items in a different order ([#​835](krisk/Fuse#835)). - **Bitap respects `minMatchCharLength` in the exact-match shortcut.** Matches shorter than `minMatchCharLength` were still reported via the exact-match fast path, so the `matches` array could contain entries it was configured to exclude ([#​831](krisk/Fuse#831)). ##### Bug Fixes - **bitap:** respect minMatchCharLength in exact-match shortcut ([dbb98b6](krisk/Fuse@dbb98b6)), closes [#​831](krisk/Fuse#831) - **fieldNorm:** count tabs and newlines as word separators ([6fe85b0](krisk/Fuse@6fe85b0)), closes [#​830](krisk/Fuse#830) - **fieldNorm:** count word-starts instead of space transitions ([2946f97](krisk/Fuse@2946f97)) - **scoring:** normalise key weights in object and keyless-logical search ([e164b61](krisk/Fuse@e164b61)), closes [#​833](krisk/Fuse#833) - **search:** keep the correct top-N under limit when scores tie ([437f8f3](krisk/Fuse@437f8f3)), closes [#​835](krisk/Fuse#835), thanks [@​spokodev](https://github.com/spokodev) for the report and the fix ##### [7.4.2](krisk/Fuse@v7.4.1...v7.4.2) (2026-06-05) ##### Bug Fixes - **types:** emit CommonJS declarations (.d.cts) for node16/nodenext ([#​780](krisk/Fuse#780)) ([33f5d29](krisk/Fuse@33f5d29)) ##### [7.4.1](krisk/Fuse@v7.4.0...v7.4.1) (2026-06-02) ##### Bug Fixes - **types:** add TypeScript declarations for fuse.js/worker-script ([6ef6c33](krisk/Fuse@6ef6c33)), closes [#​828](krisk/Fuse#828) - **types:** ship TypeScript declarations for fuse.js/worker ([572ad1e](krisk/Fuse@572ad1e)), closes [#​828](krisk/Fuse#828) </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yNjEuMiIsInVwZGF0ZWRJblZlciI6IjQzLjI2MS4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119--> Reviewed-on: https://git.oirnoir.dev/OIRNOIR/YouTube-Helper-Server/pulls/27
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
fieldNorm's word-counting loop startednumTokensat1and incremented on every space-to-non-space transition. This means a leading space was treated as a transition from an implicit word, over-counting the number of words:numTokensnumTokens" hello""hello "" hello world "" ""hello world"""The inflated count produces a lower norm value (e.g.
0.707instead of1.0for a single-word field), which in turn raises the final multiplicative score — penalising any document whose stored field value happens to contain leading or trailing whitespace.Fix
Replace the space-transition counter with a word-start counter: increment only when encountering a non-space character that follows a space or the start of the string. This is equivalent to the old logic for well-trimmed strings (same result for
"hello world","hello world", etc.) and correctly ignores stray boundary whitespace. Strings with zero real words (empty or all-whitespace) are clamped tonumTokens = 1to preserve the existing fallback.Tests
A new test — "does not count leading or trailing spaces as word boundaries" — was failing before the fix and passes after. All 371 existing tests continue to pass.