Repository navigation
fix(rag): stop the document cleaner from stripping valid characters ï, ¿, ¾ - #39215
Merged
fatelei merged 1 commit intoJul 27, 2026
Merged
Conversation
The invalid-symbol filter listed \xEF\xBF\xBE inside a regex character class. Those bytes are the UTF-8 encoding of the U+FFFE noncharacter, but the pattern is applied to a decoded str, where they are three separate code points: U+00EF, U+00BF and U+00BE, i.e. the valid characters ï, ¿ and ¾. Every ingested document therefore had those characters silently removed, turning "naïve" into "nave" and dropping the leading inverted question mark from Spanish questions such as "¿Cómo?". U+FFFE is already handled by the dedicated substitution on the next line, so removing it from the character class preserves that behaviour while keeping valid text intact. The same pattern was duplicated in IndexingRunner.filter_string; both sites are fixed and covered by tests.
winklemad
requested review from
JohnJyong,
QuantumGhost and
laipz8200
as code owners
July 18, 2026 11:36
Contributor
Pyrefly Type Coverage
|
fatelei
approved these changes
Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The default cleaner's "invalid symbol" regex listed
\xEF\xBF\xBEinside a character class. On a decoded string those are the valid code points U+00EF / U+00BF / U+00BE (ï/¿/¾), so every ingested document had them silently removed (naïve→nave,¿Cómo?→Cómo?,¾ cup→cup).CleanProcessor.cleanis the default cleaner for every document during indexing, so this corrupts any ingested corpus containing accented / Spanish / fraction characters. U+FFFE is already handled by the dedicatedre.sub("�", ...)on the next line, so removing the byte-triplet from the class preserves that behavior. The duplicated pattern inIndexingRunner.filter_stringis fixed too, with regression tests in both test files.Closes #39214