Skip to content

fix(rag): stop the document cleaner from stripping valid characters ï, ¿, ¾ - #39215

Merged
fatelei merged 1 commit into
langgenius:mainfrom
winklemad:fix/clean-processor-valid-characters
Jul 27, 2026
Merged

fatelei merged 1 commit into
langgenius:mainfrom
winklemad:fix/clean-processor-valid-characters

Conversation

@winklemad

Copy link
Copy Markdown
Contributor

The default cleaner's "invalid symbol" regex listed \xEF\xBF\xBE inside a character class. On a decoded string those are the valid code points U+00EF / U+00BF / U+00BE (ï / ¿ / ¾), so every ingested document had them silently removed (naïve → nave, ¿Cómo? → Cómo?, ¾ cup → cup).

CleanProcessor.clean is the default cleaner for every document during indexing, so this corrupts any ingested corpus containing accented / Spanish / fraction characters. U+FFFE is already handled by the dedicated re.sub("�", ...) on the next line, so removing the byte-triplet from the class preserves that behavior. The duplicated pattern in IndexingRunner.filter_string is fixed too, with regression tests in both test files.

Closes #39214

The invalid-symbol filter listed \xEF\xBF\xBE inside a regex character
class. Those bytes are the UTF-8 encoding of the U+FFFE noncharacter, but
the pattern is applied to a decoded str, where they are three separate
code points: U+00EF, U+00BF and U+00BE, i.e. the valid characters ï, ¿
and ¾.

Every ingested document therefore had those characters silently removed,
turning "naïve" into "nave" and dropping the leading inverted question
mark from Spanish questions such as "¿Cómo?". U+FFFE is already handled
by the dedicated substitution on the next line, so removing it from the
character class preserves that behaviour while keeping valid text intact.

The same pattern was duplicated in IndexingRunner.filter_string; both
sites are fixed and covered by tests.
@dosubot dosubot Bot added the size:XS This PR changes 0-9 lines, ignoring generated files. label Jul 18, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Pyrefly Type Coverage

Metric Base PR Delta
Type coverage 54.01% 54.01% -0.00%
Strict coverage 53.50% 53.50% -0.00%
Typed symbols 34,407 34,407 0
Untyped symbols 29,577 29,579 +2
Modules 3029 3029 0

@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label Jul 27, 2026
@fatelei
fatelei added this pull request to the merge queue Jul 27, 2026
Merged via the queue into langgenius:main with commit aeda37d Jul 27, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lgtm This PR has been approved by a maintainer size:XS This PR changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Default RAG cleaner strips valid characters ï, ¿, ¾ from every ingested document

2 participants