Greptile Summary

This PR adds a targeted regex (/<\|[^|]*\|>/g) to stripFinalTagsFromText() in order to strip GLM/DeepSeek-style special delimiter tokens (e.g. <|tool_call_result_begin|>, <|user|>, <|assistant|>) from all user-facing text routed through sanitizeUserFacingText(). The change is minimal, well-motivated, and correctly implemented.

• The regex is safe from ReDoS: [^|]* cannot create ambiguous paths for the engine, so matching is always linear.
• Using String.prototype.replace() with a module-level global regex is safe — replace() always resets lastIndex before scanning, so there is no statefulness hazard.
• Five new test cases adequately cover the happy paths (GLM tokens, DeepSeek tokens, multiple tokens) and the negative cases (normal angle brackets and HTML-like tags are preserved).
• Minor style note: the helper function stripFinalTagsFromText now serves a broader purpose than its name implies; a rename or an inline comment could help future readers understand its full scope.

Confidence Score: 5/5

• This PR is safe to merge — it adds a well-scoped regex strip with no side-effects on existing behaviour.
• The change is two lines of production code (a new regex constant and one additional .replace() call) backed by five targeted test cases. The regex is not susceptible to ReDoS, is used only via .replace() (no lastIndex hazard), and is confirmed not to touch normal angle brackets or HTML tags. No existing tests are modified, and all 66 tests reportedly pass.
• No files require special attention.

_{Last reviewed commit: d9d5054}

Thanks for the review feedback! Pushed fixes in 744aee2ec:

1. Narrowed regex — Replaced the broad <|...|> wildcard with an explicit allowlist of known model special tokens (endoftext, im_start, im_end, user, assistant, system, observation, EOT, begin▁of▁sentence, end▁of▁sentence, tool_call). This prevents false positives on legitimate content like F#/Elm pipeline operators.

2. Renamed function — stripFinalTagsFromText → stripSpecialMarkupFromText to reflect its expanded responsibility. Updated all call sites.

All tests pass. ✅

@codex review

Addressed Codex review feedback in bd2277f:

Gated special-token stripping behind leak detection: stripSpecialMarkupFromText now only applies MODEL_SPECIAL_TOKEN_RE when the text actually contains matching control tokens (via .test() guard). This prevents accidentally mangling normal assistant text that mentions token names in prose. Also resets lastIndex after .test() since the regex uses the /g flag.

FINAL_TAG_RE still runs unconditionally since <final> tags are never valid user-facing content.

Re: full-width DeepSeek tokens — already addressed in the prior commit with the [|｜] character class in the regex.

👋 @obviyus Quick ping — this strips leaked GLM/DeepSeek special tokens (<|tool_call_result_begin|> etc.) from user-facing text. Small targeted fix with leak-detection guard. Tests pass.

Thanks for the fix @imwyvern! I've reworked this into #42173 with a few architectural changes:

1. Generic pattern instead of allowlist — <[|｜]...[|｜]> catches all model control tokens (they all follow this convention), so we don't need to maintain an explicit list as new models add tokens
2. Moved to pi-embedded-utils.ts — next to stripMinimaxToolCallXml() which handles the same class of problem (provider-specific text leaks), keeping errors.ts focused on error rewriting
3. Added session replay path — sessions-helpers.ts has a parallel sanitization chain that also needed the fix

Your original work and investigation on #40020 made this possible — you're credited as co-author on the commit. 🙏

Superseded by #42173.

Superseded by #42173

Superseded by #42173 (merged).

Sorry about the separate PR — I had permission issues pushing to your branch when I was reworking it. You're credited as co-author on the merged commit. Next time I'll push directly to your branch. Thanks again for the fix!