fix(chunk): keep surrogate pairs whole when hard-splitting an over-long line#96671
Closed
ly-wang19 wants to merge 1 commit into
Closed
fix(chunk): keep surrogate pairs whole when hard-splitting an over-long line#96671ly-wang19 wants to merge 1 commit into
ly-wang19 wants to merge 1 commit into
Conversation
…ng line chunkByNewline length-splits a single over-long line that has no usable break point. The first head cut used a raw UTF-16 slice (lineValue.slice(0, firstLimit)), so when firstLimit landed inside a surrogate pair it emitted a chunk ending in a lone high surrogate and a next chunk starting with a lone low surrogate, which render as U+FFFD on delivery. The recursive chunkText that handles the remainder is already surrogate-safe; only this first cut was raw. Route the head cut through avoidTrailingHighSurrogateBreak (the same helper chunkText and chunkMarkdownText already use) so the cut backs off to a code-point boundary. ASCII and non-surrogate cuts are unaffected. Reproduces at the production-default 4096 limit for emoji-dense lines; chunkByNewline is the published plugin-SDK channel.text helper, reachable with arbitrary outbound text. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
Closing this PR because the author has more than 20 active PRs in this repo. Please reduce the active PR queue and reopen or resubmit once it is back under the limit. You can close your own PRs to get back under the limit. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Problem This Solves
chunkByNewline(src/auto-reply/chunk.ts) length-splits a single over-long line that has no usable break point. The first head cut used a raw UTF-16 slice —lineValue.slice(0, firstLimit)— so whenfirstLimitlanded inside a surrogate pair, it emitted a chunk ending in a lone high surrogate and a next chunk starting with a lone low surrogate. Those unpaired surrogates render asU+FFFD(�) when the message is delivered. The recursivechunkTextthat handles the remainder is already surrogate-safe (it routes cuts throughavoidTrailingHighSurrogateBreak); only this first head cut was raw.Why This Change Was Made
Surrogate-splitting is treated as a defect across this codebase —
avoidTrailingHighSurrogateBreakexists for exactly this,chunkText/chunkMarkdownTextalready use it, andchunk.test.tsasserts chunks must never end in a high surrogate or start with a low surrogate. The head cut inchunkByNewlinewas the one path that skipped it. The fix routes that cut through the same helper, backing it off to a code-point boundary. ASCII and any non-surrogate cut are unaffected (the helper is a no-op there).User Impact
chunkByNewlineis the published plugin-SDK channel helper (runtime.channel.text.chunkByNewline), used to chunk outbound channel messages. Emoji-dense lines that exceed the per-line limit no longer get a mojibake�at the split boundary. No API or config change.Evidence
oxfmt,oxlint,tsgo:coreall pass; the fullchunk.test.tssuite passes with a new regression test, and reverting the one-line fix makes that test fail (fail-before).Real behavior proof
Behavior addressed:
chunkByNewlinesplit a surrogate pair on the first hard cut of an over-long line, emitting a chunk ending in a lone high surrogate and the next starting with a lone low surrogate (renders asU+FFFDon delivery).Real environment tested: Ran the exported
chunkByNewlinefromsrc/auto-reply/chunk.tsviatsxon Node v22.22.1 (no mocks), before and after the patch, plus the colocated Vitest suite vianode scripts/run-vitest.mjs.Exact steps or command run after this patch:
chunkByNewline("😀".repeat(30), 11)andchunkByNewline("a" + "😀".repeat(5000), 4096)(the production-default 4096 limit), checking each chunk against/[\uD800-\uDBFF]$/(lone trailing high surrogate) and/^[\uDC00-\uDFFF]/(lone leading low surrogate); thennode scripts/run-vitest.mjs src/auto-reply/chunk.test.ts,node scripts/run-oxlint.mjs,pnpm tsgo:core.Evidence after fix:
Observed result after fix: no chunk ends in a lone high surrogate or starts with a lone low surrogate for emoji-dense over-long lines (at both a tiny limit and the production-default 4096); content is preserved (
join("")equals the input); ASCII/non-surrogate splitting is byte-identical to before.What was not tested: a full live channel send rendering the emoji end-to-end on a specific platform UI (no live channel in this environment); the fix is a pure text-chunking change proven at the function and unit-test level, and the surrogate-integrity contract matches the existing
chunkMarkdownTexttests.