Skip to content

fix(chunk): keep surrogate pairs whole when hard-splitting an over-long line#96671

Closed
ly-wang19 wants to merge 1 commit into
openclaw:mainfrom
ly-wang19:fix/chunk-newline-surrogate-split
Closed

fix(chunk): keep surrogate pairs whole when hard-splitting an over-long line#96671
ly-wang19 wants to merge 1 commit into
openclaw:mainfrom
ly-wang19:fix/chunk-newline-surrogate-split

Conversation

@ly-wang19

Copy link
Copy Markdown
Contributor

What Problem This Solves

chunkByNewline (src/auto-reply/chunk.ts) length-splits a single over-long line that has no usable break point. The first head cut used a raw UTF-16 slice — lineValue.slice(0, firstLimit) — so when firstLimit landed inside a surrogate pair, it emitted a chunk ending in a lone high surrogate and a next chunk starting with a lone low surrogate. Those unpaired surrogates render as U+FFFD (�) when the message is delivered. The recursive chunkText that handles the remainder is already surrogate-safe (it routes cuts through avoidTrailingHighSurrogateBreak); only this first head cut was raw.

Why This Change Was Made

Surrogate-splitting is treated as a defect across this codebase — avoidTrailingHighSurrogateBreak exists for exactly this, chunkText/chunkMarkdownText already use it, and chunk.test.ts asserts chunks must never end in a high surrogate or start with a low surrogate. The head cut in chunkByNewline was the one path that skipped it. The fix routes that cut through the same helper, backing it off to a code-point boundary. ASCII and any non-surrogate cut are unaffected (the helper is a no-op there).

User Impact

chunkByNewline is the published plugin-SDK channel helper (runtime.channel.text.chunkByNewline), used to chunk outbound channel messages. Emoji-dense lines that exceed the per-line limit no longer get a mojibake at the split boundary. No API or config change.

Evidence

oxfmt, oxlint, tsgo:core all pass; the full chunk.test.ts suite passes with a new regression test, and reverting the one-line fix makes that test fail (fail-before).

Real behavior proof

Behavior addressed: chunkByNewline split a surrogate pair on the first hard cut of an over-long line, emitting a chunk ending in a lone high surrogate and the next starting with a lone low surrogate (renders as U+FFFD on delivery).

Real environment tested: Ran the exported chunkByNewline from src/auto-reply/chunk.ts via tsx on Node v22.22.1 (no mocks), before and after the patch, plus the colocated Vitest suite via node scripts/run-vitest.mjs.

Exact steps or command run after this patch: chunkByNewline("😀".repeat(30), 11) and chunkByNewline("a" + "😀".repeat(5000), 4096) (the production-default 4096 limit), checking each chunk against /[\uD800-\uDBFF]$/ (lone trailing high surrogate) and /^[\uDC00-\uDFFF]/ (lone leading low surrogate); then node scripts/run-vitest.mjs src/auto-reply/chunk.test.ts, node scripts/run-oxlint.mjs, pnpm tsgo:core.

Evidence after fix:

BEFORE (origin/main):
  chunkByNewline("😀"x30, 11)          -> chunk[0] ENDS in lone HIGH surrogate (len 11); chunk[1] STARTS with lone LOW surrogate
  chunkByNewline("a"+"😀"x5000, 4096)  -> chunk[0] ENDS in lone HIGH surrogate (len 4096); chunk[1] STARTS with lone LOW surrogate
AFTER:
  chunkByNewline("😀"x30, 11)          -> 6 chunks, splits-surrogate = false
  chunkByNewline("a"+"😀"x5000, 4096)  -> 3 chunks, splits-surrogate = false; join("") reconstructs the input exactly
  chunkByNewline("a"x50, 20) (control) -> unchanged (no surrogates)
vitest chunk.test.ts -> exit 0 (all pass, incl. new regression test); fail-before (revert source) -> the new test fails
oxlint -> exit 0 ; tsgo:core -> exit 0

Observed result after fix: no chunk ends in a lone high surrogate or starts with a lone low surrogate for emoji-dense over-long lines (at both a tiny limit and the production-default 4096); content is preserved (join("") equals the input); ASCII/non-surrogate splitting is byte-identical to before.

What was not tested: a full live channel send rendering the emoji end-to-end on a specific platform UI (no live channel in this environment); the fix is a pure text-chunking change proven at the function and unit-test level, and the surrogate-integrity contract matches the existing chunkMarkdownText tests.

…ng line

chunkByNewline length-splits a single over-long line that has no usable break
point. The first head cut used a raw UTF-16 slice (lineValue.slice(0,
firstLimit)), so when firstLimit landed inside a surrogate pair it emitted a
chunk ending in a lone high surrogate and a next chunk starting with a lone low
surrogate, which render as U+FFFD on delivery. The recursive chunkText that
handles the remainder is already surrogate-safe; only this first cut was raw.

Route the head cut through avoidTrailingHighSurrogateBreak (the same helper
chunkText and chunkMarkdownText already use) so the cut backs off to a
code-point boundary. ASCII and non-surrogate cuts are unaffected. Reproduces at
the production-default 4096 limit for emoji-dense lines; chunkByNewline is the
published plugin-SDK channel.text helper, reachable with arbitrary outbound text.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@openclaw-barnacle openclaw-barnacle Bot added size: XS r: too-many-prs Auto-close: author has more than twenty active PRs. labels Jun 25, 2026
@openclaw-barnacle

Copy link
Copy Markdown

Closing this PR because the author has more than 20 active PRs in this repo. Please reduce the active PR queue and reopen or resubmit once it is back under the limit. You can close your own PRs to get back under the limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

r: too-many-prs Auto-close: author has more than twenty active PRs. size: XS

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant