fix(irc): chunk PRIVMSG on UTF-16 boundary to avoid lone surrogates by WeeLi-009 · Pull Request #96572 · openclaw/openclaw

WeeLi-009 · 2026-06-24T22:26:54Z

Summary: Long IRC PRIVMSG payloads were chunked on raw UTF-16 indices, so an emoji straddling a chunk boundary was torn into lone surrogates — malformed UTF-8 on the wire and a garbled glyph for the recipient. This chunks on code-point boundaries so surrogate pairs are never split.

What Problem This Solves

connectIrcClient in extensions/irc/src/client.ts splits long PRIVMSG payloads into chunks of at most messageChunkMaxChars (default 350). The split was a raw chunk.slice(0, splitAt) on the UTF-16 string. When the split index lands in the middle of an astral character — i.e. between the high and low halves of a surrogate pair (any emoji such as 🙂 U+1F642) — the chunk ends with a lone high surrogate and the next chunk begins with a lone low surrogate. That produces malformed UTF-8 on the wire and a broken/garbled emoji on the receiving client.

Trigger: a message of "xxxxxxxxx🙂rest" with messageChunkMaxChars = 10. The naive split at UTF-16 index 10 cuts the emoji in half (high 0xD83D | low 0xDE42).

Fix

Add sliceIrcPrivmsgChunk, which slices on a safe UTF-16 boundary via the shared sliceUtf16Safe helper (openclaw/plugin-sdk/text-utility-runtime) so a surrogate pair is never split. Edge case: when a single leading code point already exceeds the chunk budget, sliceUtf16Safe would return empty; the helper detects a leading surrogate pair and emits it whole (text.slice(0, 2)) so chunking still advances instead of looping. messageChunkMaxChars is also clamped to a sane Math.max(1, Math.floor(...)).

Evidence

Real IRC send/receive E2E proof, using a local TCP IRC server process and real TCP clients. This is not a node:net / node:tls mock.

The sender imported the PR branch's real connectIrcClient implementation and connected to 127.0.0.1. A separate raw net.Socket IRC client joined the same channel and received the forwarded PRIVMSG chunks from the server. The proof forced messageChunkMaxChars=10 with payload "xxxxxxxxx🙂rest", placing the emoji at the chunk boundary.

IRC PRIVMSG emoji-boundary real TCP E2E proof
[server] server process pid=1924576
[server] READY host=127.0.0.1 port=37955
receiver: real net.Socket client nick=oc_pr_receiver channel=#oc-e2e
[server] recv NICK nick=oc_pr_receiver
[server] recv USER nick=oc_pr_receiver
[server] welcome nick=oc_pr_receiver
[server] recv JOIN nick=oc_pr_receiver channel=#oc-e2e
sender: real OpenClaw connectIrcClient nick=oc_pr_sender channel=#oc-e2e
[server] recv NICK nick=oc_pr_sender
[server] recv USER nick=oc_pr_sender
[server] welcome nick=oc_pr_sender
payload: "xxxxxxxxx🙂rest"
payload codeUnits=15 messageChunkMaxChars=10
send: OpenClaw client sendPrivmsg(...) over real TCP
[server] recv JOIN nick=oc_pr_sender channel=#oc-e2e
[server] recv PRIVMSG from=oc_pr_sender target=#oc-e2e text="xxxxxxxxx" codeUnits=9 loneSurrogate=false
[server] fwd PRIVMSG from=oc_pr_sender to=oc_pr_receiver target=#oc-e2e
[server] recv PRIVMSG from=oc_pr_sender target=#oc-e2e text="🙂rest" codeUnits=6 loneSurrogate=false
[server] fwd PRIVMSG from=oc_pr_sender to=oc_pr_receiver target=#oc-e2e
raw receiver received PRIVMSG chunks: 2
recv chunk 1: "xxxxxxxxx" codeUnits=9 containsEmoji=false loneSurrogate=false
recv chunk 2: "🙂rest" codeUnits=6 containsEmoji=true loneSurrogate=false
check allChunksWellFormed: PASS
check emojiArrivedIntact: PASS
check rejoinedEqualsOriginal: PASS
check usedRealTcpServerProcess: PASS
check usedRealRawSocketReceiver: PASS
check usedOpenClawConnectIrcClient: PASS
[server] cleanup SIGTERM received
[server] cleanup server closed
cleanup: server process killed pid=1924576 signal=SIGTERM exitCode=0

Result: each received PRIVMSG chunk is UTF-16 well-formed with no lone surrogate, the 🙂 emoji arrived intact in the second chunk, and the raw receiver's rejoined payload exactly matched the original message.

Tests

extensions/irc/src/client.surrogate.test.ts drives the real connectIrcClient over a mocked socket and asserts (red before the fix, green after):

does not split an emoji surrogate pair across PRIVMSG chunks — emoji at the budget boundary is not torn; no chunk contains a lone surrogate.
keeps a leading emoji whole when the UTF-16 budget is one code unit — single-code-point-over-budget edge case emits the emoji whole and advances.
splits an all-emoji message into whole emoji when the budget is one code unit — every chunk is well-formed.
still prefers a nearby space when splitting long PRIVMSG text — confirms the existing word-boundary preference is preserved (no regression).

Each test runs the lone-surrogate regex /[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]/ over every PRIVMSG body the client writes.

clawsweeper · 2026-06-24T22:27:54Z

Codex review: needs maintainer review before merge. Reviewed July 5, 2026, 3:09 PM ET / 19:09 UTC.

Summary
The branch changes IRC connectIrcClient PRIVMSG chunking to slice on UTF-16-safe boundaries and adds socket-level tests for emoji boundary and one-unit chunk cases.

PR surface: Source +11, Tests +178. Total +189 across 2 files.

Reproducibility: yes. The reproduction path is a long IRC PRIVMSG such as xxxxxxxxx🙂rest with messageChunkMaxChars=10; current main and v2026.6.11 raw-slice at that UTF-16 index, while the PR body supplies real TCP before/after-style proof for the fixed path.

Review metrics: 1 noteworthy metric.

Runtime outbound paths affected: 2 call paths. Both transient IRC sends and monitor replies route through the changed sendPrivmsg splitter, so maintainers should treat this as shared message-delivery behavior.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Risk before merge

[P1] The shared sendPrivmsg splitter feeds both transient IRC sends and monitor replies, so a chunking regression could reshape or drop outbound IRC messages across the channel.
[P1] fix(irc): long non-ASCII messages are silently truncated at the 512-byte line limit #99138 changes the same loop for the IRC 512-byte line limit; whichever PR lands second needs to preserve both the byte-budget and UTF-16 boundary invariants.
[P1] Live PR status currently reports check-lint failed on the current head, and the log was unavailable through gh during this review, so that check needs inspection or rerun before merge.

Maintainer options:

Resolve Check And Land This Fix First (recommended)
After the current lint failure is inspected or rerun cleanly, merge this focused boundary fix and make the byte-budget PR preserve the new UTF-16 invariant during rebase.
Pause For A Combined IRC Chunker
Hold this PR if maintainers prefer to merge one branch that addresses both surrogate-safe slicing and IRC byte-budget chunking together.

Next step before merge

No ClawSweeper repair is appropriate because no code defect was found; remaining action is maintainer merge review, current check-lint handling, and landing-order coordination with the byte-budget IRC PR.

Maintainer decision needed

Question: Should this surrogate-boundary IRC chunking fix land first after check-lint is resolved, or should maintainers pause it and combine it with the byte-budget splitter work?
Rationale: Both open PRs modify the same IRC PRIVMSG loop, and green focused tests do not by themselves decide the preferred landing order for the two delivery invariants.
Likely owner: steipete — steipete is assigned on this PR and already posted maintainer verification for the final reviewed IRC splitter shape.
Options:
- Land Boundary Fix First (recommended): Resolve or rerun the current lint failure, merge this focused fix, and require the byte-budget PR to rebase over the UTF-16-safe splitter.
- Pause For Combined Splitter: Hold this PR if maintainers want one combined branch to handle surrogate boundaries and the 512-byte IRC line budget before either change lands.

Security
Cleared: The diff is limited to IRC text chunking and focused tests, with no dependency, workflow, secret, package, permission, or generated-code change.

Review details

Best possible solution:

Land this focused surrogate-boundary fix after current-head checks are resolved, then rebase the IRC byte-budget PR over it so the final splitter enforces both delivery invariants.

Do we have a high-confidence way to reproduce the issue?

Yes. The reproduction path is a long IRC PRIVMSG such as xxxxxxxxx🙂rest with messageChunkMaxChars=10; current main and v2026.6.11 raw-slice at that UTF-16 index, while the PR body supplies real TCP before/after-style proof for the fixed path.

Is this the best way to solve the issue?

Yes. sendPrivmsg is the shared protocol choke point for transient sends and monitor replies, and reusing sliceUtf16Safe is narrower and more maintainable than duplicating surrogate logic upstream.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 39b5bf38f746.

Label changes

Label justifications:

P2: This is a bounded IRC message-delivery bug fix with focused proof and limited channel-specific blast radius.
merge-risk: 🚨 message-delivery: The PR changes the shared IRC outbound PRIVMSG chunker, where mistakes can split, reshape, or drop delivered messages.
rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body includes terminal real TCP IRC sender/receiver proof showing well-formed chunks, the emoji intact, and exact reassembly after the fix.
proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes terminal real TCP IRC sender/receiver proof showing well-formed chunks, the emoji intact, and exact reassembly after the fix.

Evidence reviewed

PR surface:

Source +11, Tests +178. Total +189 across 2 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	1	14	3	+11
Tests	1	178	0	+178
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	2	192	3	+189

What I checked:

Root policy read: Read the full root AGENTS.md and applied its PR-depth, plugin-boundary, message-delivery risk, and proof requirements. (AGENTS.md:1, 39b5bf38f746)
Scoped extension policy read: The scoped extension policy allows bundled plugin production code to import through openclaw/plugin-sdk/*, matching the new text utility runtime import. (extensions/AGENTS.md:1, 39b5bf38f746)
Current main still has unsafe raw slicing: Current main still chunks PRIVMSG bodies with chunk.slice(0, splitAt).trim(), which can split an astral character between surrogate halves. (extensions/irc/src/client.ts:223, 39b5bf38f746)
Latest release still has unsafe raw slicing: The latest release tag v2026.6.11 contains the same raw PRIVMSG split, so this fix is not already shipped. (extensions/irc/src/client.ts:220, e085fa1a3ffd)
PR head fixes the split at the choke point: The PR imports sliceUtf16Safe, keeps the one-unit leading astral progress exception, clamps the chunk budget to at least one, and calls the helper at the single PRIVMSG split point. (extensions/irc/src/client.ts:14, 44f75d394d63)
Shared helper contract supports the fix: sliceUtf16Safe adjusts slice boundaries away from dangling high or low surrogate halves, which is the dependency contract this bug needs. (packages/normalization-core/src/utf16-slice.ts:16, 39b5bf38f746)

Likely related people:

vignesh07: GitHub PR metadata for feat: IRC — add first-class channel support #11482 shows this person authored the first-class IRC plugin, including the central client, monitor, send, and protocol surfaces. (role: introduced IRC channel behavior; confidence: high; commits: c323e3fb50e5, b5bdf7b26f7e, 1b1af2392a8b; files: extensions/irc/src/client.ts, extensions/irc/src/send.ts, extensions/irc/src/monitor.ts)
Takhoffman: GitHub PR metadata shows this person merged the first-class IRC support PR and also contributed final documentation/config fixes on that original branch. (role: IRC feature merger and follow-up contributor; confidence: medium; commits: a2347e498f71, ecb39622cc61, fa906b26add7; files: extensions/irc/src/client.ts, docs/channels/irc.md)
steipete: This person is assigned on the PR, posted maintainer verification, and authored the final branch commits that refined the surrogate-safe splitter and one-unit budget tests. (role: recent verifier and PR branch contributor; confidence: high; commits: df96d5c61996, 44f75d394d63; files: extensions/irc/src/client.ts, extensions/irc/src/client.surrogate.test.ts)
llagy009: This person authored the current PR and the previously merged IRC surrogate-range literal escape fix, so they have recent domain history beyond this branch proposal. (role: adjacent surrogate-handling contributor; confidence: medium; commits: 1f9f7f2541db, 537672d238fd; files: extensions/irc/src/client.ts, extensions/irc/src/client.surrogate.test.ts, extensions/irc/src/protocol.ts)
vincentkoc: Current-main blame on the unchanged raw splitter points to a broad file-carrying commit, which is useful routing context but not strong evidence of IRC feature ownership. (role: recent current-main carrier; confidence: low; commits: 61ac4ec8a0db; files: extensions/irc/src/client.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (5 earlier review cycles)

reviewed 2026-06-27T17:45:36.146Z sha 8f18551 :: needs maintainer review before merge. :: none
reviewed 2026-07-05T15:03:49.972Z sha d365ae9 :: needs maintainer review before merge. :: none
reviewed 2026-07-05T15:14:57.271Z sha d365ae9 :: needs maintainer review before merge. :: none
reviewed 2026-07-05T18:27:28.539Z sha 9055dbe :: needs maintainer review before merge. :: none
reviewed 2026-07-05T18:48:49.639Z sha b4264c0 :: needs maintainer review before merge. :: none

steipete · 2026-07-05T18:29:28Z

Maintainer verification for final rebased head 44f75d394d6353fecd3c6ee8fae1cd08e9ad879d:

Retained the UTF-16-safe split at the single IRC sendPrivmsg choke point.
Simplified the astral-only progress exception and added the missing contract test: messageChunkMaxChars: 1 still sends BMP text as one-character chunks while a leading emoji is emitted whole.
Sanitized AWS Crabbox run_5e56622ff316: 14 focused IRC client tests passed on the final reviewed head.
Sanitized AWS Crabbox run_5ed4a26396f2: extension typechecks, extension test types, lint, and changed guards passed.
Fresh Codex autoreview after fixing its low-cap finding: clean, correctness 0.98. Repo review artifacts validate with zero findings.
Exact-head hosted CI: run 28751281845, passed after retrying one runner-lost lint shard.
The PR body already carries real TCP sender/receiver proof: no lone surrogate, emoji intact, and exact reassembly.

Known gap: I did not repeat the public-network IRC run; my independent final-head proof used the real client over socket-level transport mocks in a sanitized AWS environment. #99138 is complementary UTF-8 wire-byte work, not a duplicate, and should rebase over this boundary fix.

sendPrivmsg split long messages with String.slice on a UTF-16 code-unit index, so an emoji (or other astral character) straddling the split point was cut into a lone high/low surrogate, sending broken bytes in the PRIVMSG to the IRC server. Slice each chunk with sliceUtf16Safe so a surrogate pair is never split. When the budget is too small to fit even the leading astral character, emit that character whole so chunking still makes progress. Adds a socket-level test (fake net/tls socket) asserting no PRIVMSG chunk contains a lone surrogate, including the one-code-unit budget edge case, while the existing space-preferring split is preserved.

steipete · 2026-07-05T19:14:10Z

Merged via squash.

Prepared head SHA: 44f75d394d6353fecd3c6ee8fae1cd08e9ad879d
Landed commit: 685c22dfb7e110730e51fe3cb712fc913308882b

…penclaw#96572) * fix(irc): chunk PRIVMSG on UTF-16 boundary to avoid lone surrogates sendPrivmsg split long messages with String.slice on a UTF-16 code-unit index, so an emoji (or other astral character) straddling the split point was cut into a lone high/low surrogate, sending broken bytes in the PRIVMSG to the IRC server. Slice each chunk with sliceUtf16Safe so a surrogate pair is never split. When the budget is too small to fit even the leading astral character, emit that character whole so chunking still makes progress. Adds a socket-level test (fake net/tls socket) asserting no PRIVMSG chunk contains a lone surrogate, including the one-code-unit budget edge case, while the existing space-preferring split is preserved. * refactor(irc): make surrogate-safe chunking direct * fix(irc): preserve one-unit chunk limits --------- Co-authored-by: Peter Steinberger <[email protected]>

…void lone surrogates The Mattermost inbound handler in extensions/mattermost/src/mattermost/monitor.ts truncates the message body preview with bodyText.slice(0, 200), which can split a UTF-16 surrogate pair when an emoji (e.g. 🎉, 🦞) or other astral character lands at the 200-char boundary. The resulting lone surrogate half leaks into the verbose log preview and could propagate to downstream consumers that reject ill-formed strings. Replace the raw .slice(0, 200) with the shared truncateUtf16Safe helper (from openclaw/plugin-sdk/text-utility-runtime), which backs up one code unit when the cut would land inside a surrogate pair. This matches the same pattern already applied to the Slack, Discord, Telegram, IRC, Line, MS Teams, and Feishu channel previews (openclaw#96456, openclaw#96569, openclaw#96572, openclaw#96577, openclaw#96578, openclaw#97462, openclaw#97472, openclaw#98994). The preview is only used for the verbose inbound log line so the behavioral change is cosmetic: when the 200th code unit would split an emoji, the preview now includes one fewer code unit rather than emitting a broken surrogate.

openclaw-barnacle Bot added channel: irc size: S triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. labels Jun 24, 2026

WeeLi-009 force-pushed the fix/irc-privmsg-surrogate branch from 4e945f0 to 1230b28 Compare June 25, 2026 00:23

clawsweeper Bot added the merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. label Jun 25, 2026

WeeLi-009 force-pushed the fix/irc-privmsg-surrogate branch from 1230b28 to 8f18551 Compare June 25, 2026 04:12

openclaw-barnacle Bot removed the triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. label Jun 25, 2026

WeeLi-009 mentioned this pull request Jun 29, 2026

fix(irc): guard surrogate-range codepoints in \u literal-escape decoder #97683

Merged

clawsweeper Bot mentioned this pull request Jul 1, 2026

feat(irc): support markdown messages via draft/multiline #55901

Open

yetval mentioned this pull request Jul 2, 2026

fix(irc): long non-ASCII messages are silently truncated at the 512-byte line limit #99138

Merged

WeeLi-009 force-pushed the fix/irc-privmsg-surrogate branch from 8f18551 to d365ae9 Compare July 5, 2026 14:48

steipete self-assigned this Jul 5, 2026

clawsweeper Bot added the merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. label Jul 5, 2026

steipete force-pushed the fix/irc-privmsg-surrogate branch 2 times, most recently from 4aa2692 to b4264c0 Compare July 5, 2026 18:43

Leon-SK668 and others added 3 commits July 5, 2026 19:52

refactor(irc): make surrogate-safe chunking direct

df96d5c

fix(irc): preserve one-unit chunk limits

44f75d3

steipete force-pushed the fix/irc-privmsg-surrogate branch from b4264c0 to 44f75d3 Compare July 5, 2026 18:52

steipete merged commit 685c22d into openclaw:main Jul 5, 2026
142 of 143 checks passed

steipete mentioned this pull request Jul 5, 2026

docs(changelog): record five landed fixes #100441

Merged

github-actions Bot mentioned this pull request Jul 5, 2026

📡 Upstream Digest — 2026-07-05 20:49 UTC curtismercier/openclaw-mods#1218

Open

clawsweeper Bot mentioned this pull request Jul 14, 2026

fix(irc): disconnect on oversized inbound lines #107376

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(irc): chunk PRIVMSG on UTF-16 boundary to avoid lone surrogates#96572

fix(irc): chunk PRIVMSG on UTF-16 boundary to avoid lone surrogates#96572
steipete merged 3 commits into
openclaw:mainfrom
WeeLi-009:fix/irc-privmsg-surrogate

WeeLi-009 commented Jun 24, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jun 24, 2026 •

edited

Loading

Uh oh!

steipete commented Jul 5, 2026 •

edited

Loading

Uh oh!

Uh oh!

steipete commented Jul 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

WeeLi-009 commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Fix

Evidence

Tests

Uh oh!

clawsweeper Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steipete commented Jul 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

steipete commented Jul 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

WeeLi-009 commented Jun 24, 2026 •

edited

Loading

clawsweeper Bot commented Jun 24, 2026 •

edited

Loading

steipete commented Jul 5, 2026 •

edited

Loading