fix(core): keep backend truncation UTF-16 safe by xialonglee · Pull Request #100244 · openclaw/openclaw

xialonglee · 2026-07-05T08:59:06Z

What Problem This Solves

Backend text truncation could split UTF-16 surrogate pairs in two places:

ACP parent-stream previews at the 220-character display cutoff.
ACP's 4,000-character retained buffer, where a long delta could begin the retained tail on a lone low surrogate.
Active-memory recall summaries at their configured prompt-context limit.

The malformed halves can surface as replacement characters in progress text or injected memory context.

Why This Change Was Made

Use truncateUtf16Safe for ACP preview and active-memory summary cutoffs.
Use sliceUtf16Safe when retaining the tail of a long ACP delta.
Preserve active-memory's existing word-boundary selection.
Replace duplicated surrogate scanners with exact-output regression cases.

The original chat-display and native approval changes were removed. This PR is intentionally limited to non-UI, non-approval backend paths.

This also folds the valid active-memory fix from #98711, preserving contributor credit.

User Impact

Emoji and other supplementary-plane characters remain valid when backend relay or memory text is truncated. Limits, ellipsis behavior, routing, and configuration remain unchanged.

Evidence

Prepared head: 00266be5318b07012f6b4c17b338243ef9687f88
Blacksmith Testbox: tbx_01kws0qtfp8bxaxfy381gtqt9z
- exact-head check:test-types: passed;
- exact-head focused serial proof: 192 tests passed;
- the pre-rebase repair tree passed check:changed; exact-head CI covers the final rebase.
git diff --check and four-file oxfmt --check: passed.
Fresh exact-head autoreview: clean, correctness confidence 0.89.
Raw-slice reproduction confirmed lone code units 0xD83D, 0xDE00, and 0xD83C at the three repaired boundaries.

Exact-head hosted CI: https://github.com/openclaw/openclaw/actions/runs/28741038571

clawsweeper · 2026-07-05T09:10:04Z

Codex review: needs maintainer review before merge. Reviewed July 5, 2026, 8:33 AM ET / 12:33 UTC.

Summary
The PR changes active-memory summary truncation and ACP parent-stream preview/tail retention to use existing surrogate-safe UTF-16 helpers, with focused regression tests for those backend paths.

PR surface: Source +3, Tests +63. Total +66 across 4 files.

Reproducibility: yes. Source inspection shows current main uses raw UTF-16 code-unit slicing at the ACP and active-memory boundaries, and the PR's regression cases target those exact split-surrogate positions.

Review metrics: none identified.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Risk before merge

[P1] Some exact-head CI jobs were still in progress when inspected, so merge should wait for the normal required checks to finish even though no code-level blocker was found.

Maintainer options:

Decide the mitigation before merge
Land this narrowed PR after routine exact-head CI and maintainer review, then treat the active-memory-only related PR as superseded if this broader backend fix merges.
Pause or close
Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

No ClawSweeper repair job is needed because the review found no blocking patch defect; the remaining action is normal maintainer and CI handling.

Security
Cleared: The diff only changes string truncation helper use and tests, with no dependency, workflow, secret, package, or code-execution surface changes.

Review details

Best possible solution:

Land this narrowed PR after routine exact-head CI and maintainer review, then treat the active-memory-only related PR as superseded if this broader backend fix merges.

Do we have a high-confidence way to reproduce the issue?

Yes. Source inspection shows current main uses raw UTF-16 code-unit slicing at the ACP and active-memory boundaries, and the PR's regression cases target those exact split-surrogate positions.

Is this the best way to solve the issue?

Yes. Reusing the existing UTF-16-safe helper is the narrow maintainable fix here, and the PR correctly leaves broader chat-display and native-approval surfaces out after narrowing.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against fd2e4da00651.

Label changes

Label changes:

add rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster. Replaced prior rating: 🐚 platinum hermit.
remove rating: 🐚 platinum hermit: Current PR rating is rating: 🦞 diamond lobster, so this older rating label is no longer current.

Label justifications:

P2: This is a bounded backend correctness fix for malformed truncated Unicode in agent progress and memory context text.
rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster. Replaced prior rating: 🐚 platinum hermit.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body includes exact-head Blacksmith Testbox validation, check:test-types, focused serial proof, and raw-slice boundary reproduction for the repaired behavior.
proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes exact-head Blacksmith Testbox validation, check:test-types, focused serial proof, and raw-slice boundary reproduction for the repaired behavior.

Evidence reviewed

PR surface:

Source +3, Tests +63. Total +66 across 4 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	2	9	6	+3
Tests	2	84	21	+63
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	4	93	27	+66

What I checked:

Repository policy applied: Root AGENTS.md plus scoped extensions/AGENTS.md and src/agents/AGENTS.md were read fully; the review applied the plugin-boundary and deep PR review requirements. (AGENTS.md:1, fd2e4da00651)
Current main reproduces the raw-slice boundary: Current main still uses raw String.slice in active-memory truncation and ACP preview/tail retention, so a surrogate pair can be split at those code-unit boundaries. (extensions/active-memory/index.ts:2573, c64a306f908b)
Patch uses the existing safe helper contract: The PR head imports sliceUtf16Safe/truncateUtf16Safe from the existing normalization-core helper and applies them to ACP preview truncation and retained-tail slicing. (src/agents/acp-spawn-parent-stream.ts:6, e1bb413600b6)
Active-memory behavior stays word-boundary aware: The PR head keeps the raw cutoff for word-boundary selection while returning UTF-16-safe slices, preserving the previous truncation policy without adding a new plugin seam. (extensions/active-memory/index.ts:2573, e1bb413600b6)
Focused regression coverage exists: The PR adds exact-output tests for active-memory split-surrogate cases and ACP preview/tail retention boundaries. (extensions/active-memory/index.test.ts:5370, e1bb413600b6)
Helper export contract checked: normalization-core exports the utf16-slice subpath, and the plugin SDK text-utility runtime already re-exports the safe slicing helpers for bundled plugin use. (packages/normalization-core/package.json:47, fd2e4da00651)

Likely related people:

steipete: Current-main blame attributes the raw active-memory and ACP truncation code to c64a306, merged through fix(macos): preserve PATH for SSH helpers #100214. (role: introduced current behavior; confidence: medium; commits: c64a306f908b; files: extensions/active-memory/index.ts, src/agents/acp-spawn-parent-stream.ts, packages/normalization-core/src/utf16-slice.ts)
vincentkoc: Live PR timeline shows vincentkoc assigned to the PR and force-pushed the narrowing commit that removed broader UI/native-hook changes while keeping the backend fix. (role: recent reviewer and narrowing contributor; confidence: medium; commits: e1bb413600b6; files: extensions/active-memory/index.ts, src/agents/acp-spawn-parent-stream.ts)
ZengWen-DT: ZengWen-DT authored the related active-memory-only PR that this branch explicitly folds into the broader backend truncation fix. (role: related active-memory contributor; confidence: medium; commits: 460e04d401c9; files: extensions/active-memory/index.ts, extensions/active-memory/index.test.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (4 earlier review cycles)

reviewed 2026-07-05T09:13:36.775Z sha b113104 :: needs real behavior proof before merge. :: [P1] Import the UTF-16 helper from an existing module
reviewed 2026-07-05T09:47:39.430Z sha 7ddd1aa :: needs real behavior proof before merge. :: [P1] Import the UTF-16 helper from the current main export
reviewed 2026-07-05T11:43:30.665Z sha dc114e4 :: needs real behavior proof before merge. :: none
reviewed 2026-07-05T12:20:00.345Z sha e22a266 :: needs maintainer review before merge. :: none

vincentkoc · 2026-07-05T12:34:46Z

Maintainer repair is land-ready.

Prepared head: 00266be5318b07012f6b4c17b338243ef9687f88
Scope: four backend files only. The submitted chat-display UI and native approval changes were removed.
Fix: ACP preview and retained-buffer boundaries now preserve UTF-16 surrogate pairs; active-memory recall summaries use the same safe boundary helper while retaining existing word-boundary behavior.
Folded fix(active-memory): keep recall summaries UTF-16 safe at truncation boundary #98711's active-memory fix with contributor credit.
Blacksmith Testbox tbx_01kws0qtfp8bxaxfy381gtqt9z:
- pnpm check:test-types passed on the exact head.
- pnpm test:serial src/agents/acp-spawn-parent-stream.test.ts extensions/active-memory/index.test.ts passed: 192 tests.
Exact-head CI passed: https://github.com/openclaw/openclaw/actions/runs/28741038571
git diff --check and four-file oxfmt --check passed.
Fresh exact-head autoreview: clean, correctness confidence 0.89.

The earlier check-test-types failure was an unrelated current-main fixture regression introduced by 1f484a8dbde; it was repaired on main in 931f13c before the final rebase.

…lay, and native hook relay

Co-authored-by: xialonglee <[email protected]> Co-authored-by: ZengWen-DT <[email protected]>

vincentkoc · 2026-07-05T12:48:40Z

Merged via squash.

Prepared head SHA: 00266be5318b07012f6b4c17b338243ef9687f88
Landed commit: 66081c09ee501d5edd8b0d2f0fa46218f8934833

* fix(core): use UTF-16-safe truncation for chat display, ACP stream relay, and native hook relay * fix(core): narrow UTF-16 truncation repair Co-authored-by: xialonglee <[email protected]> Co-authored-by: ZengWen-DT <[email protected]> --------- Co-authored-by: Vincent Koc <[email protected]> Co-authored-by: ZengWen-DT <[email protected]>

openclaw-barnacle Bot added gateway Gateway runtime agents Agent runtime and tooling size: S labels Jul 5, 2026

xialonglee force-pushed the fix/utf16-safe-truncation-core branch from 7ddd1aa to dc114e4 Compare July 5, 2026 11:12

vincentkoc self-assigned this Jul 5, 2026

clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 5, 2026

openclaw-barnacle Bot added docs Improvements or additions to documentation channel: imessage Channel integration: imessage scripts Repository scripts docker Docker and sandbox tooling channel: qqbot size: XL and removed gateway Gateway runtime size: S labels Jul 5, 2026

vincentkoc force-pushed the fix/utf16-safe-truncation-core branch from 8465086 to 0cb26b7 Compare July 5, 2026 11:46

openclaw-barnacle Bot added size: S and removed docs Improvements or additions to documentation channel: imessage Channel integration: imessage scripts Repository scripts docker Docker and sandbox tooling channel: qqbot size: XL labels Jul 5, 2026

vincentkoc changed the title ~~fix(core): use UTF-16-safe truncation for chat display, ACP stream relay, and native hook relay~~ fix(core): keep backend truncation UTF-16 safe Jul 5, 2026

vincentkoc force-pushed the fix/utf16-safe-truncation-core branch 2 times, most recently from bed8948 to e1bb413 Compare July 5, 2026 12:23

vincentkoc force-pushed the fix/utf16-safe-truncation-core branch from e1bb413 to e883141 Compare July 5, 2026 12:35

xialonglee and others added 2 commits July 5, 2026 05:38

fix(core): use UTF-16-safe truncation for chat display, ACP stream re…

c0843bf

…lay, and native hook relay

fix(core): narrow UTF-16 truncation repair

00266be

Co-authored-by: xialonglee <[email protected]> Co-authored-by: ZengWen-DT <[email protected]>

vincentkoc force-pushed the fix/utf16-safe-truncation-core branch from e883141 to 00266be Compare July 5, 2026 12:38

vincentkoc merged commit 66081c0 into openclaw:main Jul 5, 2026
104 checks passed

vincentkoc mentioned this pull request Jul 5, 2026

fix(active-memory): keep recall summaries UTF-16 safe at truncation boundary #98711

Closed

xialonglee deleted the fix/utf16-safe-truncation-core branch July 5, 2026 12:50

github-actions Bot mentioned this pull request Jul 5, 2026

📡 Upstream Digest — 2026-07-05 13:13 UTC curtismercier/openclaw-mods#1213

Open

clawsweeper Bot mentioned this pull request Jul 6, 2026

fix(codex): use UTF-16-safe truncation for approval display paths #100177

Merged

wangmiao0668000666 mentioned this pull request Jul 9, 2026

fix(matrix): keep HTTP error and tool-progress truncations UTF-16 safe #102395

Merged

clawsweeper Bot mentioned this pull request Jul 9, 2026

fix(active-memory): use truncateUtf16Safe for log value truncation #102551

Merged

clawsweeper Bot mentioned this pull request Jul 17, 2026

fix(acpx): preserve Unicode in catalog tool arguments #109589

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(core): keep backend truncation UTF-16 safe#100244

fix(core): keep backend truncation UTF-16 safe#100244
vincentkoc merged 2 commits into
openclaw:mainfrom
xialonglee:fix/utf16-safe-truncation-core

xialonglee commented Jul 5, 2026 •

edited by vincentkoc

Loading

Uh oh!

clawsweeper Bot commented Jul 5, 2026 •

edited

Loading

Uh oh!

vincentkoc commented Jul 5, 2026 •

edited

Loading

Uh oh!

Uh oh!

vincentkoc commented Jul 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

xialonglee commented Jul 5, 2026 • edited by vincentkoc Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Uh oh!

clawsweeper Bot commented Jul 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vincentkoc commented Jul 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

vincentkoc commented Jul 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

xialonglee commented Jul 5, 2026 •

edited by vincentkoc

Loading

clawsweeper Bot commented Jul 5, 2026 •

edited

Loading

vincentkoc commented Jul 5, 2026 •

edited

Loading