fix(memory): snippets split emoji when truncated by qingminglong · Pull Request #102478 · openclaw/openclaw

qingminglong · 2026-07-09T06:50:34Z

What Problem This Solves

Fixes an issue where users reading truncated memory excerpts or provider response error snippets could see malformed text when the truncation boundary landed inside an emoji or other UTF-16 surrogate pair.

Why This Change Was Made

This keeps memory-host snippet truncation on a UTF-16-safe boundary within the package-local string helper layer, then applies that same invariant to both response snippets and single-line memory read excerpts. The change stays inside the memory-host SDK boundary and preserves the existing continuation notice behavior, including when the safe visible prefix is empty.

User Impact

Truncated memory and response snippets no longer expose dangling surrogate halves or replacement-character-style output when text contains emoji near the character cap. Memory read results still tell users how to continue or fall back to the raw source line when the excerpt is capped.

Evidence

Claim proved: response snippets and memory read single-line excerpts truncate on UTF-16-safe boundaries and keep read continuation guidance for empty safe prefixes.
Current-head proof at 5cc3a3d: node scripts/run-vitest.mjs packages/memory-host-sdk/src/host/response-snippet.test.ts packages/memory-host-sdk/src/host/read-file-shared.test.ts passed on branch codex/fix-memory-surrogate-truncation with 2 files and 8 tests passing.
PR diff hygiene: git diff --check upstream/main...HEAD passed.
Review proof: python .agents\skills\autoreview\scripts\autoreview --mode branch --base upstream/main passed with no accepted/actionable findings.
Observed result: readResponseTextSnippet returns abc... [truncated] for an emoji split at the cap, and buildMemoryReadResultFromSlice returns a continuation notice without preserving a dangling surrogate when a leading emoji is dropped.

…rrogate-truncation

clawsweeper · 2026-07-09T06:58:20Z

Codex review: needs maintainer review before merge. Reviewed July 9, 2026, 3:10 AM ET / 07:10 UTC.

Summary
The PR adds a package-local UTF-16-safe truncation helper and uses it for memory read excerpts and response text snippets, with focused regression tests.

PR surface: Source +22, Tests +42. Total +64 across 5 files.

Reproducibility: yes. Current main visibly uses raw String.slice in both affected truncation paths, and the PR tests exercise emoji boundaries that would split a UTF-16 surrogate pair before the change.

Review metrics: none identified.

Stored data model
Persistent data-model change detected: vector/embedding metadata: packages/memory-host-sdk/src/host/read-file-shared.test.ts, vector/embedding metadata: packages/memory-host-sdk/src/host/read-file-shared.ts, vector/embedding metadata: packages/memory-host-sdk/src/host/string-utils.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🐚 platinum hermit
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

none.

Next step before merge

[P2] No repair job is needed because this PR is already the narrow implementation candidate and I found no actionable patch defect.

Security
Cleared: The diff only changes package-local string truncation and tests, with no dependency, workflow, secret, install, or code-execution surface changes.

Review details

Best possible solution:

Land this narrow helper/test fix after exact-head merge and CI gates; broader byte-decoding or grapheme-cluster improvements should be handled separately if they are observed.

Do we have a high-confidence way to reproduce the issue?

Yes. Current main visibly uses raw String.slice in both affected truncation paths, and the PR tests exercise emoji boundaries that would split a UTF-16 surrogate pair before the change.

Is this the best way to solve the issue?

Yes. The package-local helper is the narrowest maintainable fix for these host SDK call sites because it preserves the existing memory-host boundary and avoids importing the broad runtime facade into lightweight string/result helpers.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against c20ce00d19ec.

Label changes

Label changes:

add P2: This is a normal-priority memory-host bug fix for malformed truncated text with limited blast radius and focused tests.
add proof: sufficient: Contributor real behavior proof is sufficient. Sufficient for this non-visual helper fix: the PR body includes an exact current-head command and copied observed outputs for both changed behavior paths, with CI also green on the relevant shard.
add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): Sufficient for this non-visual helper fix: the PR body includes an exact current-head command and copied observed outputs for both changed behavior paths, with CI also green on the relevant shard.

Label justifications:

P2: This is a normal-priority memory-host bug fix for malformed truncated text with limited blast radius and focused tests.
rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): Sufficient for this non-visual helper fix: the PR body includes an exact current-head command and copied observed outputs for both changed behavior paths, with CI also green on the relevant shard.
proof: sufficient: Contributor real behavior proof is sufficient. Sufficient for this non-visual helper fix: the PR body includes an exact current-head command and copied observed outputs for both changed behavior paths, with CI also green on the relevant shard.

Evidence reviewed

PR surface:

Source +22, Tests +42. Total +64 across 5 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	3	25	3	+22
Tests	2	42	0	+42
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	5	67	3	+64

What I checked:

Current main response snippet behavior: Current main still appends the truncation suffix after collapsed.slice(0, maxChars), so a character cap can cut between a UTF-16 surrogate pair. (packages/memory-host-sdk/src/host/response-snippet.ts:44, c20ce00d19ec)
Current main memory read behavior: Current main still truncates single-line memory excerpts with text.slice(0, maxChars) before this PR. (packages/memory-host-sdk/src/host/read-file-shared.ts:51, c20ce00d19ec)
PR helper implementation: The PR head adds truncateUtf16Safe, which backs off one code unit when the limit would leave a high surrogate without its low surrogate. (packages/memory-host-sdk/src/host/string-utils.ts:23, 5cc3a3d9da7e)
PR read excerpt call site: The PR head routes single-line memory excerpt truncation through truncateUtf16Safe and keeps the continuation notice even when the safe prefix is empty. (packages/memory-host-sdk/src/host/read-file-shared.ts:52, 5cc3a3d9da7e)
PR response snippet call site: The PR head routes response snippet character truncation through the same package-local UTF-16-safe helper. (packages/memory-host-sdk/src/host/response-snippet.ts:46, 5cc3a3d9da7e)
Regression tests: The PR head adds tests for response snippets, normal memory read truncation, and the empty safe-prefix continuation-notice case. (packages/memory-host-sdk/src/host/read-file-shared.test.ts:38, 5cc3a3d9da7e)

Likely related people:

vincentkoc: Live commit history credits this account with introducing and maintaining response-snippet.ts, including bounded remote error bodies, bounded remote JSON responses, and abort handling. (role: recent area contributor; confidence: high; commits: 93577ad58753, 4ad875308f5c, be7d86ed8005; files: packages/memory-host-sdk/src/host/response-snippet.ts, packages/memory-host-sdk/src/host/string-utils.ts, packages/memory-host-sdk/src/host/post-json.ts)
steipete: Live commit history credits this account with recent changes to read-file-shared.ts and package documentation around the memory host SDK contract surface. (role: adjacent read-result contributor; confidence: medium; commits: 9596b7bd7a0a, 5b98f03c64d2, 4df95d3c3fd6; files: packages/memory-host-sdk/src/host/read-file-shared.ts, packages/memory-host-sdk/src/host/read-file-shared.test.ts, packages/memory-host-sdk/src/engine-storage.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

steipete · 2026-07-09T07:27:32Z

Land-ready at exact head f5154db39dc2761e65488f419e4e76edf41a5aa3.

Review improvement: replaced the newly added package-local UTF-16 truncation implementation with the canonical dependency-free @openclaw/normalization-core/utf16-slice helper. The exact tests retain coverage for memory excerpts, response snippets, and the one-unit budget where no part of the first emoji fits but the continuation notice must remain.

Proof:

node scripts/run-vitest.mjs packages/memory-host-sdk/src/host/read-file-shared.test.ts packages/memory-host-sdk/src/host/response-snippet.test.ts — 8 tests passed.
git diff --check passed.
Exact-head CI run 29001061888 passed.
Fresh exact-head autoreview found no actionable findings at 0.98 confidence.

No known proof gaps.

steipete · 2026-07-09T07:27:47Z

Merged via squash.

Prepared head SHA: f5154db39dc2761e65488f419e4e76edf41a5aa3
Landed commit: c3d567db95d0d17f36f484ad9b9c0d14800614b3

* fix(memory): snippets split emoji when truncated * refactor(memory): reuse shared UTF-16 truncation --------- Co-authored-by: Peter Steinberger <[email protected]>

fix(memory): snippets split emoji when truncated

28dd59f

openclaw-barnacle Bot added the size: S label Jul 9, 2026

qingminglong added 2 commits July 9, 2026 14:52

Merge remote-tracking branch 'upstream/main' into codex/fix-memory-su…

ea4fa39

…rrogate-truncation

Merge remote-tracking branch 'upstream/main' into codex/fix-memory-su…

5cc3a3d

…rrogate-truncation

refactor(memory): reuse shared UTF-16 truncation

f5154db

steipete merged commit c3d567d into openclaw:main Jul 9, 2026
99 checks passed

clawsweeper Bot mentioned this pull request Jul 9, 2026

fix(talk): use truncateUtf16Safe for LLM-prompt-facing text truncation #102477

Merged

clawsweeper Bot mentioned this pull request Jul 9, 2026

fix(infra): drop replacement characters when response snippets truncate UTF-8 #102532

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(memory): snippets split emoji when truncated#102478

fix(memory): snippets split emoji when truncated#102478
steipete merged 4 commits into
openclaw:mainfrom
qingminglong:codex/fix-memory-surrogate-truncation

qingminglong commented Jul 9, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jul 9, 2026 •

edited

Loading

Uh oh!

steipete commented Jul 9, 2026

Uh oh!

Uh oh!

steipete commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

qingminglong commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Uh oh!

clawsweeper Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steipete commented Jul 9, 2026

Uh oh!

Uh oh!

steipete commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

qingminglong commented Jul 9, 2026 •

edited

Loading

clawsweeper Bot commented Jul 9, 2026 •

edited

Loading