fix(talk): use truncateUtf16Safe for LLM-prompt-facing text truncation by xydt-tanshanshan · Pull Request #102477 · openclaw/openclaw

xydt-tanshanshan · 2026-07-09T06:48:16Z

What Problem This Solves

Two LLM-prompt-facing text truncation sites use raw UTF-16 .slice(0, N), which can produce lone surrogates when truncation boundaries cross emoji surrogate pairs.

Why This Change Was Made

Replace raw slice with the canonical truncateUtf16Safe helper:

src/talk/fast-context-runtime.ts — normalizeSnippet truncates memory snippets at 700 chars for realtime response prompts
src/infra/heartbeat-events-filter.ts — buildExecEventPrompt truncates command output at 8000 chars for LLM prompts

User Impact

None for typical text. Memory snippets or command output that happen to contain emoji at the truncation boundary may previously have produced malformed text in the LLM prompt.

Real Behavior Proof

Behavior addressed: Prevent lone surrogates in LLM-prompt-facing text truncation at emoji boundaries
Real environment tested: Linux, Node 22.19, commit da8182b
Exact steps or command run after this patch:

node --import tsx --input-type=module -e "
import { truncateUtf16Safe } from '@openclaw/normalization-core/utf16-slice';

// normalizeSnippet scenario: emoji at 700-char boundary
const text700 = 'x'.repeat(697) + '🚀' + 'yz';
const sliced = text700.slice(0, 699);
console.log('BEFORE slice(0,699): last code unit =', sliced.charCodeAt(698).toString(16), '(lone surrogate)');
const safe = truncateUtf16Safe(text700, 699);
console.log('AFTER  truncateUtf16Safe: length =', safe.length, '(safe)');

// buildExecEventPrompt scenario: emoji at 8000-char boundary
const text8k = 'x'.repeat(7997) + '🚀' + 'yz';
const sliced8k = text8k.slice(0, 8000);
console.log('');
console.log('BEFORE slice(0,8000): contains lone surrogate:', sliced8k.includes('\uD83D'));
const safe8k = truncateUtf16Safe(text8k, 8000);
console.log('AFTER  truncateUtf16Safe: valid text:', ![...safe8k].some(c => c >= '\uDC00' && c <= '\uDFFF'));
"

Evidence after fix:

=== normalizeSnippet scenario (700 char boundary) ===
BEFORE slice(0,699): last code unit = de80 (lone surrogate)
AFTER  truncateUtf16Safe: length = 699 (safe)

=== buildExecEventPrompt scenario (8000 char boundary) ===
BEFORE slice(0,8000): contains lone surrogate: true
AFTER  truncateUtf16Safe: valid text: true

Observed result after fix: Both truncation sites now safe at emoji boundaries. Raw slice(0, N) produces lone surrogate 0xDE80; truncateUtf16Safe drops the preceding high surrogate and produces valid text.
What was not tested: E2E agent loop with real LLM consuming emoji-rich output at exact truncation boundary (requires real model inference)

fast-context-runtime.ts normalizeSnippet and heartbeat-events-filter.ts buildExecEventPrompt used raw .slice(0, N), which can produce lone surrogates when truncation boundaries cross emoji surrogate pairs.

clawsweeper · 2026-07-09T06:55:04Z

Codex review: needs real behavior proof before merge. Reviewed July 9, 2026, 3:32 AM ET / 07:32 UTC.

Summary
The branch replaces Talk fast-context snippet truncation with truncateUtf16Safe and adds a focused regression test for the surrogate-pair boundary.

PR surface: Source +1, Tests +40. Total +41 across 2 files.

Reproducibility: yes. for source-level reproduction: current main still uses raw UTF-16 slice in the Talk fast-context prompt path, and the shared helper contract shows why that can split surrogate pairs. I did not run a live failing runtime path in this read-only review.

Review metrics: none identified.

Merge readiness
Overall: 🦪 silver shellfish
Proof: 🦪 silver shellfish
Patch quality: 🐚 platinum hermit
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

[P1] Add redacted terminal output, logs, copied live output, or a short recording that invokes resolveRealtimeVoiceFastContextConsult with an emoji at the truncation boundary and shows the actual generated prompt text after the fix.

Proof guidance:

[P1] Needs stronger real behavior proof before merge: The PR body includes terminal-style helper output, but it does not exercise the changed Talk prompt path after the fix; the added mocked regression test is useful CI proof but not contributor real-behavior proof. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

[P1] The posted proof is still insufficient because it exercises truncateUtf16Safe directly, not resolveRealtimeVoiceFastContextConsult or the produced Talk prompt, and the earlier sample boundary does not prove the claimed before-state.
[P1] I did not run tests in this read-only review; merge confidence relies on source inspection, the added regression test, and GitHub check state rather than fresh local or Crabbox execution.

Maintainer options:

Decide the mitigation before merge
Land the narrow Talk fast-context helper substitution after the contributor adds corrected redacted terminal or live output that calls the actual prompt path; keep the heartbeat fix from fix(auto-reply,infra): keep startup context and heartbeat event text UTF-16 safe #102483 on main.
Pause or close
Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

[P1] No automated repair is indicated; the contributor needs to provide corrected real-behavior proof before ordinary maintainer merge review.

Security
Cleared: The live PR diff only imports an existing internal helper and adds a focused unit test, with no dependency, workflow, secret, permission, package-resolution, or code-execution surface change.

Review details

Best possible solution:

Land the narrow Talk fast-context helper substitution after the contributor adds corrected redacted terminal or live output that calls the actual prompt path; keep the heartbeat fix from #102483 on main.

Do we have a high-confidence way to reproduce the issue?

Yes for source-level reproduction: current main still uses raw UTF-16 slice in the Talk fast-context prompt path, and the shared helper contract shows why that can split surrogate pairs. I did not run a live failing runtime path in this read-only review.

Is this the best way to solve the issue?

Yes for the code shape: using the existing truncateUtf16Safe package export is the narrowest maintainable fix and matches the already-merged heartbeat solution. The remaining blocker is proof quality, not implementation direction.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against c3d567db95d0.

Label changes

Label justifications:

P2: This is a limited-scope prompt-text correctness fix for malformed Unicode at exact truncation boundaries.
rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The PR body includes terminal-style helper output, but it does not exercise the changed Talk prompt path after the fix; the added mocked regression test is useful CI proof but not contributor real-behavior proof. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence reviewed

PR surface:

Source +1, Tests +40. Total +41 across 2 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	1	2	1	+1
Tests	1	40	0	+40
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	2	42	1	+41

What I checked:

Repository policy read: Root AGENTS.md was read fully and applied; no scoped AGENTS.md owns src/talk or src/infra directly. (AGENTS.md:1, c3d567db95d0)
Current main still has the remaining Talk issue: normalizeSnippet still truncates prompt-facing memory snippets with raw normalized.slice(0, MAX_SNIPPET_CHARS - 1) on current main. (src/talk/fast-context-runtime.ts:72, c3d567db95d0)
PR diff fixes the remaining Talk truncation site: The live PR file list shows the current merge diff only changes src/talk/fast-context-runtime.ts plus its test, replacing the raw slice with the existing UTF-16-safe helper. (src/talk/fast-context-runtime.ts:70, 4ca80bc83956)
Regression test added for the Talk path: The PR adds a mocked resolveRealtimeVoiceFastContextConsult regression test with an emoji at the fast-context snippet boundary and expects the safe prefix plus ellipsis. (src/talk/fast-context-runtime.test.ts:82, 4ca80bc83956)
Shared helper contract supports the fix: truncateUtf16Safe floors the requested limit, leaves short inputs unchanged, and delegates to sliceUtf16Safe, which adjusts cut points that would split surrogate pairs. (packages/normalization-core/src/utf16-slice.ts:44, c3d567db95d0)
Package export supports the import: @openclaw/normalization-core/utf16-slice is an exported subpath in the package contract. (packages/normalization-core/package.json:47, c3d567db95d0)

Likely related people:

chengzhichao-xydt: Shallow blame for normalizeSnippet, MAX_SNIPPET_CHARS, and the raw fast-context slice points to 9e75ce9dd1, which added the current Talk fast-context runtime and the shared UTF-16 helper in the available checkout history. (role: introduced current Talk runtime in available history; confidence: medium; commits: 9e75ce9dd1e1; files: src/talk/fast-context-runtime.ts, packages/normalization-core/src/utf16-slice.ts)
wings1029: Merged PR fix(auto-reply,infra): keep startup context and heartbeat event text UTF-16 safe #102483 fixed the heartbeat exec prompt overlap with the same helper and added heartbeat regression coverage now present on main. (role: adjacent surrogate-safe truncation contributor; confidence: high; commits: f0cc5e500f66; files: src/infra/heartbeat-events-filter.ts, src/infra/heartbeat-events-filter.test.ts)
qingminglong: Merged PR fix(memory): snippets split emoji when truncated #102478 fixed related memory snippet surrogate-safe truncation on current main, which is adjacent to the Talk memory snippet path. (role: adjacent memory truncation contributor; confidence: medium; commits: c3d567db95d0; files: packages/memory-host-sdk/src/host/read-file-shared.ts, packages/memory-host-sdk/src/host/response-snippet.ts)
steipete: The current PR includes a follow-up commit from this author adding the Talk fast-context regression test, and related merged surrogate-safe truncation PRs also show test/refactor follow-up from the same area. (role: PR follow-up test author and adjacent reviewer; confidence: medium; commits: a74cb606b9b6, b99c82b99e82, f5154db39dc2; files: src/talk/fast-context-runtime.test.ts, src/infra/heartbeat-events-filter.test.ts, packages/memory-host-sdk/src/host/read-file-shared.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (2 earlier review cycles)

reviewed 2026-07-09T06:58:54.364Z sha da8182b :: needs real behavior proof before merge. :: none
reviewed 2026-07-09T07:21:39.372Z sha da8182b :: needs real behavior proof before merge. :: none

xydt-tanshanshan · 2026-07-09T07:12:28Z

@clawsweeper re-review

clawsweeper · 2026-07-09T07:12:31Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

steipete · 2026-07-09T07:39:28Z

Land-ready at exact head 4ca80bc83956e9afc18cc19847b37423d4eda8e8.

Review improvements:

added an exact production-path Talk test with an emoji straddling the 699-unit snippet-content cutoff;
updated the branch onto current main after fix(auto-reply,infra): keep startup context and heartbeat event text UTF-16 safe #102483 landed, so the overlapping heartbeat hunk disappeared and this PR now contains only the distinct Talk fix.

Proof:

node scripts/run-vitest.mjs src/talk/fast-context-runtime.test.ts — 3 tests passed after the current-main update.
git diff --check passed.
Exact-head CI run 29001439450 passed.
Fresh exact-head autoreview found no actionable findings at 0.99 confidence.

No known proof gaps.

steipete · 2026-07-09T07:39:44Z

Merged via squash.

Prepared head SHA: 4ca80bc83956e9afc18cc19847b37423d4eda8e8
Landed commit: f0d4d4f2fccda02915cfa9d467e660f82ff9bb1d

Replace unsafe .slice() with truncateUtf16Safe to avoid splitting emoji surrogate pairs when truncating memory/session snippets for realtime voice prompts. This fixes the same issue as PR openclaw#102477 but in the fast-context runtime path.

Replace raw .slice(0, maxLength) with truncateUtf16Safe to prevent emoji surrogate pair truncation issues in assistant identity values. The coerceIdentityValue function is used to truncate assistant name, avatar, and emoji values. When these values contain emoji at the truncation boundary, the old implementation could produce lone surrogates (invalid UTF-16 text). This fix follows the same pattern as PR openclaw#102477 which addressed similar issues in talk/fast-context-runtime.ts and infra/heartbeat-events-filter.ts.

openclaw#102477) * fix(talk): use truncateUtf16Safe for snippet text truncation fast-context-runtime.ts normalizeSnippet and heartbeat-events-filter.ts buildExecEventPrompt used raw .slice(0, N), which can produce lone surrogates when truncation boundaries cross emoji surrogate pairs. * test(talk): cover UTF-16 snippet boundary --------- Co-authored-by: hailory <[email protected]> Co-authored-by: Peter Steinberger <[email protected]> Co-authored-by: Peter Steinberger <[email protected]>

fix(talk): use truncateUtf16Safe for snippet text truncation

da8182b

fast-context-runtime.ts normalizeSnippet and heartbeat-events-filter.ts buildExecEventPrompt used raw .slice(0, N), which can produce lone surrogates when truncation boundaries cross emoji surrogate pairs.

openclaw-barnacle Bot added the size: XS label Jul 9, 2026

steipete and others added 2 commits July 9, 2026 08:23

test(talk): cover UTF-16 snippet boundary

a74cb60

Merge branch 'main' into fix/surrogate-safe-llm-prompt

4ca80bc

steipete merged commit f0d4d4f into openclaw:main Jul 9, 2026
99 checks passed

clawsweeper Bot mentioned this pull request Jul 9, 2026

feat(continuation): context-pressure-aware continuation (continue_work / continue_delegate / request_compaction) #85651

Open

wangmiao0668000666 mentioned this pull request Jul 9, 2026

fix(text): keep five src-side text truncations UTF-16 safe #102630

Closed

9 tasks

SunnyShu0925 mentioned this pull request Jul 9, 2026

fix(llm-slug): use truncateUtf16Safe for LLM-prompt-facing session content truncation #102672

Closed

wangmiao0668000666 mentioned this pull request Jul 9, 2026

fix(chat,cli): keep session snippet and CLI output digest truncation UTF-16 safe #102747

Closed

9 tasks

This was referenced Jul 9, 2026

fix(talk): use UTF-16 safe truncation in fast-context-runtime #102798

Closed

fix(shared): use truncateUtf16Safe in assistant-identity-values #103025

Closed

This was referenced Jul 9, 2026

fix(shared): use truncateUtf16Safe in assistant-identity-values #103027

Closed

fix(shared): use truncateUtf16Safe in assistant-identity-values #103029

Closed

Simon-XYDT mentioned this pull request Jul 9, 2026

fix(shared): use truncateUtf16Safe in assistant-identity-values #103032

Closed

Simon-XYDT mentioned this pull request Jul 9, 2026

fix(identity): keep bounded identity values UTF-16 safe #103034

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(talk): use truncateUtf16Safe for LLM-prompt-facing text truncation#102477

fix(talk): use truncateUtf16Safe for LLM-prompt-facing text truncation#102477
steipete merged 3 commits into
openclaw:mainfrom
xydt-tanshanshan:fix/surrogate-safe-llm-prompt

xydt-tanshanshan commented Jul 9, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jul 9, 2026 •

edited

Loading

Uh oh!

xydt-tanshanshan commented Jul 9, 2026

Uh oh!

clawsweeper Bot commented Jul 9, 2026 •

edited

Loading

Uh oh!

steipete commented Jul 9, 2026

Uh oh!

Uh oh!

steipete commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

xydt-tanshanshan commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

User Impact

Real Behavior Proof

Uh oh!

clawsweeper Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xydt-tanshanshan commented Jul 9, 2026

Uh oh!

clawsweeper Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steipete commented Jul 9, 2026

Uh oh!

Uh oh!

steipete commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

xydt-tanshanshan commented Jul 9, 2026 •

edited

Loading

clawsweeper Bot commented Jul 9, 2026 •

edited

Loading

clawsweeper Bot commented Jul 9, 2026 •

edited

Loading