fix(agents,skills): keep session-identity and experience-review transcript truncation UTF-16 safe by Monkey-wusky · Pull Request #106370 · openclaw/openclaw

Monkey-wusky · 2026-07-13T12:10:32Z

What Problem This Solves

normalizeCliContextValue (cli-output.ts) and formatSkillExperienceReviewTranscript (experience-review-prompt.ts) use raw .slice(0, N) / .slice(-N) to truncate user-visible strings. When a truncation boundary falls inside a UTF-16 surrogate pair, the result contains a lone surrogate — encodeURIComponent() throws URIError, JSON serialisers may reject the string, and downstream display surfaces render broken characters.

The codebase already applies truncateUtf16Safe / sliceUtf16Safe in 30+ locations. These two truncation sites were missed.

Why This Change Was Made

Replace raw .slice(0, N) with truncateUtf16Safe, and full.slice(-tailBudget) with sliceUtf16Safe(full, tailStart). Same approach as #98644 and #102470.

User Impact

Error messages from formatCliOutputError no longer produce broken characters at emoji boundaries
Skill experience review transcripts no longer corrupt emoji at truncation boundaries

Evidence

Environment: Node.js v24.13.1, @openclaw/normalization-core/utf16-slice, Linux x86_64.

Production-function proof (through formatCliOutputError and formatSkillExperienceReviewTranscript, not raw helpers):

npx tsx ./tmp-proof.mjs

> Scenario 1: formatCliOutputError (Production function)
  Old normalizeCliContextValue(session_id): length=200 dangling=true
  Production function output: length=845 dangling=false

> Scenario 2: formatSkillExperienceReviewTranscript First message
  Old rendered[0].slice(0,6000): length=6000 last=0xd83d dangling=true
  Production function output: length=59950 dangling=false

> Scenario 3: formatSkillExperienceReviewTranscript Tail truncation
  Old full.slice(20017): first=0xdd9e dangling=true
  Production function output: dangling=false

All three scenarios: old code produces isolated surrogates, production functions are clean.

Unit tests (each pre-verifies the old .slice() dangles, then calls the production function):

Both fixtures now exceed EXPERIENCE_REVIEW_MAX_TRANSCRIPT_CHARS (60,000), so the truncation branch fires:

Test 1: rendered[0] has high surrogate at index 5,999 → .slice(0,6000) dangles → formatSkillExperienceReviewTranscript is clean
Test 2: full.slice(tailStart) starts at low surrogate → formatSkillExperienceReviewTranscript is clean
Test 3: normalizeCliContextValue raw .slice(0,200) dangles → formatCliOutputError is clean

node scripts/run-vitest.mjs src/agents/cli-output.test.ts src/skills/workshop/experience-review.test.ts

Test Files  3 passed (3)
     Tests  148 passed (148)

Diff: 4 files, +87/-14. No pnpm-lock.yaml. Branch from upstream/main.

AI-assisted.

clawsweeper · 2026-07-13T13:06:39Z

Codex review: needs maintainer review before merge. Reviewed July 15, 2026, 11:54 PM ET / July 16, 2026, 03:54 UTC.

Summary
Replaces raw UTF-16 code-unit slicing in CLI diagnostic identity fields and Skill Workshop experience-review transcripts with shared surrogate-safe helpers, with boundary regression tests.

PR surface: Source +4, Tests +108. Total +112 across 4 files.

Reproducibility: yes. —the raw current behavior is deterministically exposed by placing a surrogate pair across the 200-, 6,000-, or retained-tail boundary, and the submitted fixtures demonstrate those exact conditions. The review did not independently execute current main, so the structured status remains source-reproducible rather than reproduced.

Review metrics: none identified.

Stored data model
Persistent data-model change detected: serialized state: src/agents/cli-output.test.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: canonical
Canonical: #106370
Summary: This PR is the canonical candidate for the CLI identity and experience-review truncation boundaries; related PRs cover separate truncation owners and should remain distinct.

Members:

adjacent_distinct: fix: tool summaries preserve emoji truncation boundaries #98644 - Merged precedent for tool-description truncation, but it does not touch either affected function in this PR.
adjacent_distinct: fix(gateway): keep chat history display text truncation surrogate-safe #102470 - Merged precedent for Gateway chat-history display projection, a different owner and output path.
adjacent_distinct: fix(skills): preserve emoji in long history scans #107720 - Covers the downstream 16,000-character Skill Workshop history-scan cap rather than the earlier 60,000-character formatter or CLI context fields.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Risk before merge

[P1] The PR head is mergeable but behind current main; maintainers should refresh the branch or review the three-way merge result and confirm the focused tests on the exact landing head.

Maintainer options:

Decide the mitigation before merge
Land the shared-helper substitutions and boundary tests without broadening the change, after refreshing against current main and retaining the existing character budgets and transcript layout.
Pause or close
Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

No automated repair is needed; the remaining action is ordinary maintainer review of the refreshed landing head.

Security
Cleared: The patch only replaces local string slicing with an existing internal normalization helper and adds tests; it introduces no dependency, secret, permission, workflow, download, or package-resolution change.

Review details

Best possible solution:

Land the shared-helper substitutions and boundary tests without broadening the change, after refreshing against current main and retaining the existing character budgets and transcript layout.

Do we have a high-confidence way to reproduce the issue?

Yes—the raw current behavior is deterministically exposed by placing a surrogate pair across the 200-, 6,000-, or retained-tail boundary, and the submitted fixtures demonstrate those exact conditions. The review did not independently execute current main, so the structured status remains source-reproducible rather than reproduced.

Is this the best way to solve the issue?

Yes—the shared normalization-core helpers are the narrowest maintainable fix and match recently merged handling on sibling truncation surfaces; duplicating custom surrogate logic would be worse.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against de3f7925c4c3.

Label changes

Label justifications:

P2: The patch fixes malformed Unicode at uncommon but real truncation boundaries with limited blast radius and strong regression evidence.
rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR provides after-fix terminal output through both production functions, including negative-control boundary observations and clean patched results, with no private data visible.
proof: sufficient: Contributor real behavior proof is sufficient. The PR provides after-fix terminal output through both production functions, including negative-control boundary observations and clean patched results, with no private data visible.

Evidence reviewed

PR surface:

Source +4, Tests +108. Total +112 across 4 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	2	7	3	+4
Tests	2	108	0	+108
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	4	115	3	+112

What I checked:

CLI implementation: The branch changes session and run identity normalization from raw 200-code-unit slicing to the shared UTF-16-safe truncation helper. (src/agents/cli-output.ts:66, a3dd35108259)
Experience-review implementation: The branch applies safe prefix and tail slicing while preserving the existing transcript budget and omission layout. (src/skills/workshop/experience-review-prompt.ts:66, a3dd35108259)
Regression coverage: The latest head constructs inputs beyond the 60,000-character branch, proves the prior raw slices create isolated surrogates, and verifies the production functions return well-formed strings. (src/skills/workshop/experience-review.test.ts:344, 13b37f0fb05b)
Real behavior proof: The PR body includes terminal output from production functions showing three boundary scenarios where the old slicing dangles and the patched output does not; the repository has marked the proof sufficient. (13b37f0fb05b)
Established repository pattern: Merged UTF-16 fixes in fix: tool summaries preserve emoji truncation boundaries #98644 and fix(gateway): keep chat history display text truncation surrogate-safe #102470 use the same shared normalization-core approach on adjacent user-visible truncation surfaces. (src/agents/tool-description-summary.ts, d9d6108086da)
Related work is distinct: fix(skills): preserve emoji in long history scans #107720 explicitly covers the later 16,000-character per-session history-scan cap, not the CLI identity or 60,000-character experience-review boundaries fixed here. (src/skills/workshop/history-scan.test.ts, 1ba1110785b7)

Likely related people:

ZengWen-DT: Introduced the recently merged shared-helper fix for emoji-safe tool-summary truncation, making them relevant to consistency across user-visible truncation surfaces. (role: adjacent UTF-16 truncation contributor; confidence: medium; commits: d9d6108086da; files: src/agents/tool-description-summary.ts, src/agents/tool-description-summary.test.ts)
zw-xysk: Introduced the recently merged surrogate-safe chat-history projection fix using the same normalization-core contract. (role: adjacent UTF-16 truncation contributor; confidence: medium; commits: a6d45c46de78; files: src/gateway/server-methods/chat-history.ts, src/gateway/server-methods/server-methods.test.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (5 earlier review cycles)

reviewed 2026-07-13T13:04:42.100Z sha 17a1e9b :: needs real behavior proof before merge. :: [P2] Exercise the 60,000-character truncation branch
reviewed 2026-07-13T13:30:02.040Z sha 17a1e9b :: needs real behavior proof before merge. :: [P2] Exercise the 60,000-character truncation branch
reviewed 2026-07-14T01:52:17.658Z sha 4e64571 :: needs maintainer review before merge. :: none
reviewed 2026-07-14T02:09:29.524Z sha 4db3c3b :: needs maintainer review before merge. :: none
reviewed 2026-07-14T10:44:49.596Z sha 7c101c4 :: needs maintainer review before merge. :: none

Monkey-wusky · 2026-07-14T01:45:05Z

@clawsweeper re-review

clawsweeper · 2026-07-14T01:45:08Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review published an exact durable review comment and queued an executing target-wide serialized router scan.
Run: https://github.com/openclaw/clawsweeper/actions/runs/29299450047
Updated: 2026-07-14T01:54:43.454Z

…cript truncation UTF-16 safe normalizeCliContextValue (cli-output.ts) and formatSkillExperienceReviewTranscript (experience-review-prompt.ts) both used raw .slice(0, N) / .slice(-N) to truncate user-facing strings. When a truncation boundary falls inside a surrogate pair (emoji), the resulting string contains a lone surrogate — encodeURIComponent throws URIError, JSON serialisers may reject it, and downstream display surfaces can render garbage. Replace .slice(0, 200) with truncateUtf16Safe in normalizeCliContextValue. Replace .slice(0, 6_000) with truncateUtf16Safe and full.slice(-tailBudget) with sliceUtf16Safe(full, tailStart) in formatSkillExperienceReviewTranscript. Follows the pattern established by openclaw#98644, openclaw#102470, openclaw#101728 and the broader UTF-16 safety sweep already applied across the codebase.

…y old slices dangle Both experience-review fixtures now exceed EXPERIENCE_REVIEW_MAX_TRANSCRIPT_CHARS so the truncation branch actually fires. Pre-condition assertions prove the old raw .slice(0,6000) / .slice(tailStart) produce isolated surrogates; the production functions (formatSkillExperienceReviewTranscript, formatCliOutputError) do not. cli-output test also adds pre-condition check showing normalizeCliContextValue with raw .slice(0,200) would split a surrogate pair.

Co-authored-by: 毛宇豪0668001457 <[email protected]>

steipete · 2026-07-16T16:54:02Z

Merged via squash.

Prepared head SHA: e2d5a75ded6200abd13208ef31ef9e04e7090131
Landed commit: e6ac6c85067eed11a2e79692dcee37ee4b768731

…cript truncation UTF-16 safe (openclaw#106370) * fix(agents,skills): keep session-identity and experience-review transcript truncation UTF-16 safe normalizeCliContextValue (cli-output.ts) and formatSkillExperienceReviewTranscript (experience-review-prompt.ts) both used raw .slice(0, N) / .slice(-N) to truncate user-facing strings. When a truncation boundary falls inside a surrogate pair (emoji), the resulting string contains a lone surrogate — encodeURIComponent throws URIError, JSON serialisers may reject it, and downstream display surfaces can render garbage. Replace .slice(0, 200) with truncateUtf16Safe in normalizeCliContextValue. Replace .slice(0, 6_000) with truncateUtf16Safe and full.slice(-tailBudget) with sliceUtf16Safe(full, tailStart) in formatSkillExperienceReviewTranscript. Follows the pattern established by openclaw#98644, openclaw#102470, openclaw#101728 and the broader UTF-16 safety sweep already applied across the codebase. * chore: re-trigger CI after PR body update * test(agents,skills): construct transcripts exceeding 60k chars, verify old slices dangle Both experience-review fixtures now exceed EXPERIENCE_REVIEW_MAX_TRANSCRIPT_CHARS so the truncation branch actually fires. Pre-condition assertions prove the old raw .slice(0,6000) / .slice(tailStart) produce isolated surrogates; the production functions (formatSkillExperienceReviewTranscript, formatCliOutputError) do not. cli-output test also adds pre-condition check showing normalizeCliContextValue with raw .slice(0,200) would split a surrogate pair. * test: tighten UTF-16 truncation coverage Co-authored-by: 毛宇豪0668001457 <[email protected]> --------- Co-authored-by: Peter Steinberger <[email protected]>

openclaw-barnacle Bot added agents Agent runtime and tooling size: S triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. and removed triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. labels Jul 13, 2026

Monkey-wusky force-pushed the fix/utf16-safe-truncation-cli-output-experience-review branch from 17a1e9b to 4e64571 Compare July 14, 2026 01:44

Monkey-wusky force-pushed the fix/utf16-safe-truncation-cli-output-experience-review branch from 4e64571 to 4db3c3b Compare July 14, 2026 02:03

clawsweeper Bot added rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels Jul 14, 2026

Monkey-wusky force-pushed the fix/utf16-safe-truncation-cli-output-experience-review branch from 4db3c3b to 7c101c4 Compare July 14, 2026 04:20

Leon-SK668 mentioned this pull request Jul 14, 2026

fix(skills): preserve emoji in long history scans #107720

Merged

Monkey-wusky added 3 commits July 16, 2026 11:01

chore: re-trigger CI after PR body update

83f4a99

Monkey-wusky force-pushed the fix/utf16-safe-truncation-cli-output-experience-review branch from 7c101c4 to 13b37f0 Compare July 16, 2026 03:01

steipete and others added 2 commits July 16, 2026 09:36

chore: merge main for UTF-16 truncation fix

693eacc

test: tighten UTF-16 truncation coverage

e2d5a75

Co-authored-by: 毛宇豪0668001457 <[email protected]>

steipete merged commit e6ac6c8 into openclaw:main Jul 16, 2026
115 checks passed

github-actions Bot mentioned this pull request Jul 16, 2026

📡 Upstream Digest — 2026-07-16 17:05 UTC curtismercier/openclaw-mods#1314

Open

xydt-juyaohui mentioned this pull request Jul 20, 2026

fix(skills): keep experience review transcript UTF-16 safe at truncation boundary #108341

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(agents,skills): keep session-identity and experience-review transcript truncation UTF-16 safe#106370

fix(agents,skills): keep session-identity and experience-review transcript truncation UTF-16 safe#106370
steipete merged 5 commits into
openclaw:mainfrom
Monkey-wusky:fix/utf16-safe-truncation-cli-output-experience-review

Monkey-wusky commented Jul 13, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jul 13, 2026 •

edited

Loading

Uh oh!

Monkey-wusky commented Jul 14, 2026

Uh oh!

clawsweeper Bot commented Jul 14, 2026 •

edited

Loading

Uh oh!

Uh oh!

steipete commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

Monkey-wusky commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Uh oh!

clawsweeper Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Monkey-wusky commented Jul 14, 2026

Uh oh!

clawsweeper Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

steipete commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Monkey-wusky commented Jul 13, 2026 •

edited

Loading

clawsweeper Bot commented Jul 13, 2026 •

edited

Loading

clawsweeper Bot commented Jul 14, 2026 •

edited

Loading