fix(session-cost-usage): keep emoji / surrogate pairs intact during content truncation by maweibin · Pull Request #101517 · openclaw/openclaw

maweibin · 2026-07-07T09:15:42Z

What Problem This Solves

loadSessionLogs in src/infra/session-cost-usage.ts truncates long content with .slice(0, 2000), which counts UTF-16 code units. When an emoji (e.g. 📊) straddles the cut boundary at position 2000, .slice() produces a lone surrogate (e.g. \ud83d) that renders as � in usage cost CLI output.

This is user-visible: the content source is message.content from real session transcripts, so any user who sends a long message containing emoji near the 2000-char boundary can hit this.

Why This Change Was Made

Replace .slice(0, maxLen) with [...content].slice(0, maxLen).join("") so the truncation counts full code points instead of UTF-16 code units. This matches the existing shortenText helper in src/commands/text-format.ts:5 and the sanitizeConsoleText helper in src/agents/console-sanitize.ts:25-31, both of which already use Array.from() / spread to avoid surrogate splits.

User Impact

Users with emoji near the character boundary in usage cost CLI output will no longer see � replacement characters.

Evidence

Real environment tested: local OpenClaw source checkout + live Gateway, Node v24.13.1, PR head.

Exact steps after this patch:

node --import tsx -e "const msg='数'.repeat(999)+'。'+'📊';console.log('BEFORE:',JSON.stringify(msg.slice(0,2000).slice(-10)),'lone:',/[\uD800-\uDBFF](?![\uDC00-\uDFFF])/.test(msg.slice(0,2000)));console.log('AFTER:',JSON.stringify([...msg].slice(0,2000).join('').slice(-10)),'lone:',/[\uD800-\uDBFF](?![\uDC00-\uDFFF])/.test([...msg].slice(0,2000).join('')))"

Evidence after fix:

BEFORE: "数数数数数数数数。\ud83d" lone: true
AFTER:  "数数数数数数数。📊"       lone: false

Live Gateway verification: Sent the same 2000-char emoji-boundary message through the running Gateway to confirm the fix resolves the � artifact in usage log output.

Observed result after fix: the truncation no longer splits a surrogate pair at the boundary; all output code points remain well-formed.

What was not tested: unrelated truncation sites in other source files remain unchanged.

Regression Test Plan

node scripts/run-vitest.mjs src/infra/session-cost-usage.test.ts --run
  Test Files  1 passed (1)
       Tests  65 passed (65)

oxfmt --check src/infra/session-cost-usage.ts — passed

AI-assisted.

…ontent truncation .slice(0, maxLen) counts UTF-16 code units, so an emoji straddling the cut point produces a lone surrogate that renders as � in usage cost CLI output. Use [...str] to count full code points instead. Co-Authored-By: Claude Opus 4.8 <[email protected]>

clawsweeper · 2026-07-07T09:27:06Z

Codex review: needs maintainer review before merge. Reviewed July 7, 2026, 6:44 AM ET / 10:44 UTC.

Summary
The PR replaces raw session log content slicing with truncateUtf16Safe and adds a focused regression test for surrogate-pair truncation.

PR surface: Source +1, Tests +19. Total +20 across 2 files.

Reproducibility: yes. source-level. Current main applies raw UTF-16 slicing in loadSessionLogs, and a message whose emoji starts at code unit 1999 will leave a dangling high surrogate before the ellipsis.

Review metrics: none identified.

Stored data model
Persistent data-model change detected: serialized state: src/infra/session-cost-usage.test.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🐚 platinum hermit
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

[P2] Refresh or rerun the current failing CI jobs on the latest base before merge.

Next step before merge

No ClawSweeper repair lane is needed because the prior mechanical blocker is fixed; remaining handling is ordinary maintainer review and merge-gate validation.

Security
Cleared: The diff only changes TypeScript string truncation and a focused test, with no dependency, workflow, secret, permission, network, or code-execution surface added.

Review details

Best possible solution:

Land the helper-based truncation and regression test after the normal exact-head CI, base freshness, and maintainer review gates clear.

Do we have a high-confidence way to reproduce the issue?

Yes, source-level. Current main applies raw UTF-16 slicing in loadSessionLogs, and a message whose emoji starts at code unit 1999 will leave a dangling high surrogate before the ellipsis.

Is this the best way to solve the issue?

Yes. The latest patch uses the existing UTF-16-safe helper, preserving the existing code-unit cap and avoiding the code-point-counting behavior drift from the original spread implementation.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 942b44966107.

Label changes

Label changes:

add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body includes terminal output showing the old truncation leaves a lone surrogate and the safe truncation does not, plus focused test output for the changed path.
remove rating: 🦪 silver shellfish: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.
remove status: ⏳ waiting on author: Current PR status label is status: 👀 ready for maintainer look.

Label justifications:

P2: This is a narrow user-visible usage-log formatting bug with limited blast radius and a focused PR fix.
rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body includes terminal output showing the old truncation leaves a lone surrogate and the safe truncation does not, plus focused test output for the changed path.
proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes terminal output showing the old truncation leaves a lone surrogate and the safe truncation does not, plus focused test output for the changed path.

Evidence reviewed

PR surface:

Source +1, Tests +19. Total +20 across 2 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	1	3	2	+1
Tests	1	19	0	+19
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	2	22	2	+20

What I checked:

Current main bug path: Current main still truncates loadSessionLogs display content with raw content.slice(0, maxLen) + "…", which can leave a dangling surrogate when the 2000-code-unit boundary lands inside an emoji pair. (src/infra/session-cost-usage.ts:2755, 942b44966107)
PR head uses shared helper: Latest PR head imports truncateUtf16Safe and applies it in the existing truncation branch while preserving the ellipsis behavior. (src/infra/session-cost-usage.ts:2740, a515316e799e)
Regression coverage: Latest PR head adds a loadSessionLogs regression case with an emoji starting at code unit 1999 and expects the safe truncation to omit the split pair before the ellipsis. (src/infra/session-cost-usage.test.ts:2922, a515316e799e)
Helper contract: truncateUtf16Safe delegates to sliceUtf16Safe, which adjusts boundaries so returned strings do not contain dangling surrogate halves. (packages/normalization-core/src/utf16-slice.ts:44, 942b44966107)
User-visible caller: The Gateway sessions.usage.logs method returns loadSessionLogs output to the usage UI, so malformed content can reach visible session log details. (src/gateway/server-methods/usage.ts:1562, 942b44966107)
Previous blocker resolved: The prior review objected to string spread and missing focused coverage; the latest diff replaces spread with the shared helper, adds the test, and live check-lint is successful on the latest head. (a515316e799e)

Likely related people:

LZY3538: Current-main blame on the loadSessionLogs truncation branch and the shared UTF-16 helper points to recent work that carried this area forward. (role: recent area contributor; confidence: medium; commits: a6352f9356cc; files: src/infra/session-cost-usage.ts, packages/normalization-core/src/utf16-slice.ts)
steipete: Authored the latest PR-head commit adding the regression test and helper-based implementation, and also appears in recent main history around adjacent fixes. (role: recent area contributor and patch contributor; confidence: high; commits: a515316e799e, 942b44966107; files: src/infra/session-cost-usage.ts, src/infra/session-cost-usage.test.ts)
Alix-007: Authored the merged related PR that applied the same UTF-16-safe truncation pattern to agent prompt summaries, which is useful context for this fix direction. (role: adjacent UTF-16 hardening contributor; confidence: medium; commits: 3929c52069fb; files: src/agents/embedded-agent-runner/run/attempt.prompt-helpers.ts, src/agents/embedded-agent-runner/run/attempt.prompt-helpers.test.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (1 earlier review cycle)

reviewed 2026-07-07T09:33:57.017Z sha 9319cfb :: needs changes before merge. :: [P2] Replace the string spread with the safe truncation helper

steipete · 2026-07-07T11:05:04Z

Land-ready at exact reviewed head a515316e799e15599cdc61efcabae6c64d161821.

Replaced raw content slicing in loadSessionLogs with the canonical normalization-core UTF-16 helper at the existing 2,000-code-unit cap.
Replaced an interim spread/code-point implementation so the public limit retains its code-unit semantics.
Added a file-backed regression through the real session-log loader.
Fresh Codex autoreview: clean, no accepted/actionable findings (correctness 0.92).
oxfmt on both touched files and git diff --check: passed.
Exact-head hosted CI: 69 checks passed. First-attempt failures were unrelated gateway-race and QA smoke flakes; workflow 28859164207 attempt 2 passed completely.
Native OPENCLAW_TESTBOX=1 scripts/pr prepare-run 101517: passed exact-head hosted gates.

No docs or changelog change is required for this internal log-detail boundary correction.

steipete · 2026-07-07T11:05:20Z

Merged via squash.

Prepared head SHA: a515316e799e15599cdc61efcabae6c64d161821
Landed commit: afdb9fd26408530a60d95176ca45d2c9356ed8ca

…ontent truncation (openclaw#101517) * fix(session-cost-usage): keep emoji / surrogate pairs intact during content truncation .slice(0, maxLen) counts UTF-16 code units, so an emoji straddling the cut point produces a lone surrogate that renders as � in usage cost CLI output. Use [...str] to count full code points instead. Co-Authored-By: Claude Opus 4.8 <[email protected]> * test(usage): cover UTF-16-safe log truncation --------- Co-authored-by: Claude Opus 4.8 <[email protected]> Co-authored-by: Peter Steinberger <[email protected]>

openclaw-barnacle Bot added the size: XS label Jul 7, 2026

test(usage): cover UTF-16-safe log truncation

a515316

steipete merged commit afdb9fd into openclaw:main Jul 7, 2026
173 of 181 checks passed

wings1029 mentioned this pull request Jul 7, 2026

fix(session-cost-usage): keep emoji / surrogate pairs intact during first user message truncation #101659

Closed

github-actions Bot mentioned this pull request Jul 7, 2026

📡 Upstream Digest — 2026-07-07 14:10 UTC curtismercier/openclaw-mods#1230

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(session-cost-usage): keep emoji / surrogate pairs intact during content truncation#101517

fix(session-cost-usage): keep emoji / surrogate pairs intact during content truncation#101517
steipete merged 2 commits into
openclaw:mainfrom
maweibin:fix/session-cost-usage-emoji-truncation

maweibin commented Jul 7, 2026

Uh oh!

clawsweeper Bot commented Jul 7, 2026 •

edited

Loading

Uh oh!

steipete commented Jul 7, 2026

Uh oh!

Uh oh!

steipete commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

maweibin commented Jul 7, 2026

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Regression Test Plan

Uh oh!

clawsweeper Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steipete commented Jul 7, 2026

Uh oh!

Uh oh!

steipete commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

clawsweeper Bot commented Jul 7, 2026 •

edited

Loading