fix(memory-core): use truncateUtf16Safe for diary context truncation by lsr911 · Pull Request #102524 · openclaw/openclaw

lsr911 · 2026-07-09T07:46:34Z

What Problem This Solves

Dream-diary context was truncated with a raw UTF-16 code-unit slice. A boundary between an emoji's surrogate halves could pass malformed text into the dreaming narrative prompt.

Why This Change Was Made

The diary reader now uses the shared UTF-16-safe truncation helper through the supported plugin SDK text-runtime facade. The fix stays at the memory-core owner boundary and does not add a second truncation path.

User Impact

Long diary entries can contain emoji and other supplementary Unicode characters without producing an unpaired surrogate at the prompt boundary. ASCII and already-short entries are unchanged.

Evidence

Added a public-path regression test through readRecentDreamDiaryEntries with an emoji straddling the 20,000-code-unit boundary.
node scripts/run-vitest.mjs extensions/memory-core/src/dreaming-narrative.test.ts — 59 passed.
Focused rerun of preserves complete Unicode code points at the diary context limit — 1 passed, 58 skipped.
Fresh autoreview: clean, no accepted/actionable findings.
Exact-head CI at 5ce7d36ee027 passed after rerunning one transient artifact job with unavailable logs.

🤖 Generated with Claude Code

clawsweeper · 2026-07-09T07:54:06Z

Codex review: needs real behavior proof before merge. Reviewed July 9, 2026, 4:45 AM ET / 08:45 UTC.

Summary
The PR replaces raw diary context slicing with UTF-16-safe truncation in memory-core dreaming narrative code and adds a regression test for a surrogate pair at the clamp boundary.

PR surface: Source +1, Tests +15. Total +16 across 2 files.

Reproducibility: yes. source-reproducible: current main clamps diary context with raw .slice(0, RECENT_DIARY_CONTEXT_MAX_CHARS), so a surrogate pair crossing that boundary can be split. I did not run a live memory-core dreaming flow in this read-only review.

Review metrics: none identified.

Stored data model
Persistent data-model change detected: vector/embedding metadata: extensions/memory-core/src/dreaming-narrative.test.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🧂 unranked krab
Patch quality: 🐚 platinum hermit
Result: blocked until real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

[P1] Add redacted after-fix proof, such as terminal output or logs from a memory-core diary context run showing the surrogate-boundary case.

Proof guidance:

[P1] Needs real behavior proof before merge: The PR body and comments do not include after-fix terminal output, logs, screenshot, recording, or linked artifact showing memory-core diary context truncation with surrogate-pair input. The contributor should add redacted proof, update the PR body so ClawSweeper can re-review automatically, and ask a maintainer for @clawsweeper re-review only if it does not rerun.

Risk before merge

[P1] External real behavior proof is still absent; tests and CI are useful but do not show the contributor’s after-fix memory-core diary context path in a real setup.

Maintainer options:

Decide the mitigation before merge
Merge the narrow SDK-facade truncation fix after the contributor adds redacted terminal output, logs, or another real behavior artifact showing the diary context surrogate-boundary case.
Pause or close
Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

[P1] The code blocker from the previous cycle appears fixed, but ClawSweeper should wait for contributor-supplied real behavior proof rather than queueing an automated repair.

Security
Cleared: No concrete security or supply-chain concern found; the diff changes one string truncation call and adds a focused unit test without dependency, workflow, secret, or package-resolution changes.

Review details

Best possible solution:

Merge the narrow SDK-facade truncation fix after the contributor adds redacted terminal output, logs, or another real behavior artifact showing the diary context surrogate-boundary case.

Do we have a high-confidence way to reproduce the issue?

Yes, source-reproducible: current main clamps diary context with raw .slice(0, RECENT_DIARY_CONTEXT_MAX_CHARS), so a surrogate pair crossing that boundary can be split. I did not run a live memory-core dreaming flow in this read-only review.

Is this the best way to solve the issue?

Yes for the code shape: the current PR head uses the existing openclaw/plugin-sdk/text-utility-runtime facade and adds focused regression coverage. It still needs external real behavior proof before merge.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against ecc2cffad8aa.

Label changes

Label changes:

remove merge-risk: 🚨 compatibility: Current PR review selected no merge-risk labels.

Label justifications:

P2: This is a normal-priority memory-core correctness fix with limited blast radius, with merge readiness currently blocked only by the real behavior proof gate.
rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🐚 platinum hermit.
status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR body and comments do not include after-fix terminal output, logs, screenshot, recording, or linked artifact showing memory-core diary context truncation with surrogate-pair input. The contributor should add redacted proof, update the PR body so ClawSweeper can re-review automatically, and ask a maintainer for @clawsweeper re-review only if it does not rerun.

Evidence reviewed

PR surface:

Source +1, Tests +15. Total +16 across 2 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	1	2	1	+1
Tests	1	15	0	+15
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	2	17	1	+16

What I checked:

Current main raw truncation: Current origin/main still clamps normalized diary context with raw .slice(0, RECENT_DIARY_CONTEXT_MAX_CHARS).trimEnd() before appending ..., so the central bug is not already implemented on main. (extensions/memory-core/src/dreaming-narrative.ts:458, ecc2cffad8aa)
Latest PR head uses SDK facade: The latest PR diff imports truncateUtf16Safe from openclaw/plugin-sdk/text-utility-runtime, resolving the prior direct package-boundary finding, and applies it at the diary context clamp. (extensions/memory-core/src/dreaming-narrative.ts:15, 5ce7d36ee027)
Regression coverage added: The PR adds a focused test that writes a diary entry with a surrogate pair crossing the 360-code-unit boundary and expects the recent diary entry to truncate before the pair. (extensions/memory-core/src/dreaming-narrative.test.ts:436, 5ce7d36ee027)
Helper contract: text-utility-runtime exports truncateUtf16Safe, and the underlying implementation backs off when a slice would leave a dangling high surrogate. (src/plugin-sdk/text-utility-runtime.ts:21, ecc2cffad8aa)
Extension boundary policy: Scoped extension guidance requires extension production code to import through openclaw/plugin-sdk/*; the current PR head follows that boundary. (extensions/AGENTS.md:27, ecc2cffad8aa)
Live PR proof state: Live PR metadata shows the prior ClawSweeper proof request is still the only comment, the dedicated Real behavior proof check failed, and the PR body/comments contain no after-fix terminal output, logs, screenshot, recording, or linked artifact. (5ce7d36ee027)

Likely related people:

steipete: Blame for the current diary context clamp points to 00e323c, and the current PR head includes steipete-authored commits adding the regression test and switching the import to the plugin SDK facade. (role: recent area contributor and PR follow-up author; confidence: high; commits: 00e323c9902c, d882e399c535, 5ce7d36ee027; files: extensions/memory-core/src/dreaming-narrative.ts, extensions/memory-core/src/dreaming-narrative.test.ts)
Vincent Koc: History shows b78713a moved dreaming narrative imports onto SDK seams, directly relevant to the plugin boundary used by the current PR head. (role: SDK seam contributor; confidence: medium; commits: b78713a36391; files: extensions/memory-core/src/dreaming-narrative.ts, src/plugin-sdk/memory-core-host-engine-foundation.ts)
Vignesh Natarajan: History shows 61e61cc introduced the dreaming diary surface that owns this diary context behavior. (role: feature introducer; confidence: medium; commits: 61e61ccc182f; files: extensions/memory-core/src/dreaming-narrative.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (1 earlier review cycle)

reviewed 2026-07-09T07:58:21.837Z sha 132c746 :: needs real behavior proof before merge. :: [P2] Use the plugin SDK facade for truncation

steipete · 2026-07-09T08:49:09Z

Maintainer review complete at exact head 5ce7d36ee0277fa3243af874f0504efe2a68f3d8.

What changed during review:

moved the shared UTF-16 helper behind the supported plugin SDK text-runtime facade;
added a public-path regression through readRecentDreamDiaryEntries at the 20,000-code-unit boundary;
refreshed the PR problem, impact, and evidence sections.

Validation:

node scripts/run-vitest.mjs extensions/memory-core/src/dreaming-narrative.test.ts — 59 passed;
focused UTF-16 boundary rerun — 1 passed, 58 skipped;
fresh autoreview — clean, no accepted/actionable findings;
exact-head hosted CI/Testbox gates — passed (one artifact job rerun after GitHub returned no logs).

Known proof gaps: none for this bounded prompt-truncation fix.

steipete · 2026-07-09T08:49:26Z

Merged via squash.

Prepared head SHA: 5ce7d36ee0277fa3243af874f0504efe2a68f3d8
Landed commit: 6a3834d41d0c29b7cb8659552eb9a80fa2714ad9

lsr911 · 2026-07-09T08:49:43Z

@steipete this follows the same UTF-16 truncation pattern as the recently-merged #102522, #102500, #102512, #102527, and others. Patch applies truncateUtf16Safe to prevent surrogate pair splitting. Proof is in the PR body. Could you take a look?

…penclaw#102524) * fix(memory-core): use truncateUtf16Safe for diary context truncation * test(memory-core): cover utf16 diary truncation * refactor(memory-core): use plugin sdk text runtime --------- Co-authored-by: Peter Steinberger <[email protected]>

fix(memory-core): use truncateUtf16Safe for diary context truncation

132c746

openclaw-barnacle Bot added extensions: memory-core Extension: memory-core size: XS triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. labels Jul 9, 2026

steipete added 2 commits July 9, 2026 09:15

test(memory-core): cover utf16 diary truncation

d882e39

refactor(memory-core): use plugin sdk text runtime

5ce7d36

clawsweeper Bot removed the merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. label Jul 9, 2026

openclaw-barnacle Bot removed the triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. label Jul 9, 2026

steipete merged commit 6a3834d into openclaw:main Jul 9, 2026
153 of 158 checks passed

wangmiao0668000666 mentioned this pull request Jul 9, 2026

fix(text): keep five src-side text truncations UTF-16 safe #102630

Closed

9 tasks

OpenClawKobian99 mentioned this pull request Jul 14, 2026

[Bug]: memory-core dreaming narrative fails with "no low surrogate in string" API 400 — clampDiaryContextEntry truncates mid-surrogate-pair (fix on main, missing from v2026.7.1) #107815

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(memory-core): use truncateUtf16Safe for diary context truncation#102524

fix(memory-core): use truncateUtf16Safe for diary context truncation#102524
steipete merged 3 commits into
openclaw:mainfrom
lsr911:fix/utf16-dreaming-narrative

lsr911 commented Jul 9, 2026 •

edited by steipete

Loading

Uh oh!

clawsweeper Bot commented Jul 9, 2026 •

edited

Loading

Uh oh!

steipete commented Jul 9, 2026

Uh oh!

Uh oh!

steipete commented Jul 9, 2026

Uh oh!

lsr911 commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

lsr911 commented Jul 9, 2026 • edited by steipete Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Uh oh!

clawsweeper Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steipete commented Jul 9, 2026

Uh oh!

Uh oh!

steipete commented Jul 9, 2026

Uh oh!

lsr911 commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lsr911 commented Jul 9, 2026 •

edited by steipete

Loading

clawsweeper Bot commented Jul 9, 2026 •

edited

Loading