[AI] fix(memory): use truncateUtf16Safe for dreaming snippet truncation by xydt-tanshanshan · Pull Request #101946 · openclaw/openclaw

xydt-tanshanshan · 2026-07-08T01:28:30Z

What Problem This Solves

Memory-core dreaming truncated daily-note and session-corpus snippets with raw UTF-16 code-unit slices. A limit through an emoji or another supplementary Unicode character could persist a lone surrogate in staged memory or corpus output.

Why This Change Was Made

Reuse the existing truncateUtf16Safe Plugin SDK surface at all five dreaming-owned truncation boundaries.
Cover daily headings, daily items, joined daily chunks, normalized session snippets, and rendered session lines through the existing production-path dreaming suite.
Remove the standalone proof script and duplicate 191-line test harness.

User Impact

Dreaming ingestion keeps valid Unicode when a configured snippet cap lands inside a surrogate pair. Ordinary text and existing limits are unchanged.

Evidence

Blacksmith Testbox: env CI=true corepack pnpm test extensions/memory-core/src/dreaming-phases.test.ts
- 1 file passed
- 51 tests passed
- Run: https://github.com/openclaw/openclaw/actions/runs/28994107855
git diff --check
Fresh structured autoreview: clean, no accepted/actionable findings (confidence 0.98)

AI-assisted: Yes

clawsweeper · 2026-07-08T01:30:18Z

Codex review: needs real behavior proof before merge. Reviewed July 9, 2026, 12:37 AM ET / 04:37 UTC.

Summary
The PR replaces memory-core dreaming raw .slice truncation with truncateUtf16Safe and adds focused dreaming-phase coverage for daily and session surrogate-boundary snippets.

PR surface: Source +7, Tests +67. Total +74 across 2 files.

Reproducibility: yes. source-reproducible: current main and v2026.6.11 use raw .slice at the dreaming truncation points, so a surrogate pair straddling the limit can leave a lone surrogate. I did not run a live current-main repro in this read-only review.

Review metrics: none identified.

Stored data model
Persistent data-model change detected: serialized state: extensions/memory-core/src/dreaming-phases.test.ts, unknown-data-model-change: extensions/memory-core/src/dreaming-phases.test.ts, unknown-data-model-change: extensions/memory-core/src/dreaming-phases.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🦪 silver shellfish
Proof: 🦪 silver shellfish
Patch quality: 🦞 diamond lobster
Result: blocked until real behavior proof from a real setup is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

[P1] Add redacted real workspace dreaming output showing daily markdown and session corpus contain zero lone surrogates.
Update the PR body after adding proof; if re-review does not trigger automatically, ask a maintainer to comment @clawsweeper re-review.

Proof guidance:

[P1] Needs real behavior proof before merge: The PR body provides focused test/CI/autoreview evidence only; add redacted terminal output, logs, copied live output, or a recording from a real OpenClaw workspace dreaming run, then update the PR body to trigger re-review.

Risk before merge

[P1] Contributor proof is still test/CI-only; before merge, the PR needs redacted output from a real OpenClaw workspace dreaming run showing generated daily snippets and session corpus output contain no lone surrogates.

Maintainer options:

Decide the mitigation before merge
Merge the SDK-facade truncation fix after a real workspace dreaming run proves daily snippets and session corpus output stay valid Unicode.
Pause or close
Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

[P1] Human follow-up is needed because the contributor must add real behavior proof; there is no concrete code defect for ClawSweeper to repair.

Security
Cleared: No concrete security or supply-chain concern found; the diff adds no dependencies, workflows, permissions, secrets handling, lockfiles, lifecycle hooks, or downloaded-code execution.

Review details

Best possible solution:

Merge the SDK-facade truncation fix after a real workspace dreaming run proves daily snippets and session corpus output stay valid Unicode.

Do we have a high-confidence way to reproduce the issue?

Yes, source-reproducible: current main and v2026.6.11 use raw .slice at the dreaming truncation points, so a surrogate pair straddling the limit can leave a lone surrogate. I did not run a live current-main repro in this read-only review.

Is this the best way to solve the issue?

Yes for the code change: reusing the existing Plugin SDK truncateUtf16Safe facade is the narrow owner-boundary fix. Merge readiness still depends on real behavior proof from a real setup.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against e2a4973edf3b.

Label changes

Label justifications:

P2: This is a normal-priority memory-core correctness fix with limited blast radius and no evidence of widespread user-facing breakage.
rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🦞 diamond lobster.
status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR body provides focused test/CI/autoreview evidence only; add redacted terminal output, logs, copied live output, or a recording from a real OpenClaw workspace dreaming run, then update the PR body to trigger re-review.

Evidence reviewed

PR surface:

Source +7, Tests +67. Total +74 across 2 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	1	12	5	+7
Tests	1	69	2	+67
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	2	81	7	+74

What I checked:

Repository policy read: Root AGENTS.md and extensions/AGENTS.md were read fully; the extension boundary rule supports using the Plugin SDK facade import rather than a core-internal import. (AGENTS.md:1, e2a4973edf3b)
Current main still has raw daily truncation: Current main truncates daily headings, list snippets, and chunk snippets with raw .slice, so the central fix is not already implemented on main. (extensions/memory-core/src/dreaming-phases.ts:188, e2a4973edf3b)
Current main still has raw session truncation: Current main also raw-slices normalized session corpus snippets and rendered session corpus lines, which are the session-side boundaries this PR targets. (extensions/memory-core/src/dreaming-phases.ts:678, e2a4973edf3b)
PR head uses existing SDK helper: PR head imports truncateUtf16Safe from the memory-core Plugin SDK facade and applies it to the five affected dreaming truncation sites. (extensions/memory-core/src/dreaming-phases.ts:6, f08f11342823)
Helper contract matches the bug: truncateUtf16Safe clamps through sliceUtf16Safe, which backs off when the cut would leave a high surrogate without its paired low surrogate. (packages/normalization-core/src/utf16-slice.ts:44, e2a4973edf3b)
SDK facade exports helper: The memory-core host engine foundation facade exports truncateUtf16Safe, matching the extension boundary rule and an existing memory-core manager-search usage pattern. (src/plugin-sdk/memory-core-host-engine-foundation.ts:57, e2a4973edf3b)

Likely related people:

vignesh07: Authored the dreaming sweep/session ingestion hardening and checkpointing commits that own the session corpus path this PR changes. (role: feature-history contributor; confidence: high; commits: 5291a2cfd1be, 61e61ccc182f, 695176542fcf; files: extensions/memory-core/src/dreaming-phases.ts, extensions/memory-core/src/dreaming-phases.test.ts)
mbelinky: Authored and merged the daily dreaming chunk ingestion PR that introduced the chunking surface now using safe truncation. (role: daily dreaming ingestion contributor; confidence: high; commits: 4661bf66c4c4; files: extensions/memory-core/src/dreaming-phases.ts)
jalehman: Authored the accessor-backed transcript corpus refactor touching the memory transcript/corpus access boundary adjacent to this session ingestion path. (role: recent adjacent owner; confidence: medium; commits: c24d266b2d09; files: extensions/memory-core/src/dreaming-phases.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (19 earlier review cycles; latest 8 shown)

reviewed 2026-07-08T09:52:02.752Z sha d77856e :: needs real behavior proof before merge. :: [P2] Place the session emoji at the corpus cut point
reviewed 2026-07-08T09:58:54.405Z sha d77856e :: needs real behavior proof before merge. :: [P2] Place the session emoji at the actual corpus cut point
reviewed 2026-07-09T00:45:29.102Z sha 709d8d3 :: needs real behavior proof before merge. :: none
reviewed 2026-07-09T00:51:45.384Z sha 709d8d3 :: needs real behavior proof before merge. :: none
reviewed 2026-07-09T01:34:45.447Z sha 709d8d3 :: needs real behavior proof before merge. :: none
reviewed 2026-07-09T01:41:57.773Z sha 709d8d3 :: needs real behavior proof before merge. :: none
reviewed 2026-07-09T04:09:38.492Z sha 709d8d3 :: needs real behavior proof before merge. :: none
reviewed 2026-07-09T04:18:05.193Z sha 709d8d3 :: needs real behavior proof before merge. :: none

Replace .slice(0, N) with truncateUtf16Safe() at 5 call sites in dreaming-phases.ts so complex emoji and surrogate pairs near the truncation boundary are not split into lone surrogates. truncateUtf16Safe is the standard SDK helper, already used in session-cost-usage, cron, exec-approval, and node-host modules for the same purpose.

xydt-tanshanshan · 2026-07-08T01:42:08Z

@clawsweeper re-review

clawsweeper · 2026-07-08T01:42:11Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

xydt-tanshanshan · 2026-07-08T01:47:13Z

@clawsweeper re-review

xydt-tanshanshan · 2026-07-08T07:34:04Z

@clawsweeper re-review

…rrogate-proof test Replace direct @openclaw/normalization-core/utf16-slice import with openclaw/plugin-sdk/memory-core-host-engine-foundation SDK facade, matching manager-search.ts in the same extension. Add surrogate-proof.test.ts that runs a full dreaming ingestion sweep with emoji at the 280-char boundary and verifies zero lone surrogates.

… assert session corpus file

…script

xydt-tanshanshan · 2026-07-09T03:58:31Z

@clawsweeper re-review

clawsweeper · 2026-07-09T03:58:33Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

steipete · 2026-07-09T04:40:11Z

Land-ready review complete for exact head f08f1134282326ad18d4038b6458b54d7ba5e7a3.

Maintainer review kept all five useful memory-core fixes and replaced 280 lines of standalone proof scaffolding with exact cases in the existing dreaming production harness. Coverage now isolates daily headings, daily items, joined daily chunks, normalized session snippets, and rendered session lines.

Proof:

Blacksmith Testbox: env CI=true corepack pnpm test extensions/memory-core/src/dreaming-phases.test.ts — 51/51 passed (run 28994107855);
exact-head hosted CI — passed (run 28994221181);
git diff --check — passed;
fresh structured autoreview after final edits — clean, no accepted/actionable findings (confidence 0.98).

Known proof gaps: none for this bounded owner path.

steipete · 2026-07-09T04:40:26Z

Merged via squash.

Prepared head SHA: f08f1134282326ad18d4038b6458b54d7ba5e7a3
Landed commit: 1d4d8474da853845687ed0d0a5b18870b3f0ddd3

…on (openclaw#101946) * [AI] fix(memory): use truncateUtf16Safe for dreaming snippet truncation Replace .slice(0, N) with truncateUtf16Safe() at 5 call sites in dreaming-phases.ts so complex emoji and surrogate pairs near the truncation boundary are not split into lone surrogates. truncateUtf16Safe is the standard SDK helper, already used in session-cost-usage, cron, exec-approval, and node-host modules for the same purpose. * [AI] fix(memory): use SDK facade import for truncateUtf16Safe, add surrogate-proof test Replace direct @openclaw/normalization-core/utf16-slice import with openclaw/plugin-sdk/memory-core-host-engine-foundation SDK facade, matching manager-search.ts in the same extension. Add surrogate-proof.test.ts that runs a full dreaming ingestion sweep with emoji at the 280-char boundary and verifies zero lone surrogates. * [AI] fix(memory-proof): use relative imports in surrogate-proof test * [AI] fix(memory-proof): place emoji at post-prefix 280-char boundary, assert session corpus file * [AI] fix(memory-proof): avoid unnecessary String callback in oxlint * [AI] fix(memory-proof): add curly braces per eslint curly rule * [AI] feat(proof): add standalone dreaming surrogate truncation proof script * [AI] chore(proof): remove unused SQLite-dependent proof script * [AI] fix(proof): correct emoji boundary placement for all 5 call sites * [AI] fix(proof): add curly braces per eslint curly rule in proof script * [AI] fix(proof): session emoji at actual corpus 280-char boundary * test: streamline memory dreaming UTF-16 coverage --------- Co-authored-by: hailory <[email protected]> Co-authored-by: Peter Steinberger <[email protected]>

openclaw-barnacle Bot added extensions: memory-core Extension: memory-core size: XS labels Jul 8, 2026

xydt-tanshanshan force-pushed the fix/dreaming-emoji-safe-truncation branch from 668ce1e to 9ae8803 Compare July 8, 2026 01:38

xydt-tanshanshan changed the title ~~[AI] fix(memory): keep emoji / surrogate pairs intact during dreaming snippet truncation~~ [AI] fix(memory): use truncateUtf16Safe for dreaming snippet truncation Jul 8, 2026

clawsweeper Bot removed the merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. label Jul 8, 2026

openclaw-barnacle Bot added triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. and removed triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. labels Jul 8, 2026

clawsweeper Bot added the merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. label Jul 8, 2026

openclaw-barnacle Bot added size: M and removed size: XS labels Jul 8, 2026

hailory added 4 commits July 8, 2026 16:30

[AI] fix(memory-proof): use relative imports in surrogate-proof test

d9ac5e1

[AI] fix(memory-proof): place emoji at post-prefix 280-char boundary,…

da68715

… assert session corpus file

[AI] fix(memory-proof): avoid unnecessary String callback in oxlint

34c1f45

[AI] fix(memory-proof): add curly braces per eslint curly rule

7876800

hailory added 2 commits July 8, 2026 17:19

[AI] feat(proof): add standalone dreaming surrogate truncation proof …

76e0f92

…script

[AI] chore(proof): remove unused SQLite-dependent proof script

a02499c

openclaw-barnacle Bot added the scripts Repository scripts label Jul 8, 2026

[AI] fix(proof): correct emoji boundary placement for all 5 call sites

26bc079

clawsweeper Bot removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 8, 2026

[AI] fix(proof): add curly braces per eslint curly rule in proof script

d77856e

clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 8, 2026

[AI] fix(proof): session emoji at actual corpus 280-char boundary

709d8d3

test: streamline memory dreaming UTF-16 coverage

f08f113

openclaw-barnacle Bot added size: S and removed scripts Repository scripts size: M labels Jul 9, 2026

steipete merged commit 1d4d847 into openclaw:main Jul 9, 2026
104 of 108 checks passed

This was referenced Jul 9, 2026

fix(text): keep five src-side text truncations UTF-16 safe #102630

Closed

fix(chat,cli): keep session snippet and CLI output digest truncation UTF-16 safe #102747

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[AI] fix(memory): use truncateUtf16Safe for dreaming snippet truncation#101946

[AI] fix(memory): use truncateUtf16Safe for dreaming snippet truncation#101946
steipete merged 12 commits into
openclaw:mainfrom
xydt-tanshanshan:fix/dreaming-emoji-safe-truncation

xydt-tanshanshan commented Jul 8, 2026 •

edited by steipete

Loading

Uh oh!

clawsweeper Bot commented Jul 8, 2026 •

edited

Loading

Uh oh!

xydt-tanshanshan commented Jul 8, 2026

Uh oh!

clawsweeper Bot commented Jul 8, 2026 •

edited

Loading

Uh oh!

xydt-tanshanshan commented Jul 8, 2026

Uh oh!

xydt-tanshanshan commented Jul 8, 2026

Uh oh!

xydt-tanshanshan commented Jul 9, 2026

Uh oh!

clawsweeper Bot commented Jul 9, 2026 •

edited

Loading

Uh oh!

steipete commented Jul 9, 2026

Uh oh!

Uh oh!

steipete commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

xydt-tanshanshan commented Jul 8, 2026 • edited by steipete Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Uh oh!

clawsweeper Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xydt-tanshanshan commented Jul 8, 2026

Uh oh!

clawsweeper Bot commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xydt-tanshanshan commented Jul 8, 2026

Uh oh!

xydt-tanshanshan commented Jul 8, 2026

Uh oh!

xydt-tanshanshan commented Jul 9, 2026

Uh oh!

clawsweeper Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steipete commented Jul 9, 2026

Uh oh!

Uh oh!

steipete commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

xydt-tanshanshan commented Jul 8, 2026 •

edited by steipete

Loading

clawsweeper Bot commented Jul 8, 2026 •

edited

Loading

clawsweeper Bot commented Jul 8, 2026 •

edited

Loading

clawsweeper Bot commented Jul 9, 2026 •

edited

Loading