Skip to content

fix(agent-core,memory-core): keep compaction summary and memory snippet truncation UTF-16 safe#102542

Merged
steipete merged 2 commits into
openclaw:mainfrom
wings1029:fix/utf16-safe-compaction-memory-summaries
Jul 9, 2026
Merged

fix(agent-core,memory-core): keep compaction summary and memory snippet truncation UTF-16 safe#102542
steipete merged 2 commits into
openclaw:mainfrom
wings1029:fix/utf16-safe-compaction-memory-summaries

Conversation

@wings1029

@wings1029 wings1029 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

What Problem This Solves

Compaction and short-term memory snippets could split a UTF-16 surrogate pair when a length boundary landed inside an emoji or another supplementary Unicode character.

Why This Change Was Made

Compaction now uses the shared normalization helper and reports the omitted count from the actual safe prefix. Memory-core reaches the same helper through the supported plugin SDK facade. The reviewed fix keeps helpers private and tests the public serialization, recall-store, and promotion paths rather than exporting internals for tests.

User Impact

Compaction summaries and memory snippets remain valid Unicode at truncation boundaries. The output and limits for ASCII and already-short text remain unchanged.

Evidence

  • node scripts/run-vitest.mjs packages/agent-core/src/harness/compaction/utils.test.ts — 3 passed.
  • Focused memory-core extension run for UTF-16 snippet bounds — 2 passed, 91 skipped.
  • Regression assertions cover exact serialized output, exact omitted count, persisted recall snippets, and promoted MEMORY.md content.
  • Source-blind node --import tsx runtime probe exercised actual compaction serialization plus SQLite-backed recall persistence and promotion output: all three reported hasUnpairedSurrogate: false; exact safe prefixes and omitted count matched.
  • Fresh autoreview: clean; patch judged correct with 0.98 confidence.
  • Agent-core now declares its direct normalization-core workspace dependency; memory-core uses openclaw/plugin-sdk/text-utility-runtime.

🤖 Generated with Claude Code

@clawsweeper

clawsweeper Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed July 9, 2026, 4:54 AM ET / 08:54 UTC.

Summary
The PR changes agent compaction and memory-core snippet truncation to use UTF-16-safe truncation helpers, adds focused regression coverage, and declares a new internal workspace dependency for agent-core.

PR surface: Source +3, Tests +80, Config +1, Other +3. Total +87 across 6 files.

Reproducibility: yes. from source: current main truncates with raw string slice at the compaction and memory snippet boundaries, so a supplementary Unicode character whose high surrogate lands at the limit can produce a dangling surrogate. I did not run a live repro in this read-only review.

Review metrics: 2 noteworthy metrics.

  • Truncation boundaries changed: 3 call sites changed. The patch touches compaction summary truncation, short-term recall snippets, and promoted MEMORY.md display snippets, which are the behavior surfaces reviewers should verify.
  • Dependency graph surfaces: 1 manifest dependency added, 1 lockfile importer changed. agent-core now directly depends on the internal normalization-core package, so the dependency guard and package-boundary review are relevant before merge.

Stored data model
Persistent data-model change detected: unknown-data-model-change: extensions/memory-core/src/short-term-promotion.test.ts, vector/embedding metadata: extensions/memory-core/src/short-term-promotion.test.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🦪 silver shellfish
Proof: 🦪 silver shellfish
Patch quality: 🐚 platinum hermit
Result: blocked until real behavior proof from a real setup is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P1] Add redacted terminal or live-output proof that exercises the after-fix compaction serialization and memory recall/promotion paths outside unit-test assertions.
  • Rerun or wait for dependency-guard after the /allow-dependencies-change comment and confirm CI is green on the exact head.

Proof guidance:

  • [P1] Needs real behavior proof before merge: The PR body reports focused tests only; add redacted terminal/live output or logs showing the after-fix compaction and memory behavior, then update the PR body to trigger a fresh ClawSweeper review or ask a maintainer to comment @clawsweeper re-review.

Risk before merge

  • [P1] Real behavior proof is only focused test output; the PR still needs redacted terminal/live output or logs showing the after-fix compaction and memory paths.
  • [P1] The dependency graph change has a maintainer authorization comment, but dependency-guard still showed failure at review time and needs rerun/settling on the exact head.
  • [P1] One CI shard was failing while logs were unavailable because the run was still in progress, so validation is not fully settled.

Maintainer options:

  1. Decide the mitigation before merge
    Land the shared UTF-16-safe truncation fix after adding real behavior proof and letting dependency guard plus CI settle on the exact head.
  2. Pause or close
    Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

  • [P1] No automated code repair is indicated; the PR needs contributor real-behavior proof plus exact-head guard/CI follow-through.

Security
Cleared: No third-party dependency, secret, permission, or code-execution expansion was found; the internal workspace dependency has an explicit maintainer authorization comment pending guard rerun.

Review details

Best possible solution:

Land the shared UTF-16-safe truncation fix after adding real behavior proof and letting dependency guard plus CI settle on the exact head.

Do we have a high-confidence way to reproduce the issue?

Yes from source: current main truncates with raw string slice at the compaction and memory snippet boundaries, so a supplementary Unicode character whose high surrogate lands at the limit can produce a dangling surrogate. I did not run a live repro in this read-only review.

Is this the best way to solve the issue?

Yes for the implementation shape: core agent code uses the shared normalization-core helper directly, while memory-core uses the plugin SDK facade as required by the extension boundary. It is not merge-ready until proof and guard status are settled.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 85aa26d34f8b.

Label changes

Label changes:

  • add P2: This is a bounded correctness fix for invalid Unicode in agent/memory text truncation, with limited blast radius and no evidence of an urgent production outage.
  • add rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.

Label justifications:

  • P2: This is a bounded correctness fix for invalid Unicode in agent/memory text truncation, with limited blast radius and no evidence of an urgent production outage.
  • rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
Evidence reviewed

PR surface:

Source +3, Tests +80, Config +1, Other +3. Total +87 across 6 files.

View PR surface stats
Area Files Added Removed Net
Source 2 7 4 +3
Tests 2 80 0 +80
Docs 0 0 0 0
Config 1 1 0 +1
Generated 0 0 0 0
Other 1 3 0 +3
Total 6 91 4 +87

What I checked:

Likely related people:

  • steipete: Authored the current PR head commit and previously led agent-core extraction and normalization-core package work that define the affected boundary. (role: recent area contributor and PR head committer; confidence: high; commits: 35c4f99efc05, bb46b79d3c14, 062f88e3e3af; files: packages/agent-core/src/harness/compaction/utils.ts, packages/agent-core/package.json, packages/normalization-core/src/utf16-slice.ts)
  • vincentkoc: Recent commits changed and refactored the memory-core promotion area, including short-term promotion internals. (role: recent memory-core area contributor; confidence: medium; commits: 6765eb0166fc, c9abc7588493, abd8a46b0a52; files: extensions/memory-core/src/short-term-promotion.ts)
  • LZY3538: Recent history for compaction utils shows the nested tool-result serialization behavior came through their merged work. (role: recent compaction behavior contributor; confidence: medium; commits: 413acc2e634d; files: packages/agent-core/src/harness/compaction/utils.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@steipete
steipete requested a review from a team as a code owner July 9, 2026 08:46
@github-actions github-actions Bot added the dependencies-changed PR changes dependency-related files label Jul 9, 2026
@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Dependency Guard

This PR changes dependency-related files. Maintainers should confirm these changes are intentional.

Changed files:

  • packages/agent-core/package.json
  • pnpm-lock.yaml

Maintainer follow-up:

  • Review whether the dependency changes are intentional.
  • Inspect resolved package deltas when lockfile, shrinkwrap, or workspace dependency policy changes are present.
  • Treat package-lock.json and npm-shrinkwrap.json diffs as security-review surfaces.
  • Run pnpm deps:changes:report -- --base-ref origin/main --markdown /tmp/dependency-changes.md --json /tmp/dependency-changes.json locally for detailed release-style evidence.

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Dependency graph change authorized

This PR includes dependency graph changes. A repository admin or member of @openclaw/openclaw-secops authorized this exact head SHA with /allow-dependencies-change.

  • Approved SHA: 35c4f99efc05a47f5dae719b16246409d8307706
  • Approved by: @steipete

A later push changes the PR head SHA and requires a fresh security approval.

@steipete

steipete commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

/allow-dependencies-change

@steipete

steipete commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Additional real-behavior proof at exact head 35c4f99efc05a47f5dae719b16246409d8307706:

I ran a source-blind node --import tsx probe against the actual production functions, outside Vitest. It exercised:

  • serializeConversation with an emoji straddling the 2,000-code-unit tool-result boundary;
  • recordShortTermRecalls through the SQLite-backed memory-core state runtime with an emoji straddling the 800-code-unit recall boundary;
  • rankShortTermPromotionCandidates plus applyShortTermPromotions, then read the generated MEMORY.md line with an emoji straddling the promoted-snippet budget.

Observed output:

{
  "compaction": {
    "hasUnpairedSurrogate": false,
    "exactOmittedMarker": true,
    "retainedPrefixChars": true
  },
  "recallStore": {
    "hasUnpairedSurrogate": false,
    "storedLength": 799,
    "exactSafePrefix": true
  },
  "promotion": {
    "hasUnpairedSurrogate": false,
    "exactSafePrefix": true
  }
}

The sole CI failure was an unrelated src/tui/tui-pty-local.e2e.test.ts timeout in the whole-repo shard; failed jobs are rerunning. The exact-head internal workspace dependency was reviewed and explicitly authorized with /allow-dependencies-change.

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@steipete

steipete commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Maintainer review complete at exact head 35c4f99efc05a47f5dae719b16246409d8307706.

Review improvements:

  • corrected compaction's omitted count to use the actual safe prefix;
  • kept private truncation helpers private and replaced direct-helper tests with public serialization, recall-store, and promotion assertions;
  • routed memory-core through the supported plugin SDK facade;
  • declared agent-core's intentional direct internal workspace dependency.

Validation:

  • compaction tests — 3 passed;
  • focused memory-core UTF-16 tests — 2 passed, 91 skipped;
  • source-blind runtime proof for compaction, SQLite recall persistence, and generated MEMORY.mdall boundaries valid;
  • fresh autoreview — clean, 0.98 confidence;
  • exact-head hosted CI/Testbox and dependency guard — passed after one unrelated TUI PTY timeout rerun.

Known proof gaps: none for these three bounded text paths.

@steipete
steipete merged commit f7cc6eb into openclaw:main Jul 9, 2026
166 of 168 checks passed
@steipete

steipete commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Merged via squash.

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 10, 2026
…et truncation UTF-16 safe (openclaw#102542)

* fix(agent-core,memory-core): keep compaction summary and memory snippet truncation UTF-16 safe

* fix: make compaction and memory truncation UTF-16 safe

---------

Co-authored-by: Peter Steinberger <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies-changed PR changes dependency-related files extensions: memory-core Extension: memory-core size: S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants