fix(agent-core,memory-core): keep compaction summary and memory snippet truncation UTF-16 safe#102542
Conversation
…et truncation UTF-16 safe
|
Codex review: needs real behavior proof before merge. Reviewed July 9, 2026, 4:54 AM ET / 08:54 UTC. Summary PR surface: Source +3, Tests +80, Config +1, Other +3. Total +87 across 6 files. Reproducibility: yes. from source: current main truncates with raw string slice at the compaction and memory snippet boundaries, so a supplementary Unicode character whose high surrogate lands at the limit can produce a dangling surrogate. I did not run a live repro in this read-only review. Review metrics: 2 noteworthy metrics.
Stored data model Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Proof guidance:
Risk before merge
Maintainer options:
Next step before merge
Security Review detailsBest possible solution: Land the shared UTF-16-safe truncation fix after adding real behavior proof and letting dependency guard plus CI settle on the exact head. Do we have a high-confidence way to reproduce the issue? Yes from source: current main truncates with raw string slice at the compaction and memory snippet boundaries, so a supplementary Unicode character whose high surrogate lands at the limit can produce a dangling surrogate. I did not run a live repro in this read-only review. Is this the best way to solve the issue? Yes for the implementation shape: core agent code uses the shared normalization-core helper directly, while memory-core uses the plugin SDK facade as required by the extension boundary. It is not merge-ready until proof and guard status are settled. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 85aa26d34f8b. Label changesLabel changes:
Label justifications:
Evidence reviewedPR surface: Source +3, Tests +80, Config +1, Other +3. Total +87 across 6 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
Dependency GuardThis PR changes dependency-related files. Maintainers should confirm these changes are intentional. Changed files:
Maintainer follow-up:
|
Dependency graph change authorizedThis PR includes dependency graph changes. A repository admin or member of
A later push changes the PR head SHA and requires a fresh security approval. |
|
/allow-dependencies-change |
|
Additional real-behavior proof at exact head I ran a source-blind
Observed output: {
"compaction": {
"hasUnpairedSurrogate": false,
"exactOmittedMarker": true,
"retainedPrefixChars": true
},
"recallStore": {
"hasUnpairedSurrogate": false,
"storedLength": 799,
"exactSafePrefix": true
},
"promotion": {
"hasUnpairedSurrogate": false,
"exactSafePrefix": true
}
}The sole CI failure was an unrelated @clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
Maintainer review complete at exact head Review improvements:
Validation:
Known proof gaps: none for these three bounded text paths. |
|
Merged via squash.
|
…et truncation UTF-16 safe (openclaw#102542) * fix(agent-core,memory-core): keep compaction summary and memory snippet truncation UTF-16 safe * fix: make compaction and memory truncation UTF-16 safe --------- Co-authored-by: Peter Steinberger <[email protected]>
What Problem This Solves
Compaction and short-term memory snippets could split a UTF-16 surrogate pair when a length boundary landed inside an emoji or another supplementary Unicode character.
Why This Change Was Made
Compaction now uses the shared normalization helper and reports the omitted count from the actual safe prefix. Memory-core reaches the same helper through the supported plugin SDK facade. The reviewed fix keeps helpers private and tests the public serialization, recall-store, and promotion paths rather than exporting internals for tests.
User Impact
Compaction summaries and memory snippets remain valid Unicode at truncation boundaries. The output and limits for ASCII and already-short text remain unchanged.
Evidence
node scripts/run-vitest.mjs packages/agent-core/src/harness/compaction/utils.test.ts— 3 passed.UTF-16 snippet bounds— 2 passed, 91 skipped.MEMORY.mdcontent.node --import tsxruntime probe exercised actual compaction serialization plus SQLite-backed recall persistence and promotion output: all three reportedhasUnpairedSurrogate: false; exact safe prefixes and omitted count matched.openclaw/plugin-sdk/text-utility-runtime.🤖 Generated with Claude Code