fix(memory-lancedb): keep memory recall/text preview truncation UTF-16 safe by lsr911 · Pull Request #101623 · openclaw/openclaw

lsr911 · 2026-07-07T11:47:04Z

What Problem This Solves

The memory-lancedb plugin already imports truncateUtf16Safe for recall-query normalization, but two call sites still use raw .slice(0, N):

Memory create/store: text.slice(0, 100) in the Stored confirmation message
Memory search/list: r.entry.text.slice(0, 60) in formatted search result lines

When the boundary lands mid-surrogate-pair, the dangling high surrogate renders as U+FFFD in agent-visible output.

Why This Change Was Made

truncateUtf16Safe is already imported in this file at line 33 — the two raw .slice() calls were simply missed when the recall-query path was fixed.

Evidence

Branch prerun (commit `38e0e34`)

=== memory stored @100 boundary ===
Raw .slice ends: "mm\ud83e" (dangling high surrogate)
truncateUtf16Safe: "mmm" | len: 99 | no U+FFFD

=== memory list @60 boundary ===
Raw .slice ends: "nn\ud83d" (dangling high surrogate)
truncateUtf16Safe: "nnn" | len: 59 | no U+FFFD

Files Changed

File	Change
`extensions/memory-lancedb/index.ts`	2 calls: `.slice(0, 100)` → `truncateUtf16Safe`, `.slice(0, 60)` → `truncateUtf16Safe`

…6 safe Replace two raw .slice(0,) calls with truncateUtf16Safe in memory create/store (100-code-unit preview) and search/list (60-code-unit preview) paths, preventing dangling surrogates when memory text contains emoji or CJK characters at the boundary.

clawsweeper · 2026-07-07T12:04:39Z

Codex review: needs real behavior proof before merge. Reviewed July 7, 2026, 8:15 AM ET / 12:15 UTC.

Summary
The PR replaces two memory-lancedb preview .slice calls in memory_store and memory_forget candidate output with truncateUtf16Safe, with incidental formatter wrapping.

PR surface: Source +24. Total +24 across 1 file.

Reproducibility: yes. for the underlying string bug: current main's exact preview slice operations can be exercised with crafted surrogate-pair boundaries and produce dangling high surrogates. A full real memory plugin run has not been provided.

Review metrics: none identified.

Stored data model
Persistent data-model change detected: unknown-data-model-change: extensions/memory-lancedb/index.ts, vector/embedding metadata: extensions/memory-lancedb/index.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🦪 silver shellfish
Proof: 🦪 silver shellfish
Patch quality: 🐚 platinum hermit
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

[P1] Add redacted real behavior proof from memory_store and memory_forget candidate runs with emoji at the truncation boundary.
Consider adding focused regressions in extensions/memory-lancedb/index.test.ts for both preview strings.

Proof guidance:

[P1] Needs stronger real behavior proof before merge: The copied terminal-style boundary output is useful but does not show actual after-fix memory_store or memory_forget output; add redacted terminal output, logs, a screenshot, or a recording from those real paths and update the PR body for re-review.

Risk before merge

[P1] The PR body contains copied boundary-output text, but it does not yet show an after-fix memory_store or memory_forget candidate run, so the changed user-visible tool output remains unproven in a real setup.

Maintainer options:

Decide the mitigation before merge
Land the narrow helper reuse after the contributor adds redacted real memory tool output; focused regression coverage for both preview paths would make the fix more durable.
Pause or close
Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

[P1] Keep this open for contributor-provided real behavior proof; there is no narrow ClawSweeper repair because the remaining blocker is evidence from the contributor's changed setup, not a code defect.

Security
Cleared: Cleared: the diff only changes in-plugin preview string truncation and formatter wrapping, with no dependency, workflow, secret, config, or storage schema changes.

Review details

Best possible solution:

Land the narrow helper reuse after the contributor adds redacted real memory tool output; focused regression coverage for both preview paths would make the fix more durable.

Do we have a high-confidence way to reproduce the issue?

Yes for the underlying string bug: current main's exact preview slice operations can be exercised with crafted surrogate-pair boundaries and produce dangling high surrogates. A full real memory plugin run has not been provided.

Is this the best way to solve the issue?

Yes, the proposed code path is the narrowest maintainable fix because it reuses the existing UTF-16-safe helper already imported by the plugin. The remaining blocker is proof, not a different implementation layer.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 2ba622ca3019.

Label changes

Label justifications:

P3: This is a small preview-output correctness fix in one bundled memory plugin, without evidence of data loss, security impact, or a broken core workflow.
rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The copied terminal-style boundary output is useful but does not show actual after-fix memory_store or memory_forget output; add redacted terminal output, logs, a screenshot, or a recording from those real paths and update the PR body for re-review.

Evidence reviewed

PR surface:

Source +24. Total +24 across 1 file.

View PR surface stats

Area	Files	Added	Removed	Net
Source	1	32	8	+24
Tests	0	0	0	0
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	1	32	8	+24

What I checked:

Repository policy read: Root and scoped extension policies were read and applied; the review used the OpenClaw PR proof gate and extension-boundary guidance. (AGENTS.md:1, 2ba622ca3019)
Current main still has the raw preview slices: Current main returns text.slice(0, 100) for the stored-memory confirmation and r.entry.text.slice(0, 60) for memory_forget candidate previews, so the central change is not already implemented. (extensions/memory-lancedb/index.ts:1661, 2ba622ca3019)
PR head applies the intended helper: The PR head changes the two preview sites to truncateUtf16Safe(text, 100) and truncateUtf16Safe(r.entry.text, 60). (extensions/memory-lancedb/index.ts:1669, 38e0e342b5d2)
Helper contract matches the fix: truncateUtf16Safe delegates to sliceUtf16Safe(input, 0, limit), whose boundary checks drop dangling surrogate halves at slice edges. (packages/normalization-core/src/utf16-slice.ts:43, 2ba622ca3019)
Source-level reproduction: A Node expression using the same current-main slice boundaries ends with lone high surrogates at the 100- and 60-code-unit preview cuts. (2ba622ca3019)
Adjacent tests do not cover this boundary: Existing memory-lancedb tests exercise store and forget candidate formatting, but the current assertions do not cover surrogate-safe truncation at the two preview boundaries. (extensions/memory-lancedb/index.test.ts:2921, 2ba622ca3019)

Likely related people:

steipete: Recent memory-lancedb history includes repeated fixes for recall limits, timeouts, validation, and the current UTF-16 helper history also points to steipete-authored reliability work. (role: recent area contributor; confidence: high; commits: 9e2bd8b2f7eb, 87eae7f811da, b22c36f1125c; files: extensions/memory-lancedb/index.ts, extensions/memory-lancedb/index.test.ts, packages/normalization-core/src/utf16-slice.ts)
amittell: Authored recent memory-lancedb envelope/sludge sanitizer work and tests around memory text cleanup, which is adjacent to the affected preview text paths. (role: adjacent feature contributor; confidence: medium; commits: 945faf8e6798, 1d4c1ba56dcc; files: extensions/memory-lancedb/index.ts, extensions/memory-lancedb/index.test.ts)
pgondhi987: Authored a recent memory-lancedb recall output guard touching the same plugin output-cleanliness area. (role: adjacent recall-output contributor; confidence: medium; commits: 03a8d18cd434; files: extensions/memory-lancedb/index.ts, extensions/memory-lancedb/index.test.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (1 earlier review cycle)

reviewed 2026-07-07T12:08:07.724Z sha 38e0e34 :: needs real behavior proof before merge. :: none

steipete · 2026-07-07T12:45:55Z

Thanks @lsr911. The useful behavior from this PR was consolidated with the sibling UTF-16 boundary fixes into #101654 and landed on main in a9582a1bb62aa70bb0b0ceb72536d0d76f08eab8.

The landed fix uses the existing truncateUtf16Safe helper, preserves this caller's prior code-unit limit and output shape, and adds caller-level surrogate-boundary coverage. Closing this PR as superseded by the canonical landed change.

openclaw-barnacle Bot added extensions: memory-lancedb Extension: memory-lancedb size: XS triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. labels Jul 7, 2026

openclaw-barnacle Bot removed the triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. label Jul 7, 2026

steipete mentioned this pull request Jul 7, 2026

fix: keep bounded text truncation UTF-16 safe #101654

Merged

steipete closed this Jul 7, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(memory-lancedb): keep memory recall/text preview truncation UTF-16 safe#101623

fix(memory-lancedb): keep memory recall/text preview truncation UTF-16 safe#101623
lsr911 wants to merge 1 commit into
openclaw:mainfrom
lsr911:fix/memory-utf16-safe-truncation

lsr911 commented Jul 7, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jul 7, 2026 •

edited

Loading

Uh oh!

steipete commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

lsr911 commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

Evidence

Branch prerun (commit 38e0e34)

Files Changed

Uh oh!

clawsweeper Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steipete commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lsr911 commented Jul 7, 2026 •

edited

Loading

Branch prerun (commit `38e0e34`)

clawsweeper Bot commented Jul 7, 2026 •

edited

Loading