fix(memory-host): use truncateUtf16Safe for QMD stderr context truncation by lsr911 · Pull Request #102547 · openclaw/openclaw

lsr911 · 2026-07-09T08:20:38Z

What Problem This Solves

QMD stderr excerpts were truncated with a raw UTF-16 slice. When the cutoff landed between an emoji's surrogate halves, the resulting error text contained an unpaired surrogate.

Why This Change Was Made

Use the repository's existing truncateUtf16Safe boundary helper at the QMD parser's owned stderr limit, preserving the same maximum and suffix while keeping output valid Unicode.

User Impact

QMD parse errors remain readable when stderr contains emoji or other supplementary Unicode characters at the truncation boundary.

Evidence

Added public parser regression coverage with an emoji crossing the 117-code-unit boundary.
node scripts/run-vitest.mjs packages/memory-host-sdk/src/host/qmd-query-parser.test.ts — 9 tests passed.
Fresh autoreview: clean, no accepted/actionable findings (0.99).

Files Changed

File	Change
`packages/memory-host-sdk/src/host/qmd-query-parser.ts`	Replace the unsafe stderr slice with `truncateUtf16Safe`.
`packages/memory-host-sdk/src/host/qmd-query-parser.test.ts`	Add boundary regression coverage.

🤖 Generated with Claude Code

…tion

clawsweeper · 2026-07-09T08:45:47Z

Codex review: needs real behavior proof before merge. Reviewed July 9, 2026, 4:49 AM ET / 08:49 UTC.

Summary
The branch imports truncateUtf16Safe into the QMD query parser and uses it when summarizing long stderr context.

PR surface: Source +1. Total +1 across 1 file.

Reproducibility: yes. source inspection plus a focused Node check shows the current .slice(0, 117) path can produce a dangling UTF-16 surrogate when stderr crosses that boundary. I did not run the full QMD parser test suite in this read-only review.

Review metrics: none identified.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🧂 unranked krab
Patch quality: 🦐 gold shrimp
Result: blocked until real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

[P1] Add redacted real behavior proof for the after-fix QMD stderr parse-error path; redact private paths, endpoints, IPs, keys, and other sensitive details.
[P1] Add a focused parser regression test with long stderr crossing a surrogate boundary.

Proof guidance:

[P1] Needs real behavior proof before merge: No after-fix real behavior proof is present; the contributor should add redacted terminal output, logs, or a short reproducible transcript showing the QMD stderr truncation path, and updating the PR body should trigger re-review. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

[P1] No contributor-provided real behavior proof shows the QMD stderr parse-error path after the change.
[P1] No focused regression test currently proves long QMD stderr with a surrogate boundary stays well-formed.

Maintainer options:

Decide the mitigation before merge
Keep the narrow helper change, add focused parser coverage for the stderr boundary, and require redacted terminal/log proof before merge.
Pause or close
Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

[P1] Contributor proof is the next gate; there is no concrete patch defect for ClawSweeper to repair automatically.

Security
Cleared: The diff only adds an existing internal helper import and changes diagnostic string truncation; it does not touch dependencies, workflows, secrets, or code execution surfaces.

Review details

Best possible solution:

Keep the narrow helper change, add focused parser coverage for the stderr boundary, and require redacted terminal/log proof before merge.

Do we have a high-confidence way to reproduce the issue?

Yes, source inspection plus a focused Node check shows the current .slice(0, 117) path can produce a dangling UTF-16 surrogate when stderr crosses that boundary. I did not run the full QMD parser test suite in this read-only review.

Is this the best way to solve the issue?

Yes, using the existing shared truncateUtf16Safe helper at the current stderr-summary call site is the narrowest maintainable fix. The missing pieces are proof and focused regression coverage, not a different implementation layer.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against ccdb60e5db57.

Label changes

Label changes:

add P3: This is a low-risk diagnostic-text correctness fix with limited user impact and no evidence of a blocked workflow.
add rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🦐 gold shrimp.
add status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: No after-fix real behavior proof is present; the contributor should add redacted terminal output, logs, or a short reproducible transcript showing the QMD stderr truncation path, and updating the PR body should trigger re-review. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Label justifications:

P3: This is a low-risk diagnostic-text correctness fix with limited user impact and no evidence of a blocked workflow.
rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🦐 gold shrimp.
status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: No after-fix real behavior proof is present; the contributor should add redacted terminal output, logs, or a short reproducible transcript showing the QMD stderr truncation path, and updating the PR body should trigger re-review. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence reviewed

PR surface:

Source +1. Total +1 across 1 file.

View PR surface stats

Area	Files	Added	Removed	Net
Source	1	2	1	+1
Tests	0	0	0	0
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	1	2	1	+1

What I checked:

Current main still uses unsafe stderr slicing: origin/main at ccdb60e still summarizes long QMD stderr with raw.slice(0, 117), which can return a dangling UTF-16 surrogate before appending the ellipsis. (packages/memory-host-sdk/src/host/qmd-query-parser.ts:83, ccdb60e5db57)
PR diff applies the shared helper at the right call site: The PR changes only the stderr summary truncation expression from .slice(0, 117) to truncateUtf16Safe(raw, 117). (packages/memory-host-sdk/src/host/qmd-query-parser.ts:82, 5a1c4b00f874)
Shared helper contract is surrogate-safe truncation: truncateUtf16Safe floors the limit, clamps below zero, and delegates to sliceUtf16Safe, which avoids cutting surrogate pairs at the slice edge. (packages/normalization-core/src/utf16-slice.ts:44, ccdb60e5db57)
Memory host package already uses this import path: packages/memory-host-sdk already imports @openclaw/normalization-core/utf16-slice in response-snippet.ts, so the PR is not adding a new package-boundary pattern. (packages/memory-host-sdk/src/host/response-snippet.ts:2, ccdb60e5db57)
Adjacent parser tests miss this edge: The existing QMD parser tests cover JSON, noisy stdout, no-result markers, and invalid stdout, but no test asserts long stderr truncation or surrogate boundaries. (packages/memory-host-sdk/src/host/qmd-query-parser.test.ts:5, ccdb60e5db57)
Source-level reproduction of the current bug: A 116-character prefix followed by an emoji makes native slice(0, 117) end on code unit d83d, a dangling high surrogate, matching the current implementation's boundary. (packages/memory-host-sdk/src/host/qmd-query-parser.ts:84, ccdb60e5db57)

Likely related people:

steipete: Recent path history shows multiple memory-host SDK refactors and docs/source contract updates around qmd-query-parser.ts, plus the broader normalization-core extraction that made the helper subpath available. (role: recent memory-host contributor; confidence: medium; commits: 4df95d3c3fd6, 5b98f03c64d2, c42664f9b226; files: packages/memory-host-sdk/src/host/qmd-query-parser.ts, packages/normalization-core/src/utf16-slice.ts)
vincentkoc: Authored and merged the recent QMD parser line-metadata work in fix(memory): use explicit qmd snippet line metadata #58181, touching both parser code and adjacent parser tests. (role: QMD parser contributor; confidence: high; commits: 075645f5cbfd, 5707038e6c5a; files: packages/memory-host-sdk/src/host/qmd-query-parser.ts, packages/memory-host-sdk/src/host/qmd-query-parser.test.ts, extensions/memory-core/src/memory/qmd-manager.ts)
lsr911: Previously authored the merged helper-pattern PR fix(tool-policy-audit): use truncateUtf16Safe for audit field truncation #102464 for another unsafe diagnostic truncation site, making them relevant to this narrow UTF-16-safe truncation sweep. (role: recent adjacent UTF-16 truncation contributor; confidence: medium; commits: 472e5167d38a; files: src/agents/tool-policy-audit.ts, src/agents/tool-policy-pipeline.test.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

steipete · 2026-07-09T09:38:32Z

Land-ready at exact head f07ef572a47e8b87a50b7ea6d481fdd132c19650.

Maintainer fixup added exported-parser regression coverage for an emoji crossing the 117-code-unit QMD stderr boundary. Focused proof: node scripts/run-vitest.mjs packages/memory-host-sdk/src/host/qmd-query-parser.test.ts (9 tests passed), targeted oxfmt, and git diff --check. Fresh autoreview completed clean with no accepted/actionable findings (0.99). The repository landing wrapper confirms exact-head hosted CI/Testbox gates passed.

Best-fix verdict: this keeps the existing parser-owned size policy and routes it through the shared UTF-16-safe primitive; no broader parser or error-contract change is needed.

steipete · 2026-07-09T09:38:48Z

Merged via squash.

Prepared head SHA: f07ef572a47e8b87a50b7ea6d481fdd132c19650
Landed commit: bb1fa4012ecffa3ed37f6b08cccf030a5789c6e5

…tion (openclaw#102547) * fix(memory-host): use truncateUtf16Safe for QMD stderr context truncation * test(memory): cover UTF-16-safe qmd stderr summary --------- Co-authored-by: Peter Steinberger <[email protected]>

fix(memory-host): use truncateUtf16Safe for QMD stderr context trunca…

5a1c4b0

…tion

openclaw-barnacle Bot added size: XS triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. labels Jul 9, 2026

test(memory): cover UTF-16-safe qmd stderr summary

f07ef57

openclaw-barnacle Bot removed the triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. label Jul 9, 2026

steipete merged commit bb1fa40 into openclaw:main Jul 9, 2026
105 of 106 checks passed

steipete mentioned this pull request Jul 9, 2026

fix(memory-host-sdk): use truncateUtf16Safe for qmd stderr truncation #102589

Closed

github-actions Bot mentioned this pull request Jul 9, 2026

📡 Upstream Digest — 2026-07-09 10:56 UTC curtismercier/openclaw-mods#1242

Open

OpenClawKobian99 mentioned this pull request Jul 14, 2026

[Bug]: memory-core dreaming narrative fails with "no low surrogate in string" API 400 — clampDiaryContextEntry truncates mid-surrogate-pair (fix on main, missing from v2026.7.1) #107815

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(memory-host): use truncateUtf16Safe for QMD stderr context truncation#102547

fix(memory-host): use truncateUtf16Safe for QMD stderr context truncation#102547
steipete merged 2 commits into
openclaw:mainfrom
lsr911:fix/utf16-qmd-parser

lsr911 commented Jul 9, 2026 •

edited by steipete

Loading

Uh oh!

clawsweeper Bot commented Jul 9, 2026 •

edited

Loading

Uh oh!

steipete commented Jul 9, 2026

Uh oh!

Uh oh!

steipete commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

lsr911 commented Jul 9, 2026 • edited by steipete Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Files Changed

Uh oh!

clawsweeper Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steipete commented Jul 9, 2026

Uh oh!

Uh oh!

steipete commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lsr911 commented Jul 9, 2026 •

edited by steipete

Loading

clawsweeper Bot commented Jul 9, 2026 •

edited

Loading