Skip to content

fix(agents): keep chunkString and buildResumeMessage truncation UTF-16 safe#102085

Merged
steipete merged 2 commits into
openclaw:mainfrom
wm0018:fix/chunk-resume-utf16-safe-truncation
Jul 9, 2026
Merged

fix(agents): keep chunkString and buildResumeMessage truncation UTF-16 safe#102085
steipete merged 2 commits into
openclaw:mainfrom
wm0018:fix/chunk-resume-utf16-safe-truncation

Conversation

@wm0018

@wm0018 wm0018 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

What Problem This Solves

Two agent-owned prefix cuts could split UTF-16 surrogate pairs: streamed bash output chunking and truncated task text in orphan-recovery resume messages. The initial chunking fix also dropped a whole emoji when the configured limit was one UTF-16 unit.

Why This Change Was Made

  • Make chunkString choose a surrogate-safe boundary while always emitting at least one whole code point.
  • Permit a two-unit code point to exceed a one-unit chunk limit because it is indivisible.
  • Use the shared UTF-16-safe truncation helper in orphan recovery.
  • Assert exact chunk arrays and exercise resume-message truncation through the real gateway recovery path.
  • Remove the helper-only resume test.

User Impact

Agent output chunking no longer corrupts or loses emoji at transport boundaries. Restarted subagents receive valid Unicode task prefixes in synthetic resume messages.

Evidence

AI-assisted: Yes

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: S labels Jul 8, 2026
@clawsweeper

clawsweeper Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs changes before merge. Reviewed July 8, 2026, 8:58 PM ET / 00:58 UTC.

Summary
The branch replaces raw slices in bash output chunking and subagent resume task truncation with UTF-16-safe helpers and adds focused regression tests.

PR surface: Source +11, Tests +70. Total +81 across 4 files.

Reproducibility: yes. source-reproducible. Current main uses raw code-unit slicing at both reported boundaries, and the current PR head still exposes a focused chunkString limit-1 data-loss case.

Review metrics: none identified.

Stored data model
Persistent data-model change detected: migration/backfill/repair: src/agents/task-resume-utf16-safe.test.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🦞 diamond lobster
Patch quality: 🦐 gold shrimp
Result: needs maintainer review before merge.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P2] Fix the tiny positive-limit chunkString case and add an assertion such as chunkString("🚀x", 1).join("") === "🚀x".

Risk before merge

  • [P1] The exported chunkString helper can still lose text when called with a tiny positive limit, even though the default production chunk size path is improved.

Maintainer options:

  1. Decide the mitigation before merge
    Keep the UTF-16 helper approach, but make chunkString preserve all input for every positive limit it accepts or explicitly reject/clamp unsupported tiny limits, with a limit-1 regression test.
  2. Pause or close
    Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

  • [P2] A narrow mechanical repair remains on the PR branch for chunkString’s empty-safe-slice path.

Security
Cleared: No security or supply-chain concern found; the diff is limited to TypeScript text handling and focused tests.

Review findings

  • [P3] Preserve tiny-limit surrogate chunks — src/agents/bash-tools.shared.ts:145-146
Review details

Best possible solution:

Keep the UTF-16 helper approach, but make chunkString preserve all input for every positive limit it accepts or explicitly reject/clamp unsupported tiny limits, with a limit-1 regression test.

Do we have a high-confidence way to reproduce the issue?

Yes, source-reproducible. Current main uses raw code-unit slicing at both reported boundaries, and the current PR head still exposes a focused chunkString limit-1 data-loss case.

Is this the best way to solve the issue?

No, not fully as submitted. Reusing the existing UTF-16 helpers is the right owner-boundary fix, but the empty-safe-slice branch should preserve the full code point or reject/clamp tiny limits instead of skipping input.

Full review comments:

  • [P3] Preserve tiny-limit surrogate chunks — src/agents/bash-tools.shared.ts:145-146
    The current head still advances past an empty safe slice without emitting anything. For chunkString("🚀x", 1), sliceUtf16Safe(input, 0, 1) returns "", this branch increments i by 2, and the joined chunks become "x", so the exported helper drops user output for a positive limit. Please emit the full code point, clamp/reject unsupported tiny limits, or otherwise preserve input.
    Confidence: 0.86

Overall correctness: patch is correct
Overall confidence: 0.84

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against eae3fc1a79dd.

Label changes

Label justifications:

  • P2: This is a focused agent-runtime correctness fix for output and resume-message text handling with limited blast radius.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦞 diamond lobster and patch quality is 🦐 gold shrimp.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body includes terminal output from a standalone production-path proof plus focused tests showing the before/after UTF-16 boundary behavior.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes terminal output from a standalone production-path proof plus focused tests showing the before/after UTF-16 boundary behavior.
Evidence reviewed

PR surface:

Source +11, Tests +70. Total +81 across 4 files.

View PR surface stats
Area Files Added Removed Net
Source 2 15 4 +11
Tests 2 71 1 +70
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 4 86 5 +81

Acceptance criteria:

  • [P1] node scripts/run-vitest.mjs src/agents/bash-tools.shared.test.ts src/agents/task-resume-utf16-safe.test.ts src/agents/subagent-orphan-recovery.test.ts.

What I checked:

  • Current main bash chunking: Current main chunks sanitized bash stdout/stderr through chunkString, and chunkString still uses raw input.slice(i, i + limit), which can split a surrogate pair at the fixed transport boundary. (src/agents/bash-tools.shared.ts:140, eae3fc1a79dd)
  • Current main resume truncation: Current main builds the orphaned-subagent resume task text with task.slice(0, maxTaskLen), so a task ending with a split surrogate pair at the 2000-code-unit boundary can produce a replacement character. (src/agents/subagent-orphan-recovery.ts:75, eae3fc1a79dd)
  • UTF-16 helper contract: sliceUtf16Safe backs out a slice endpoint that would expose only part of a surrogate pair, and existing tests assert sliceUtf16Safe(input, 0, 1) returns an empty string for a leading emoji pair. (packages/normalization-core/src/utf16-slice.ts:33, eae3fc1a79dd)
  • PR head remaining defect: At PR head, an empty safe slice with a positive tiny limit advances i without pushing the skipped code point, so chunkString("🚀x", 1).join("") would lose the leading emoji. (src/agents/bash-tools.shared.ts:145, 99b58c454ea7)
  • PR test gap: The added chunkString tests cover the default 8192-code-unit boundary and a limit-2 case, but not the limit-1 empty-safe-slice case that still drops data. (src/agents/bash-tools.shared.test.ts:97, 99b58c454ea7)
  • Contributor proof: The PR body includes focused Vitest output and terminal production-path proof for the default chunk and resume-message UTF-16 boundary behavior; the live PR also carries the proof: sufficient label. (99b58c454ea7)

Likely related people:

  • vincentkoc: Blame and PR metadata show the current affected helper files and normalization helper locations were localized by PR refactor(line): localize internal declarations #102037, authored and merged by this account. (role: recent area contributor; confidence: high; commits: 63192752325b; files: src/agents/bash-tools.shared.ts, src/agents/subagent-orphan-recovery.ts, packages/normalization-core/src/utf16-slice.ts)
  • steipete: History shows this account merged the original UTF-16 truncation fix and recently refactored adjacent bash exec helper code in PR refactor: consolidate markdown code fences, error coercion, and byte-identical helper pairs #99932. (role: adjacent owner; confidence: medium; commits: 63f5fa47deb6, 49cc59b1e88e; files: src/utils.ts, src/agents/bash-tools.exec-runtime.ts)
  • amittell: Recent merged work on preserving subagent task text during restart redispatch touched the same orphan-recovery flow adjacent to buildResumeMessage. (role: recent area contributor; confidence: medium; commits: 7b5d86e89a5f; files: src/agents/subagent-orphan-recovery.ts, src/agents/subagent-registry-run-manager.ts)
  • jalehman: Recent transcript-reader refactor work touched the subagent recovery flow around last-message extraction and resume context. (role: adjacent contributor; confidence: medium; commits: d216f7c876dd; files: src/agents/subagent-orphan-recovery.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (2 earlier review cycles)
  • reviewed 2026-07-08T07:24:24.458Z sha 52d7059 :: needs changes before merge. :: [P3] Preserve tiny-limit surrogate chunks
  • reviewed 2026-07-09T00:46:05.896Z sha 99b58c4 :: needs changes before merge. :: [P3] Preserve tiny-limit surrogate chunks

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P2 Normal backlog priority with limited blast radius. labels Jul 8, 2026
@wm0018
wm0018 force-pushed the fix/chunk-resume-utf16-safe-truncation branch from 52d7059 to 99b58c4 Compare July 9, 2026 00:40
@steipete

steipete commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Land-ready proof for exact head 9bab14415a285a63799fdd30eb60a3c9d4773f32:

  • Improved the submitted chunker so an astral code point larger than a one-unit limit is emitted whole instead of being skipped; every iteration now preserves input and makes progress.
  • Kept the recovery-message fix on the production buildResumeMessage path and replaced helper-only coverage with an exact gateway-delivery assertion.
  • Blacksmith Testbox run 28994579755: 31/31 focused tests passed across bash-tools.shared.test.ts and subagent-orphan-recovery.test.ts.
  • Exact-head CI run 28994671278: passed.
  • git diff --check: passed.
  • Fresh structured autoreview: clean, no accepted/actionable findings, confidence 0.97.

The reviewed implementation is the best bounded fix: one canonical chunk path, no data loss, no introduced surrogate halves, and owner-path regression coverage.

@steipete
steipete merged commit 9b4cbe4 into openclaw:main Jul 9, 2026
110 of 114 checks passed
@steipete

steipete commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Merged via squash.

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 9, 2026
…6 safe (openclaw#102085)

* fix(agents): keep chunkString and buildResumeMessage truncation UTF-16 safe

* fix: preserve code points in agent output chunks

---------

Co-authored-by: Peter Steinberger <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. size: XS status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants