fix(web-fetch): keep spill content truncation UTF-16 safe by Alix-007 · Pull Request #101312 · openclaw/openclaw

Alix-007 · 2026-07-07T03:11:13Z

Summary

AI-assisted with Codex; I inspected the changed production path and can explain the change.
web_fetch spill-file content truncation now preserves UTF-16 boundaries at WEB_FETCH_SPILL_MAX_CHARS.
The visible spill footer now reports the actual safe-sliced character count when the helper backs off at a surrogate boundary.
Intentionally out of scope: unrelated truncation sites, response-read bounding, config changes, and behavior outside this specific web_fetch spill formatting boundary.

Linked context

No linked issue.
Related to the existing OpenClaw UTF-16-safe truncation hardening pattern.

Real behavior proof (required for external PRs)

Behavior addressed: web_fetch recoverable spill files should not contain malformed text when a response crosses WEB_FETCH_SPILL_MAX_CHARS inside an emoji/surrogate pair.
Real environment tested: local OpenClaw source checkout on Node v22.22.0; PR head 9152389df4c611962d53f9d81937360026a35539.
Exact steps or command run after this patch:
- node scripts/run-vitest.mjs src/agents/tools/web-tools.fetch.test.ts
- Standalone node --import tsx --input-type=module invocation of the actual createWebFetchTool().execute path with a deterministic text response payload.
Evidence after fix:

$ node scripts/run-vitest.mjs src/agents/tools/web-tools.fetch.test.ts
[test] starting test/vitest/vitest.agents.config.ts
RUN  v4.1.9 /tmp/openclaw-utf16-prs/web-fetch-spill
Test Files  1 passed (1)
Tests  28 passed (28)
[test] passed 1 Vitest shard in 16.22s

Standalone tool-execution transcript from the same head, not a Vitest assertion:

$ node --import tsx --input-type=module  # invokes createWebFetchTool().execute
node=v22.22.0
head=9152389df4c611962d53f9d81937360026a35539
tool=web_fetch createWebFetchTool().execute
inputChars=2000005
spilledChars=1999999
spillTruncated=true
footerHasActualCount=true
spillFileHasEmoji=false
spillFileHasLoneSurrogate=false

Observed result after fix: the actual web_fetch tool path spills WEB_FETCH_SPILL_MAX_CHARS - 1 characters for the surrogate-boundary case, the footer says Spilled first 1999999 chars., and the spilled file has no dangling surrogate.
What was not tested: full production deployment and unrelated provider/channel end-to-end delivery were not run; this proof is scoped to the touched web_fetch spill path.
Proof limitations or environment constraints: the standalone proof uses a deterministic local fetch payload to avoid external-network flake or private credentials; the production createWebFetchTool().execute path, spill writer, footer formatter, and spill-file readback were exercised.
Before evidence (optional but encouraged): raw JavaScript string slicing at this cap can cut between a high and low surrogate, leaving malformed text in the recoverable spill file.

Tests and validation

node scripts/run-vitest.mjs src/agents/tools/web-tools.fetch.test.ts
Standalone node --import tsx --input-type=module tool-execution transcript shown above.
Added focused regression coverage for the UTF-16 boundary case and footer count.
No known local failures from this change.

Risk checklist

Did user-visible behavior change? (Yes/No)

Yes. Malformed truncated spill text is now avoided, and the footer reports the actual safe-sliced character count.

Did config, environment, or migration behavior change? (Yes/No)

No.

Did security, auth, secrets, network, or tool execution behavior change? (Yes/No)

No.

What is the highest-risk area?

web_fetch private spill-file content formatting.

How is that risk mitigated?

The patch is limited to the existing truncation boundary, uses the existing UTF-16 helper, preserves the existing cap behavior, and is covered by focused test plus standalone tool execution proof.

Current review state

What is the next action?

ClawSweeper re-review and maintainer review.

What is still waiting on author, maintainer, CI, or external proof?

Nothing is waiting on the author after this update; waiting on CI/ClawSweeper/maintainer review.

Which bot or reviewer comments were addressed?

Addressed ClawSweeper's footer-count finding and needs-proof feedback with a code fix, a focused regression assertion, and standalone web_fetch tool-execution proof from the current PR head.

clawsweeper · 2026-07-07T03:54:01Z

Codex review: needs maintainer review before merge. Reviewed July 7, 2026, 1:27 AM ET / 05:27 UTC.

Summary
The PR makes web_fetch spill-file truncation UTF-16 safe, reports the actual safe-sliced spill count, and adds focused regression coverage.

PR surface: Source +2, Tests +32. Total +34 across 2 files.

Reproducibility: yes. from source inspection: current main raw-slices spill content at WEB_FETCH_SPILL_MAX_CHARS, so an emoji crossing that UTF-16 boundary can leave a dangling surrogate in the recoverable spill file. I did not execute tests in this read-only review.

Review metrics: none identified.

Stored data model
Persistent data-model change detected: serialized state: src/agents/tools/web-tools.fetch.test.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: canonical
Canonical: #101312
Summary: This PR is the canonical remaining fix for UTF-16 safety at the web_fetch recoverable spill-file cap; the merged broader hardening only partially overlaps and does not implement this path.

Members:

partial_overlap: fix(text): keep bounded outputs UTF-16 safe #101355 - Merged broad bounded-output UTF-16 hardening updated sibling truncation sites, including web-fetch-utils, but did not touch spillWebFetchContent in src/agents/tools/web-fetch.ts.
same_root_cause: fix(web-fetch): keep extracted text truncation UTF-16 safe #101297 - Closed unmerged sibling PR targeted UTF-16-safe web_fetch extracted/returned text truncation rather than recoverable spill-file content.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Next step before merge

[P2] No repair job is needed; the contributor branch is narrow, maintainer-editable, sufficiently proven, and has no concrete code findings.

Security
Cleared: Security review cleared: the diff uses an existing internal normalization helper and adds a focused test without changing dependencies, secrets, permissions, CI, package metadata, or network policy.

Review details

Best possible solution:

Land this narrow spill-path fix after exact-head CI and maintainer review; keep broader bounded-output UTF-16 hardening tracked through the already-merged related PR.

Do we have a high-confidence way to reproduce the issue?

Yes from source inspection: current main raw-slices spill content at WEB_FETCH_SPILL_MAX_CHARS, so an emoji crossing that UTF-16 boundary can leave a dangling surrogate in the recoverable spill file. I did not execute tests in this read-only review.

Is this the best way to solve the issue?

Yes: applying truncateUtf16Safe at spillWebFetchContent is the narrow owner-boundary fix because model-visible truncateText is already hardened separately and the spill-file cap remains a distinct path.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 53580e13a483.

Label changes

Label justifications:

P2: This is a normal-priority correctness fix for malformed recoverable web_fetch spill content with limited blast radius and no config, migration, or provider-routing change.
rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body includes after-fix terminal proof from the actual createWebFetchTool().execute path plus focused Vitest output for the changed spill boundary.
proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes after-fix terminal proof from the actual createWebFetchTool().execute path plus focused Vitest output for the changed spill boundary.

Evidence reviewed

PR surface:

Source +2, Tests +32. Total +34 across 2 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	1	5	3	+2
Tests	1	32	0	+32
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	2	37	3	+34

What I checked:

Current main still has the bug surface: spillWebFetchContent on current main slices recoverable spill content with value.slice(0, WEB_FETCH_SPILL_MAX_CHARS), so a surrogate pair crossing that cap can leave a dangling high surrogate in the spill file. (src/agents/tools/web-fetch.ts:317, 53580e13a483)
Existing helper is the right primitive: truncateUtf16Safe floors the limit and delegates to sliceUtf16Safe, which backs off when the end would split a surrogate pair. (packages/normalization-core/src/utf16-slice.ts:44, 53580e13a483)
PR applies the helper at the spill cap: The PR replaces the raw spill-file slice with truncateUtf16Safe and derives spilledChars from the actual content length, fixing the footer-count mismatch when the safe slice backs off. (src/agents/tools/web-fetch.ts:315, 9152389df4c6)
Regression coverage targets the boundary: The added test creates a payload where an emoji starts at WEB_FETCH_SPILL_MAX_CHARS - 1 and asserts the spill count, footer, and file content do not include a dangling high surrogate. (src/agents/tools/web-tools.fetch.test.ts:503, 9152389df4c6)
Related merged hardening is only partial overlap: The merged UTF-16 hardening in fix(text): keep bounded outputs UTF-16 safe #101355 updated web-fetch-utils truncateText and other bounded-output sites, but its file list did not include src/agents/tools/web-fetch.ts spillWebFetchContent. (src/agents/tools/web-fetch-utils.ts:126, 84e5327720b4)
Proof in PR body exercises the production tool path: The PR body includes a focused test run and a standalone createWebFetchTool().execute transcript showing spilledChars=1999999, footerHasActualCount=true, and spillFileHasLoneSurrogate=false on head 9152389. (9152389df4c6)

Likely related people:

obviyus: GitHub commit history and associated PR metadata show the recoverable web_fetch spill-file path and raw WEB_FETCH_SPILL_MAX_CHARS slice were introduced in the merged spill-output change. (role: introduced current spill path; confidence: high; commits: 235f18def56b; files: src/agents/tools/web-fetch.ts, src/agents/tools/web-tools.fetch.test.ts)
vincentkoc: Authored and merged the related bounded-output UTF-16 hardening that moved web-fetch-utils truncateText to truncateUtf16Safe but did not cover this spill path. (role: recent adjacent hardening owner; confidence: medium; commits: 84e5327720b4; files: src/agents/tools/web-fetch-utils.ts, src/agents/tools/web-fetch-utils.test.ts, packages/normalization-core/src/utf16-slice.ts)
steipete: Recent GitHub path history shows repeated web_fetch work around max chars, response bounding, proxy/fallback behavior, and agent tool tests near this surface. (role: web_fetch feature-history owner; confidence: medium; commits: 44dc29f397af, aff6d079d35b, 66336bf7c846; files: src/agents/tools/web-fetch.ts, src/agents/tools/web-tools.fetch.test.ts)
Alix-007: Besides authoring this PR, Alix-007 is credited as co-author on the merged broader UTF-16 hardening, so they have adjacent context beyond only opening this branch. (role: recent adjacent contributor; confidence: low; commits: 84e5327720b4; files: src/agents/tools/web-fetch-utils.ts, src/agents/tools/web-fetch-utils.test.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Alix-007 · 2026-07-07T05:02:08Z

@clawsweeper re-review

I addressed the spill footer count finding and refreshed the real behavior proof with a standalone createWebFetchTool().execute transcript from the current head.

clawsweeper · 2026-07-07T05:02:11Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Alix-007 · 2026-07-07T05:18:15Z

@clawsweeper re-review

Current head/proof body was refreshed after the last rebase; please re-review the latest head.

clawsweeper · 2026-07-07T05:18:18Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

steipete · 2026-07-07T09:56:04Z

Maintainer review complete on exact head 9152389df4c611962d53f9d81937360026a35539.

Confirmed the real createWebFetchTool().execute spill path now uses the shared UTF-16-safe helper and reports the actual written code-unit count.
Regression reads back the private spill artifact at the split-surrogate boundary and verifies metadata/footer behavior.
Exact-head hosted CI, Real behavior proof, git diff --check, and fresh Codex autoreview all pass.
No config, dependency, auth, or public API surface changed.

Ready to land.

steipete · 2026-07-07T09:56:18Z

Merged via squash.

Prepared head SHA: 9152389df4c611962d53f9d81937360026a35539
Landed commit: 240487179b69724a3e30909704844ae8337f7841

…01312) * Fix UTF-16-safe web fetch spill truncation * fix(web-fetch): report actual spill character count

openclaw-barnacle Bot added agents Agent runtime and tooling size: XS labels Jul 7, 2026

clawsweeper Bot mentioned this pull request Jul 7, 2026

fix(web-fetch): keep extracted text truncation UTF-16 safe #101297

Closed

Alix-007 force-pushed the alix/utf16-web-fetch-spill branch 4 times, most recently from 328bb56 to 8087092 Compare July 7, 2026 05:00

Alix-007 added 2 commits July 7, 2026 13:14

Fix UTF-16-safe web fetch spill truncation

6c8a1ea

fix(web-fetch): report actual spill character count

9152389

Alix-007 force-pushed the alix/utf16-web-fetch-spill branch from 8087092 to 9152389 Compare July 7, 2026 05:15

steipete self-assigned this Jul 7, 2026

steipete merged commit 2404871 into openclaw:main Jul 7, 2026
132 of 135 checks passed

github-actions Bot mentioned this pull request Jul 7, 2026

📡 Upstream Digest — 2026-07-07 10:48 UTC curtismercier/openclaw-mods#1229

Open

This was referenced Jul 7, 2026

fix(infra): bound response body read in non-streaming fallback paths #101313

Closed

[Bug]: web_fetch reports exact-size responses as truncated #101837

Closed

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 8, 2026

fix(web-fetch): keep spill content truncation UTF-16 safe (openclaw#1…

e5824b8

…01312) * Fix UTF-16-safe web fetch spill truncation * fix(web-fetch): report actual spill character count

This was referenced Jul 8, 2026

fix: don't mark exact-size web_fetch responses as truncated (#101837) #102111

Closed

fix: web_fetch reports exact-size responses as truncated #102233

Closed

giodl73-repo pushed a commit to giodl73-repo/openclaw that referenced this pull request Jul 8, 2026

fix(web-fetch): keep spill content truncation UTF-16 safe (openclaw#1…

9aea267

…01312) * Fix UTF-16-safe web fetch spill truncation * fix(web-fetch): report actual spill character count

This was referenced Jul 9, 2026

fix(agents): keep truncation surrogate-safe #102332

Merged

fix(web_fetch): exact-size responses no longer reported as truncated #102335

Closed

fix(web-fetch): report exact-limit response bodies as complete, not truncated [AI-assisted] #102389

Merged

wangmiao0668000666 mentioned this pull request Jul 9, 2026

fix(matrix): keep HTTP error and tool-progress truncations UTF-16 safe #102395

Merged

This was referenced Jul 9, 2026

fix: don't mark exact-size responses as truncated in readResponseText #102404

Closed

fix(web-shared): treat exact-limit response reads as complete #102103

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(web-fetch): keep spill content truncation UTF-16 safe#101312

fix(web-fetch): keep spill content truncation UTF-16 safe#101312
steipete merged 2 commits into
openclaw:mainfrom
Alix-007:alix/utf16-web-fetch-spill

Alix-007 commented Jul 7, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jul 7, 2026 •

edited

Loading

Uh oh!

Alix-007 commented Jul 7, 2026

Uh oh!

clawsweeper Bot commented Jul 7, 2026 •

edited

Loading

Uh oh!

Alix-007 commented Jul 7, 2026

Uh oh!

clawsweeper Bot commented Jul 7, 2026 •

edited

Loading

Uh oh!

steipete commented Jul 7, 2026

Uh oh!

Uh oh!

steipete commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

Alix-007 commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Linked context

Real behavior proof (required for external PRs)

Tests and validation

Risk checklist

Current review state

Uh oh!

clawsweeper Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Alix-007 commented Jul 7, 2026

Uh oh!

clawsweeper Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Alix-007 commented Jul 7, 2026

Uh oh!

clawsweeper Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steipete commented Jul 7, 2026

Uh oh!

Uh oh!

steipete commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Alix-007 commented Jul 7, 2026 •

edited

Loading

clawsweeper Bot commented Jul 7, 2026 •

edited

Loading

clawsweeper Bot commented Jul 7, 2026 •

edited

Loading

clawsweeper Bot commented Jul 7, 2026 •

edited

Loading