fix(boundary-safety): guard response reads and UTF-16 truncation#97774
fix(boundary-safety): guard response reads and UTF-16 truncation#97774mushuiyu886 wants to merge 15 commits into
Conversation
|
Codex review: needs maintainer review before merge. Reviewed July 7, 2026, 5:47 AM ET / 09:47 UTC. Summary PR surface: Source +52, Tests +2003, Config +2, Other +631. Total +2688 across 18 files. Reproducibility: yes. the source path is reproducible from current main and the PR’s targeted tests: current helpers show raw slice and older snippet semantics, while the PR tests exercise exact byte caps, UTF-8 boundaries, and guard detection. I did not execute tests in this read-only review. Review metrics: 1 noteworthy metric.
Stored data model Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Risk before merge
Maintainer options:
Next step before merge
Maintainer decision needed
Security Review detailsBest possible solution: Land the runtime helper fixes and validator only after a maintainer explicitly accepts the baseline-backed boundary-safety guard as repository CI policy. Do we have a high-confidence way to reproduce the issue? Yes, the source path is reproducible from current main and the PR’s targeted tests: current helpers show raw slice and older snippet semantics, while the PR tests exercise exact byte caps, UTF-8 boundaries, and guard detection. I did not execute tests in this read-only review. Is this the best way to solve the issue? Mostly yes for the shared helper fixes; the validator is the right centralized shape if maintainers want this invariant enforced in CI. The only unresolved question is policy acceptance of the baseline-backed gate. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 6125dd95e682. Label changesLabel changes:
Label justifications:
Evidence reviewedPR surface: Source +52, Tests +2003, Config +2, Other +631. Total +2688 across 18 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
Review history (12 earlier review cycles; latest 8 shown)
|
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
b519d34 to
af0b243
Compare
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
b66a540 to
2916324
Compare
|
Maintainer decision: closing this broad guard proposal. The useful runtime direction is already being handled through focused owner-path fixes and the shared UTF-16 / bounded-response helpers. This PR instead couples those fixes to a new required repository policy: a 628-line name-heuristic AST scanner, a 1,472-line baseline containing 210 known exceptions, and new full/changed-check gates, plus unrelated ACP and HTTP runtime changes. That creates substantial false-positive, baseline-drift, and contributor-CI maintenance cost for a heuristic that cannot prove the underlying string/Response types. Supported alternative: keep landing narrow caller-path fixes with regression tests, and keep response-read changes in their owning helper PRs. If a repository-wide guard is proposed again, it should be a separate maintainer-owned policy change, advisory first, with a typed/high-confidence rule set and without bundling runtime behavior changes. Thank you for the extensive investigation and test work, @mushuiyu886. The technical evidence was useful even though we are not adopting this CI gate. |
Summary
Responsebody reads, cloned/aliased responses, and full-bufferblob()/formData()reads.boundary-safety-ignorecomments with reasons.Responseread andreadResponseTextSnippet(...)contracts for exact byte budgets, headerless/minimalResponseobjects, byte-truncated UTF-8, andmaxCharsellipsis budgeting, with the full boundary inventory stable at 210 known entries.What Problem This Solves
OpenClaw already had canonical helpers for UTF-16-safe text truncation and bounded response reads, but the repository invariant was incomplete: new production code could still add raw head truncation or direct unbounded
Responsebody reads in source shapes the original guard missed.This update keeps the guardrail value while lowering reviewer noise. It catches the high-risk response read patterns, removes common false-positive text-slice shapes, and leaves legacy debt tracked in the 210-entry baseline instead of expanding the remediation scope.
User Impact
Future runtime/channel/model-adjacent changes are less likely to reintroduce corrupted surrogate-pair snippets or unbounded
Responsebody buffering. Diagnostic snippets are also more accurate: exactContent-Lengthbodies at the byte cap no longer get a false ellipsis, headerless/minimalResponseobjects still enforce overflow handling, byte-truncated UTF-8 no longer emits�, and the ellipsis stays inside the configured character budget.Origin / follow-up
Follow-up to #97620. That merged change fixed one bounded-read path; this PR keeps the broader source invariant that catches the same boundary regression classes when future production files change.
Competition / linked PR analysis
This is an update to the existing #97774 branch, not a replacement PR. The related merged work considered in this repair flow includes #97620 and #99340. Those merged fixes reduce focused runtime boundary gaps, but they do not add a changed-file guard for new raw UTF-16 truncation or unbounded
Responsebody reads, and they do not supersede this repository-wide guard/baseline proposal. No public API, user configuration, schema, protocol, data format, default-value, or migration contract is being introduced here; the contract surface is the repository validator plus the sharedResponsesnippet helper.Maintainer-ready notes
Responsebody reads and raw UTF-16 text truncation while reducing known false-positive classes. The boundedResponsehelpers now preserve exact byte-budget snippets without a false ellipsis, keep headerless/minimalResponseobjects on the overflow-enforced path, drop incomplete UTF-8 tails instead of emitting�, and count the ellipsis withinmaxChars.Responsehelper.Responsesnippet semantics covered by focused tests, priorcheck:changed, and latest diff hygiene. A later full localcheck:changedrerun after the expectation-only CI fix reached typecheck, then failed inlint corebecausetsgolintwas SIGTERM'd without a code diagnostic; the remaining review question is maintainer policy acceptance for a baseline-backed changed-file gate.Responsereads.scripts/check-boundary-safety.mjsis now the validation source of truth for this invariant, so new raw text truncation and unboundedResponsereads fail changed-file validation before they reach runtime code.src/infra/http-body.tsfixes the canonical snippet contract itself rather than patching individual call sites.Responsesnippet semantics. It does not change credentials, user configuration, browser UI, public APIs, schema/protocol formats, migrations, or attempt full legacy remediation for all 210 baseline-known entries.scripts/check-boundary-safety.mjsis the validator andtest/fixtures/boundary-safety-inventory.jsonis the baseline contract.src/infra/http-body.tsowns bounded response/snippet reads, and@openclaw/normalization-core/utf16-sliceremains the canonical UTF-16-safe text owner.Responsepath keeps a narrow comment-only ignore because maxBytes enforcement happens immediately after the only available read path.test/scripts/check-boundary-safety.test.ts;src/infra/http-body.response.test.ts;src/agents/runtime/proxy.test.ts;src/agents/anthropic-transport-stream.test.ts.Responsereads are flagged; collection/range/suffix/byte slices are not treated as user-visible text truncation; spoofed ignore strings do not suppress violations; bounded response reads handle headerless/minimalResponseobjects; snippet output handles exact byte caps plus UTF-8/max-char boundaries correctly.Real behavior proof
Responsebody reads and raw UTF-16 text truncation while reducing known false-positive classes. The boundedResponsehelpers now preserve exact byte-budget snippets without a false ellipsis, keep headerless/minimalResponseobjects on the overflow-enforced path, drop incomplete UTF-8 tails instead of emitting�, and count the ellipsis withinmaxChars.pnpm check:changed/node scripts/check-boundary-safety.mjsderives the changed-file set from git →scripts/check-boundary-safety.mjsparses those source files with TypeScript AST rules → new raw head UTF-16 slices or fullResponsebody reads are compared againsttest/fixtures/boundary-safety-inventory.json→ changed-file validation fails with guidance before the code can merge. Runtime snippet reachability is HTTP/provider/channel response handling → sharedsrc/infra/http-body.tsbounded reader → diagnostic/error snippet text returned to callers.Responsestream boundaries, including unknown/headerless responses, exactContent-Length, clone/alias response reads, byte-truncated UTF-8, and stalled/overflowing body streams.truncateUtf16Safe,sliceUtf16Safe,readResponseWithLimit,readResponseTextSnippet, andreadProviderJsonResponse) instead of adding provider-specific policy. No provider credentials, routing, schema, model selection, or channel protocol constraint changed./media/vdb/code/ai/aispace/openclaw-worktrees/pr-97774, branchrepair/pr-97774, afterdaily_fix.sh sync-mainonto currentorigin/main. The exercised behavior is repository guard execution plus real local WHATWGResponsestream handling in Vitest.Responsereads, response clones/aliases,blob()/formData(), generic display truncation, comment-only ignores, and false-positive boundaries. Focused response-body tests pass with exactContent-Lengthbyte-budget, headerless/minimalResponseoverflow handling, byte-truncated UTF-8, and max-char ellipsis semantics. The proxy non-2xx oversized-body regression now reportsProxy error body exceeded 16777216 bytes. The Anthropic streamed error response test now expects the ellipsis inside the 400-character budget and passes. The ACP cancellation tests requested by ClawSweeper pass. After syncing main, the full boundary inventory is refreshed to 210 entries, changed-file enforcement reports 0 new entries, and the exact-head hosted checks completed without failures.tool-searchabort-signal failure was checked with its targeted local test and passed locally; this PR does not claim that local result as a hosted CI pass. Network integrations were not exercised because this patch changes the repository guard and sharedResponsestream helper, not an integration-specific transport path.Review findings addressed
gh pr checks 97774 --repo openclaw/openclaw --watchexited 0; key checks includingchecks-node-compact-large-2,check-lint,check-guards,check-prod-types,check-test-types,Real behavior proof,Security High (process-exec-boundary), andCritical Quality (agent-runtime-boundary)passed.cancelledcompatibility: Addressed by preserving the ACPcancelledterminal status compatibility in the repaired branch and verifying the relevant ACP manager tests.daily_fix.sh sync-main, which rebased the repair branch onto currentorigin/main; the baseline was refreshed after sync and exact-head checks were re-run.src/acp/control-plane/manager.background-task.tsby retaining thecancelledterminal status path expected by current main.pnpm test src/acp/control-plane/manager.cancel-session.test.ts src/acp/control-plane/manager.turn-results.test.tspassed after sync.pnpm check:changed: Addressed through prior localcheck:changedsuccess, post-sync targeted validation, and exact-head hosted checks. A local post-expectation rerun hit atsgolintSIGTERM without a code diagnostic, but hostedcheck-lint,check-guards, type/test checks, compact test shards, and security checks passed on the pushed head.src/agents/anthropic-transport-stream.test.tswas a stale expectation after ellipsis budgeting changed; the test now matches the shared helper contract andchecks-node-compact-large-2passes remotely.Regression Test Plan
test/scripts/check-boundary-safety.test.ts;src/infra/http-body.response.test.ts;src/agents/runtime/proxy.test.ts;src/agents/anthropic-transport-stream.test.ts.Responsereads, generic display truncation, actual-comment ignores, spoofed ignore strings, and collection/range/suffix/byte false-positive boundaries. Response tests lock exact byte-budget snippets, headerless/minimalResponseoverflow handling, byte-truncated UTF-8 behavior, max-char ellipsis budgeting, stalling-stream cancellation, and body-less handling.test/fixtures/boundary-safety-inventory.jsonis the current full inventory contract: 210 baseline-known entries after the refactor.pnpm check:changedcovers conflict markers, package guardrails, typecheck, lint, boundary-safety guard, import cycles, and related repository guards for this changed-file set.Merge risk
Responsebody reads. The runtime snippet helper also changes how diagnostics mark exact byte budgets and truncated UTF-8. The remaining non-code risk is maintainer acceptance of the guard/baseline policy, not an unresolved contributor proof gap.truncateUtf16Safe,sliceUtf16Safe,readResponseWithLimit,readResponseTextSnippet,readProviderJsonResponse, or a narrow comment-only ignore with a reason. Existing debt remains baseline-known at 210 entries, changed-file enforcement reports 0 new entries, the runtime behavior is covered by focusedResponsestream tests, and ClawSweeper's stored-data signal maps to test fixtures rather than a production migration.Risk / Compatibility
Responsereads. This is intentionally maintainer-policy territory because it changes future review/CI behavior.readResponseTextSnippet(...)still cancels stalling exact-budget streams when size is not proven. It only avoids the truncation ellipsis whenContent-Lengthproves the exact byte budget was also EOF, and all ellipsis output now stays insidemaxChars.What was not changed
CHANGELOG.mdchanges were made.Root Cause
Responsebody reads even though canonical helpers existed. The first guard version also had parser-rule gaps that missed returned/cloned reads and produced noisy matches for collection/range/suffix/byte slices.scripts/check-boundary-safety.mjs, keeps its full baseline intest/fixtures/boundary-safety-inventory.json, and fixessrc/infra/http-body.tswhere the shared snippet contract is owned. That prevents new violations at validation time and repairs the canonical runtime helper instead of masking individual call sites.