fix(infra): bound ClawHub fetchJson and error response bodies by Alix-007 · Pull Request #95226 · openclaw/openclaw

Alix-007 · 2026-06-20T04:40:15Z

Summary

ClawHub is an external marketplace and therefore an untrusted source. Every
ClawHub response reader in src/infra/clawhub.ts previously consumed the body
without a byte cap, so a hostile or malfunctioning ClawHub host could exhaust
process memory by streaming an unbounded response:

fetchJson (shared by search/detail/version/artifact-resolver/verify/security
metadata) called await response.json() — the entire success body was
buffered before parsing, with no upper bound.
readErrorBody called await response.text() — the entire error body was
buffered into the diagnostic/error message, with no upper bound.
fetchClawHubSkillInstallResolution called a bare await response.json() for
both the OK and the structured 403/409/410/423 install bodies — the one
remaining uncapped ClawHub JSON reader.

This is the symmetric, harder-justified counterpart to the error-stream
hardening landed in #95108 (which bounded the Anthropic Messages error stream):
here the source is a third-party marketplace, not first-party transport.

Changes

fetchJson now delegates to a small parseClawHubJsonBody helper that reads
the success body through the existing readResponseWithLimit helper under a
16 MiB cap (CLAWHUB_JSON_MAX_BYTES), then decodes and JSON.parses the
bounded buffer. Overflow and idle-timeout cancel the stream (releasing the
socket/buffer) and throw a descriptive error; malformed JSON keeps throwing
the existing returned malformed JSON error.
readErrorBody now uses readResponseTextSnippet (the same helper fix(agents): bound Anthropic error streams #95108
used) under an 8 KiB byte cap and 400-char display cap, collapsing whitespace
and truncating with …. The statusText / HTTP <status> fallback chain is
preserved.
fetchClawHubSkillInstallResolution now reuses the same
parseClawHubJsonBody helper instead of its own bare response.json(), so
every ClawHub JSON success/structured-block body is bounded by the identical
cap. This is pure reuse of the helper introduced in this PR — no new
abstraction — and closes the last uncapped ClawHub JSON path that the review
flagged.
Regression tests cover: an oversized 200 JSON body via searchClawHubSkills
(asserts the cap error + stream cancellation), an oversized error body
(asserts the message is collapsed/truncated and stays far below the raw body),
and an oversized install-resolution body via
fetchClawHubSkillInstallResolution (asserts the cap error + cancellation).

Out of scope (intentionally focused): the MAX_SAFE_INTEGER default on
readClawHubResponseBytes for artifact/archive downloads — those are length-
checked binary fetches with their own integrity handling, not in-memory JSON
parses, and belong in a separate change.

Why this is worth merging

Untrusted-source memory-exhaustion vector. ClawHub is an external
marketplace; an unbounded response.json()/response.text() lets a hostile or
malfunctioning host stream an arbitrarily large body straight into process
memory. The narrowest maintainable fix is to route reads through the existing
response-limit helper, which is exactly what this does.
Complete, not scoped. A prior review noted one sibling ClawHub JSON path
(fetchClawHubSkillInstallResolution) was still uncapped; it is now bounded
through the same helper, so there is no remaining direct response.json() /
response.text() on any ClawHub response path.

How the `merge-risk: availability` is bounded

The label flags that this changes response-body reading for live ClawHub
requests. The change is purely additive on the success path: the bounded
reader streams chunks and only diverges from prior behaviour when a body exceeds
16 MiB (JSON) / 8 KiB (error snippet). Real ClawHub responses are kilobytes
(a live ?q=calendar&limit=5 search is ~2 KB), orders of magnitude under the
cap, so normal traffic parses byte-for-byte as before — verified live below.
Beyond the cap, the previous behaviour was unbounded buffering (the actual
availability/OOM risk); the new behaviour is a fast, descriptive failure with
the stream cancelled. So the cap narrows an availability risk rather than adding
one.

Real behavior proof

A local tsx harness (proof.mts) drove the real exported
searchClawHubSkills and fetchClawHubSkillInstallResolution (no mocks of the
code under test). For injected cases it streams an oversized body through a
ReadableStream whose cancel() hook is observed, so a cancelled: true
readback proves the production code released the stream instead of draining it.
For the normal path it hits the live clawhub.ai marketplace over the network.

Behavior addressed: unbounded buffering of untrusted ClawHub JSON and
error response bodies (memory-exhaustion / availability vector).
Real environment tested: Node v22.22.0, tsx, against the patched
src/infra/clawhub.ts; vitest via node scripts/run-vitest.mjs.
Exact steps / commands run after this patch:
- ./node_modules/.bin/tsx proof.mts
- node scripts/run-vitest.mjs run --config test/vitest/vitest.infra.config.ts src/infra/clawhub.test.ts
Evidence after fix:
- LIVE normal path (real network, searchClawHubSkills →
  live https://clawhub.ai/api/v1/search?q=calendar): HTTP 200,
  5 skill(s) parsed through the bounded parseClawHubJsonBody
  (first.slug = calendar, displayName present). Production JSON parses
  correctly after response.json() was replaced by the bounded reader.
- fetchJson cap (injected 17 MiB): threw
  ClawHub /api/v1/search response exceeded 16777216 bytes (17301504 bytes received),
  stream cancelled: true.
- install-resolution cap (injected 17 MiB, sibling path): threw
  ClawHub /api/v1/skills/weather/install response exceeded 16777216 bytes (17301504 bytes received),
  stream cancelled: true.
- vitest: Test Files 1 passed (1), Tests 55 passed | 2 skipped (57)
  (the 2 skips are pre-existing live-network-gated tests).
Observed result after fix: live production JSON parses unchanged; oversized
bodies on both the shared and the install-resolution JSON paths stop at the
cap and cancel the underlying stream.
Negative control: the harness re-runs the pre-fix behaviour — an
unbounded Response.arrayBuffer() drain of the same 17 MiB stream. It reads
the full 17825792 bytes (~17 MiB), stream cancelled: false, and never
stops early. This confirms the cap is load-bearing: without it the stream is
fully buffered into memory, with it the reader cancels at 16 MiB.
What was not tested: a live oversized ClawHub response (the live host
returns small bodies; oversized behaviour is proven with injected streams that
drive the same production reader), and the out-of-scope artifact-download
default cap noted above.

AI-assisted (human-run real behavior proof included).

clawsweeper · 2026-06-20T04:41:28Z

Codex review: needs maintainer review before merge. Reviewed June 22, 2026, 5:28 PM ET / 21:28 UTC.

Summary
The PR replaces direct ClawHub metadata/error response reads with bounded media-core readers and adds oversized-response cancellation/truncation tests.

PR surface: Source +22, Tests +89. Total +111 across 2 files.

Reproducibility: yes. Source inspection shows current main and v2026.6.9 still use direct response.text()/response.json() on ClawHub metadata/error paths, and the PR body includes after-fix live output plus oversized-stream cancellation proof.

Review metrics: 1 noteworthy metric.

ClawHub Body Readers: 2 JSON readers capped, 1 error reader capped, 0 direct response.json/text readers remain at PR head. This shows the shared metadata path and install-resolution sibling path now share bounded helpers before maintainers decide on the cap values.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

Get the exact head green or identify the runtime-config failure as an unrelated flake.
[P2] Have a maintainer accept or tune the ClawHub response-size and diagnostic caps before merge.

Risk before merge

[P1] The new 16 MiB JSON cap, 8 KiB error-byte cap, 400-character diagnostic cap, and 30s body idle window intentionally turn oversized or stalled ClawHub metadata/error responses into fast failures or truncated diagnostics; maintainers should accept or tune those limits before merge.
[P1] Exact-head CI currently has one failing runtime-config shard outside the touched ClawHub files, so merge still needs green checks or a maintainer-known-flake determination.

Maintainer options:

Accept The Fail-Fast Caps After Green CI (recommended)
Maintainers can land this as the intended ClawHub response-size boundary once the current head has green required checks.
Tune The Response Limits First
If legitimate ClawHub metadata or diagnostics may exceed these limits, adjust the constants and regression expectations before merge.
Pause If ClawHub Needs A Broader Contract
If artifact downloads, metadata JSON, and diagnostics should share one documented marketplace response policy, pause this PR and settle that contract first.

Next step before merge

[P2] Human review is needed for cap acceptance or tuning and the current exact-head CI failure; there is no narrow ClawSweeper repair finding in the changed code.

Security
Cleared: The diff narrows an untrusted ClawHub response memory-exhaustion surface and adds no dependencies, scripts, workflows, permissions, or credential handling changes.

Review details

Best possible solution:

Land the bounded ClawHub JSON/error readers after maintainers accept or tune the limits and exact-head CI is green; keep artifact-download sizing as a separate binary-download contract.

Do we have a high-confidence way to reproduce the issue?

Yes. Source inspection shows current main and v2026.6.9 still use direct response.text()/response.json() on ClawHub metadata/error paths, and the PR body includes after-fix live output plus oversized-stream cancellation proof.

Is this the best way to solve the issue?

Yes, with maintainer cap acceptance. Centralizing ClawHub metadata JSON and diagnostics on media-core bounded readers is the narrow owner-boundary fix; binary artifact downloads are a separate contract.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 88c3bb539189.

Label changes

Label justifications:

P2: The PR addresses a focused ClawHub marketplace availability hardening issue with limited blast radius.
merge-risk: 🚨 availability: The patch changes live ClawHub response-body handling so oversized or stalled metadata/error responses now fail fast or truncate diagnostics instead of buffering indefinitely.
rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body includes after-fix live ClawHub search output, injected oversized-stream cancellation for both JSON paths, error-body truncation output, a negative control, and focused Vitest output.
proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes after-fix live ClawHub search output, injected oversized-stream cancellation for both JSON paths, error-body truncation output, a negative control, and focused Vitest output.

Evidence reviewed

PR surface:

Source +22, Tests +89. Total +111 across 2 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	1	31	9	+22
Tests	1	89	0	+89
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	2	120	9	+111

What I checked:

Current main still has unbounded ClawHub body readers: Current main reads ClawHub error bodies with response.text(), shared metadata JSON with response.json(), and install-resolution JSON with another response.json(). (src/infra/clawhub.ts:676, 88c3bb539189)
Latest release has the same unbounded readers: v2026.6.9 still contains response.text() for ClawHub error bodies and response.json() for both shared metadata and install-resolution JSON. (src/infra/clawhub.ts:673, c645ec4555c0)
PR head bounds the shared and install-resolution JSON paths: At PR head, fetchJson delegates to parseClawHubJsonBody, the parser uses readResponseWithLimit with a 16 MiB cap, readErrorBody uses readResponseTextSnippet, and fetchClawHubSkillInstallResolution reuses the same parser. (src/infra/clawhub.ts:685, eb81ab8ec105)
No direct ClawHub response.json/text readers remain at PR head: A PR-head search shows readResponseTextSnippet/readResponseWithLimit and parseClawHubJsonBody usages, with no response.json() or response.text() matches in src/infra/clawhub.ts. (src/infra/clawhub.ts:687, eb81ab8ec105)
Dependency helper contract supports the fix: readResponseWithLimit reads a capped prefix, cancels on overflow or idle timeout, throws the supplied overflow error, and readResponseTextSnippet collapses/truncates diagnostic text. (packages/media-core/src/read-response-with-limit.ts:56, 88c3bb539189)
PR regression coverage targets the affected paths: The PR adds tests for oversized shared ClawHub JSON cancellation, oversized error-body truncation, and oversized install-resolution JSON cancellation. (src/infra/clawhub.test.ts:804, eb81ab8ec105)

Likely related people:

steipete: Peter Steinberger introduced the native ClawHub install flows in commit 91b2800, and current blame in the shallow checkout points the current ClawHub helpers and media-core response-limit import at commit 37714f1. (role: feature owner and recent area contributor; confidence: high; commits: 91b2800241c1, 37714f185fea, 08cee3316d06; files: src/infra/clawhub.ts, src/infra/clawhub.test.ts, packages/media-core/src/read-response-with-limit.ts)
vincentkoc: Vincent Koc authored the related Anthropic error-stream hardening in merged PR fix(agents): bound Anthropic error streams #95108 and has recent ClawHub auth/token work in the same infra module. (role: recent adjacent hardening contributor; confidence: medium; commits: d6cefe26f499, 071de383ffa9, c645ec4555c0; files: src/agents/anthropic-transport-stream.ts, src/infra/clawhub.ts, src/infra/clawhub.test.ts)
mappel-nv: Commit c6b5731 added ClawHub archive integrity verification across the same infra and plugin install surfaces that consume ClawHub responses. (role: adjacent ClawHub install security contributor; confidence: medium; commits: c6b5731c5d82; files: src/infra/clawhub.ts, src/infra/clawhub.test.ts, src/plugins/clawhub.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Alix-007 · 2026-06-20T22:28:35Z

@clawsweeper re-review — Addressed both P1 blockers: (1) added a live normal-path proof (real clawhub.ai search returning 5 skills, parsed through the bounded reader) and (2) the sibling install-resolution path now also uses the bounded parseClawHubJsonBody helper — no remaining raw response.json() readers. Negative control confirms the 16 MiB cap is load-bearing (unbounded drains 17 MiB, bounded cancels). Vitest 55 pass.

clawsweeper · 2026-06-20T22:28:38Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

github-actions · 2026-06-23T20:13:59Z

Dependency Guard

This PR changes dependency-related files. Maintainers should confirm these changes are intentional.

Changed files:

extensions/acpx/npm-shrinkwrap.json
extensions/acpx/package.json
pnpm-lock.yaml

Maintainer follow-up:

Review whether the dependency changes are intentional.
Inspect resolved package deltas when lockfile, shrinkwrap, or workspace dependency policy changes are present.
Treat package-lock.json and npm-shrinkwrap.json diffs as security-review surfaces.
Run pnpm deps:changes:report -- --base-ref origin/main --markdown /tmp/dependency-changes.md --json /tmp/dependency-changes.json locally for detailed release-style evidence.

github-actions · 2026-06-23T20:14:15Z

Dependency graph changes are blocked

OpenClaw does not accept dependency graph changes through PRs unless a repository admin or security explicitly authorizes the current head SHA. Dependency updates are generated internally by maintainers so external PRs cannot change the resolved graph.

Detected dependency graph changes:

extensions/acpx/npm-shrinkwrap.json changed.
pnpm-lock.yaml changed.
extensions/acpx/package.json changed dependencies.

Auto-scrub was not attempted because this PR changes package manifest dependency graph fields:

extensions/acpx/package.json changed dependencies.

Dependency graph changes must be reviewed by security or handled by maintainers internally. Please remove lockfile changes manually if they are not needed.

To remove lockfile changes, restore them from the target branch:

git fetch origin
git checkout 'origin/main' -- 'extensions/acpx/npm-shrinkwrap.json' 'pnpm-lock.yaml'
git commit -m 'chore: remove dependency lockfile change'
git push

If this PR intentionally needs a dependency graph change, ask a repository admin or member of @openclaw/openclaw-secops to comment:

/allow-dependencies-change

The action will approve the current head SHA (8fb5055b781b06aef0e91d34009f69c43a033be2) when it reruns. A later push requires a fresh approval.

steipete · 2026-06-23T20:23:21Z

Maintainer verification complete on exact head 8fb5055b781b06aef0e91d34009f69c43a033be2.

Reviewed both generic ClawHub JSON and install-resolution JSON paths, error diagnostics, caller timeout propagation, and the shared cancelling reader.
Accepted the 16 MiB JSON ceiling: live public ClawHub search/package/detail payloads sampled at 1.9-12.2 KiB. Error diagnostics remain capped at 8 KiB / 400 characters.
node scripts/run-vitest.mjs src/infra/clawhub.test.ts passed 60 tests, including stalled successful JSON and stalled error-body cancellation.
Blacksmith Testbox tbx_01kvv12qas5ahk62knrterykmj / workflow run 28053229153 passed remote pnpm check:changed.
Fresh Codex autoreview completed with no accepted/actionable findings (0.96 confidence).

The final stack preserves each caller's resolved timeout through body reads and removes PR-specific production-comment lore. No remaining proof gaps.

steipete · 2026-06-23T20:23:52Z

Merged via rebase.

Prepared head SHA: 8fb5055b781b06aef0e91d34009f69c43a033be2
Landed commit: cf67d8dded290f1859fa2e4d3c8764efe7aaefde

Alix-007 · 2026-06-24T04:11:40Z

Thanks @steipete for the detailed ClawHub verification and rebase merge.

openclaw-barnacle Bot added the size: S label Jun 20, 2026

clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. labels Jun 20, 2026

github-actions Bot mentioned this pull request Jun 20, 2026

🦞 OpenClaw 生态日报 2026-06-20 litang9/big_model_radar#90

Open

clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. and removed rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. labels Jun 22, 2026

Alix-007 force-pushed the fix/bound-clawhub-fetchjson branch from 7c96a81 to eb81ab8 Compare June 22, 2026 08:54

openclaw-barnacle Bot removed the channel: mattermost Channel integration: mattermost label Jun 22, 2026

cxbAsDev mentioned this pull request Jun 24, 2026

fix(video-generation): bound DashScope JSON response reads #96248

Closed

solodmd mentioned this pull request Jun 24, 2026

test(provider-usage): cover bounded shared response reads #96360

Merged

github-actions Bot mentioned this pull request Jul 1, 2026

chore: bump openclaw/openclaw version from v2026.6.10 to v2026.6.11 Docker-Hub-sirmark/docker-openclaw#35

Merged

ttomiczek mentioned this pull request Jul 5, 2026

Skill and Read tools truncating output at ~600-700 chars in 2026.6.11 #100458

Open

github-actions Bot mentioned this pull request Jul 6, 2026

chore(openclaw): bump pinned version v2026.5.28 → v2026.6.11 Evans-Software-Solutions-Limited/axel-saas#121

Open

This was referenced Jul 13, 2026

fix(discord): bound stalled probe response reads #106036

Merged

fix(http-error-body): log response body snippet read errors instead of silently returning empty string #106393

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(infra): bound ClawHub fetchJson and error response bodies#95226

fix(infra): bound ClawHub fetchJson and error response bodies#95226
steipete merged 3 commits into
openclaw:mainfrom
Alix-007:fix/bound-clawhub-fetchjson

Alix-007 commented Jun 20, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jun 20, 2026 •

edited

Loading

Uh oh!

Alix-007 commented Jun 20, 2026

Uh oh!

clawsweeper Bot commented Jun 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 23, 2026

Uh oh!

github-actions Bot commented Jun 23, 2026

Uh oh!

steipete commented Jun 23, 2026

Uh oh!

steipete commented Jun 23, 2026

Uh oh!

Alix-007 commented Jun 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

Alix-007 commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Why this is worth merging

How the merge-risk: availability is bounded

Real behavior proof

Uh oh!

clawsweeper Bot commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Alix-007 commented Jun 20, 2026

Uh oh!

clawsweeper Bot commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 23, 2026

Dependency Guard

Uh oh!

github-actions Bot commented Jun 23, 2026

Dependency graph changes are blocked

Uh oh!

steipete commented Jun 23, 2026

Uh oh!

steipete commented Jun 23, 2026

Uh oh!

Alix-007 commented Jun 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Alix-007 commented Jun 20, 2026 •

edited

Loading

How the `merge-risk: availability` is bounded

clawsweeper Bot commented Jun 20, 2026 •

edited

Loading

clawsweeper Bot commented Jun 20, 2026 •

edited

Loading