fix(provider-usage): bound usage JSON response reads to prevent OOM by cxbAsDev · Pull Request #97610 · openclaw/openclaw

cxbAsDev · 2026-06-29T01:30:58Z

Summary

Replace unbounded response.json() with readProviderJsonResponse in the readUsageJson shared helper to cap provider usage API response bodies at 16 MiB. This prevents unbounded buffering of responses from all seven providers that call this helper: Claude, Gemini, Codex, Z.AI, DeepSeek, MiniMax, and OpenAI-compatible embeddings.

Real behavior proof (required for external PRs)

Behavior addressed: Unbounded response.json() replaced with readProviderJsonResponse (16 MiB cap).

Real environment tested: Linux, Node 22, OpenClaw main @ 87db23e

Exact steps or command run after fix:

node --import tsx proof-readUsageJson.mts

Proof harness:

// Creates a streaming 64 MiB body with 1 MiB chunks.
// The bounded reader must stop at the 16 MiB cap, cancel
// the stream, and return { ok: false, snapshot }.
const { response, state } = makeOversizedStream();
const result = await readUsageJson("anthropic", response);

Evidence after fix:

cap: 16777216 bytes (16 MiB)
chunk: 1 MiB, total stream: 64 MiB
chunks enqueued: 17
stream cancelled: true
result.ok: false
snapshot.error: Malformed usage response
bytes buffered: 17825792 (17.0 MiB)
PASS: reader stopped at 17.0 MiB, stream cancelled before full 64 MiB buffering

The harness creates a streaming 64 MiB body with 1 MiB chunks. The bounded reader pulls only 17 chunks (stopping at the 16 MiB cap), cancels the stream, and returns the error snapshot. Before the fix, response.json() would have buffered the entire 64 MiB before parsing.

Observed result after fix: readProviderJsonResponse caps provider usage API responses at 16 MiB. Streaming bodies are cancelled at the cap instead of being fully buffered. The existing { ok: false, snapshot } error contract is preserved for both overflow and malformed-JSON errors.

All existing tests pass: 51 tests across 5 caller test files (claude, codex, deepseek, gemini, zai) plus the shared helpers test file.

What was not tested: Live provider usage API calls against Anthropic, Gemini, Codex, Z.AI, DeepSeek, or MiniMax usage endpoints were not tested.

Risk

Low. The 16 MiB cap matches the convention in provider-http-errors.ts. The existing try-catch in readUsageJson preserves the { ok: false, snapshot } contract — overflow and JSON parse errors both return "Malformed usage response", matching the existing behavior for malformed-JSON responses.

Replace unbounded response.json() with readProviderJsonResponse in readUsageJson to cap provider usage API response bodies at 16 MiB, preventing unbounded buffering from all seven providers (claude, gemini, codex, z.ai, deepseek, minimax, and openai-compatible). Co-Authored-By: Claude <[email protected]>

clawsweeper · 2026-06-29T01:32:07Z

Thanks for the context here. I swept through the related work, and this is now duplicate or superseded.

Close as duplicate/superseded because #96360 already owns the same shared provider-usage JSON OOM fix with focused tests, sufficient proof, and a ready-for-maintainer-review ClawSweeper verdict; this branch's useful helper-reuse idea can be folded there instead of keeping two landing candidates open.

Root-cause cluster
Relationship: duplicate
Canonical: #96360
Summary: The current PR and #96360 both address the same unbounded shared provider-usage JSON helper; the older PR is the canonical open landing path because it has focused tests, sufficient proof, and a ready-for-maintainer-review status.

Members:

canonical: test(provider-usage): cover bounded shared response reads #96360 - It is an open proof-positive PR changing the same shared helper with tests for normal, malformed, BOM-prefixed, and oversized responses.
adjacent_distinct: fix(inworld): bound TTS audio, voices, and error response reads to prevent OOM #95416 - It is a merged adjacent response-limit PR for Inworld TTS/voices, not the shared provider-usage helper.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Canonical path: Close this duplicate and concentrate review on #96360, optionally adopting this branch's readProviderJsonResponse call there if maintainers prefer that helper shape.

So I’m closing this here and keeping the remaining discussion on #96360.

Review details

Best possible solution:

Close this duplicate and concentrate review on #96360, optionally adopting this branch's readProviderJsonResponse call there if maintainers prefer that helper shape.

Do we have a high-confidence way to reproduce the issue?

Yes. Source inspection shows current main and v2026.6.10 still use direct response.json() in readUsageJson, and the PR body supplies terminal output from a 64 MiB streaming harness showing bounded cancellation after the change.

Is this the best way to solve the issue?

No. The helper reuse is a reasonable implementation detail, but the best solution is to use the canonical PR with focused tests and proof, folding this helper call into that branch only if maintainers prefer it.

Security review:

Security review cleared: The diff reuses an existing internal bounded JSON reader and does not change secrets, auth, dependencies, workflows, package resolution, or code execution surfaces.

AGENTS.md: found and applied where relevant.

What I checked:

PR diff: This PR changes only readUsageJson in the shared provider-usage helper by importing readProviderJsonResponse and replacing the direct response.json() call; it adds no tests on this branch. (src/infra/provider-usage.fetch.shared.ts:70, 1f1c87e6fcbd)
Current main behavior: Current main still parses successful shared provider usage responses with direct response.json(), so the problem is not implemented on main and needs one provider-usage PR to land. (src/infra/provider-usage.fetch.shared.ts:70, 561c713bb1f3)
Sibling provider gap: Current main also keeps MiniMax successful usage responses on direct res.json(), which is why the broader all-provider usage-response sweep should stay concentrated in one canonical PR rather than split across duplicate branches. (src/infra/provider-usage.fetch.minimax.ts:421, 561c713bb1f3)
Canonical overlapping PR: test(provider-usage): cover bounded shared response reads #96360 is open, has proof: sufficient and status: 👀 ready for maintainer look, changes the same shared helper, and adds focused readUsageJson tests for normal, BOM-prefixed, malformed, oversized, and provider-name cases. (src/infra/provider-usage.fetch.shared.test.ts:149, 24c8d081bea2)
Release behavior check: The latest release tag v2026.6.10 still has the unbounded shared response.json() and MiniMax res.json() paths, so this is not a shipped fix; the close reason is duplicate/superseded by an open canonical PR, not implemented-on-main. (src/infra/provider-usage.fetch.shared.ts:70, aa69b12d0086)
Bounded reader contract: readProviderJsonResponse reads through readResponseWithLimit with the provider JSON cap, and readResponseWithLimit cancels streams on overflow before throwing. (src/agents/provider-http-errors.ts:318, 561c713bb1f3)

Likely related people:

vincentkoc: Live commit metadata shows vincentkoc added the readUsageJson malformed-JSON wrapper and recently touched the shared provider-usage helper and MiniMax usage path for response-body cancellation. (role: recent area contributor; confidence: high; commits: a118e114fe5f, 0c565f3b0ef9; files: src/infra/provider-usage.fetch.shared.ts, src/infra/provider-usage.fetch.minimax.ts, src/infra/provider-usage.fetch.codex.ts)
steipete: Commit metadata shows steipete deduplicated provider usage fetch logic into the shared helper structure and originally added the MiniMax usage snapshot path. (role: feature-history owner; confidence: high; commits: badafdc7b3e4, 18b4575e4dcd; files: src/infra/provider-usage.fetch.shared.ts, src/infra/provider-usage.fetch.minimax.ts, src/infra/provider-usage.fetch.codex.ts)
Alix-007: Commit metadata shows Alix-007 introduced the adjacent readProviderJsonResponse bounded JSON helper used by this PR. (role: adjacent hardening contributor; confidence: medium; commits: 2592f8a51a4e; files: src/agents/provider-http-errors.ts, src/agents/provider-http-errors.test.ts)
IVY-AI-gif: Merged PR metadata shows IVY-AI-gif recently improved MiniMax usage parsing and tests in the sibling path that remains separate from the shared helper. (role: recent MiniMax usage contributor; confidence: medium; commits: 9eb8184f368a; files: src/infra/provider-usage.fetch.minimax.ts, src/infra/provider-usage.fetch.minimax.test.ts)

Codex review notes: model internal, reasoning high; reviewed against 561c713bb1f3.

clawsweeper · 2026-06-29T04:34:34Z

ClawSweeper applied the proposed close for this PR.

Action: closed this PR.
Close reason: duplicate or superseded.
Evidence: durable ClawSweeper review.
Coverage proof: PR B clearly owns the same shared provider-usage JSON OOM fix and carries the material behavior and review concerns from PR A with stronger focused tests, so PR A has no independent remaining work that needs separate review. Covering PR: test(provider-usage): cover bounded shared response reads #96360.

vincentkoc · 2026-06-29T04:57:47Z

Follow-up closeout: the landed replacement is #97659, merged as db2786b.

Reason for replacement instead of editing this branch: maintainer push was rejected on the fork, and the original diff only covered the shared helper path; the landed replacement also bounds the MiniMax success path and Claude OAuth error-body parsing, with focused tests for both. Credit preserved from this PR in the replacement PR body.

clawsweeper · 2026-06-29T05:13:16Z

🦞👀
ClawSweeper could not autoclose this item.

Reason: structured ClawSweeper close marker: close-required (sha=1f1c87e6fcbd4b59ff75176db959bd291da17e75)

Usage: /autoclose <maintainer close reason>. I will close this item and any open same-repo items explicitly referenced in the command text.

openclaw-barnacle Bot added the size: XS label Jun 29, 2026

clawsweeper Bot closed this Jun 29, 2026

vincentkoc mentioned this pull request Jun 29, 2026

fix(provider-usage): bound usage JSON response reads #97659

Merged

clawsweeper Bot mentioned this pull request Jun 29, 2026

test(provider-usage): cover bounded shared response reads #96360

Merged

cxbAsDev deleted the fix/bound-usage-json-response branch July 15, 2026 03:32

pitcockchris79 mentioned this pull request Jul 18, 2026

[Bug]: /status reports MiniMax "Unsupported response shape" — fetcher does not recognize current MiniMax API payload (no chat-model entry, remaining percent fields ignored) #110887

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(provider-usage): bound usage JSON response reads to prevent OOM#97610

fix(provider-usage): bound usage JSON response reads to prevent OOM#97610
cxbAsDev wants to merge 1 commit into
openclaw:mainfrom
cxbAsDev:fix/bound-usage-json-response

cxbAsDev commented Jun 29, 2026

Uh oh!

clawsweeper Bot commented Jun 29, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jun 29, 2026

Uh oh!

vincentkoc commented Jun 29, 2026

Uh oh!

clawsweeper Bot commented Jun 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

cxbAsDev commented Jun 29, 2026

Summary

Real behavior proof (required for external PRs)

Risk

Uh oh!

clawsweeper Bot commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clawsweeper Bot commented Jun 29, 2026

Uh oh!

vincentkoc commented Jun 29, 2026

Uh oh!

clawsweeper Bot commented Jun 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

clawsweeper Bot commented Jun 29, 2026 •

edited

Loading