Skip to content

Bound proxy error response JSON parsing to avoid loading huge bodies#97351

Merged
sallyom merged 1 commit into
openclaw:mainfrom
zhangguiping-xydt:fix/problem-proxy-bounded-error-response
Jun 28, 2026
Merged

Bound proxy error response JSON parsing to avoid loading huge bodies#97351
sallyom merged 1 commit into
openclaw:mainfrom
zhangguiping-xydt:fix/problem-proxy-bounded-error-response

Conversation

@zhangguiping-xydt

@zhangguiping-xydt zhangguiping-xydt commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

Summary

What problem does this PR solve?

  • Fix classification: Root-cause fix for the agent-runtime proxy error-response parser reading unbounded JSON bodies.
  • Root cause: The root cause was that the non-OK /api/stream proxy response path called Response.json() directly, so the parser had to read and decode the complete error body before the proxy client could fall back to the HTTP status message.
  • Why this is root-cause fix: This fixes the parser/source invariant by routing proxy error JSON through readProviderJsonResponse, the existing bounded JSON response reader backed by readResponseWithLimit, instead of adding a downstream fallback after the unbounded read already happened.
  • Patch quality notes: The code diff remains unchanged from the submitted PR: an XS scoped proxy error-branch change, reuse of the canonical bounded reader, preserved status/statusText fallback for malformed or oversized bodies, and deterministic regression coverage for the oversized-body path.

Why does this matter now?

  • Why it matters / User impact: Users and operators keep the same visible proxy error behavior for normal failures, while a bad proxy or intermediary can no longer force the client process to buffer an arbitrary-size error body before surfacing the failure.

What is the intended outcome?

  • Maintainer-ready confidence: High; the code is unchanged from the previous submitted head, local focused tests pass, a new real local HTTP proxy proof directly addresses ClawSweeper RF-001 through RF-004, and local verification is bound to the current PR head.

What is intentionally out of scope?

  • What did NOT change: This does not change auth credentials, provider configuration, session state, schema, migrations, proxy protocol, successful SSE stream parsing, or normal small { "error": "..." } error extraction.
  • Out of scope: Successful streaming response buffering and live external proxy network behavior are not changed; this PR only fixes and proves the non-OK proxy error JSON parsing branch.

What does success look like?

The proxy client reports the existing Proxy error: 502 Bad Gateway fallback for an oversized malformed JSON error response, and the real local HTTP proxy observes the client closing the response before the full oversized body is sent.

What should reviewers focus on?

  • Architecture / source-of-truth check: readProviderJsonResponse is already the source-of-truth helper for bounded provider JSON parsing, so this proxy path now follows the same response-size invariant instead of maintaining a separate limit.
  • Related open PR / same-contract scan: check-upstream-fixed found origin/main has not covered this PR and has no overlapping files with the PR diff; newer main changes are unrelated media/plugin/state work.

Linked context

Which issue does this close?

No linked issue number; this PR is based on the reported proxy error-response path.

Which issues, PRs, or discussions are related?

Related: ClawSweeper comment requiring real proxy/gateway proof for PR #97351.

Was this requested by a maintainer or owner?

No maintainer request was linked in the local task context.

Real behavior proof (required for external PRs)

  • Behavior or issue addressed: A non-OK /api/stream proxy error response with a very large JSON body should return the proxy error result without reading the whole error stream into memory.

  • Real environment tested: Local OpenClaw agent runtime loaded from this PR head, using the real streamProxy implementation and a real local HTTP proxy server listening on 127.0.0.1 for /api/stream.

  • Exact steps or command run after this patch: node --import tsx <local real-proxy proof script>

  • Evidence after fix (screenshot, recording, terminal capture, console output, redacted runtime log, linked artifact, or copied live output):

    Real proxy proof: local HTTP proxy returned oversized non-OK JSON body
    request_path=/api/stream
    response_status=502 Bad Gateway
    client_error_message=Proxy error: 502 Bad Gateway
    chunks_planned=24
    chunk_size_bytes=1048576
    chunks_sent_before_close=18
    bytes_sent_before_close=18874368
    connection_closed_before_full_body=true
    bounded_fallback_without_full_read=true
    
  • Observed result after fix: The real local proxy planned to stream 24 MiB but observed the client connection close after 18 MiB, while the client surfaced Proxy error: 502 Bad Gateway; this shows the bounded reader stopped the oversized non-OK body before the full response was consumed.

  • What was not tested: No external hosted proxy service was called; the proof uses a real local HTTP proxy endpoint to avoid credentials and non-public infrastructure while exercising the real client fetch path.

  • Proof limitations or environment constraints: The proof is a localhost proxy/gateway simulation for the client-side parser boundary, not a full deployed OpenClaw gateway run; it directly covers the failure ClawSweeper requested: oversized non-OK proxy body, bounded fallback, and early connection close.

  • Before evidence (optional but encouraged): The regression test failed before the fix with expected 20 to be less than 20, showing the old path read all twenty 1 MiB chunks in the controlled streaming Response case.

Tests and validation

Which commands did you run?

  • Target test file: src/agents/runtime/proxy.test.ts via pnpm exec vitest run src/agents/runtime/proxy.test.ts.

What regression coverage was added or updated?

  • Scenario locked in: A 502 Bad Gateway proxy response with Content-Type: application/json streams oversized chunks; the proxy must return the fallback error and stop reading before the complete body is consumed.

What failed before this fix, if known?

  • Why this is the smallest reliable guardrail: The test exercises the actual streamProxy non-OK response branch and response-reader boundary directly, while the new real local proxy proof exercises the same branch through a real HTTP server and Node fetch path.

If no test was added, why not?

A regression test was added in src/agents/runtime/proxy.test.ts; the current update adds PR-visible proof only and does not change production code.

Risk checklist

Did user-visible behavior change? (Yes/No)

No for normal proxy errors; oversized or malformed JSON error bodies now stop at the bounded reader and keep the existing status/statusText fallback.

Did config, environment, or migration behavior change? (Yes/No)

No.

Did security, auth, secrets, network, or tool execution behavior change? (Yes/No)

No; the touched code is proxy/provider-adjacent, but it only changes how non-OK error response bodies are parsed after fetch returns.

What is the highest-risk area?

  • Risk labels considered: auth-provider, compatibility, session-state.
  • Risk explanation: The auth-provider signal comes from touching the agent proxy/provider-adjacent path. Compatibility and session-state are not touched by this PR's two changed files.

How is that risk mitigated?

  • Why acceptable: The committed diff only changes src/agents/runtime/proxy.ts and src/agents/runtime/proxy.test.ts, does not touch config/schema/session/auth storage, preserves existing fallback messages, and now includes a real local proxy proof showing early close before full oversized body consumption.

Current review state

What is the next action?

Ready for ClawSweeper/maintainer re-review after this PR body update with real proxy proof.

What is still waiting on author, maintainer, CI, or external proof?

RF-001, RF-002, and RF-003 are addressed by the real local HTTP proxy proof above; RF-004 is addressed because no code repair job is needed and this update only supplies contributor proof.

Which bot or reviewer comments were addressed?

Addressed ClawSweeper comment #97351 (comment): added redacted real local proxy output showing bounded fallback and early connection close for an oversized non-OK response body.

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: S labels Jun 28, 2026
@clawsweeper

clawsweeper Bot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 28, 2026, 12:42 AM ET / 04:42 UTC.

Summary
The branch routes non-OK streamProxy error JSON through readProviderJsonResponse and adds a regression test for oversized streamed proxy error bodies.

PR surface: Source +2, Tests +51. Total +53 across 2 files.

Reproducibility: yes. Current main and v2026.6.10 still call response.json() for non-OK proxy errors, and the PR adds focused regression coverage plus copied live local proxy output; this read-only review did not rerun tests locally.

Review metrics: 1 noteworthy metric.

  • Open Proxy Hardening Overlap: 3 open related PRs. Maintainers should notice the active overlap before choosing the final merge order for the streamProxy read-bound changes.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Risk before merge

  • [P1] The same streamProxy loop has three open related hardening PRs, so maintainers should choose whether this narrow error-body fix lands separately or is folded into the broader proxy guard sequence.

Maintainer options:

  1. Decide the mitigation before merge
    Land this narrow bounded error-body fix as a discrete proxy hardening step, or deliberately fold it into one broader proxy guard PR before merging.
  2. Pause or close
    Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

  • No automated repair is needed; maintainers should review the clean narrow patch and decide merge order with the overlapping proxy-hardening PRs.

Security
Cleared: The diff reduces an unbounded response-body read and does not change credentials, auth routing, dependency sources, workflows, package metadata, or code execution surfaces.

Review details

Best possible solution:

Land this narrow bounded error-body fix as a discrete proxy hardening step, or deliberately fold it into one broader proxy guard PR before merging.

Do we have a high-confidence way to reproduce the issue?

Yes. Current main and v2026.6.10 still call response.json() for non-OK proxy errors, and the PR adds focused regression coverage plus copied live local proxy output; this read-only review did not rerun tests locally.

Is this the best way to solve the issue?

Yes. Reusing readProviderJsonResponse is the narrow maintainable fix because it moves the proxy error path onto the existing bounded reader instead of adding a proxy-specific duplicate limit.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against c6ade83a5ccb.

Label changes

Label justifications:

  • P2: This is a normal-priority agent runtime proxy hardening bug fix with limited blast radius and no config, migration, or persisted-state change.
  • rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body includes after-fix copied live output from a real local HTTP proxy exercising production streamProxy and showing early close before the full oversized non-OK body was sent.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes after-fix copied live output from a real local HTTP proxy exercising production streamProxy and showing early close before the full oversized non-OK body was sent.
Evidence reviewed

PR surface:

Source +2, Tests +51. Total +53 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 1 6 4 +2
Tests 1 51 0 +51
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 2 57 4 +53

What I checked:

  • Root policy read: Root AGENTS.md was read fully and applied; it requires whole-path review, proof assessment, and owner/history checks for agent runtime PRs. (AGENTS.md:1, c6ade83a5ccb)
  • Scoped agent policy read: The scoped agent guide was read and supports focused behavior-oriented runtime tests that avoid broad runtime loading. (src/agents/AGENTS.md:1, c6ade83a5ccb)
  • Current main behavior: Current main still enters the non-OK proxy response branch and calls response.json() before falling back to the status/statusText proxy error. (src/agents/runtime/proxy.ts:181, c6ade83a5ccb)
  • Latest release behavior: Release v2026.6.10 has the same unbounded non-OK proxy response.json() path, so the fix is not shipped already. (src/agents/runtime/proxy.ts:181, aa69b12d0086)
  • Proposed source change: The PR head replaces the direct JSON read with readProviderJsonResponse and preserves fallback behavior when parsing or overflow fails. (src/agents/runtime/proxy.ts:185, b2f0b703335b)
  • Bounded helper contract: readProviderJsonResponse reads through readResponseWithLimit under the provider JSON cap and labels overflow or malformed JSON errors for callers. (src/agents/provider-http-errors.ts:312, c6ade83a5ccb)

Likely related people:

  • kevinslin: Git blame and PR metadata tie the current main proxy runtime file carrying the unbounded non-OK branch to the merged always-plugin-approval import commit. (role: current-main path carrier; confidence: medium; commits: c5d34c8376f8; files: src/agents/runtime/proxy.ts, src/agents/runtime/proxy.test.ts)
  • Alix-007: Merged PR 95218 added the bounded readProviderJsonResponse helper that this PR reuses for the proxy error path. (role: bounded helper contributor; confidence: high; commits: a15f8e3aaac5; files: src/agents/provider-http-errors.ts, src/agents/provider-http-errors.test.ts)
  • wangmiao0668000666: Live related PRs by this contributor touch the same streamProxy loop for success-body and SSE parser bounds, so they are relevant for merge sequencing. (role: adjacent proxy hardening contributor; confidence: medium; commits: 87d72e2e20ee, caa27dc8b4ea; files: src/agents/runtime/proxy.ts, src/agents/runtime/proxy.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 28, 2026
@sallyom sallyom self-assigned this Jun 28, 2026
@sallyom

sallyom commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

Merge-ready for b2f0b70.

Local review and ClawSweeper review clean, CI green, best narrow fix, and follows the established bounded provider JSON response-read pattern. This is the right small precursor to #97235: it only bounds the non-OK proxy error JSON branch, preserves the existing status/statusText fallback for malformed or oversized bodies, and leaves the broader proxy SSE/body/idle-timeout policy decisions for #97235 to consider next. No maintainer edits needed.

@sallyom
sallyom merged commit 27c1685 into openclaw:main Jun 28, 2026
161 of 174 checks passed
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jun 29, 2026
QiuYuang pushed a commit to QiuYuang/openclaw that referenced this pull request Jul 1, 2026
chenyangjun-xy pushed a commit to chenyangjun-xy/openclaw that referenced this pull request Jul 1, 2026
chenyangjun-xy pushed a commit to chenyangjun-xy/openclaw that referenced this pull request Jul 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. size: S status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants