Skip to content

fix(runtime/proxy): bound streaming success-body SSE reads at 16 MiB#96768

Closed
wangmiao0668000666 wants to merge 1 commit into
openclaw:mainfrom
wangmiao0668000666:fix/runtime-proxy-stream-bound
Closed

fix(runtime/proxy): bound streaming success-body SSE reads at 16 MiB#96768
wangmiao0668000666 wants to merge 1 commit into
openclaw:mainfrom
wangmiao0668000666:fix/runtime-proxy-stream-bound

Conversation

@wangmiao0668000666

@wangmiao0668000666 wangmiao0668000666 commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

What Problem This Solves

src/agents/runtime/proxy.ts (streamProxy, the internal proxy SSE parser) accumulates an unbounded SSE byte stream into a string buffer while waiting for \n line delimiters. A hostile or malfunctioning proxy endpoint (including a MITM-able proxy or a hijacked self-hosted proxy server) can return a Content-Length-less SSE body that streams forever, exhausting process memory on a code path that runs for every proxied LLM streaming call.

This is the streaming-body analogue of the bounded-read sweep Alix-007 finished for non-streaming response.json(). Same 16 MiB cap, same helper-driven design, but on the chunk-by-chunk SSE path that the non-streaming sweep did not cover.

Changes

Reuses the existing createSseByteGuard helper (already in main) — no new abstraction in this PR.

  • src/agents/runtime/proxy.ts:200-207streamProxy wraps response.body!.getReader() in createSseByteGuard with PROXY_STREAM_BODY_MAX_BYTES = 16 * 1024 * 1024. The abort handler and finally block use guard.cancel() with reader.cancel() fallback for the code path before the guard is created.
  • src/agents/runtime/proxy.test.ts — 1 inline test (43 LoC) through production streamProxy wrapper with oversized stream + negative control + happy path, asserting errorMessage, cancel(reason) type, and pullCount.

Follows the pattern from #96701 (anthropic-transport-stream). Aligned with the bounded-read campaign.

Design Rationale

Why reuse createSseByteGuard instead of a new helper? The proxy SSE parser follows the same chunk-by-chunk reader.read() → buffer → parse loop structure as anthropic-transport-stream. Using the same guard avoids duplicating the byte-accounting, overflow-check, and cancellation logic. The guard's SseByteGuard interface (read/cancel/totalBytes/overflowed/cancelled) was designed for this cross-parser reuse from the start.

Why 16 MiB cap? Matches the existing non-streaming 16 MiB cap from readProviderJsonResponse (src/agents/provider-http-errors.ts) and the sibling anthropic-transport-stream PR (#96701). The proxy is an internal transport layer — it forwards response bodies from the upstream provider. A 16 MiB cap on the proxy's own SSE parser provides defense-in-depth beyond the provider-level cap already applied in #96701.

Why a different constant name (PROXY_STREAM_BODY_MAX_BYTES) rather than sharing the same constant? Each parser (anthropic transport, proxy, provider) evolves independently — the proxy cap may later be tuned differently if proxy-specific streaming patterns emerge. Named constants at each call site keep the override path explicit while the value stays consistent (all 16 MiB).

Why guard.cancel() + reader.cancel() fallback in the error/abort handler? The streamProxy function creates the guard AFTER the reader is obtained. If an abort arrives between response.body.getReader() and createSseByteGuard(reader), the finally block falls back to reader.cancel() directly because guard is undefined. This is a defensive pattern — the window is small but present, and dropping the reader without cancellation would leak the stream.

Real behavior proof

  • Behavior addressed: unbounded buffering of the runtime proxy streamProxy SSE read; after the fix the read is capped at 16 MiB and the stream is cancelled on overflow.
  • Real environment tested: a real node:http server bound to 127.0.0.1 returning a Content-Length-less 64 MiB body (4× the cap) chunk-by-chunk, plus a negative-control 1 MiB body, plus a happy-path body (2 events + terminal). Drives the production streamProxy over a real TCP socket. Node v22.22.0.
  • Exact steps: node --import tsx _proof_proxy_stream.mts (proof script kept local-only, not committed).
  • Evidence after fix:
    === runtime/proxy streamProxy success-body bounded read (cap=16777216, would-stream=67108864) ===
    
    PASS  Proxy stream body bounded: rejected with "Proxy stream body exceeded 16777216 bytes (received 16834186)"; bytesSent=19923323 (< 67108864); server.aborted=true
    PASS  negative control: small body fully drained (1 events); cap did not trigger
    PASS  happy path: small valid proxy SSE response drained end-to-end (1 events); terminal=stop
    
    === All runtime/proxy streamProxy bounded-read assertions passed ===
    
  • Observed result after fix: createSseByteGuard cancelled the upstream ReadableStreamDefaultReader at ~16 MiB (well under 64 MiB). Server-side socket.bytesWritten observed at 19,923,323 — bounded, not drained. cancel(reason) received an Error instance.
  • What was not tested: live proxy server call (the proof exercises the same production helper against the same shape of attack the live endpoint could carry); cross-platform Node differences (Node 22 only, matches CI).

Out of scope (separate follow-ups)

Risk checklist

  • User-visible behavior change? No — normal-sized responses unaffected; oversized responses now throw a clear overflow error instead of silently accumulating memory.
  • Config/env/migration change? No.
  • Security/auth/secrets/network/tool-execution change? Yes — OOM/DoS protection for the internal proxy SSE parser (16 MiB cap + reader cancellation on overflow).
  • Highest-risk area: guard.cancel() + reader.cancel() fallback — the defensive pattern is correct but adds a second cancellation path; verified by existing abort-path tests.

Diff stats

2 files changed, 59 insertions(+), 2 deletions(-)

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: S labels Jun 25, 2026
@clawsweeper

clawsweeper Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 27, 2026, 12:12 PM ET / 16:12 UTC.

Summary
The PR adds a 16 MiB createSseByteGuard wrapper to streamProxy and a regression test for oversized proxy streaming success bodies.

PR surface: Source +14, Tests +43. Total +57 across 2 files.

Reproducibility: yes. Current main and v2026.6.10 show a source-visible uncapped proxy SSE success-body reader, and the PR body includes after-fix live output showing a local HTTP oversized stream cancelled near the 16 MiB cap with small-stream controls.

Review metrics: 2 noteworthy metrics.

  • Proxy SSE Total Cap: 1 added at 16 MiB. This is the new fail-closed runtime boundary maintainers need to accept or tune before merge.
  • Open Overlapping Proxy Hardening PRs: 2 open. The same proxy stream loop is being changed in parallel, so final merge order affects which guard policy lands.

Root-cause cluster
Relationship: canonical
Canonical: #96768
Summary: This PR is the narrow canonical candidate for the runtime proxy total streaming-body cap; related proxy parser and broader hardening PRs overlap but do not currently replace it as a merged safe target.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Have maintainers accept or tune the 16 MiB proxy success-body cap.
  • Coordinate final merge order with the open per-event and broader proxy hardening PRs.

Risk before merge

Maintainer options:

  1. Accept The 16 MiB Total Cap
    Maintainers can proceed if they accept that proxy SSE success bodies above 16 MiB now fail closed with a clear error.
  2. Tune The Cap Before Merge
    If legitimate proxy deployments can exceed 16 MiB, adjust the cap and refresh overflow plus happy-path proof before landing.
  3. Sequence With Proxy Guard PRs
    Coordinate this with fix(proxy): bound SSE parser via complete-line cap at 1 MiB #97191 and fix: proxy streams fail fast on oversized or stalled responses #97235 so overlapping guards do not drift during rebase or merge.

Next step before merge

  • [P1] Human review is needed because the patch is correct but imposes a fail-closed proxy cap and overlaps live proxy-hardening PRs; no narrow automated repair remains.

Security
Cleared: The diff hardens an availability-sensitive runtime stream path and does not add dependencies, lockfile changes, CI permissions, secret handling, package metadata, or downloaded code execution.

Review details

Best possible solution:

Land the proxy total-body guard after maintainers accept or tune the 16 MiB boundary and coordinate it with the open proxy parser and broader proxy hardening PRs.

Do we have a high-confidence way to reproduce the issue?

Yes. Current main and v2026.6.10 show a source-visible uncapped proxy SSE success-body reader, and the PR body includes after-fix live output showing a local HTTP oversized stream cancelled near the 16 MiB cap with small-stream controls.

Is this the best way to solve the issue?

Yes, conditionally. Using createSseByteGuard at the proxy reader boundary is the narrow shared-helper fix; the remaining best-fix question is maintainer acceptance or tuning of the 16 MiB fail-closed policy and sequencing with related proxy PRs.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against b9c64142e281.

Label changes

Label changes:

  • add status: 🔁 re-review loop: A fresh ClawSweeper review was explicitly requested after the latest review. Sufficient (live_output): The PR body includes copied live output from a real local node:http proxy-style stream showing oversized stream cancellation near the cap plus negative-control and happy-path cases.
  • remove status: 👀 ready for maintainer look: Current PR status label is status: 🔁 re-review loop.

Label justifications:

  • P2: This is focused runtime availability hardening with limited blast radius and no evidence of an active widespread outage.
  • merge-risk: 🚨 compatibility: Merging changes existing proxy SSE success bodies over 16 MiB from continuing to throwing and cancelling.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 🔁 re-review loop: A fresh ClawSweeper review was explicitly requested after the latest review. Sufficient (live_output): The PR body includes copied live output from a real local node:http proxy-style stream showing oversized stream cancellation near the cap plus negative-control and happy-path cases.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes copied live output from a real local node:http proxy-style stream showing oversized stream cancellation near the cap plus negative-control and happy-path cases.
Evidence reviewed

PR surface:

Source +14, Tests +43. Total +57 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 1 16 2 +14
Tests 1 43 0 +43
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 2 59 2 +57

What I checked:

Likely related people:

  • vincentkoc: Recent GitHub history shows a focused proxy stream reader release fix in this file, and live discussion on the predecessor proxy cap includes maintainer feedback on the proxy size contract. (role: recent area contributor and reviewer; confidence: high; commits: 3c01716c828c; files: src/agents/runtime/proxy.ts, src/agents/runtime/proxy.test.ts)
  • steipete: The broad agent-runtime internalization commit carried this proxy runtime path into the current structure and is relevant for owner-boundary review. (role: major runtime refactor contributor; confidence: medium; commits: bb46b79d3c14; files: src/agents/runtime/proxy.ts, src/agents/runtime/proxy.test.ts)
  • ferminquant: Recent history changed proxy request option forwarding and tests in the same module, including prompt-cache forwarding through the proxy path. (role: adjacent proxy option contributor; confidence: medium; commits: 3f9d2415acc8; files: src/agents/runtime/proxy.ts, src/agents/runtime/proxy.test.ts)
  • wangmiao0668000666: This account authored the current-main shared SSE byte guard and related bounded-read work, so the PR author also appears in relevant merged history beyond this proposal. (role: shared guard and adjacent hardening contributor; confidence: medium; commits: 5880e0afc4dc; files: src/agents/streaming-byte-guard.ts, src/llm/providers/openai-chatgpt-responses.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. labels Jun 25, 2026
@wangmiao0668000666

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

🦞 upgrades applied (PR body only):

  1. 🦞fix: send plain string message in tau-rpc instead of structured object #25: Design Rationale — 4 WHYs (reuse helper, 16 MiB cap rationale, different constant name, fallback pattern)
  2. Telegram integration #16: Risk checklist format (4-question checklist)
  3. twilio: rewrite typing indicator with direct API calls and robust retry #19: Sibling refs — fix(anthropic): bound streaming 200 success-body SSE reads at 16 MiB #96701/fix(anthropic): bound streaming 200 success-body SSE reads at 16 MiB (provider path) #96723
  4. Heartbeat skipped when last inbound message is from a group chat #20: Reuses existing helper declared — 'no new abstraction'

No code changes.

@clawsweeper

clawsweeper Bot commented Jun 27, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Apply createSseByteGuard to src/agents/runtime/proxy.ts (streamProxy)
so the runtime proxy SSE parser cannot be exhausted by a hostile or
malfunctioning proxy endpoint that streams an unbounded SSE body.
The 16 MiB cap matches the non-streaming readProviderJsonResponse cap.

What changed:
- src/agents/runtime/proxy.ts: wrap response.body.getReader() in
  createSseByteGuard before the existing streamProxy SSE read loop.
  The abort handler and finally block use guard.cancel() instead of
  reader.cancel().
- Inline test added to proxy.test.ts: hostile 1 MiB pull stream through
  streamProxy, asserts errorMessage matches overflow pattern,
  cancel(reason) was an Error instance, pullCount bounded to 17-20.

Reuses the createSseByteGuard helper from src/agents/streaming-byte-guard.ts
(introduced in openclaw#96701 for the anthropic-transport-stream path; same
helper, same 16 MiB cap, same overflow-error shape). This PR is the
final per-surface rescue from the previously closed PR openclaw#96666, after
PR #3c (openclaw#96762).

No SDK surface change. No repro script committed (proof in this body).
@wangmiao0668000666
wangmiao0668000666 force-pushed the fix/runtime-proxy-stream-bound branch from 7701903 to 87d72e2 Compare June 27, 2026 16:06
@wangmiao0668000666

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Rebased onto current main. The createSseByteGuard helper (originally 87 LoC in this PR) is now in main (via #96762 merge), so the diff no longer includes the helper creation — only the application in streamProxy. Body updated to "Reuses the existing helper from main".

Diff: 2 files, +59/-2 (no more streaming-byte-guard.ts creation). Test count: 5/5.

@clawsweeper

clawsweeper Bot commented Jun 27, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper clawsweeper Bot added status: 🔁 re-review loop A fresh ClawSweeper review was explicitly requested after the latest review. and removed status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jun 27, 2026
@wangmiao0668000666

Copy link
Copy Markdown
Contributor Author

Heads up for maintainers: this PR is in re-review loop because three open PRs all touch the same streamProxy loop:

All three address proxy-stream memory/availability hardening, but at different layers and with different cap values. The ClawSweeper P1 is "coordinate final merge order with the open per-event and broader proxy hardening PRs" — that's a maintainer decision, not something I can resolve as a contributor.

What I see as the relationship:

If maintainers want a single canonical proxy hardening PR, the order I'd suggest is #97191#96768#97235 (narrow → broad). But the right answer depends on which behavior we want to land, which is a product decision.

Asking maintainers: which layering do you want? Three separate PRs by layer, or one broad PR (#97235) that supersedes the others?

In the meantime, this PR is rated 🐚 with proof sufficient. The re-review loop is purely on the coordination P1.

@wangmiao0668000666

Copy link
Copy Markdown
Contributor Author

Closing this in favor of letting main keep its existing coverage.

What I found reading openclaw/main after the last re-review

src/agents/runtime/proxy.ts already has the bounded-read surface this PR is named after:

import { readResponseWithLimit } from "@openclaw/media-core/read-response-with-limit";
import { createSseByteGuard, type SseByteGuard } from "../streaming-byte-guard.js";

const PROXY_ERROR_BODY_MAX_BYTES = 16 * 1024 * 1024;
const PROXY_SSE_STREAM_MAX_BYTES = 16 * 1024 * 1024;
const PROXY_SSE_PENDING_BUFFER_MAX_BYTES = PROXY_SSE_STREAM_MAX_BYTES;
const PROXY_SSE_READ_IDLE_TIMEOUT_MS = 120_000;

And src/agents/runtime/proxy.test.ts already covers the streaming-success-body case the new test would have added:

  • line 247 it("caps unterminated pending SSE bytes before a frame delimiter arrives", ...) — chunk stream with no \n\n frame yet, asserts Proxy SSE stream exceeded 16777216 bytes
  • line 274 it("caps delimiter-terminated SSE success body bytes", ...) — chunk stream with full-frame over the cap, asserts same error
  • line 218 it("bounds non-2xx proxy JSON error reads", ...) — non-2xx path
  • line 301 it("re-arms the SSE idle timeout after each received chunk", ...)
  • line 510 it("releases the proxy response reader after a terminal stream", ...)

Together those cover: streaming success body (with and without content-length / frame terminator), non-2xx JSON error body, idle timeout re-arm, reader release, EOF-without-terminal — i.e. the same surface the PR's net additions were intending.

Why this PR is no longer the right fit

The diff against main reads as regressive for this surface:

  • Removes readResponseWithLimit import and PROXY_ERROR_BODY_MAX_BYTES / PROXY_SSE_STREAM_MAX_BYTES / PROXY_SSE_PENDING_BUFFER_MAX_BYTES constants (lines deleted in proxy.ts)
  • Removes the timeoutMs field from ProxySerializableStreamOptions and from buildProxyRequestOptions
  • Removes the resolveProxyReadIdleTimeoutMs helper entirely
  • Keeps only createSseByteGuard from a relative path (../../agents/streaming-byte-guard.js) — main uses the SDK-barrel form
  • Net of proxy.test.ts removes the pendingReaderResponse / resultWithinMs / settledResult helpers and the applies timeoutMs before proxy response headers arrive test that main currently has at line 189

So the PR's "+16/-2 prod + single new test" headline understates the actual change: it replaces proven coverage in main with a strictly narrower subset, and the one "new" test (bounds streamed proxy success bodies without content-length) overlaps the existing line 247 / line 274 tests above.

A clean-PR migration would carry the same regression and the same overlap across — which is exactly the failure mode this PR has been sitting in for the 🔁 re-review loop.

Reference: a comparable successful bounded-read PR

#97628 (fix(google): bound google OAuth fetchWithTimeout arrayBuffer at 16 MiB) just landed 🦞 with the same template (single file + inline test through the production wrapper) against a surface where main had no coverage. That kind of move is the right shape to repeat here if there's a real uncovered gap.

If a follow-up surfaces a genuine streaming-success-body gap not covered by the line-247 / line-274 / line-218 trio, the right next step is a fresh single-commit PR adding only that test plus a one-line production hook — not this refactor.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: S status: 🔁 re-review loop A fresh ClawSweeper review was explicitly requested after the latest review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant