Skip to content

fix(fal): bound music/video generation response reads#96886

Merged
vincentkoc merged 1 commit into
openclaw:mainfrom
Alix-007:fix/bound-fal
Jun 29, 2026
Merged

fix(fal): bound music/video generation response reads#96886
vincentkoc merged 1 commit into
openclaw:mainfrom
Alix-007:fix/bound-fal

Conversation

@Alix-007

@Alix-007 Alix-007 commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

What Problem This Solves

Resolves a problem where the fal image, music, and video generation providers read a successful fal API JSON response through an unbounded await response.json() after the HTTP status check. A hostile, compromised, or buggy fal endpoint can stream an arbitrarily large success body with no Content-Length, forcing the runtime to buffer the entire payload in memory before parsing. This risks out-of-memory crashes or hangs on the media-generation path — affecting any agent or workflow that generates images, music, or video through the fal provider.

This is the success path: the existing assertOkOrThrowHttpError / bounded error-body reader only guards non-2xx responses, so the oversized-body risk is on 2xx replies that reach response.json().

Why This Change Was Made

All three success-path reads are routed through the shared readProviderJsonResponse helper (already used by other media providers, e.g. byteplus). It reads the body through the same 16 MiB bounded reader used for binary responses and cancels the upstream stream on overflow, then parses JSON. No new abstraction is introduced — this reuses existing plugin-SDK infrastructure.

  • extensions/fal/image-generation-provider.ts (success read at the generation call)
  • extensions/fal/music-generation-provider.ts (success read after postJsonRequest)
  • extensions/fal/video-generation-provider.ts (fetchFalJson, the shared submit/status/result read)

Boundary preserved: the video path keeps its existing FAL_VIDEO_MALFORMED_RESPONSE mapping for malformed JSON while letting the new overflow error propagate unchanged, so existing behavior for malformed payloads is unaffected.

User Impact

No change to normal behavior: well-formed fal responses parse exactly as before. When a fal endpoint returns an oversized success body, the provider now fails fast with a clear bounded error and tears down the connection instead of buffering unbounded memory. Operators running fal-backed media generation are protected from an OOM/hang failure mode.

Evidence

Behavior addressed: success-path untrusted JSON body is now bounded to 16 MiB; overflow throws and the TCP stream is cancelled.

Real environment tested: a real node:http loopback server streams a 20 MiB JSON success body with no Content-Length; the real exported provider functions (generateImage, generateMusic, generateVideo) are driven against it over real TCP. Only the credential/config resolution and the fetch-guard transport are stubbed to point at loopback — the bounded read under test (readProviderJsonResponse) runs for real.

Exact steps: drive each provider against the oversize server; assert it throws ... JSON response exceeds 16777216 bytes, that fewer than the full 20 MiB hit the wire (bytesSent < 20971520), and that the socket is torn down. Plus a negative control (the pre-fix response.json() buffers the whole body) and a happy-path control (small JSON still parses).

Observed result (real terminal output):

[image] bounded throw OK; bytesSent=19464238 (<20971520=full); socketTornDown=true
 ✓ image generation bounds an oversize success body and tears down the socket 362ms
[video] bounded throw OK; bytesSent=17825838 (<20971520=full); socketTornDown=true
 ✓ video submit bounds an oversize success body and tears down the socket 139ms
[music] bounded throw OK; bytesSent=18546734 (<20971520=full); socketTornDown=true
 ✓ music generation bounds an oversize success body and tears down the socket 436ms
[negative] unbounded json() buffered fullBytes=20971566 (>=20971520); junkLen=20971520 (>cap=16777216)
 ✓ NEGATIVE CONTROL: the old unbounded response.json() buffers the whole oversize body past the cap 237ms
[happy] small JSON parsed OK through bounded reader; bytesSent=38
 ✓ HAPPY PATH: a small JSON success body still parses through the bounded reader 9ms

 Test Files  1 passed (1)
      Tests  5 passed (5)

The three numbers above show only ~17–19 MiB reached the wire before the bounded reader cancelled the stream, versus the full 20 MiB the unbounded path buffered in the negative control.

Existing suite: the fal provider tests pass after updating the test doubles to stream real Response bodies (the old doubles only implemented .json()):

 Test Files  3 passed (3)
      Tests  49 passed (49)

Typecheck: tsgo over the extensions + test projects reports 0 errors.

What was not tested: no live call against the real fal API (requires credentials); the proof exercises the bounded read over a real TCP loopback socket instead.

Label: security

AI-assisted.


Post-rebase scope update: main has since landed the fal image response bound, so the image hunk has been dropped from this branch. This PR is now narrowed to the fal music and video success-path reads only — music-generation-provider.ts and video-generation-provider.ts (via fetchFalJson), both routed through the shared readProviderJsonResponse (16 MiB cap), with the video path preserving its existing FAL_VIDEO_MALFORMED_RESPONSE mapping. The image-generation-provider.ts change and the [image] … line in the Evidence block above no longer correspond to this PR's diff and can be disregarded.

@clawsweeper

clawsweeper Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 26, 2026, 8:37 PM ET / 00:37 UTC.

Summary
This PR routes fal music and video successful JSON parsing through readProviderJsonResponse and updates the provider tests to use real Response bodies.

PR surface: Source +8, Tests -8. Total 0 across 4 files.

Reproducibility: yes. Current main and v2026.6.10 still use response.json() on fal music/video success responses, and the PR body includes loopback TCP terminal proof with a negative control for the old path.

Review metrics: 1 noteworthy metric.

  • Successful Fal JSON Reads: 2 changed from raw response.json() to shared 16 MiB capped reads. These are the semantic merge surface because both changed reads can now fail closed on oversized successful fal JSON metadata.

Root-cause cluster
Relationship: canonical
Canonical: #96886
Summary: This PR is the viable open fal music/video continuation; the merged image-generation PR fixed fal image, and closed fal-only attempts are superseded by this broader proof-positive branch.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Maintainers should explicitly accept the 16 MiB successful-response cap for fal music/video metadata before landing.
  • Run or request a credentialed fal smoke only if loopback proof is not enough vendor compatibility confidence.

Risk before merge

  • [P1] The shared 16 MiB cap intentionally changes successful fal music/video metadata responses over the cap from fully buffered parses into failures, so maintainers should accept that compatibility tradeoff before merge.
  • [P1] No credentialed live fal API smoke is included; the loopback TCP proof covers the hostile oversized-body behavior, while real vendor compatibility relies on unchanged request/response shapes plus existing fal tests.

Maintainer options:

  1. Accept The Fal Metadata Cap (recommended)
    Land the PR if maintainers agree successful fal music/video JSON metadata over 16 MiB should fail fast rather than be buffered whole.
  2. Require Vendor Smoke First
    Ask for a credentialed fal smoke if loopback proof and unchanged response-shape tests are not enough compatibility evidence.
  3. Pause For Broader Provider Sweep
    Defer only if maintainers want the remaining fal music/video response-limit work folded into a wider provider hardening pass.

Next step before merge

  • [P1] The remaining action is maintainer acceptance of the fail-closed cap plus normal merge/check handling, not an automated code repair.

Security
Cleared: The diff hardens untrusted fal response reads and does not add dependencies, workflow permissions, package-resolution changes, secret handling, or downloaded-code execution.

Review details

Best possible solution:

Land the focused hardening once maintainers accept the 16 MiB successful-response metadata cap, with a credentialed fal smoke only if they need extra vendor compatibility confidence.

Do we have a high-confidence way to reproduce the issue?

Yes. Current main and v2026.6.10 still use response.json() on fal music/video success responses, and the PR body includes loopback TCP terminal proof with a negative control for the old path.

Is this the best way to solve the issue?

Yes. Reusing readProviderJsonResponse is the narrow maintainable fix because it applies the existing SDK byte cap at the affected fal success reads without adding fal-local parsing, configuration, or a second cap mechanism.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 65fec9d787e3.

Label changes

Label justifications:

  • P2: This is a focused hardening fix for optional fal media provider paths with limited blast radius and no active outage report.
  • merge-risk: 🚨 compatibility: The new bounded reader can reject unusually large successful fal music/video metadata JSON that previous versions would attempt to parse.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body includes after-fix terminal output from real loopback TCP runs through exported fal providers, with negative and happy-path controls for the bounded read.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes after-fix terminal output from real loopback TCP runs through exported fal providers, with negative and happy-path controls for the bounded read.
Evidence reviewed

PR surface:

Source +8, Tests -8. Total 0 across 4 files.

View PR surface stats
Area Files Added Removed Net
Source 2 13 5 +8
Tests 2 41 49 -8
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 4 54 54 0

What I checked:

  • Root policy read: Read the full root AGENTS.md and applied the extension-boundary, review-depth, proof, and compatibility-risk guidance to this PR. (AGENTS.md:1, 65fec9d787e3)
  • Scoped extension policy read: The scoped extensions policy allows bundled plugin production code to import public openclaw/plugin-sdk/* helpers, which matches this PR's use of readProviderJsonResponse. (extensions/AGENTS.md:1, 65fec9d787e3)
  • Current music success read is unbounded: Current main still checks fal music HTTP status and then calls response.json() on the successful provider-controlled response before extracting generated music candidates. (extensions/fal/music-generation-provider.ts:164, 65fec9d787e3)
  • Current video success read is unbounded: Current main's fetchFalJson helper still calls response.json() for fal video submit/status/result success bodies and maps parse failures to FAL_VIDEO_MALFORMED_RESPONSE. (extensions/fal/video-generation-provider.ts:483, 65fec9d787e3)
  • Latest release has same remaining gap: The latest release tag v2026.6.10 still shows the same fal music and video response.json() success reads, so this PR is not obsolete due to a shipped release. (extensions/fal/music-generation-provider.ts:164, aa69b12d0086)
  • Shared bounded reader contract: readProviderJsonResponse delegates to readResponseWithLimit, applies the 16 MiB default cap, throws labelled overflow errors, and wraps malformed JSON under the caller label. (src/agents/provider-http-errors.ts:312, 65fec9d787e3)

Likely related people:

  • steipete: GitHub commit history shows Peter Steinberger introduced fal/OpenRouter music generation and the fal video provider path, then repeatedly maintained fal video queue/model behavior. (role: fal media feature introducer and recent area contributor; confidence: high; commits: f45390416565, 932194b7d58c, 6d34a1c814b7; files: extensions/fal/music-generation-provider.ts, extensions/fal/video-generation-provider.ts, extensions/fal/video-generation-provider.test.ts)
  • Alix-007: Merged history shows Alix-007 added the shared readProviderJsonResponse helper that this PR reuses, so they are relevant beyond authoring this branch. (role: shared bounded-reader contributor; confidence: high; commits: 2592f8a51a4e; files: src/agents/provider-http-errors.ts, src/plugin-sdk/provider-http.ts)
  • vincentkoc: Recent history shows Vincent Koc shared fal HTTP auth and bounded generated media downloads near the same fal media provider surface. (role: adjacent fal response hardening contributor; confidence: medium; commits: 667c03f87e71, 7d5dd8aad26d, 036298fbaec0; files: extensions/fal/music-generation-provider.ts, extensions/fal/video-generation-provider.ts)
  • harjothkhara: Recent history shows Harjoth Khara changed fal completed queue result parsing in the same video provider path this PR hardens. (role: recent adjacent fal video contributor; confidence: medium; commits: 34b81e935dbd; files: extensions/fal/video-generation-provider.ts, extensions/fal/video-generation-provider.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P2 Normal backlog priority with limited blast radius. labels Jun 26, 2026
@clawsweeper clawsweeper Bot added the merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. label Jun 26, 2026
The image, music, and video generation providers read successful fal API
JSON bodies via an unbounded await response.json() after the HTTP status
check. A hostile or buggy fal endpoint can stream an arbitrarily large
success body with no Content-Length, forcing the runtime to buffer the
entire payload in memory before parsing and risking OOM/hang.

Route all three success-path reads through the shared
readProviderJsonResponse helper, which reads the body through the same
16 MiB bounded reader used for binary responses and cancels the upstream
stream on overflow. The video path preserves its existing
FAL_VIDEO_MALFORMED_RESPONSE mapping for malformed JSON while letting the
overflow error propagate. Tests updated to stream real Response bodies.
@Alix-007 Alix-007 changed the title fix(fal): bound image/music/video generation response reads fix(fal): bound music/video generation response reads Jun 27, 2026
@vincentkoc vincentkoc self-assigned this Jun 29, 2026
@vincentkoc
vincentkoc merged commit ce1217a into openclaw:main Jun 29, 2026
92 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

extensions: fal merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: S status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants