fix(firecrawl): reject malformed 2xx response envelopes by Alix-007 · Pull Request #111210 · openclaw/openclaw

Alix-007 · 2026-07-19T04:30:12Z

What Problem This Solves

Fixes an issue where Firecrawl Search and Scrape could receive a successful HTTP response with a null or array JSON body, then either return an invalid empty result or throw a raw TypeError instead of a stable provider error.

Why This Change Was Made

The shared Firecrawl response reader now uses readProviderJsonObjectResponse, which enforces the top-level object envelope already required by Firecrawl v2. The existing bounded body reader, endpoint guards, error labels, cache behavior, and valid payload handling remain unchanged.

User Impact

Malformed upstream responses fail with the existing provider-owned malformed JSON response boundary. Valid Firecrawl responses continue through the same Search and Scrape paths.

Evidence

AI-assisted implementation: Codex.
Base: main at 5e51c4bbcca3c3af767ebf1d74e2fcfd0c398195.
Head: e48d9f1247446226a5b96d5396b338f4d0ded99f.
Focused tests: node scripts/run-vitest.mjs extensions/firecrawl/src/firecrawl-tools.test.ts extensions/firecrawl/src/firecrawl-client.test.ts --run -> 2 files, 101/101 tests passed.
Formatting and static hygiene: targeted oxfmt and git diff --check passed.
Controlled HTTP after-fix proof (trusted candidate checkout, loopback server only, no credentials): the server returned HTTP 200 with null for /v2/search and HTTP 200 with [] for /v2/scrape; the real runFirecrawlSearch and runFirecrawlScrape entry paths produced these exact redacted outputs:
```
MALFORMED_SHAPE search-null error="Firecrawl Search API error: malformed JSON response"
MALFORMED_SHAPE scrape-array error="Firecrawl fetch failed: malformed JSON response"
```
Reproducible public-run capture: proof workflow run 29673678349 checked the immutable candidate head and reported 2 files / 101 tests passed plus the same two MALFORMED_SHAPE lines.
Keyless production probe ran the real runFirecrawlScrape entry against https://api.firecrawl.dev/v2/scrape with no API key and returned an object result with extractor=firecrawl, extractMode=markdown, textLength=304, and the expected Example Domain content.
The same proof command also checked the live endpoint contract and emitted: LIVE_API status=200 topLevel=object successType=boolean dataType=object; the successful production entry emitted: LIVE_PRODUCTION resultType=object extractor=firecrawl extractMode=markdown textLength=304 exampleDomain=true.
Firecrawl's official Search response and Scrape response contracts both describe a 200 JSON object with a boolean success and object data; null and arrays are outside that contract.
Codex autoreview was attempted with the configured Sol and documented Terra fallback, but the account endpoint returned HTTP 401 before a review bundle could be evaluated. I completed a frozen-diff manual audit of the changed helper, both callers, the shared provider contract, and the focused tests; no actionable findings remained.

clawsweeper · 2026-07-19T04:37:11Z

Codex review: needs changes before merge. Reviewed July 19, 2026, 11:38 PM ET / July 20, 2026, 03:38 UTC.

Summary
The branch switches Firecrawl Search and Scrape to the shared top-level-object JSON reader and adds regressions for null and array HTTP 200 bodies.

PR surface: Source 0, Tests +69. Total +69 across 2 files.

Reproducibility: yes. from source: current main’s generic reader accepts parsed null and arrays while Firecrawl callers expect object properties; the PR body also supplies controlled entry-path output for both malformed envelope shapes.

Review metrics: none identified.

Merge readiness
Overall: 🦪 silver shellfish
Proof: 🦞 diamond lobster
Patch quality: 🦪 silver shellfish
Result: blocked by patch quality or review findings.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

Extend the shared object reader to forward optional byte-limit options.
[P2] Add a focused limit-preservation regression and rerun the Firecrawl tests and changed typecheck lane.
[P2] Refresh the controlled malformed-envelope proof on the repaired exact head if the repair changes runtime helper behavior.

Risk before merge

[P1] The current patch cannot typecheck because the object-reader API has no opts argument. A superficial removal of that argument would lower v2/scrape from its intentional 64 MiB allowance to the generic 16 MiB limit, causing valid larger scrape responses to fail.

Maintainer options:

Forward the existing Scrape byte cap (recommended)
Add an optional maxBytes parameter to the shared object-envelope reader, pass it to the generic reader, and cover the Firecrawl Scrape limit before merging.
Pause rather than narrow the cap
Do not merge a version that drops the Scrape-specific limit merely to satisfy the current two-argument helper signature.

Copy recommended automerge instruction

@clawsweeper automerge

Special instructions:
Extend `readProviderJsonObjectResponse` to accept and forward the optional byte-limit options, preserve Firecrawl Scrape’s 64 MiB limit, add focused coverage, and rerun the Firecrawl tests plus the changed typecheck lane.

Next step before merge

[P2] A narrow mechanical repair can preserve the shared object-envelope design and the existing Scrape response-size contract without product or API-direction judgment.

Security
Cleared: The diff reuses an existing SDK reader and adds test coverage; it introduces no dependency, permission, secret, artifact-execution, or supply-chain change.

Review findings

[P1] Forward the Scrape byte-limit option through the object reader — extensions/firecrawl/src/firecrawl-client.ts:75

Review details

Best possible solution:

Extend the shared object-envelope reader with the same optional byte-limit argument as the generic reader, forward it unchanged, and retain Firecrawl’s 64 MiB Scrape limit while rejecting non-object successful envelopes.

Do we have a high-confidence way to reproduce the issue?

Yes, from source: current main’s generic reader accepts parsed null and arrays while Firecrawl callers expect object properties; the PR body also supplies controlled entry-path output for both malformed envelope shapes.

Is this the best way to solve the issue?

No, not as written: using the shared object reader is the best boundary, but it must preserve the caller’s existing byte-limit option rather than call an incompatible two-argument API.

Full review comments:

[P1] Forward the Scrape byte-limit option through the object reader — extensions/firecrawl/src/firecrawl-client.ts:75
readProviderJsonObjectResponse currently accepts only two parameters, so this three-argument call fails typechecking. Removing opts to make it compile would also discard Scrape’s intentional 64 MiB allowance and fall back to the generic 16 MiB cap; extend the shared helper to accept and forward the option, then add a regression for that preserved limit. (github.com)
Confidence: 0.99

Overall correctness: patch is incorrect
Overall confidence: 0.99

AGENTS.md: unclear because the file could not be read completely.

Codex review notes: model internal, reasoning high; reviewed against 200653bd60c8.

Label changes

Label changes:

add merge-risk: 🚨 availability: Changing Scrape from its explicit 64 MiB response allowance to the shared 16 MiB default would make otherwise valid larger responses fail at runtime.
add rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦞 diamond lobster and patch quality is 🦪 silver shellfish.
add status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (terminal): The PR body provides controlled loopback output from the real Search and Scrape entry paths, a public workflow capture, and a keyless production-path probe; redact private endpoints or credentials in any future refreshed capture.
remove rating: 🐚 platinum hermit: Current PR rating is rating: 🦪 silver shellfish, so this older rating label is no longer current.
remove status: 👀 ready for maintainer look: Current PR status label is status: ⏳ waiting on author.

Label justifications:

P2: This is a bounded optional-plugin correctness repair with a clear merge blocker but no demonstrated broad production outage.
merge-risk: 🚨 availability: Changing Scrape from its explicit 64 MiB response allowance to the shared 16 MiB default would make otherwise valid larger responses fail at runtime.
rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦞 diamond lobster and patch quality is 🦪 silver shellfish.
status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (terminal): The PR body provides controlled loopback output from the real Search and Scrape entry paths, a public workflow capture, and a keyless production-path probe; redact private endpoints or credentials in any future refreshed capture.
proof: sufficient: Contributor real behavior proof is sufficient. The PR body provides controlled loopback output from the real Search and Scrape entry paths, a public workflow capture, and a keyless production-path probe; redact private endpoints or credentials in any future refreshed capture.

Evidence reviewed

PR surface:

Source 0, Tests +69. Total +69 across 2 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	1	2	2	0
Tests	1	76	7	+69
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	2	78	9	+69

Acceptance criteria:

[P1] node scripts/run-vitest.mjs extensions/firecrawl/src/firecrawl-tools.test.ts extensions/firecrawl/src/firecrawl-client.test.ts --run.
[P1] pnpm check:changed -- src/agents/provider-http-errors.ts extensions/firecrawl/src/firecrawl-client.ts extensions/firecrawl/src/firecrawl-tools.test.ts.

What I checked:

Current-main behavior: Current main’s Firecrawl client still calls the generic JSON reader in readFirecrawlJsonResponse; that reader parses null and arrays despite the nominal Record cast, so the requested object-envelope enforcement is not already implemented. (extensions/firecrawl/src/firecrawl-client.ts:71, 200653bd60c8)
Shared helper contract: readProviderJsonObjectResponse accepts only (response, label) and invokes the generic reader without an options parameter. The PR’s readProviderJsonObjectResponse(response, label, opts) call has an extra argument, which TypeScript rejects; its existing 64 MiB scrape cap cannot be preserved without extending the helper. (github.com) (src/agents/provider-http-errors.ts:1634, 200653bd60c8)
Caller and behavior boundary: The two Firecrawl entry paths use the same local reader: Search consumes the envelope before result extraction, while Scrape explicitly supplies its 64 MiB body limit before parsing and then caches only parsed successful results. (github.com) (extensions/firecrawl/src/firecrawl-client.ts:2605, 200653bd60c8)
External provider contract: Firecrawl’s current v2 Search and Scrape documentation shows successful 200 responses as JSON objects containing boolean success and object data; rejecting top-level null and arrays is therefore consistent with the provider contract. (docs.firecrawl.dev)
Extension boundary: The extension imports the shared provider HTTP SDK surface rather than a core internal, matching the applicable extension guidance to reuse SDK helpers for shared provider behavior. (github.com) (extensions/firecrawl/src/firecrawl-client.ts:2, 66afbea4507c)

Likely related people:

vincentkoc: Release history credits this person for the Firecrawl web-fetch migration into the plugin-owned provider boundary, which is the surrounding ownership seam for this client behavior. (role: adjacent feature owner; confidence: medium; files: extensions/firecrawl/src/firecrawl-client.ts, extensions/firecrawl/src/config.ts)
steipete: The current PR head includes a commit authored by this person that updates the Firecrawl test fixture to the canonical plugin configuration shape, making them a relevant routing candidate for the exact test and SDK-contract repair. (role: recent area contributor; confidence: medium; commits: 66afbea4507c; files: extensions/firecrawl/src/firecrawl-tools.test.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (5 earlier review cycles)

reviewed 2026-07-19T04:33:21.525Z sha e48d9f1 :: needs real behavior proof before merge. :: none
reviewed 2026-07-19T04:42:34.540Z sha e48d9f1 :: needs maintainer review before merge. :: none
reviewed 2026-07-19T05:25:06.589Z sha e48d9f1 :: needs maintainer review before merge. :: none
reviewed 2026-07-20T03:09:03.659Z sha d9d653f :: needs maintainer review before merge. :: none
reviewed 2026-07-20T03:30:17.106Z sha c71727a :: needs maintainer review before merge. :: none

Alix-007 · 2026-07-19T04:39:12Z

@clawsweeper re-review

clawsweeper · 2026-07-19T04:39:15Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The durable review comment was published and its verdict was queued for target-serialized routing.
Run: https://github.com/openclaw/clawsweeper/actions/runs/29673734488
Updated: 2026-07-19T04:46:37.390Z

Co-authored-by: Alix-007 <[email protected]>

steipete · 2026-07-20T03:42:05Z

Merged via squash.

Prepared head SHA: 66afbea4507cddecbcb854e0274c9047d686f642
Landed commit: 990916c6ac877e62cfa9e07ba562d58eb6402487

) * fix(firecrawl): reject malformed JSON envelopes * test(firecrawl): use canonical plugin config fixture Co-authored-by: Alix-007 <[email protected]> --------- Co-authored-by: Peter Steinberger <[email protected]>

openclaw-barnacle Bot added extensions: firecrawl size: S labels Jul 19, 2026

Alix-007 force-pushed the alix/firecrawl-response-shape branch 2 times, most recently from 7884d71 to d9d653f Compare July 20, 2026 03:05

clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. and removed rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. labels Jul 20, 2026

Alix-007 and others added 2 commits July 20, 2026 11:33

fix(firecrawl): reject malformed JSON envelopes

7433074

test(firecrawl): use canonical plugin config fixture

66afbea

Co-authored-by: Alix-007 <[email protected]>

Alix-007 force-pushed the alix/firecrawl-response-shape branch from c71727a to 66afbea Compare July 20, 2026 03:34

steipete merged commit 990916c into openclaw:main Jul 20, 2026
91 of 97 checks passed

clawsweeper Bot added the merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. label Jul 20, 2026

github-actions Bot mentioned this pull request Jul 20, 2026

🦞 OpenClaw 生态日报 2026-07-20 96loveslife/big_model_radar#188

Open

github-actions Bot mentioned this pull request Jul 20, 2026

📡 Upstream Digest — 2026-07-20 06:38 UTC curtismercier/openclaw-mods#1352

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(firecrawl): reject malformed 2xx response envelopes#111210

fix(firecrawl): reject malformed 2xx response envelopes#111210
steipete merged 2 commits into
openclaw:mainfrom
Alix-007:alix/firecrawl-response-shape

Alix-007 commented Jul 19, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jul 19, 2026 •

edited

Loading

Uh oh!

Alix-007 commented Jul 19, 2026

Uh oh!

clawsweeper Bot commented Jul 19, 2026 •

edited

Loading

Uh oh!

Uh oh!

steipete commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

Alix-007 commented Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Uh oh!

clawsweeper Bot commented Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Alix-007 commented Jul 19, 2026

Uh oh!

clawsweeper Bot commented Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

steipete commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Alix-007 commented Jul 19, 2026 •

edited

Loading

clawsweeper Bot commented Jul 19, 2026 •

edited

Loading

clawsweeper Bot commented Jul 19, 2026 •

edited

Loading