feat(firecrawl): add keyless scrape support by vincentkoc · Pull Request #94551 · openclaw/openclaw

vincentkoc · 2026-06-18T10:55:46Z

Summary

Enable Firecrawl scraping and explicitly selected web_fetch fallback without an API key; send a bearer token only when one is configured for higher limits.
Selecting Firecrawl in the existing web-search onboarding/configure flow also selects Firecrawl for web_fetch, unless another fetch provider is already explicit.
Keep anonymous Firecrawl opt-in: existing credential-based auto-detection remains intact and empty configurations do not route outbound traffic to Firecrawl.

Release note: Firecrawl scrape and selected web_fetch fallback now support keyless starter access. Configured keys still apply higher account limits.

Credit: @developersdigest independently surfaced the keyless Firecrawl capability in #93590; this narrow canonical implementation preserves the existing firecrawl provider instead of introducing a parallel provider model.

Linked Context

Real Behavior Proof

Behavior addressed: keyless Firecrawl scrape and explicitly selected Firecrawl web-fetch fallback.
Real environment tested: live hosted Firecrawl endpoint on June 18, 2026.
Exact steps or command run after this patch: unauthenticated POST https://api.firecrawl.dev/v2/scrape with {"url":"https://example.com","formats":["markdown"]}.
Evidence after fix: HTTP 200 with success: true; no Authorization header was sent.
Observed result after fix: Firecrawl accepts starter scrape access without a configured key, while configured keys continue to be sent.
What was not tested: a full Gateway agent turn using a configured model; Firecrawl search remains key-required by design.

Verification

node scripts/run-vitest.mjs extensions/firecrawl/src/firecrawl-tools.test.ts src/secrets/runtime-web-tools.test.ts src/web-fetch/runtime.test.ts
node scripts/check-docs-mdx.mjs docs README.md
node scripts/docs-link-audit.mjs
node scripts/check-docs-i18n-glossary.mjs
node scripts/check-src-extension-import-boundary.mjs --json
node scripts/crabbox-wrapper.mjs run --provider aws --idle-timeout 90m --ttl 240m --timing-json -- env OPENCLAW_CHECK_CHANGED_REMOTE_CHILD=1 OPENCLAW_CHANGED_LANES_RAW_SYNC=1 corepack pnpm check:changed
- AWS Crabbox: cbx_516dca401270, run run_152e7d0e1f5e, exit 0.
.agents/skills/autoreview/scripts/autoreview --mode branch --base origin/main --no-web

Risk Checklist

User-visible behavior: yes, Firecrawl scrape and selected fallback can run without a key.
Config/upgrade behavior: existing empty configurations keep their current no-provider behavior; selecting Firecrawl is explicit.
Network behavior: the selected Firecrawl path makes anonymous hosted requests; existing target and base-URL guards remain in place.
Highest risk: Firecrawl starter-access policy can change upstream. The runtime preserves configured-key behavior and clear provider selection.

clawsweeper · 2026-06-18T10:57:25Z

Codex review: needs maintainer review before merge. Reviewed June 18, 2026, 12:16 PM ET / 16:16 UTC.

Summary
The PR changes Firecrawl scrape/fetch runtime behavior, web-search setup/configure selection, docs/tests, and undici pins so explicitly selected Firecrawl web_fetch can scrape without an API key while direct scrape and search remain credentialed.

PR surface: Source +87, Tests +278, Docs +12, Config 0, Other 0. Total +377 across 28 files.

Reproducibility: not applicable. as a bug reproduction; this is a feature PR. The feature premise is high-confidence because PR-head tests cover no-Authorization keyless scrape behavior and a fresh live unauthenticated Firecrawl scrape returned HTTP 200.

Review metrics: 2 noteworthy metrics.

Credential Requirement Surface: 1 changed. Firecrawl web_fetch now advertises optional credentials, which changes provider readiness and selection semantics.
Generated Config Side Effect: 1 added. Firecrawl search setup/configure can now populate the fetch provider when unset, which affects upgrade and operator expectations before merge.

Stored data model
Persistent data-model change detected: serialized state: extensions/firecrawl/src/firecrawl-tools.test.ts, vector/embedding metadata: extensions/firecrawl/src/firecrawl-tools.test.ts, vector/embedding metadata: src/secrets/runtime-web-tools.test.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: canonical
Canonical: #94551
Summary: This PR is the narrowed maintainer-owned implementation path for keyless Firecrawl scrape/fetch support; related external PRs overlap but include broader provider/search direction.

Members:

partial_overlap: feat(firecrawl): keyless firecrawl-free web search + scrape (opt-in) #93590 - It proposes keyless Firecrawl search and fetch via a separate firecrawl-free provider; this PR covers the narrower selected web_fetch scrape path on the existing provider.
superseded: feat(firecrawl): zero-config keyless Firecrawl search + scrape (no API key) #93462 - It was a closed earlier keyless Firecrawl attempt that made the keyless provider a zero-config default, later narrowed by the newer related PRs.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

[P2] Record maintainer acceptance of the anonymous hosted Firecrawl scrape/fallback boundary before merge.
Confirm the undici dependency floor update is intentional and acceptable.
[P2] Wait for exact-head required CI and CodeQL checks to complete before landing.

Risk before merge

[P2] Selected Firecrawl fallback can now send requested URLs and extracted page content to hosted Firecrawl without configured account credentials, so maintainers should explicitly accept the privacy, support, rate-limit, and upstream-policy boundary.
[P1] Selecting Firecrawl search now also writes tools.web.fetch.provider: "firecrawl" when unset, which changes setup/configure-generated config for users who choose Firecrawl search.
[P1] The PR includes a dependency floor update for undici; the lockfile diff looks bounded, but dependency graph changes remain a supply-chain review surface before merge.
[P1] Exact-head CI and CodeQL checks were still queued in the live PR status, so landing should wait for required checks to finish.

Maintainer options:

Accept The Narrowed Anonymous Boundary (recommended)
Maintainers can land this shape after explicitly accepting that selected Firecrawl fallback may send URLs and extracted content to hosted Firecrawl without configured account credentials.
Keep Account-Only Firecrawl Behavior
If anonymous hosted Firecrawl requests are not acceptable product policy, pause or close this PR and keep the current credential-required scrape/fallback behavior.
Remove The Setup Side Effect
If selecting Firecrawl search should not also select Firecrawl fetch, require users to set tools.web.fetch.provider explicitly while keeping the runtime keyless path.

Next step before merge

[P2] Manual review is needed because this maintainer-authored protected-label PR changes anonymous hosted network behavior, setup-generated config, and dependency pins; there is no narrow automated repair to queue.

Security
Needs attention: The diff intentionally broadens selected Firecrawl fallback to anonymous hosted requests and changes dependency pins, so maintainer security review is still needed even though no discrete patch bug was found.

Review details

Best possible solution:

Land the narrowed existing-provider implementation only after maintainers accept the anonymous hosted Firecrawl boundary, generated-config side effect, dependency floor update, and exact-head checks are green.

Do we have a high-confidence way to reproduce the issue?

Not applicable as a bug reproduction; this is a feature PR. The feature premise is high-confidence because PR-head tests cover no-Authorization keyless scrape behavior and a fresh live unauthenticated Firecrawl scrape returned HTTP 200.

Is this the best way to solve the issue?

Yes for the narrowed scrape/fetch scope: preserving the existing firecrawl provider and requiring explicit fetch selection is narrower than the related parallel firecrawl-free provider PR. The remaining blocker is maintainer product/security acceptance, not a better code location.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 508594a620f2.

Label changes

Label justifications:

P2: This is a normal-priority Firecrawl provider feature with bounded setup, docs, dependency, and network-boundary impact.
merge-risk: 🚨 compatibility: The PR changes Firecrawl web_fetch credential readiness and setup/configure-generated fetch-provider behavior for users selecting Firecrawl search.
merge-risk: 🚨 security-boundary: The PR intentionally enables selected anonymous outbound Firecrawl scrape/fallback requests without configured credentials.
rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body includes structured after-change live hosted Firecrawl proof, and an independent unauthenticated v2 scrape request returned HTTP 200 with no Authorization header in the request command.
proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes structured after-change live hosted Firecrawl proof, and an independent unauthenticated v2 scrape request returned HTTP 200 with no Authorization header in the request command.

Evidence reviewed

PR surface:

Source +87, Tests +278, Docs +12, Config 0, Other 0. Total +377 across 28 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	10	135	48	+87
Tests	6	286	8	+278
Docs	6	30	18	+12
Config	5	5	5	0
Generated	0	0	0	0
Other	1	22	22	0
Total	28	478	101	+377

Security concerns:

[medium] Anonymous Firecrawl Network Boundary — extensions/firecrawl/src/firecrawl-client.ts:208
Selected Firecrawl fallback can send requested URLs and extracted page content to hosted Firecrawl without configured account credentials; existing target and base-URL guards reduce SSRF risk, but maintainers should explicitly accept the privacy, support, rate-limit, and upstream-policy implications.
Confidence: 0.86

What I checked:

Repository policy applied: Root AGENTS.md plus docs/extensions scoped guides were read; their plugin boundary, config compatibility, dependency, docs, and external API proof guidance apply to this PR. (AGENTS.md:1, 508594a620f2)
Current main is credential-required: Current main rejects Firecrawl scrape without an API key and sends Authorization unconditionally in the shared Firecrawl POST helper, so the requested behavior is not already implemented. (extensions/firecrawl/src/firecrawl-client.ts:187, 508594a620f2)
Latest release is also credential-required: v2026.6.8 has the same required apiKey type, unconditional Authorization header, and missing-key scrape error, so the PR is not already shipped. (extensions/firecrawl/src/firecrawl-client.ts:187, 844f405ac1be)
PR head narrows keyless access to web_fetch: PR head makes the POST apiKey optional but only bypasses the missing-key error when the caller passes access: keyless; the Firecrawl web_fetch provider is the caller that sets that access mode. (extensions/firecrawl/src/firecrawl-client.ts:188, a0a9c896fa8c)
Explicit selection avoids no-key auto-detect: PR-head runtime code requires an auto-detect credential for keyless fetch providers while preserving explicit provider selection, matching the added tests for configured keyless Firecrawl and no-key omitted-provider behavior. (src/web-fetch/runtime.ts:148, a0a9c896fa8c)
Live external API proof: A fresh unauthenticated POST to hosted Firecrawl v2 scrape for example.com returned HTTP 200 with success true and no Authorization header in the request command.

Likely related people:

vincentkoc: Recent GitHub path history shows work on web_fetch runtime helpers and Firecrawl error handling, and this PR continues the same Firecrawl/runtime surface. (role: recent area contributor and current feature-history owner; confidence: high; commits: e122cd09ab0f, b962c53e785c, 8659eabba76a; files: src/web-fetch/runtime.ts, extensions/firecrawl/src/firecrawl-client.ts, src/flows/search-setup.ts)
steipete: GitHub path history ties this person to Firecrawl target safety, self-hosted endpoint policy, web-fetch runtime, and Firecrawl plugin docs around the affected boundary. (role: introduced behavior and adjacent provider-flow owner; confidence: high; commits: 189ab9f5d120, b66459e3c2d2, e4c127e678fc; files: extensions/firecrawl/src/firecrawl-client.ts, extensions/firecrawl/web-search-shared.ts, src/web-fetch/runtime.ts)
davemorin: Recent merged work kept key-free web search providers opt-in in the same runtime selection helper that this PR extends for Firecrawl fetch behavior. (role: recent adjacent policy contributor; confidence: medium; commits: c48b36a2556e; files: src/secrets/runtime-web-tools.shared.ts, src/flows/search-setup.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Co-authored-by: Developers Digest <[email protected]>

github-actions · 2026-06-18T16:06:28Z

Dependency Guard

This PR changes dependency-related files. Maintainers should confirm these changes are intentional.

Changed files:

extensions/browser/package.json
extensions/discord/package.json
extensions/qa-matrix/package.json
extensions/telegram/package.json
package.json
pnpm-lock.yaml

Maintainer follow-up:

Review whether the dependency changes are intentional.
Inspect resolved package deltas when lockfile, shrinkwrap, or workspace dependency policy changes are present.
Treat package-lock.json and npm-shrinkwrap.json diffs as security-review surfaces.
Run pnpm deps:changes:report -- --base-ref origin/main --markdown /tmp/dependency-changes.md --json /tmp/dependency-changes.json locally for detailed release-style evidence.

github-actions · 2026-06-18T16:06:29Z

Dependency graph changes noted

This PR includes dependency graph changes. The dependency guard is informational because the PR author is a repository admin or a member of @openclaw/openclaw-secops.

Current SHA: a0a9c896fa8c5d4881619d9d3716688e206a8de1
Trusted actor: @vincentkoc
Trusted role: pull request author; openclaw-secops

Security review is still recommended before merge when the dependency graph change is intentional.

vincentkoc · 2026-06-18T16:23:39Z

Landing under maintainer approval while the exact-head CI workflow is queue-bound, not failing.

Verification completed:

focused Firecrawl/onboarding/runtime Vitest suite: 156 passing tests
node scripts/pre-commit/pnpm-audit-prod.mjs --audit-level=high
AWS Crabbox pnpm check:changed: cbx_516dca401270, run_152e7d0e1f5e
native review artifacts validated for a0a9c896fa8c5d4881619d9d3716688e206a8de1

The prior security-fast failure was from current undici advisories; this PR raises the affected production floors to 8.5.0 and 7.28.0. The replacement workflow is still queued at merge time.

Merged under maintainer approval while exact-head CI was queue-bound. Co-authored-by: Developers Digest <[email protected]> Co-authored-by: Vincent Koc <[email protected]>

openclaw-barnacle Bot added docs Improvements or additions to documentation gateway Gateway runtime extensions: firecrawl size: M maintainer Maintainer-authored PR labels Jun 18, 2026

vincentkoc marked this pull request as ready for review June 18, 2026 10:56

vincentkoc requested a review from a team as a code owner June 18, 2026 10:56

clawsweeper Bot mentioned this pull request Jun 18, 2026

feat(firecrawl): keyless firecrawl-free web search + scrape (opt-in) #93590

Closed

vincentkoc force-pushed the feat/firecrawl-keyless branch from fdba6fe to d5ad417 Compare June 18, 2026 14:30

openclaw-barnacle Bot added the commands Command implementations label Jun 18, 2026

vincentkoc force-pushed the feat/firecrawl-keyless branch from d5ad417 to a4e675b Compare June 18, 2026 14:38

vincentkoc and others added 8 commits June 19, 2026 00:02

feat(firecrawl): add keyless scrape support

6c037ee

Co-authored-by: Developers Digest <[email protected]>

fix(configure): preserve selected fetch provider

879cb74

fix(onboarding): apply keyless provider selection

bd9f4a9

fix(onboarding): disable unconfigured search

0380190

docs(firecrawl): select keyless fetch fallback

ddd5539

fix(firecrawl): restrict keyless scraping to web fetch

636ebc3

fix(firecrawl): keep direct scraping credentialed

be07b52

docs(firecrawl): explain keyless scrape boundary

f9181c0

vincentkoc and others added 2 commits June 19, 2026 00:02

fix(configure): reenable credential-ready search

a608b8a

fix(deps): update undici security floor

a0a9c89

vincentkoc force-pushed the feat/firecrawl-keyless branch from a4e675b to a0a9c89 Compare June 18, 2026 16:06

vincentkoc self-assigned this Jun 18, 2026

openclaw-barnacle Bot added channel: discord Channel integration: discord channel: telegram Channel integration: telegram labels Jun 18, 2026

github-actions Bot added the dependencies-changed PR changes dependency-related files label Jun 18, 2026

vincentkoc merged commit beea31a into main Jun 18, 2026
46 of 48 checks passed

vincentkoc deleted the feat/firecrawl-keyless branch June 18, 2026 16:23

github-actions Bot mentioned this pull request Jun 18, 2026

📡 Upstream Digest — 2026-06-18 16:51 UTC curtismercier/openclaw-mods#1100

Open

This was referenced Jun 26, 2026

feat(firecrawl): add keyless Firecrawl Search (Free) provider + richer firecrawl_search options #97078

Merged

test(firecrawl): add unit tests for firecrawl client functions #97714

Merged

This was referenced Jul 6, 2026

[Bug]: Keyless web_fetch providers are skipped during auto-detection if no credentials exist #100659

Closed

fix(web-fetch): auto-detect keyless providers without credentials #100937

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

feat(firecrawl): add keyless scrape support#94551

feat(firecrawl): add keyless scrape support#94551
vincentkoc merged 10 commits into
mainfrom
feat/firecrawl-keyless

vincentkoc commented Jun 18, 2026

Uh oh!

clawsweeper Bot commented Jun 18, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

vincentkoc commented Jun 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

vincentkoc commented Jun 18, 2026

Summary

Linked Context

Real Behavior Proof

Verification

Risk Checklist

Uh oh!

clawsweeper Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 18, 2026

Dependency Guard

Uh oh!

github-actions Bot commented Jun 18, 2026

Dependency graph changes noted

Uh oh!

vincentkoc commented Jun 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

clawsweeper Bot commented Jun 18, 2026 •

edited

Loading