Skip to content

feat(firecrawl): keyless firecrawl-free web search + scrape (opt-in)#93590

Closed
developersdigest wants to merge 4 commits into
openclaw:mainfrom
developersdigest:feat/firecrawl-free-keyless
Closed

feat(firecrawl): keyless firecrawl-free web search + scrape (opt-in)#93590
developersdigest wants to merge 4 commits into
openclaw:mainfrom
developersdigest:feat/firecrawl-free-keyless

Conversation

@developersdigest

@developersdigest developersdigest commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Firecrawl, zero-setup. Adds a keyless firecrawl-free provider for both web_search and web_fetch — clean Firecrawl results and page scrapes with no API key, no account. Previously Firecrawl was keyed-only.
  • Now possible because Firecrawl shipped a keyless tier (/v2/search and /v2/scrape answer anonymously).
  • Opt-in, not default. Selectable via tools.web.{search,fetch}.provider: firecrawl-free (or onboarding). autoDetectOrder keeps it behind the existing keyless defaults, so a fresh install's behavior is unchanged. No default/auto-selection change.
  • No paid-credential coupling. The keyless providers advertise an empty credentialPath and requiresCredential: false, so secret resolution never targets the paid Firecrawl key (this adds genuine keyless support to the web-fetch provider contract, matching web-search).
  • Additive and safe: the keyed firecrawl providers and firecrawl_search / firecrawl_scrape tools are unchanged; any configured keyed provider still wins; no new config keys.

What should reviewers focus on?

  • The keyless path in firecrawl-client.ts (no Authorization, openclaw-firecrawl UA, key optional); keyless credential handling (empty credentialPath, requiresCredential: false); and provider registration in the public-artifact barrels + manifest.

Linked context

Was this requested by a maintainer or owner? No — external contribution.

Real behavior proof (required for external PRs)

  • Behavior addressed: keyless web_search and web_fetch return live results with no API key.
  • Real environment tested: macOS, built from source; live calls to api.firecrawl.dev/v2/search and /v2/scrape with no Authorization header.
  • Exact steps or command run after this patch: pnpm build, then exercised the real provider path keyless — runFirecrawlSearch({ keyless: true, query: "openclaw github" }) and runFirecrawlScrape({ keyless: true, url: "https://example.com" }) — plus the firecrawl-tools tests and the web-fetch provider contract suite.
  • Evidence after fix:
    • search → 3 real results (openclaw.ai, github, reddit), provider: firecrawl-free, 200, no key
    • scrape → # Example Domain markdown, 200, no key
    • default selection: with no keys, the auto-detect sort picks firecrawl-free for both search and fetch
  • Observed result after fix: keyless requests resolve to firecrawl-free and return real, safety-wrapped content.
  • What was not tested: the agent/LLM deciding to invoke the tool (needs a configured model — orthogonal to this change); cross-OS beyond macOS.
  • Proof limitations or environment constraints: free keyless results vary over time; per-Firecrawl rate limits apply.
  • Before evidence: keyless requests returned 401; the providers threw "needs a Firecrawl API key".

Tests and validation

  • firecrawl-tools unit tests (35 passed), web-fetch provider contract suite (3 passed incl. firecrawl-free), pnpm build, live keyless search/scrape.
  • Regression: keyless path asserts no Authorization + the openclaw-firecrawl UA; registration contract updated for the new firecrawl-free search + fetch ids.

Risk checklist

  • User-visible behavior change? No default changefirecrawl-free is opt-in (selectable), not auto-selected; the existing keyless default is unchanged.
  • Config / migration change? No — no new config; reuses tools.web.{search,fetch}.provider; registered as non-migrated in doctor.
  • Security / network change? Yes, only when explicitly selected — anonymous outbound to api.firecrawl.dev (already an allowed host), via existing web-tools guards; no credentials sent; keyless providers never target the paid credential path.
  • Highest risk + mitigation: a new external-network path to Firecrawl when a user opts in. Mitigated by being opt-in (no default change), keyless (no credentials), and routed through existing web-tools endpoint guards.

Current review state

  • Next action: maintainer review, with attention to the keyless-default change.
  • Outstanding: full-app (TUI/agent) selection proof pending a configured model; provider-level live proof included above.

Adds a keyless firecrawl-free provider for both web_search and web_fetch via
Firecrawl's keyless tier (no API key, no account). Selectable opt-in
(tools.web.{search,fetch}.provider); not the default. Keyless providers carry
no credential path (requiresCredential: false), so secret resolution never
targets the paid Firecrawl key; adds genuine keyless support to the web-fetch
provider contract, mirroring web-search. Keyed firecrawl providers unchanged.
@openclaw-barnacle openclaw-barnacle Bot added commands Command implementations extensions: firecrawl size: M proof: supplied External PR includes structured after-fix real behavior proof. labels Jun 16, 2026
@clawsweeper

clawsweeper Bot commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Thanks for the context here. I swept through the related work, and this is now duplicate or superseded.

Close as superseded: the useful Firecrawl keyless scrape/fetch work is now tracked by the narrower maintainer PR, while this branch adds a broader parallel firecrawl-free provider model and leaves keyless search as an unsponsored product direction.

Root-cause cluster
Relationship: superseded
Canonical: #94551
Summary: The related maintainer PR is the canonical path for Firecrawl keyless scrape/fetch support and explicitly credits this contribution while narrowing the implementation.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Canonical path: Use #94551 as the canonical Firecrawl keyless scrape/fetch path, and handle keyless Firecrawl search only with explicit maintainer sponsorship.

So I’m closing this here and keeping the remaining discussion on #94551.

Review details

Best possible solution:

Use #94551 as the canonical Firecrawl keyless scrape/fetch path, and handle keyless Firecrawl search only with explicit maintainer sponsorship.

Do we have a high-confidence way to reproduce the issue?

Not applicable as a bug reproduction: this is a feature PR. The external Firecrawl premise is high-confidence because live unauthenticated /v2/search and /v2/scrape calls returned HTTP 200.

Is this the best way to solve the issue?

No; the broader parallel firecrawl-free provider model is no longer the best path. The narrower maintainer PR preserves the existing Firecrawl provider contract and carries proof for the accepted keyless scrape/fetch surface.

Security review:

Security review needs attention: The diff intentionally adds anonymous Firecrawl network paths; the superseding maintainer PR narrows that boundary to selected scrape/fetch.

  • [medium] Anonymous Firecrawl search/fetch network boundary — extensions/firecrawl/src/firecrawl-free-search-provider.ts:28
    The new provider can send user search queries, requested URLs, and fetched page content to hosted Firecrawl without credentials, so maintainer policy needs to accept that broader privacy and support boundary before any equivalent path lands.
    Confidence: 0.86

AGENTS.md: found and applied where relevant.

What I checked:

Likely related people:

  • steipete: Commit history shows the Firecrawl onboarding/search plugin and docs were introduced in the original Firecrawl plugin commit. (role: introduced Firecrawl plugin; confidence: high; commits: aa28d1c71138; files: extensions/firecrawl/src/firecrawl-client.ts, extensions/firecrawl/src/firecrawl-search-provider.ts, docs/tools/firecrawl.md)
  • vincentkoc: History ties this person to the web-fetch provider boundary, recent current-main touches, and the open canonical Firecrawl keyless PR. (role: web-fetch provider boundary and canonical follow-up owner; confidence: high; commits: 38d2faee20af, 9b082cb1bb1a, fdba6feae720; files: src/web-fetch/runtime.ts, src/secrets/runtime-web-tools.ts, extensions/firecrawl/src/firecrawl-client.ts)
  • davemorin: History shows recent key-free web-search opt-in policy work that this PR attempts to mirror. (role: recent adjacent policy contributor; confidence: medium; commits: c48b36a2556e; files: src/web-search/runtime.ts, src/secrets/runtime-web-tools.ts, src/secrets/runtime-web-tools.shared.ts)

Codex review notes: model internal, reasoning high; reviewed against b0375f25b31b.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. labels Jun 16, 2026
mirror the web_search opt-in policy so the keyless firecrawl-free fetch
provider is never auto-detected for no-key installs and is only used when
explicitly selected via tools.web.fetch.provider. addresses clawsweeper P1
security-boundary review on openclaw#93590.
@developersdigest
developersdigest requested a review from a team as a code owner June 16, 2026 18:40
@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation and removed proof: sufficient ClawSweeper judged the real behavior proof convincing. labels Jun 16, 2026
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jun 16, 2026
@openclaw-barnacle openclaw-barnacle Bot added channel: discord Channel integration: discord channel: whatsapp-web Channel integration: whatsapp-web app: web-ui App: web-ui gateway Gateway runtime agents Agent runtime and tooling channel: feishu Channel integration: feishu extensions: minimax extensions: amazon-bedrock labels Jun 16, 2026
@developersdigest

Copy link
Copy Markdown
Contributor Author
image

@vincentkoc vincentkoc self-assigned this Jun 18, 2026
@clawsweeper clawsweeper Bot added the proof: 📸 screenshot Contributor real behavior proof includes screenshot evidence. label Jun 18, 2026
@vincentkoc

Copy link
Copy Markdown
Member

The canonical scoped implementation has landed in #94551 at beea31a.

It keeps the existing firecrawl provider and limits unauthenticated hosted access to explicitly selected web_fetch; direct Firecrawl scrape and search remain credentialed. The squash commit preserves your co-author credit.

Closing this broader parallel firecrawl-free provider implementation as superseded. A future keyless-search proposal should be separately sponsored and scoped around its privacy and routing policy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling app: web-ui App: web-ui channel: discord Channel integration: discord channel: feishu Channel integration: feishu channel: whatsapp-web Channel integration: whatsapp-web commands Command implementations docs Improvements or additions to documentation extensions: amazon-bedrock extensions: firecrawl extensions: minimax gateway Gateway runtime merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. P2 Normal backlog priority with limited blast radius. proof: 📸 screenshot Contributor real behavior proof includes screenshot evidence. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. size: L status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants