HTTP scraping toolkit: one typed
fetch()over a reactive httpx → curl_cffi → Patchright fallback chain — TLS/JA3 impersonation, JS rendering, interactive browser sessions with DevTools capture (console, network, HAR, screenshots, video), structured-source discovery, and a typed error taxonomy behind a singleResponse.
Claude Code's built-in WebFetch exposes no header parameters, so callers cannot set User-Agent, Accept, or Referer. And headers alone often aren't enough: on a 2026-07-06 re-probe, hamiltoncompany.com returned 403 to curl even with a full browser UA, but 200 to polyfetch's tiers — the block keys on the client fingerprint, not the UA string. polyfetch is the next rung — browser-shape headers in the cheap tier, real TLS impersonation in the middle, headless Chromium with anti-detection patches as the fallback — escalating only when a tier is actually blocked.
uv add git+https://github.com/qte77/polyfetch-scrape # not published to PyPI
uv run patchright install chromium # one-off; only for the patchright tierOr borrow it without installing — see USING.md. On Alpine/musl the browser tier can't run (no musllinux wheel); see CLI-only commands.
from polyfetch_scrape import fetch
r = fetch("https://example.com")
print(r.status, r.backend) # e.g. 200 httpx — which tier answers depends on the target- One call, typed result.
fetch(url)returns a typedResponsewhichever backend succeeded — no per-site tool picking. - Reactive three-tier fallback.
httpx→curl_cffi(Chrome TLS/JA3 impersonation) → Patchright (headless Chromium), escalating only on a real block; pin or bound it withtier/min_tier/max_tier. - Typed error taxonomy. Terminal statuses (
401/404/407/410/451) raise typed exceptions;429and500/502/503/504retry, honouringRetry-After. - Browser tier with DevTools capture. Waits, screenshots, scripted actions, device/locale/colour-scheme emulation, video and HAR recording, headed launch, and console/network/JS-error capture — opt-in on
fetch, always on in the interactiverender_session. - Requests and politeness. Conditional GET (
etag/last_modified),json/contentrequest bodies, a per-hostThrottle, and an opt-in robots.txt check. - Structured-first discovery.
discover(url)reports sitemaps, RSS/Atom/iCal feeds,llms.txtand JSON-LD@types, so consumers parse structured data instead of HTML. - Library, CLI, or env-borrow from another repo or agent (
USING.md).
Full options: API reference. Empirical anti-bot findings: scraping landscape.
Screencast — polyfetch's render_session driving a headless browser through a live site: browse pagination → open the login form → fill and submit → land logged-in (one frame per step)
Regenerate with make screencast (examples/navigate_screencast.py) — uses only the public render_session API.
- The engine — the stable, typed surface:
fetch,render_sessionanddiscover. Most callers only need this. - The scripting substrate —
render_session(url).pagehands you the live, instrumented stealth-PatchrightPage(full DevTools/CDP surface) for app-specific flows. polyfetch owns install, launch/teardown, capture and the SSRF guard; you own the app-specific steps.
Which side a knob belongs on is settled by the ownership line; worked .page recipes live in the scripting cookbook, and examples/ shows the pattern (examples, not stable API).
I am a: Library user | CLI user | Script author | Agent / sideload | Contributor
polyfetch fetch https://example.com --json
polyfetch fetch https://quotes.toscrape.com/js/ --tier patchright --screenshot viewport --screenshot-out shot.png
polyfetch fetch https://example.com --max-tier curl_cffi # cap escalation — never launch a browser
polyfetch bulk urls.txt --workers 4 --delay 0.5 # ≥0.5 s between same-host requests
polyfetch discover https://example.com --json # sitemaps / feeds / llms.txt / JSON-LD
polyfetch doctor --fix # verify (and install) the browser-tier ChromiumEvery command and flag (request bodies, device emulation, video/HAR, --headed, …) and the --json schema: USING.md → Commands.
| Tier | Backend | When it engages |
|---|---|---|
| 1 | httpx |
every request first |
| 2 | curl_cffi (chrome impersonation) |
tier 1 returns 403 or hits a TLS error |
| 3 | Patchright (headless Chromium) | tier 2 also blocked |
Response.backend says which tier answered. make demo_tiers shows what each tier uniquely provides; details in architecture.
fetch, render_session, RenderOptions, RenderAction, Screenshot, Response, RetryPolicy, Throttle and the FetchError hierarchy, plus utils.discovery.discover and utils.robots.is_allowed (imported by module path) — full signatures in the API reference. _backends/ is private; contrib/ extras (e.g. easter_hunt) are unsupported and never imported by core.
make setup_dev && make validate (plus make audit). Full workflow, command reference and release pipeline: CONTRIBUTING.md; agent rules: AGENTS.md.
| Instead of… | polyfetch gives you |
|---|---|
httpx / requests alone |
the same simple call, plus automatic TLS impersonation and a browser fallback when a site blocks the plain client |
curl_cffi / cloudscraper alone |
TLS impersonation and a real headless browser behind one typed Response — no manual tier-picking |
| raw Playwright / Patchright | the browser install, launch/teardown, capture, retries and SSRF guard already wired; drop to .page only when you need to |
| a hosted scraping API | a dependency you run in-process: no per-request cost, no data leaving your machine, no vendor account |
Each row is a trade-off, not a claim of superiority. See the scraping landscape for dated, empirical probes.
- Out of scope (use a dedicated tool): proxy / residential-IP rotation, LLM-ready HTML→markdown extraction, a hosted scraping service, automated CAPTCHA solving, and domain API wrappers (arXiv, patents, …) — those belong in downstream packages that consume
fetch(). - Not in the engine, but scriptable on
render_session().page: accessibility snapshots, multi-step walks, ad-hoc element screenshots, live viewport changes — see Two layers.
- Public API reference — full signatures for
fetch,RenderOptions,Response,RetryPolicy, and the exception hierarchy - Scripting cookbook — worked
render_session().pagerecipes (DevTools capture, aria_snapshot, walks, live emulation) - Using without installing — call polyfetch from another repo/agent via
uv run --directory(env-borrow contract: invocation, JSON schema, errors, stable surface) - Consuming across the estate — the estate model, the ownership line, and the rule for when a consumer need becomes core
- Architecture — fallback-chain data flow, component responsibilities, invariants
- Roadmap — delivery history + core directions ahead
- User stories — who it serves and what each need maps to
- Scraping landscape — tool comparison + empirical findings
- web-recon-kit — web/API recon harness; its browser tier builds on
render_session()(optional git-dependency extra) - gha-rxiv-feed-action — fetch arXiv/bioRxiv/medRxiv feeds (open APIs; polyfetch's fallback chain isn't needed for these)
- Changelog — release notes (Keep a Changelog format)
- Codespaces reference — cross-qte77 Codespaces auth, signing and devcontainer setup
Licensed under the Apache-2.0 license (SPDX: Apache-2.0); see also NOTICE.