An MCP server that scrapes and maps web pages over a loopback HTTP endpoint, escalating through progressively heavier fetch tiers only when a lighter one fails.
English · 简体中文
Report a bug · Request a feature
Table of Contents
crawl4ai-mcp exposes scrape, crawl, map and diagnose as MCP tools over
HTTP, so an assistant can fetch a page without the caller choosing how. The
choosing is the point: src/crawl4ai_mcp/cascade.py starts at the cheapest
tier a URL has previously worked with and escalates only on failure, through a
plain HTTP client, a headless browser, Camoufox and finally a paid provider,
so most pages cost one request and the expensive path is reserved for the
ones that need it. crawl walks a site breadth-first, same-origin only, up to
a page and depth limit.
It binds to 127.0.0.1 and refuses hosts it was not configured for, which makes
it a local tool rather than a service: there is no authentication layer, because
nothing off the machine is meant to reach it.
See the open issues for planned features and known issues.
- Python 3.12 or newer, the floor declared in
pyproject.toml - A browser runtime for the middle tiers: Chromium via
crawl4ai, plus Camoufox if the optionalcamoufoxextra is installed - For the packaged service, a Linux host with systemd user sessions, since
systemd/crawl4ai-mcp.serviceinstalls under the user manager rather than the system one - API credentials only if the paid tiers are enabled; see
.env.example
git clone https://github.com/anyingiit/crawl4ai-mcp.git
cd crawl4ai-mcp
python -m venv .venv && . .venv/bin/activate
pip install -e ".[test]"
crawl4ai-setupcrawl4ai-setup downloads the Chromium build crawl4ai drives; skip it only
if CRAWL4AI_MODE=api is set and no browser-backed tier will run.
To run it as a background service on Linux instead, first create .env from
.env.example and restrict it to owner-only, which scripts/install-user-service.sh
requires even when no paid provider is enabled:
cp .env.example .env
chmod 600 .envThen scripts/install-user-service.sh installs the systemd unit and waits for
the health endpoint to answer.
Start the server in the background, then check that it is up:
crawl4ai-mcp &
sleep 1
curl -fsS http://127.0.0.1:11236/healthThe MCP endpoint is served at /mcp on the same port. Point an MCP client at
http://127.0.0.1:11236/mcp and call scrape with a URL; pass max_tier to cap
how far the cascade is allowed to escalate.
Contributions are welcome. Read CONTRIBUTING.md for how to open an issue or a pull request, and CODE_OF_CONDUCT.md for the standards expected of everyone taking part.
Please do not report security issues in public issues or pull requests. SECURITY.md explains how to report them privately.
Distributed under the MIT License. See LICENSE for details.
Project link: https://github.com/anyingiit/crawl4ai-mcp