Spider Developer Platform

Collect web data at scale. Spider handles crawling, rendering, proxy rotation, and anti-bot evasion, you get clean data back through a single API.

Developer quickstartInstall the SDK, export your API key, and make your first request.3 mins
Request
import requests, json

headers = {
    'Authorization': 'Bearer $SPIDER_API_KEY',
    'Content-Type': 'application/json',
}

json_data = {"limit":5,"url":"https://example.com"}

response = requests.post('https://api.spider.cloud/crawl', 
  headers=headers, json=json_data, stream=True)

with response as r:
    r.raise_for_status()
    
    buffer = b""

    for chunk in response.iter_content(chunk_size=8192):
        if chunk:
            buffer += chunk
            try:
                data = json.loads(buffer.decode('utf-8'))
                print(data)
                buffer = b""
            except json.JSONDecodeError:
                continue
CrawlFollow links across entire sites. Set depth, limit, and domain scope.
ScrapeFetch a single page as HTML, markdown, text, or structured JSON.
SearchSearch the web and scrape the results in one request.
ScreenshotCapture full-page screenshots with Chrome rendering.
StreamingProcess pages as they finish instead of waiting for the full result.
AI ExtractionExtract structured data from any page using AI or CSS selectors.
Data ConnectorsStream results directly to S3, Google Cloud, Azure Blob, Sheets, or Supabase.
Anti-Bot BypassAutomatic fingerprint rotation, stealth mode, and retry engine to bypass bot protection.
Proxy ModeIntelligent geo-routing across residential, ISP, and mobile proxies in 100+ countries.
Browser CloudFull cloud browsers via CDP WebSocket. Playwright and Puppeteer compatible with stealth, proxies, and recording.
Fetch API (Alpha)AI-configured per-website scrapers. Discovers selectors automatically, caches and reuses.

How It Works

Every request goes through three stages: fetch (retrieve the page using HTTP or headless Chrome), process (render JavaScript, rotate proxies, handle anti-bot challenges), and deliver (convert to your chosen format and return). Spider's Rust-based engine runs all stages concurrently, so a 500-page crawl takes seconds, not hours.

API Endpoints

All endpoints accept JSON and return JSON. Authenticate with a Bearer token.

MethodPathDescription
POST/crawlStart from a URL and follow links to discover and fetch multiple pages.
POST/scrapeFetch a single page and return its content in any format.
POST/searchSearch the web and optionally scrape the results.
POST/screenshotCapture a full-page screenshot as base64 PNG.
POST/fetch/{domain}/{path}AI-configured per-website scraper with cached configs. (Alpha)
GET/data/scraper-directoryBrowse optimized scraper configs for popular websites.
HTTPproxy.spider.cloudRoute requests through intelligent residential, ISP, or mobile proxies.
WS/v1/browserConnect a Playwright or Puppeteer client to a cloud browser via CDP.

Request Modes

Choose how Spider fetches each page. smart (default) automatically picks between HTTP and Chrome based on the page. Use http for static HTML, it is the fastest and cheapest. Use chrome when you need JavaScript rendering, SPA support, or real browser fingerprints for bot-protected sites. See Concepts for details.

Proxy Mode

Route any Spider request through proxy.spider.cloud for intelligent proxy management. Spider automatically selects the best proxy pool, rotates IPs, and handles geo-routing based on the target site — no manual configuration required. Choose from residential (real-user IPs across 100+ countries), ISP (stable datacenter IPs, highest throughput), or mobile (real 4G/5G device IPs for maximum stealth). Use the country_code parameter for geolocation targeting and proxy to select a pool. Proxy Mode works with Crawl, Scrape, Screenshot, Search, and Links. See the Proxy-Mode API reference for pricing and configuration.

Browser Cloud

Spider also provides full cloud browsers accessible over a CDP WebSocket at wss://browser.spider.cloud/v1/browser?token=YOUR-API-KEY. Connect any Playwright or Puppeteer client with connectOverCDP() for full page control, AI extraction, and automation. Sessions include built-in stealth, proxy rotation, and optional session recording. 100 concurrent browsers on all plans. See the Browser API reference for examples and configuration. Use the spider-browser npm package for a ready-made client.

Credits

Usage is measured in credits at $1 / 10,000 credits. Each page costs a base amount, with additional credits for Chrome rendering, proxy usage, and AI extraction. Every response includes a costs object with a per-request breakdown. Monitor your balance on the usage page.

Build now

JavaScript
npm install
Python
pip install
Rust
cargo add
CLI
terminal

Explore our guides

  • A practical walkthrough for collecting web data with Spider, from your first crawl to production pipelines.

    Jeff Mendez ·

  • An overview of Spider's API capabilities, endpoints, request modes, output formats, and how to get started.

    Jeff Mendez ·