Extract web data with natural language

Describe what you need in plain English. Spider's AI endpoints crawl, scrape, search, and automate the web. Get structured data back, ready for your application.

Get Started Free API Documentation

5AI Endpoints

JSONStructured Output

REST+ Python, Node, Go

HOW IT WORKS

Three steps to structured data

Send a prompt

Pass a URL and describe what you want to extract in plain English. No selectors or XPath needed.

AI processes the page

Spider renders the page in a real browser, then runs your prompt through AI to extract matching data.

Get structured JSON

Receive clean, structured data ready for your database, API, or AI pipeline.

ENDPOINTS

Five ways to extract with AI

Each endpoint is purpose-built for a specific extraction pattern, all driven by natural language prompts.

AI Crawl

POST /ai/crawl

Crawl websites intelligently using natural language prompts. Let AI guide the crawler to find and extract exactly what you need.

Natural language crawl guidanceSmart page discoveryAI-powered content filteringStructured data extraction

prompt: "Find all product pages and extract pricing information"

View Documentation

AI Scrape

POST /ai/scrape

Extract structured data from any webpage using plain English. Describe what you need, and AI handles the parsing.

Plain language data extractionAutomatic schema detectionMulti-format outputContext-aware parsing

prompt: "Extract all article titles, authors, and publish dates"

View Documentation

AI Search

POST /ai/search

Search the web with AI-enhanced queries. Get smarter, more relevant results with natural language understanding.

Semantic search queriesIntent understandingRelevance rankingStructured search results

prompt: "Find recent news about AI startups in healthcare"

View Documentation

AI Browser

POST /ai/browser

Automate browser interactions using natural language. Configure complex web automation without writing scripts.

Natural language automationSmart element detectionAction sequencingDynamic page handling

prompt: "Click the login button, fill the email field, and submit"

View Documentation

AI Links

POST /ai/links

Extract and filter links intelligently using natural language. Find specific types of links without complex CSS selectors.

AI-guided link filteringCategory detectionRelevance rankingDeep link discovery

prompt: "Find all links to product pages and documentation"

View Documentation

INTEGRATION

One API call, structured results

Add a prompt field to any request. Works with all Spider SDKs and your existing API key.

Request

$ curl -X POST \
  https://api.spider.cloud/ai/scrape \
  -H "Authorization: Bearer $API_KEY" \
  -d '{
    "url": "https://example.com",
    "prompt": "Extract product names and prices"
  }'

Response

[
  {
    "name": "Widget Pro",
    "price": "$49.99"
  },
  {
    "name": "Gadget Plus",
    "price": "$29.99"
  },
  {
    "name": "Tool Max",
    "price": "$79.99"
  }
]

Python, Node.js, Go SDKs

Uses your existing API key

Pay-per-use pricing

WHY SPIDER AI

Built for developers

No selectors needed

Forget XPath, CSS selectors, and brittle scraping scripts. Describe your data in natural language and let AI handle the extraction.

Works with any site

JavaScript-rendered pages, SPAs, and dynamic content. Spider uses full browser rendering before AI extraction.

Structured output

Get clean JSON with exactly the fields you asked for. Define a schema or let AI infer the structure from your prompt.

Scale on demand

From a single page to crawling entire sites. The same API handles it all with built-in concurrency and rate limiting.

Start extracting in minutes

Get Started Free View Pricing