Agno Integration

Use Spider as a tool inside Agno agents. Agents can scrape, crawl, and search the web with a single tool call.

Install

Install Agno, Spider, and OpenAI:

Install the Python Spider client

pip install agno spider-client openai

Scrape a URL

Pass a URL to the agent and it scrapes the page, returning content in markdown.

Simple Scrape Example in Markdown

from agno.agent import Agent from agno.tools.spider import SpiderTools agent = Agent(tools=[SpiderTools(optional_params={"proxy_enabled": True})]) # Optionally pass in extra parameters agent.print_response('Scrape https://www.cnn.com for the latest news"', markdown=True)
Agno Agent Result scraping CNN.com

Search and Scrape

The agent searches the web, then scrapes the top result and returns the content.

Search and Scrape Results Example

from agno.agent import Agent from agno.tools.spider import SpiderTools agent = Agent(tools=[SpiderTools()]) agent.print_response('Scrape the first result from a search on "current news in the USA"', markdown=True)
Agno search and scrape results for 'current news in the USA'. The response shows a breakdown from CNN's news section.

Toolkit Parameters

Configure SpiderTools behavior.

  • max_results (int): The maximum number of search results to return
  • url (str): The URL to be scraped or crawled

Toolkit Functions

Available methods the agent can call.

  • search: Searches the web for the given query
  • scrape: Scrapes the given URL
  • crawl: Crawls the given URL