apify / crawlee-python Public

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

crawlee.dev/python/

Apache-2.0 license

5.5k stars 370 forks Branches Tags Activity

Notifications

Error
Looks like something went wrong!

About

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

crawlee.dev/python/

python crawler scraper automation web-crawler headless scraping crawling pip web-scraping beautifulsoup web-crawling hacktoberfest headless-chrome apify playwright

Apache-2.0 license

Custom properties

Report repository

Releases 38

Packages

No packages published

Used by 113

Contributors 31

+ 17 contributors

Languages