Skip to content
@commoncrawl

Common Crawl Foundation

Common Crawl provides an archive of webpages going back to 2007.

Pinned Loading

  1. cc-pyspark cc-pyspark Public

    Process Common Crawl data with Python and Spark

    Python 424 89

  2. cc-crawl-statistics cc-crawl-statistics Public

    Statistics of Common Crawl monthly archives mined from URL index files

    Python 177 11

  3. cc-index-table cc-index-table Public

    Index Common Crawl archives in tabular format

    Java 115 9

  4. cc-warc-examples cc-warc-examples Public

    Forked from Smerity/cc-warc-examples

    CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop

    Java 38 18

  5. cc-citations cc-citations Public

    Scientific articles using or citing Common Crawl data

    Jupyter Notebook 19 3

  6. cc-notebooks cc-notebooks Public

    Various Jupyter notebooks about Common Crawl data

    Jupyter Notebook 51 9

Repositories

Showing 10 of 66 repositories
  • wac2025-webgraph-workshop Public

    Introduction to WebGraphs - Workshop at the IIPC Web Archiving Conference 2025

    commoncrawl/wac2025-webgraph-workshop’s past year of commit activity
    2 MIT 0 0 0 Updated Apr 2, 2025
  • cc-webgraph Public

    Tools to construct and process Common Crawl webgraphs

    commoncrawl/cc-webgraph’s past year of commit activity
    Java 88 Apache-2.0 5 2 (1 issue needs help) 0 Updated Apr 2, 2025
  • cc-webgraph-statistics Public

    Statistics of Common Crawl monthly Web Graphs

    commoncrawl/cc-webgraph-statistics’s past year of commit activity
    Python 3 Apache-2.0 0 0 1 Updated Apr 2, 2025
  • cc-notebooks Public

    Various Jupyter notebooks about Common Crawl data

    commoncrawl/cc-notebooks’s past year of commit activity
    Jupyter Notebook 51 Apache-2.0 9 0 0 Updated Apr 1, 2025
  • nutch Public Forked from Aloisius/nutch

    Common Crawl fork of Apache Nutch

    commoncrawl/nutch’s past year of commit activity
    Java 32 Apache-2.0 1,258 6 (1 issue needs help) 0 Updated Apr 1, 2025
  • web-languages Public

    Crowd-sourced lists of urls to help Common Crawl crawl under-resourced languages. See https://github.com/commoncrawl/web-languages-code/ for the code

    commoncrawl/web-languages’s past year of commit activity
    38 40 0 0 Updated Apr 1, 2025
  • cc-citations Public

    Scientific articles using or citing Common Crawl data

    commoncrawl/cc-citations’s past year of commit activity
    Jupyter Notebook 19 3 0 0 Updated Mar 30, 2025
  • cc-crawl-statistics Public

    Statistics of Common Crawl monthly archives mined from URL index files

    commoncrawl/cc-crawl-statistics’s past year of commit activity
    Python 177 Apache-2.0 11 0 0 Updated Mar 29, 2025
  • robotstxt-experiments Public

    How is the Robots Exclusion Protocol (robots.txt) used in the WWW? This projects tries to get some insights mining Common Crawl's robots.txt captures of the years 2016 – 2024.

    commoncrawl/robotstxt-experiments’s past year of commit activity
    Jupyter Notebook 0 MIT 0 0 0 Updated Mar 27, 2025
  • cc-downloader Public

    A polite and user-friendly downloader for Common Crawl data

    commoncrawl/cc-downloader’s past year of commit activity
    Rust 36 Apache-2.0 1 2 (1 issue needs help) 0 Updated Mar 22, 2025
View all repositories

Top languages

Loading…