Common Crawl Foundation

At #EMNLP2026 in Budapest? Come see our poster at the Web-as-Corpus workshop (WaC-13) on 29 Oct. Counting documents is not counting text. 📄 arxiv.org/abs/2608.16390 @commoncrawl.bsky.social #WebData #Crawl

We are absolutely thrilled to announce the release of our September 2026 Web Graph, comprising data from July, August, and September 2026. commoncrawl.org/blog/host--a...

We are pleased to announce the release of the September 202 6 crawl archive, consisting of 2.17 billion pages (or 361.4 TiB of uncompressed content.) commoncrawl.org/blog/septemb...

We made parts of our crawl archive available experimentally on a Hugging Face Storage Bucket. This blog post explains how to use the data on Hugging Face, with tools and examples on which you can build. commoncrawl.org/blog/getting...

Notes from the CommonsDB final conference in Alicante: a registry of 6.5 million open works keyed on ISCC, and the EUIPO's push for federated copyright infrastructure. commoncrawl.org/blog/commons...

We are publishing an experimental dataset of graph embeddings built from the Common Crawl Web Graph: a 128-dimensional vector for each of 52.9 million web hosts, learned from hyperlinks alone, along with two interactive Hugging Face Spaces. commoncrawl.org/blog/web-gra...

We analyzed the contents of 584,107 llms.txt files from the July 2026 crawl. Two thirds are produced by a plugin, half follow the structure the specification defines, and a few files even contain prompt injections. commoncrawl.org/blog/a-conte...

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of June, July, and August 2026. commoncrawl.org/blog/host--a...

We are pleased to announce that the crawl archive for August 2026 is now available, containing 2.14 billion web pages or 360 TiB of uncompressed content. commoncrawl.org/blog/august-...

The Common Crawl team attended the 64th Annual Meeting of the Association for Computational Linguistics in San Diego, California, presenting recent published work, and strengthening ties with the research community. commoncrawl.org/blog/common-...

Over a year ago we released cc-downloader, an experimental tool to politely download Common Crawl data. Today we're releasing its first stable version, with a Rust library and Python bindings. commoncrawl.org/blog/announc...

Two weeks in Basel and Vienna, at the HTTP Workshop and IETF 126. Protocol adoption measured across the whole web, and an attempt to define what "machine readable" actually means. commoncrawl.org/blog/notes-f...

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of May, June, and July 2026. commoncrawl.org/blog/host--a...

The crawl archive for July 2026 is now available. The data was crawled between July 7th and July 25th, and contains 2.14 billion web pages (or 364.01 TiB of uncompressed content). We also announce some improvements and changes. commoncrawl.org/blog/july-20...

Common Crawl has joined Project Tapestry, a global initiative led by the AI Alliance to advance open, sovereign AI. We will contribute our expertise in responsible web data, multilingual coverage and culturally informed AI development. commoncrawl.org/blog/common-...

How can we measure how many pages we’ve crawled from a particular website? The answer is a lot more complicated than you might think. commoncrawl.org/blog/measuri...

We probed the top 1,000,000 web hosts for IPv6 from five vantage points on three continents. 31.7% work from everywhere, one vantage point turns out to be enough for the headline rate, and 5,530 hosts reveal why it isn't enough for the rest of the story. commoncrawl.org/blog/is-one-...

Turning 30,000 Arabic Domains Into a Better Crawl: How we filtered, geolocated and categorised a donation of Arabic seed domains commoncrawl.org/blog/turning...

The Common Crawl team attended the 16th International Conference on Language Resources and Evaluation in Palma, Mallorca, co-organizing a tutorial, presenting recent published work, and strengthening links with the research community. commoncrawl.org/blog/common-...

The latest @commoncrawl.bsky.social crawl indexes way more than HTML pages. 20.9M PDFs. Plus calendars, BibTeX, markdown... The whole index now lives in a @hf.co Bucket, so I pulled this with one SQL query straight over it (new S3 API). 2.1B rows, nothing downloaded, $0 to read.

The WaC-13 workshop invites research submissions on web data, corpus building, and linguistic analysis. commoncrawl.org/blog/13th-we...

The May 2026 crawl archive (CC-MAIN-2026-21) is now also available on our HF bucket. 🤗 huggingface.co/buckets/comm...

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of April, May, and June 2026. commoncrawl.org/blog/host--a...

We are happy to announce the release of the June 2026 crawl archive, consisting of 2.10 billion web pages, or 354.59 TiB of uncompressed content. commoncrawl.org/blog/june-20...

CommonLID, a community-built language ID benchmark, has a new website and interactive leaderboard. Its paper was accepted to ACL 2026, with a poster session on 7 July. Source code, a PyPI package, and the dataset are now available. commoncrawl.org/blog/commonl...

Common Crawl was well represented with contributions at the 2026 IIPC Web Archiving Conference and General Assembly. commoncrawl.org/blog/common-...

We are very happy to announce our next seminar: Pedro Ortiz Suarez @pjox.bsky.social ( @commoncrawl.bsky.social ) "Expanding Linguistic and Cultural Coverage in Common Crawl" on Friday 12th June 2026, 11am CEST. Details here 👉 almanach.inria.fr/seminars-en....

The Columnar Index Is Now the URL Index! We have renamed the Columnar Index to the URL Index, to be clearer about its purpose and to pave the way for more datasets in a columnar format. commoncrawl.org/blog/the-col...

Introducing the AI Visibility Audit! A free guide for SEOs and GEOs on how to check whether AI systems can actually reach a site, and how to stay visible in the crawl that trains them. commoncrawl.org/blog/introdu...

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of March, April, and May 2026. commoncrawl.org/blog/host--a...

We are happy to announce the release of the May 2026 crawl archive, consisting of 2.16 billion web pages, or 365.56 TiB of uncompressed content. www.commoncrawl.org/blog/may-202...

As an early experiment in distributing Common Crawl data through another channel, the April 2026 crawl archive is now available in a Hugging Face Storage Bucket, alongside its existing home on AWS S3. commoncrawl.org/blog/april-2...

Browsers can now fetch Common Crawl data directly, no backend needed. Build SQL explorers, snapshot viewers and diff tools as static pages. commoncrawl.org/blog/you-can...

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of February, March, and April 2026. commoncrawl.org/blog/host--a...

We are pleased to announce that the crawl archive for April 2026 is now available, containing 2.19 billion web pages or 379.2 TiB of uncompressed content. commoncrawl.org/blog/april-2...

Check out our newsletter for April 2026, with updates on what we've been up to. www.commoncrawl.org/blog/april-2...

Common Crawl is switching to reporting dataset sizes in nibbles. As an organisation dedicated to data preservation, we feel it would be remiss to allow this underrepresented unit to fall out of use. Our latest crawl now exceeds 689 tebibbles. commoncrawl.org/blog/announc...

We are pleased to announce a new release of host-level and domain-level web graphs based on the crawls of January, February, and March 2026. commoncrawl.org/blog/host--a...

We are pleased to announce the release of the March 2026 crawl, containing 1.97 billion web pages, or 344.64 TiB of uncompressed content. We also observed a dramatic increase in fetches over IPv6, explained by the enabling of Happy Eyeballs in the OkHttp library. commoncrawl.org/blog/march-2...

Blocking the Internet Archive Won’t Stop AI, But It Will Erase the Web’s Historical Record www.eff.org/deeplinks/20... @eff.org

We probed the 100,000 most-linked web hosts for IPv6 support using the Common Crawl Web Graph. Only 36.9% are fully reachable over IPv6, with adoption ranging from 71% among the top 100 to 32% in the long tail. commoncrawl.org/blog/ipv6-ad...

Our Web Graph Statistics site has been updated with interactive charts, a domain lookup tool for tracking harmonic centrality and PageRank over time, mobile improvements, unified rank tables with OR filtering, and merged degree plots. commoncrawl.org/blog/web-gra...

@thomvaughan.bsky.social did a WCAG colour contrast audit of 240 top domains using Common Crawl's February 2026 archive. You can read more about his study in the thread below

Introducing the second installment in our Whirlwind Tour series, covering crawl structure, index access, and content extraction, giving developers a practical foundation for building Java-based data workflows. commoncrawl.org/blog/announc...

We're happy to announce the release of the Web Graphs for December 2025 and January/February 2026, consisting of 288.6 million nodes and 12.4 billion edges at the host level, and 134.2 million nodes and 5.4 billion edges at the domain level. www.commoncrawl.org/blog/host--a...

We've replaced our old Examples and Use Cases pages with a single searchable, filterable browser. 119 resources from 115 contributors, all in one place. Search, filter by type or language, sort, and share links. We welcome community submissions. blog.commoncrawl.org/blog/introdu...

We are pleased to announce the release of the February 2026 crawl, consisting of 2.1 billion web pages (or 363 TiB of uncompressed content). Captures are from 45.5 million hosts or 37.1 million registered domains. blog.commoncrawl.org/blog/februar...

Preserving The Web Is Not The Problem. Losing It Is. Mark Graham, Director of the Wayback Machine at @archive.org, walks us through the importance of preserving the Web in this recent post: www.techdirt.com/2026/02/17/p...