Common Crawl

Common Crawl operates as a 501(c)(3) non-profit organization dedicated to maintaining an open repository of web crawl data. Since its founding in 2008 by computer scientist Gil Elbaz, the group has systematically crawled the internet and made petabytes of raw HTML, metadata, and extracted text freely available to the public. The data is hosted on Amazon Web Services public S3 buckets; users pay only for their own compute and storage costs rather than licensing fees.

The organization's corpus stands as the largest openly accessible web dataset, serving researchers, journalists, startups, and major AI labs. In the 2020s it became the primary training source for large language models including GPT-3, GPT-4, and LLaMA. A November 2025 investigation by The Markup and Wired examined how AI companies use the data, sparking debates about consent and copyright that led to new transparency measures. [تعديل]

Common Crawl aims to democratize access to web data by crawling the internet and providing the resulting content free of charge to anyone. The goal is to lower barriers to web-scale data analysis, foster innovation, and preserve an open historical record of the web. [تعديل]

Gil Elbaz began building the first crawler in 2007, driven by the belief that open web data was essential for a healthy internet. The first public crawl dataset, containing about 2 billion pages, was released in 2008. Early infrastructure relied on donated servers and grants. In 2012, Common Crawl partnered with Amazon Web Services to host the data in public S3 buckets, making access free and scalable. The News Crawl dataset launched in 2015, providing a continuously updated feed of news articles. By 2017 the corpus had grown to 3.5 billion pages, the largest publicly available web dataset at the time. The AI boom of the early 2020s dramatically increased demand; Common Crawl became the de facto training corpus for large language models. In response to copyright and consent concerns raised by a 2025 investigation, the organization announced plans to develop an opt-out system and better provenance tracking. [تعديل]