Expanding the Language and Cultural Coverage of Common Crawl

EXPANDING THE LANGUAGE AND CULTURAL COVERAGE OF COMMON CRAWL P. Ortiz Suarez∗, G. Lindahl, T. Vaughan, S. Nagel Common Crawl Foundation, Beverly Hills , United States of America The Common Crawl Foundation is a nonprofit organization that has been operating since 2007. Its mission is to preserve and freely share samples of the public Internet. Common Crawl is a key partner to the AI community, as well as many other research communities. Our over ten-petabyte archive provides most of the web data used to train LLMs. Our crawling has always been polite and ethical, and strictly obeying robots.txt . We thus believe that improving Common Crawl’s language diversity as well as its cultural and community diversity, will directly benefit everyone from the AI to the crawling and archiving communities. The Common Crawl Foundation has already been working on linguistic diversity with academic and industry partners, including Occiglot, HPLT, MLCommons, the Allen Institute (Ai2), the AI Alliance, the Linux Foundation AI and Data Foundation, and many more. However, while these efforts have already contributed to the cultural and linguistic coverage of our corpus, from our own statistics, we know that our data has always been biased towards English content making our dataset difficult to use for individuals and organizations from smaller linguistic communities. We have always wanted to make Common Crawl as representative as possible of the Open Web, so we present here two projects on which we have been working and that we hope will allow us to expand the language and cultural coverage of our crawls, making it more representative of the actual linguistic and cultural diversity found on the web. Both projects will require input from the community, as our team is small and we speak but a handful of languages, and as we also believe that the languages and the content written in them belong in the end to their respective linguistic communities. The first initiative that we introduce here is the Web Languages project 1 , which asks culturally-literate speakers to work together to make a list of important websites for different languages, cultures, and communities. We have asked for input for nearly 8,000 languages. These curated lists are then used by our web crawler to find clusters of linked websites which are important to the given culture or community. Even languages with very few web pages can be effectively crawled using this methodology. This type of human collaboration and curation is a mature idea, and Common Crawl’s team has successfully used this approach in the past. Success of this project depends upon collaborating with a wide range of people, recruited in collaboration with universities, companies, governments, and other organizations. ∗[email protected] 1https://github.com/commoncrawl/web-languages Figure 1: Annotation interface for the LangID Project The second project is an annotation campaign for Language Identification (LangID) 2 that we are conducting in collaboration with MLCommons. In this campaign we are asking participants to annotate a subset of Common Crawl data. We would like as many annotations, and cover as many languages as possible, in order to create the first webbased LangID dataset. Our goal is to train a small language classifier that would help us steer our crawling towards underrepresented languages at crawl time. This concept is a mature idea with which we have already experimented on a small list of languages. Success of this project world-wide depends on collaborating with a wide range of people, who can be recruited in the same way as the first project. These two projects are interconnected, and mutually complementary. The first finds communities, including regional communities that mainly have web content in a national language. The second project uses a different mechanism that can find web pages with under-represented language content wherever they are on the Internet, even if they are not connected to the main community and cultural clusters found by the first project. With these two initiatives we hope to expand the access to research and technologies that our dataset has already made possible for high-resource linguistic and cultural communities, and make them available to communities all around the world. We will present the findings and challenges that we have encountered while conducting these two projects and how our crawl coverage has evolved since we started working on these initiatives. 2https://dynabench.org/tasks/text-language-identification https://doi.org/10.5281/zenodo.17238236