Apple AI released a $7 billion open-source language model trained with 2.5 trillion tokens on open datasets

Language models (LMs) have become central to natural language processing (NLP) as they enable text generation, translation, and sentiment analysis tasks. These models require large amounts of training data to perform accurately and efficiently. However, the quality and curation of these datasets are critical to the performance of LMs. This field focuses on refining data collection and preparation methods to improve the effectiveness of models.

A major challenge in developing effective language models is improving training datasets. High-quality datasets are essential for training models that generalize well across a variety of tasks, but creating such datasets is complex. It involves filtering out irrelevant or harmful content, removing duplicates, and selecting the most useful data sources.

Existing methods for dataset curation typically involve heuristic-based filtering, deduplication, and obtaining data from extensive web crawls. While these methods have shown some success, they often require more standardized benchmarks, leading to consistency in evaluating the performance of language models. This variability makes it difficult to determine the most effective data curation strategies, hampering progress in the field.

Researchers from Apple, the University of Washington and many other institutions have presented DataComp for Language Models (DCLM) To address these issues, they have recently open sourced the DCIM models and datasets on the Hugging Face platform. The open source version includes DCLM-7B, DCLM-1B, dclm-7b-it, DCLM-7B-8k, dclm-baseline-1.0, and dclm-baseline-1.0-parquet. This innovative testbed enables controlled experiments on large datasets to improve language models. The DCLM framework includes a comprehensive corpus of 240 trillion Common Crawl tokens, effective pre-training recipes based on the OpenLM framework, and a rich set of 53 post-evaluations. This setup provides a standardized approach to dataset curation, enabling consistent and comparable experiments.

DCLM offers a structured workflow for researchers. Participants can choose scales ranging from 412M to 7B parameters and experiment with data curation strategies such as deduplication, filtering, and data shuffling. Researchers can train models on selected datasets using a standardized training recipe and specific hyperparameters. The performance of these models is then evaluated on a set of downstream tasks, providing a clear measure of dataset quality. This systematic approach helps identify the most effective data curation strategies.

The introduction of DCLM has led to notable improvements in language model training. For example, a benchmark dataset built using DCLM enabled the training of a 7B parameter language model from scratch. This model achieved a 5-shot accuracy of 64% on the MMLU benchmark with 2.6 trillion training tokens. This performance represents a 6.6 percentage point improvement over the previous state-of-the-art open data language model, MAP-Neo, while using 40% fewer computational resources. The DCLM benchmark also performed comparably to Mistral-7B-v0.3 and Llama 3 8B, which required significantly more computational resources.

The effectiveness of the DCLM framework is further demonstrated by its scalability. The researchers conducted extensive experiments at different scales, from 400 million to over 7 billion parameters, using DCLM-Pool, a corpus of 240 trillion tokens derived from Common Crawl. These experiments highlighted the critical role of model-based filtering in assembling high-quality training sets. The DCLM benchmark dataset, created through this rigorous process, consistently outperformed other open-source datasets such as RefinedWeb and RedPajama in various evaluations.

The research team also explored the impact of several data curation techniques. They compared text extraction methods such as resilipse and trafilature and found that these approaches significantly improved subsequent performance compared to pre-extracted text from Common Crawl. The team investigated several model-based quality filtering strategies and ultimately determined that the fastText OH-2.5 + ELI5 classifier was the most effective, providing a substantial increase in accuracy.

In conclusion, the introduction of DCLM enables researchers to conduct controlled experiments and identify the most effective strategies for improving language models by providing a standardized and systematic approach to dataset curation. The DCLM framework sets a new benchmark for dataset quality and demonstrates the potential to achieve significant performance improvements with reduced computational resources.