OLMo 2: fully open source basic model

OLMo 2 models are Ai2's fully open source language models. They have dense autoregressive architectures with optimized training, pre-training data combinations, and advanced instruction tuning techniques. By addressing training stability and improving per-token efficiency, OLMo 2 sets a benchmark in performance and transparency. The introduction of Dolmino Mix 1124, a specialized data blend for late-stage curricular training, further enhances downstream capabilities. Together with the best practices of Tülu 3, OLMo 2-Instruct achieves impressive results, competing against Llama 3.1 and Qwen 2.5. Let's learn more about these models!

<figure class="wp-block-embed is-type-rich is-provider-twitter wp-block-embed-twitter“>

2 OLAMo 2 Furious

OLMo 2 builds on the foundation laid by its predecessors and offers completely open language models with parameter sizes of 7 billion and 13 billion. Unlike many industry peers, OLMo 2 ensures complete transparency, publishing training data, codes, recipes, and even intermediate checkpoints. This commitment not only accelerates academic and industrial research, but also fosters a collaborative ai development ecosystem.

These models compete strongly with industry giants such as Llama 3.1 and Qwen 2.5 and use fewer computational resources. Their performance places them on the Pareto frontier, where efficiency meets excellence, making them invaluable for various downstream applications.

You can find everything about the model in this research paper: 2 OLAMo 2 Furious.

Key Features of OLMo 2 Models

Improved training stability

Training large-scale language models often encounters instabilities such as loss spikes. OLMo 2 addresses these challenges through:

• Data curation: Filtering repeated n-grams to minimize gradient spikes and loss.
• Improved initialization: Switch to a standardized initialization scheme that maintains stability between layers.
• Regularization Techniques: Incorporating z-loss to stabilize the output logits.

These adjustments result in a smoother training process, allowing models to handle larger data sets more efficiently.

Optimized data mixes

OLMo 2 pre-training incorporates a two-stage approach:

• Pre-training stage: It uses a combination of high-quality web data totaling 5 billion tokens.
• Intermediate stage of training: Introduces domain-specific data sets, particularly in the mathematics and STEM fields, to reinforce specialized capabilities. The Dolmino Mix 1124 dataset exemplifies this strategy, combining curated and web-sourced data for targeted performance improvements.

Architectural advances

OLMo 2 integrates modern innovations to improve your transformer architecture, including:

• RMSnorm: A stable normalization method for activations.
• Reordered layer rule: Normalize attention outputs and feedback layers, improving stability.
• Higher positional coding resolution: Adoption of rotating positional embeddings with higher resolution for better sequence handling.

These features collectively increase the scalability and efficiency of the model.

Post-training excellence

OLMo 2's post-training process, inspired by Tülu 3's recipe, focuses on instruction tuning and reinforcement learning. Key components include:

• Supervised Tuning (SFT): Leverage high-quality prompts to hone instruction-following capabilities.
• Reinforcement Learning with Verifiable Rewards (RLVR): Optimize performance on specific tasks such as mathematics and factual reasoning by rewarding correct results.

This approach has resulted in OLMo 2-Instruct models that excel on benchmarks such as GSM8K for mathematical reasoning and MMLU for multitasking language understanding.

Efficiency meets transparency

OLMo 2 stands out for its efficient use of computational resources. By reducing FLOPs (floating point operations) during training, you achieve high performance with lower environmental impact. Detailed reporting on energy consumption and carbon emissions underlines the project's commitment to sustainability.

Infrastructure as a research catalyst

The success of the project is also attributed to Ai2's advanced infrastructure:

• High performance clusters: Leverage cutting-edge hardware, including NVIDIA H100 GPUs, across multiple data centers.
• Beaker Workload Management: Ensure smooth workload distribution and monitoring.

These infrastructure investments have significantly reduced training interruptions and increased resource utilization.

OLMo 2 vs Qwen 2.5 vs Llama 3.1 vs Others

To further illustrate its impact, OLMo 2 benchmarks often outperform Qwen 2.5 and Llama 3.1 on specific tasks. The inclusion of Dolmino Mix 1124 has significantly improved performance on STEM and math-based tests. Additionally, OLMo 2 demonstrates notable efficiency gains, using up to 20% less FLOP and achieving comparable or superior results.

Let's try OLMo 2

To access the model you can visit here. You can use it without logging in.

Immediate: You are in a hurry to work. You pour yourself a cup of black coffee, but it's too hot. You intend to add a set amount of cold milk, but you know that even after that, the coffee will need to cool for a few minutes before you can drink it.
In which case the coffee gets colder:
1) Add milk immediately and then wait a few minutes before drinking.
2) Wait a few minutes and then add the milk just before drinking.

Production:

Observation: The response to my message is correct. OLMo 2 was able to understand the problem and give the correct answer. DeepSeek V3 was unable to resolve this correctly in my previous article on DeepSeek V3 vs Claude Sonnet 3.5.

You can also use this model locally, just follow the mentioned instructions. here.

Important links

Conclusion

OLMo 2 showcases the remarkable potential of open source ai, setting new standards in transparency and innovation. By publishing your code, data, and knowledge, you democratize access to cutting-edge technology, fostering collaboration and progress. With Ai2's commitment to openness, OLMo 2 enables researchers and developers to innovate freely, expanding the possibilities for social and industrial impact while driving the future of ai applications.

If you want to learn how these models work, check out our Pinnacle Generative ai program.