The non-profit research institution AI2 has recently unveiled the fully open-source model OLMo 2, which achieves the best performance among models of the same size. Notably, it not only opens up the model weights but also discloses the training data and methodologies. The OLMo 2 series includes two models, 7B and 13B, outperforming other open-source models like Llama 3.1 and Qwen 2.5, with fewer FLOPS computations, opening new possibilities for open-source LLMs.
Below is the formatted content for a wordpress blog post:
Outstanding Performance and Transparency
OLMo 2 demonstrates remarkable generalization and adaptability across multiple downstream tasks. On 10 benchmarks, OLMo-2-13B outperforms Llama-2-13B comprehensively, and OLMo-2-8B also surpasses Llama-3.1-8B in average benchmark scores. Unlike other projects that only open-source model weights, AI2 has shared the training process, data, code, and procedures of OLMo 2, providing invaluable resources for LLM research and applications.
Training Phases of OLMo 2
The training process of OLMo 2 is divided into three stages: pre-training, intermediate training, and instruction tuning. The pre-training data is a mix of high-quality web pages, code, and academic paper data. The intermediate training phase utilizes high-quality domain-specific data and synthetic data. The final instruction tuning phase sees researchers developing the OLMo 2-Instruct model based on Tülu 3’s instruction tuning methods.
Sustainable Training Practices
In a remarkable effort to reduce training costs, the OLMo 2 team implemented various energy-saving measures, such as using a water-cooling system to lower GPU temperatures and power consumption. This resulted in a significant reduction in electricity usage — the OLMo 2 7B training consumed only 131MWh of electricity, compared to the 1022MWh consumed by the training of the same-sized Llama 3.1 model.
Significance of OLMo 2 Launch
The launch of OLMo 2 marks a significant milestone in the continuous advancement of open-source LLMs, establishing a new ecosystem for research in the field. In this ecosystem, new training methods and technologies need to be understood and shared. The release of OLMo 2 is not just a technological achievement but a beacon of hope for researchers and practitioners, promising to propel the field forward with its transparency and innovation.
