Interpretability

Olmo + Dolma

open olmo model + open dolma dataset
- Open Language Model (OLMo) is a framework intentionally designed to provide access to data, training code, models, and evaluation code necessary to advance AI through open research by empowering academics and researchers to study the science of language models collectively.
- Dolma: Information about pretraining corpora used to train the current best-performing language models is seldom discussed: commercial models rarely detail their data, and even open models are often released without accompanying training data or recipes to reproduce them. As a result, it is challenging to conduct and advance scientific research on language modeling, such as understanding how training data impacts model capabilities and limitations. To facilitate scientific research on language model pretraining, we curate and release Dolma, a three-trillion-token English corpus, built from a diverse mixture of web content, scientific papers, code, public-domain books, social media, and encyclopedic materials. We extensively document Dolma, including its design principles, details about its construction, and a summary of its contents. We present analyses and experimental results on intermediate states of Dolma to share what we have learned about important data curation practices. Finally, we open-source our data curation toolkit to enable reproduction of our work as well as support further research in large-scale data curation.

pythia (opens in a new tab)
- The hub for EleutherAI's work on interpretability and learning dynamics
- This repository is for EleutherAI's project Pythia which combines interpretability analysis and scaling laws to understand how knowledge develops and evolves during training in autoregressive transformers.
- Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (opens in a new tab) arxiv
- pdf paper (opens in a new tab)