Ai2 Shares OLMo Hybrid Research at COLM 2026 Highlighting Efficiency Gains
Ai2 presented hybrid transformer-recurrent models at COLM 2026 that match OLMo 3 7B MMLU accuracy with 49% fewer tokens while releasing training infrastructure and advancing…
- OLMo Hybrid combines attention and linear recurrent layers reaching similar MMLU performance to OLMo 3 7B using 49 percent fewer training tokens in controlled experiments.
- Ai2 released OLMo-core 3 for training large mixture-of-experts models to enable open research alongside hybrid architecture checkpoints and reports.
- The nonprofit lab also published byte-level Bolmo work in Nature and continues developing the open-weights AstaBrief model for cited scientific reports.
Hybrid Model Research at COLM 2026
At COLM 2026 in San Francisco Ai2 presented its paper examining language models that integrate transformer attention for retrieving specific prior details with linear recurrent layers that maintain a compact state updated across tokens.
The controlled training experiments connect theoretical analysis of architectural representational power to practical results. The hybrid design reached the same MMLU accuracy as OLMo 3 7B while using 49 percent fewer training tokens.
These findings are guiding the next OLMo model now in pre-training which adopts a hybrid mixture-of-experts architecture to further improve efficiency by routing tokens through selected components.
Open Infrastructure Releases
Ahead of the conference Ai2 released OLMo-core 3 a redesigned system for training large mixture-of-experts models. The open code checkpoints and technical reports allow researchers to build and investigate their own variants within the OLMo ecosystem.
Google's recent reproduction of the OLMo 3 7B training run on Cloud TPUs using MaxText demonstrates the reproducibility supported by full openness of models data code and methods.
During COLM the lab also published its byte-level Bolmo research in Nature and released additional checkpoints. Bolmo processes raw bytes instead of subword tokens after a short additional training phase on existing models.
Agentic Science Tools and Collaboration
Ai2 is developing Asta an agentic platform created with domain experts to fit real scientific workflows. The recently released AstaBrief open-weights model generates cited reports from research questions and retrieved literature and can run in Asta's Fast mode or on local infrastructure.
Senior research scientist Bodhisattwa Prasad Majumder presented related papers exploring limitations of general-purpose models for science and requirements for future open models.
A conference conversation between NLP research director Noah A. Smith and comms lead Kyle Wiggers covered progress on the next OLMo iteration and agentic scientific models.
Ai2's Open Research Mission
As a nonprofit laboratory Ai2 treats sharing of models data code and methods as fundamental to its approach rather than an afterthought. The COLM presentations and releases reflect this commitment to artifacts that the wider community can study challenge and extend.
The work underscores the value of open collaboration for advancing both fundamental understanding and practical AI tools for science.
