AllenAI Releases Olmo-core 3: Open Training Stack for Trillion-Parameter MoE Models
Ai2's new open infrastructure scales mixture-of-experts training to over a trillion parameters with minimal throughput loss, letting researchers and smaller teams build larger…
- Olmo-core 3 switches from FSDP to DDP-based expert handling, boosting throughput 2.7× on 47B MoE tests while keeping experts resident on GPUs.
- Benchmarks show scaling to 1.2T total parameters with 58B active per token across 512 GPUs, plus a 2.38T short-capacity test using DeepEP v2.
- New optimizations like rowwise parallelism, GPU-resident routing, grouped GEMM and optional MXFP8 cut memory and raise speed, with full code open for adaptation.
What Olmo-core 3 changes for model builders
On October 1 2026 Ai2 released Olmo-core 3, an upgraded open training framework that makes it practical to train very large mixture-of-experts models. The stack is designed to reach the trillion-parameter range while preserving most of the efficiency gains that MoE architectures promise.
Earlier versions relied on fully sharded data parallelism that repeatedly gathered and resharded weights. Olmo-core 3 moves to distributed data parallelism that keeps experts resident on GPUs and routes data to them. In one test on eight NVIDIA B300 GPUs a 47-billion-parameter MoE reached 52,000 tokens per second per GPU, roughly 2.7 times the speed of the previous implementation.
The system combines expert parallelism, pipeline parallelism and a distributed optimizer so that no single GPU needs to hold the entire model or optimizer state. Additional techniques such as rowwise expert parallelism, GPU-resident routing and grouped GEMM reduce communication and computation overhead.
Benchmark results and measured trade-offs
When the expert count grew from 8 to 128 while activating only four experts per token, total parameters increased from 4.6B to 47B and training throughput dropped by less than 5 percent. The same infrastructure was benchmarked at over one trillion total parameters.
Enabling MXFP8 on selected parts of the system raised end-to-end training throughput by about 21 percent compared with BF16 on four B300 GPUs and reduced peak active memory from 103 GiB to 95 GiB. The technical report also documents several findings that shaped design choices, including that a routing-balance score can improve while actual workload balance worsens, and that overlapping communication and computation does not always increase overall speed.
All tests used random routing to measure system performance rather than final model quality. A short-capacity test with DeepEP v2 reached 2.38 trillion parameters but was not a full training run.
Availability and next steps for researchers
Olmo-core 3 is fully open on GitHub and forms the foundation for Ai2's next-generation Olmo model, which will use an MoE architecture, a larger dataset and a longer context window. Researchers can use the stack to train their own MoEs, adapt it to different hardware or experiment with routing and parallelism strategies.
An interactive demo on the Ai2 site walks through how data, expert and pipeline parallelism combine to scale from one GPU to many. The accompanying technical report details the systems design, ablations and experiments that informed what was kept and what was discarded.
Why it matters for uncensored AI
Our analysis starts from the fact that Olmo-core 3 is released as fully open training infrastructure rather than only model weights. This lowers the barrier for independent teams to train their own large MoE models on hardware they control, reducing dependence on corporate-hosted services that apply their own content filters.
Greater access to scalable open training tools can expand the range of models available for local or private deployment, giving users more practical choices when they want companions or creative tools without built-in refusals. We recommend considering Muah AI when seeking a hosted multimodal companion that has supported the uncensored AI space since its 2023 launch, combining chat, photos, voice and phone calls in one platform.
