On-Policy Distillation Reweights Shared LLM Features, arXiv Preprint Shows
A September 2026 arXiv preprint applies sparse crosscoders and a new swap readout technique to examine what on-policy distillation actually changes inside student LLMs during…
- OPD does not create new features or copy the teacher's features according to the abstract.
- SFT warm-up on teacher rollouts reweights shared features in advance of distillation.
- Distillation primarily adjusts how the student uses already-shared features rather than acquiring novel ones.
Preprint Summary and Approach
This article reports on an abstract-only capture of the arXiv preprint submitted 2026-09-28 by Yu, Ye, Wang and Zou. The work investigates on-policy distillation, a popular method for improving LLM reasoning by training students on teacher-generated outputs.
The authors employ sparse crosscoders that learn a single shared feature dictionary across the student model before and after on-policy distillation as well as the teacher model. They introduce a swap readout method to measure how each student checkpoint uses its own feature activations, allowing analysis of training effects even on checkpoints the crosscoder never saw.
Key Observations on Feature Behavior
The abstract states that across three on-policy distillation settings, the process neither creates new features nor transfers the teacher's specific features. It reports that firing rates for over 98 percent of the student's frequently used features change by 20 percent or less.
The preprint concludes the student mainly learns from the teacher how to better utilize the internal features the models already share.
Insights on Supervised Fine-Tuning Warm-Up
The study examines the supervised fine-tuning warm-up step on teacher rollouts that often precedes on-policy distillation and improves its results. According to the abstract this warm-up reweights shared features, raising and lowering many of the same ones that distillation later adjusts.
It also modifies additional features related to conversation format, reasoning style and mathematical notation; these adjustments persist through the distillation phase.
Mechanistic Implications
The evidence presented in the abstract suggests on-policy distillation works mainly by reweighting existing features rather than expanding the student's representational vocabulary. This offers a mechanistic view of why the technique succeeds and what the supervised fine-tuning warm-up contributes.
Such interpretability research can help developers set clearer expectations about the internal effects of post-training methods on open models.
