Preprint Offers Theory for Why Self-Distillation Preserves Knowledge in LLM Continual Reasoning
arXiv preprint models LLM reasoning as search over a directed acyclic graph and provides a unified analysis of on-policy self-distillation and supervised fine-tuning dynamics…
- OPSD using hints from correct outputs supports continual improvement without overwriting prior knowledge
- SFT on correct paths risks catastrophic forgetting through dense gradient updates
- Diverse pre-training data helps models reach correct outputs even when starting from intermediate reasoning states
Theoretical Framework
Authors Shinichi Uemura and Taiji Suzuki treat large language model reasoning as search over a directed acyclic graph. This abstraction allows theoretical examination of how post-training methods interact with structures formed during pre-training.
The work focuses on on-policy self-distillation and supervised fine-tuning within continual learning scenarios where models must acquire new reasoning skills while retaining earlier capabilities. The analysis supplies an optimization guarantee linking observed empirical outcomes to underlying dynamics.
Main Insights from the Analysis
The preprint presents three core findings. First, on-policy self-distillation that incorporates hints from correct outputs leads to sparse yet effective gradient descent updates. These updates allow refinement of reasoning paths without disrupting previously acquired knowledge.
Second, supervised fine-tuning on correct reasoning paths tends to produce dense updates along those paths. Such density can overwrite information learned in prior stages, resulting in catastrophic forgetting.
Third, diversity during pre-training proves essential. It enables a post-trained model to arrive at correct outputs when a reasoning rollout begins from an intermediate state rather than the initial prompt.
Implications for Continual Post-Training
The results suggest that the pattern of updates during post-training interacts critically with the reasoning structure built in pre-training. This interplay determines whether continual reasoning remains reliable across tasks.
For developers focused on adapting open models locally, the theory underscores the advantage of methods that avoid dense overwriting when preserving prior capabilities is important. The preprint emphasizes that reliable performance depends on these interactions rather than any single training stage in isolation.
Scope and Limitations
This article is based solely on the arXiv abstract and metadata dated October 4 2026. The full paper may include detailed proofs, additional experiments, or qualifications not captured here.
It remains unknown how these theoretical predictions hold at the scale of the largest frontier models or whether real-world data distributions match the assumptions of the directed acyclic graph model.
