CAST Preprint Proposes Credit Alignment for Cross-Tokenizer Distillation
arXiv preprint dated 2026-10-09 introduces CAST, a shift-or-stay framework that redistributes chunk-level advantages using predictive uncertainty and teacher-signal smoothness…
- CAST solves credit alignment by weighting student tokens according to uncertainty and local smoothness, exactly solving a KL-regularized allocation while preserving total chunk advantage.
- For unaligned positions it applies selective KL anchoring to the initial policy, preventing unconstrained drift where teacher supervision is absent.
- Abstract reports average gains of 2.45 points and 1.32 points over strongest baseline on six math, science and code benchmarks for cross-tokenizer transfer from Qwen3-8B to Mistral-7B and Llama-3.1-8B respectively.
The Credit Alignment Problem
In cross-tokenizer on-policy distillation, decoded-text alignment matches teacher and student output at the chunk level. This supplies an overall signal for how much a span should change but does not indicate how that signal should be distributed across individual student tokens.
Tokens without direct teacher correspondence remain unconstrained and can drift. The preprint identifies this unresolved distribution task as the credit alignment problem.
CAST Framework
The authors propose CAST, Chunk-aware credit Assignment with Selective Token Anchoring. For aligned chunks it reallocates the advantage according to the student's predictive uncertainty and the local smoothness of teacher feedback, emphasizing regions with consistent signals while exactly solving a KL-regularized credit-allocation problem.
For unaligned positions CAST applies a selective KL penalty that anchors the student toward its initial policy, limiting drift. Because credit assignment is decoupled from chunk construction, the method also works with punctuation-defined or fixed-length chunks under shared tokenization.
Reported Results
Across six mathematics, science and code benchmarks the preprint states that CAST improves average performance by 2.45 points over the strongest baseline for cross-tokenizer transfer from Qwen3-8B to Mistral-7B and by 1.32 points to Llama-3.1-8B. Shared-tokenizer transfer to Qwen3-1.7B yields a 2.10-point gain.
These numbers derive solely from the 2026-10-09 arXiv abstract. The capture provides no full paper, experimental details, hyperparameters or independent replication.
Implications for Open Models
The technique supplies a practical adjustment for distillation pipelines that use mismatched tokenizers, a common situation when pairing strong open-weight teachers with differently vocabularized students. Applicability to simpler chunking strategies may reduce experimentation overhead.
The abstract makes no claim of universal superiority or gains outside the tested domains. Whether the uncertainty-and-smoothness weighting generalizes remains an open question.
