SPACE: Geometric Calibration Refines SFT Updates Post-Hoc
arXiv preprint proposes SPACE, a lightweight data-free post-processing technique that uses pre-SFT weight geometry to selectively adjust supervised fine-tuning updates…
- SPACE projects SFT updates relative to the principal subspace of pre-SFT weights, reverting disruptive components while preserving orthogonal ones using only two checkpoints.
- On mathematical SFT, abstract reports average ID/OOD gains of 2.85/2.26 points on Qwen2.5-7B and 2.41/1.52 points on Qwen3-14B-Base versus uncalibrated SFT.
- Results indicate spectral geometry shapes how fine-tuning affects prior knowledge, motivating geometry-aware updates for model adaptation.
Core Geometric Insight
The preprint studies supervised fine-tuning through weight spectral structure. It finds that SFT often creates dense updates perturbing the principal directions of pre-SFT weights, which can disrupt previously learned knowledge.
This leads to the idea of Geometric Calibration, using the geometry of pre-SFT weights as a reference to refine the learned updates.
SPACE Post-Processing Method
SPACE, or Subspace Projection for Adapted Checkpoint Enhancement, is a training-free and data-free approach requiring only the pre-SFT and post-SFT checkpoints.
It selectively reverts components of the SFT updates that lie within the principal subspace of the pre-SFT weights, while keeping those in the orthogonal complement intact.
Evaluation Summary from Abstract
The abstract describes tests on Qwen2.5, Qwen3, Qwen3.5, Llama-3.1, and OLMo-3 models from 2B to 14B parameters across mathematical, coding, and medical SFT tasks as well as variants.
SPACE is reported to mitigate out-of-domain capability regression while improving in-domain performance over standard SFT. Specific math results cite the listed average point gains.
Available evidence is the preprint abstract only; full methodological details, experimental conditions, and complete results are not included here.
Broader Implications
The work underscores how the geometric relationship between SFT updates and pre-trained weights influences the adapted model's capabilities.
It encourages further development of geometry-aware parameter updates to support more stable continual model improvement.
