Activation Alignment Bridges Context Gap in Tabular In-Context Learning
A new arXiv preprint proposes activation alignment, a lightweight linear method trained on synthetic data to map partial-context activations toward full-context behavior in…
- Activation alignment uses a simple linear transformation on synthetic unlabeled data to align student partial-context activations with a full-context teacher, trainable on commodity hardware without a GPU.
- Evaluated on 38 TabArena classification datasets with TabPFN-3 and TabFM, the aligned student shows improvements over the unaligned baseline across context budgets.
- In low-data regimes the approach recovers a notable share of the teacher's predictive gain, offering a practical tradeoff between speed and accuracy for tabular ICL.
The Context Gap in Tabular Foundation Models
Tabular foundation models perform in-context learning by conditioning predictions on labeled training examples provided directly in the input context. Unlike conventional models that separate training from inference, these systems must process the full set of examples in every forward pass, causing inference cost to scale with context size.
Limiting the number of examples reduces computational expense but leads to clear drops in predictive performance. The preprint from Yoel Zeldes, dated October 5 2026, introduces a method to retain the benefits of richer context without paying the full inference price.
How Activation Alignment Works
The technique trains a lightweight linear transformation using synthetic unlabeled data. This aligner maps intermediate activations from a data-constrained student model that sees only a subset of examples toward those produced by a full-context teacher model.
Training requires no GPU and finishes in seconds to minutes on standard hardware. At inference the aligned student can therefore approximate much of the teacher's behavior while using only compact context.
Results from the TabArena Benchmark
The abstract reports evaluation on 38 classification datasets from the TabArena benchmark using the leading tabular foundation models TabPFN-3 and TabFM. Across tested context budgets the aligned student shows improvements over the unaligned baseline for both models.
In low-data regimes the method recovers a substantial fraction of the full teacher's predictive advantage. The preprint positions this as a low-cost way to achieve faster inference while narrowing much of the performance difference.
Evidence Limitations and Practical Value
All findings are drawn from the preprint abstract alone; the full paper is required to assess exact implementation details, statistical tests, hyper-parameters and generalization beyond the reported setting.
For users handling tabular data the approach suggests a lightweight post-hoc adjustment that avoids retraining the base model. Research into model behavior under context constraints highlights the value of flexible tools that let practitioners balance speed and accuracy according to their needs.
