Kernelized Parameter Sensitivity Atlases for Amortized Conditional Interpretability
An arXiv preprint dated 8 October 2026 introduces KPSA, a kernel-based method derived from the empirical neural tangent kernel that approximates input-dependent parameter…
- KPSA recovers 39–85 % of full backpropagation perturbation susceptibility using only 0.02–0.1 % of parameters across vision, forecasting, and protein-modeling tasks.
- On activation attribution for transformer MLP features, it retains 81 % of the oracle-to-baseline gap on vision and 68 % on protein modeling while increasing throughput from 551 to 2,041 images/s and cutting peak GPU memory from 4.55 GB to 0.44 GB.
- The compact kernel atlas approach enables efficient input-dependent parameter sensitivity estimation, with theoretical bounds separating approximation from representation error.
Method Overview
Gradient-based attribution supplies rich input-specific information yet repeated backpropagation across many samples is computationally expensive. Static scores avoid this cost at the price of discarding input dependence.
The authors derive Kernelized Parameter Sensitivity Atlas (KPSA) from a block decomposition of the empirical neural tangent kernel. It measures normalized sensitivity of parameter groups, embeds those groups in an explicit query space, and applies a lightweight kernel to estimate sensitivities for new inputs.
Theoretical Bounds
The preprint supplies error bounds that distinguish finite-source approximation error from irreducible representation error and provides margin-based guarantees for top-k recovery of the most sensitive parameter groups.
These bounds clarify when the atlas is expected to remain faithful and when larger source sets or higher-capacity kernels may be needed.
Experimental Findings
Across computer vision, time-series forecasting, and protein modeling, KPSA identifies sparse sensitivity circuits spanning multiple layers that vary systematically with the input. Selected groups recover 39–85 % of the perturbation susceptibility of a first-order backpropagation oracle across a five-fold range of parameter budgets.
Applied to activation attribution inside transformer MLP features, the method retains 81 % of the performance gap between the backpropagation oracle and a constant-sensitivity baseline on vision tasks and 68 % on the protein-modeling task. An ablation shows nearest-neighbor retrieval in the learned embedding space remains competitive.
Performance Gains
The approach dramatically improves batched query-time throughput from 551 to 2,041 images per second on vision while reducing peak GPU memory from 4.55 GB to 0.44 GB. These gains demonstrate that compact kernel atlases can store useful input-dependent sensitivities and amortize repeated gradient attribution.
Limitations
This article is based solely on the arXiv abstract and metadata; the full paper, detailed methods, hyperparameter settings, and complete results tables are not yet available for independent review.
It remains unknown how well KPSA generalizes beyond the three evaluated domains or to substantially larger transformer architectures. The linked GitHub repository will be important for reproduction.
