Preprint Questions Brain Alignment and Cross-Lingual Scores in Language Models
An October 2026 arXiv preprint by Saman Rahbar shows that similarity scores used to claim brain alignment or cross-lingual transfer can remain high even after the underlying…
- Grammaticality probes transfer worse between distant languages, yet probe accuracy itself drops to chance in four of the five most distant languages, halving explained variance when removed.
- Brain-alignment training raises similarity to fMRI responses from 0.10 to 0.34, but targets with correspondence destroyed still reach 0.31, close to the trained result.
- A shuffled-target baseline can be computed before training and scores roughly k/n against the real target, underscoring the need to test what survives without actual correspondence.
Cross-Lingual Grammaticality Probe Findings
The preprint examines transfer of a grammaticality probe across seventeen languages. Transfer performance correlates negatively with language distance at r = -0.66. However, the probe's own accuracy falls along the same axis and reaches chance level in four of the five most distant languages.
Dropping those four languages halves the explained variance. Because it also narrows the range of distances, the experimental design cannot cleanly separate a genuine distance-graded steering effect from instrument failure.
Statistical Independence and Significance
Treating the 272 language pairs as independent observations yields p = 0.0006 for the steering effect. When the seventeen languages are instead treated as the proper unit of analysis, the p-value rises to 0.155 and the result is no longer statistically significant.
This illustrates a common risk in multilingual studies where pairwise counts can inflate degrees of freedom and produce misleading significance.
Brain Alignment Experiments
In the brain-alignment setting, a training objective improves similarity to human fMRI responses from 0.10 to 0.34, against a reliability ceiling of 0.54. Yet deliberately destroying the brain correspondence in the target data still allows the training to reach 0.31.
A simple shuffled-target baseline, computable before any model is trained, scores roughly k/n against the real target where k is rank and n the number of sentences. The work finds measurable improvements from both interventions but detects no clear distance-graded steering effect or syntactic benefit from alignment.
Methodological Recommendation
The preprint, available only as an abstract, concludes that similarity scores should not be read as evidence of shared structure until researchers first measure how much of the score survives when the correspondence itself is removed. This caution applies to both brain-alignment and cross-lingual evaluations.
The evidence here is limited to the supplied abstract and does not include the full paper or an independent replication. Before interpreting any correspondence score, the author advises quantifying the contribution that remains after structure is eliminated.
