FaithAlign Preprint Introduces Fine-Grained Benchmark for Theory-Scale Autoformalization Faithfulness
arXiv preprint presents FaithAlign, a 460-item dataset with 4982 annotated phrase-level links, plus FaithAlign-Gap, to test context-aware faithfulness evaluation in…
- FaithAlign frames faithfulness as a context-aware align-then-label task that identifies phrase correspondences and labels them under explicit assumptions.
- GPT-6 Astra reaches 47.8% defect F1 yet fully identifies all defects in only 4.5% of multi-defect items across eight models tested.
- On FaithAlign-Gap, Astra flags just 28.2% of items as undecidable when informal context is withheld, exposing limits in recovering theory-scale meaning.
Faithfulness Beyond Statement-Level Verdicts
Autoformalization translates informal mathematics into formal statements for proof assistants. The preprint stresses that true faithfulness requires preserving meaning drawn from surrounding theory, unstated conventions and contextual interpretations.
Current evaluation approaches usually output a single verdict per statement. This hides which specific phrases correspond and which assumptions support the decision. The authors argue such coarse methods fall short for theory-scale work where statements depend on broader mathematical material.
FaithAlign and FaithAlign-Gap Datasets
The preprint formulates evaluation as an align-then-label problem performed under explicitly supplied context, formal dependencies and assumptions. FaithAlign supplies 460 statement pairs together with relevant context and 4982 human-annotated phrase-level links.
Its companion set, FaithAlign-Gap, contains 146 items that deliberately withhold informal context and assumptions. This tests whether models can detect when a faithful judgment is undecidable. The preprint identifies this resource as a testbed for fine-grained, context-sensitive faithfulness research.
Model Results and Persistent Difficulties
Across eight models evaluated on the preprint's benchmark, GPT-6 Astra records the highest defect F1 of 47.8%. It nevertheless identifies every defect in only 4.5% of items containing two or more defects.
On FaithAlign-Gap, Astra correctly recognizes only 28.2% of undecidable items as such. These figures illustrate the challenge of recovering context-dependent correspondences and of knowing when information is insufficient.
Implications for Research and Model Evaluation
The evidence, drawn solely from the arXiv abstract, shows that existing models struggle with the nuanced, theory-scale nature of faithfulness. The benchmark therefore supplies a concrete yardstick for future improvements in automated evaluation techniques.
This preprint, dated 13 October 2026, does not claim to resolve the problem but establishes a structured challenge that highlights the gap between compilation accuracy and genuine meaning preservation.
