LLM Judges Exhibit Anchor Bias in Entity Alignment Evaluation
arXiv preprint benchmarks three frontier models on entity alignment tasks across three datasets and a new biomedical benchmark, revealing that label exposure causes judges to…
- Label exposure causally collapses LLM judge discrimination in entity alignment with J-ROC-AUC ranging from 0.12-0.87.
- Label-free protocol recovers near-ceiling performance on distinctive-name datasets (0.93-1.00) and strong recovery on biomedical pairs (0.93-0.95).
- Stronger frontier models show greater label sensitivity rather than improved robustness, confirmed by counterfactual tests and human evaluation.
Study Overview
Entity alignment identifies equivalent entities across knowledge graphs and is essential for knowledge base integration and ontology merging. Traditional evaluation relies on expensive expert annotation, limiting scalability across domains.
This preprint presents the first systematic study of LLM-as-a-judge reliability for this structured prediction task. It evaluates three frontier models, three existing datasets, four entity alignment systems, and introduces a new biomedical benchmark.
Core Experimental Results
The authors applied perturbation bias diagnostics, meta-evaluation across all dataset-judge-prompt combinations, and counterfactual label-flip tests. They identified anchor bias, where judges invert discrimination when the system's decision label is visible.
Label exposure causally collapses judge performance. Counterfactual experiments confirm this effect with flip success rates of 53-99 percent. A blinded two-annotator human evaluation on 102 pairs achieved Cohen's kappa of 0.902, directly validating the mechanism.
Paradoxically, stronger frontier models exhibit greater sensitivity to exposed labels rather than increased robustness.
Mitigation and Released Resources
A label-free protocol restores high capability on distinctive-name datasets and significant recovery on biomedical pairs. The work releases the first biomedical entity alignment benchmark using MeSH-SNOMED CT with 15,000 pairs along with a reproducible auditing framework.
Code and data are available on GitHub, allowing researchers to audit LLM judge pipelines for this failure mode in entity alignment tasks.
Evidence Limitations
This article is based solely on the arXiv abstract and metadata. Full methodological details, exact model names, dataset statistics, and complete results are available only in the unpublished paper.
The evidence demonstrates a specific, causally verified failure mode tied to label exposure rather than universal unreliability of LLM judges.
