Label-Free Rubric Repair Boosts LLM Judge Consensus but Not Human Agreement
arXiv preprint tests a disagreement-driven method that rewrites only contested rubric clauses without human labels. It increases inter-judge reliability on one dataset yet…
- Disagreement-ranked rubric editing raised Krippendorff's alpha by 0.069 on SummEval versus the original and by 0.074 versus random-clause controls.
- No detectable change occurred in Kendall tau agreement with humans on SummEval; point estimates for human agreement were directionally lower after targeted edits on all three datasets.
- Pre-registered hypotheses, multi-provider judge panel, and planned code release strengthen the preprint's design, though results remain dataset-specific.
Core Method and Evidence Limits
This arXiv preprint, available only as metadata and abstract, presents a label-free procedure for rubric clarification. It decomposes rubrics into sub-criteria, ranks them by cross-judge disagreement on a development split, and uses an editor model to rewrite solely the most contested clauses.
Human labels are never used to select or edit text; they serve only for final evaluation. An attempted stage converting text-checkable sub-criteria to deterministic checkers transferred none in the experiments. The available evidence is the abstract, not a full-paper review or independent replication.
Effects on Inter-Judge Agreement
On the SummEval dataset the procedure improved inter-judge Krippendorff's alpha by 0.069 (95% CI 0.046 to 0.092) over the original rubric and by 0.074 over a control that applied identical edits to randomly chosen clauses.
Improvements on HealthBench and USR Topical-Chat did not meet the authors' pre-registered effect criteria. Selecting clauses via disagreement produced greater consensus than editing arbitrary ones.
Alignment With Human Expert Ratings
Agreement with expert human ratings on SummEval showed no detectable change (Kendall tau difference -0.007, 95% CI -0.037 to 0.021).
In the pre-registered secondary comparison, human agreement after editing disagreement-selected clauses was lower in point estimate than after control edits across all three datasets, though no confidence interval excluded zero. Added judge consensus therefore did not bring judges closer to human annotators in this setting.
Study Design and Release Plans
Four hypotheses were pre-specified in a version-controlled protocol. The authors evaluated six rubric conditions using a panel of five judges from three providers, issuing 129770 judge requests in total.
They plan to release code, frozen rubrics, item identifiers, and judge scores. These details allow others to examine exactly which rubric changes were made and how the panel performed.
