Evaluation & Judgment / EACL 2027

Turning Community Preferences Into Evaluation Standards

MERIT learns which criteria expert communities use, how they score answers, and how they weigh tradeoffs.

MERIT: From Preference Traces to Evaluation Standards for Community-Aligned LLM Evaluation

Overview

A community's highest-ranked answer can tell us which response people preferred. It rarely says whether the deciding factor was accuracy, safety, clarity, or a tradeoff among them. MERIT uses those rankings to recover reusable standards for evaluating answers. A trained judge then reports how an answer scores on each criterion and how those scores contribute to its overall assessment.

Key Findings

Recover the criteria behind a ranking

A vote ordering identifies which answer was preferred, while several different quality judgments could explain that ordering. MERIT starts with a question and three ranked answers. It asks a model to identify the differences among them, consolidates recurring distinctions into community-level metrics, and includes a human review step to merge redundant or overly narrow criteria. The manuscript organizes the training communities into 19 Stack Exchange communities and 20 Stack Overflow tag communities.

MERIT discovers community metrics, filters and labels preference traces, learns aggregation weights, and trains a judge to generate a structured evaluation.
MERIT discovers community metrics, filters and labels preference traces, learns aggregation weights, and trains a judge to generate a structured evaluation.

The metric inventory differs by community. Figure 3 separates many technical criteria from qualities more common in discussion communities. Appendix Table 7 gives examples. A standard for financial advice would lose information if it inherited a programming rubric that accounts for correctness and efficiency but never asks about scam risk.

Example communitySelected induced criteria from Appendix Table 7
MoneyFinancial analysis, scam risk, alternative approaches, and evidence
Math educatorsRigor, pedagogy, examples, and relevance
AcademiaNuance, personalization, empathy, and evidence
Ask UbuntuTroubleshooting, specificity, correctness, and completeness
The matrix shows which of the 12 most prevalent induced metrics occur in each community. Technical and discussion communities share some criteria and differ on others.
The matrix shows which of the 12 most prevalent induced metrics occur in each community. Technical and discussion communities share some criteria and differ on others.

Filter rankings that the metric scores can explain

The training data start with observed answer orderings. The filter asks gpt-oss-120b to infer per-metric rankings and an overall ordering, then retains a triplet when that overall ordering matches the observed one. In the supplied manuscript, 53,574 of 190,206 triplets pass, a retention rate of 28.17%. Qwen3-32B then writes five-point rubrics and assigns scores informed by the per-metric comparisons.

The filter prioritizes examples with a recoverable ordering. Agreement on the final rank alone leaves room for two judges to assign different per-metric explanations. Rankings also reflect who saw or voted on an answer, rather than a controlled evaluation of every answer. Filtering makes the supervision more internally consistent while preserving those limits.

Fit the tradeoff between metrics and train a single-answer judge

The observed rank gives the aggregation step a target. In Appendix E, rank categories run from 1 for the worst answer to 3 for the best. For answer ii from community cc, the proportional-odds model uses metric scores xicx_{ic} to fit thresholds θc,r\theta_{c,r} and coefficients βc\beta_c.

log⁡Pr⁡(Yic≤r∣xic)Pr⁡(Yic>r∣xic)=θc,r−xic⊤βc,r∈{1,2}.\log\frac{\Pr(Y_{ic}\le r\mid x_{ic})}{\Pr(Y_{ic}>r\mid x_{ic})} =\theta_{c,r}-x_{ic}^{\top}\beta_c,\qquad r\in\{1,2\}.

The paper normalizes the fitted coefficients to obtain weights β~k\widetilde\beta_k. Its scoring rule first computes the weighted sum and then rescales it to 0–100 using a reference score range.

si=∑k=1Kβ~kxik,FinalScorei=100 si−smin⁡smax⁡−smin⁡.s_i=\sum_{k=1}^{K}\widetilde\beta_k x_{ik}, \qquad \mathrm{FinalScore}_i=100\,\frac{s_i-s_{\min}}{s_{\max}-s_{\min}}.

The same metric scores can yield different judgments when communities weight them differently. Qwen3-8B and Qwen3-32B are fine-tuned on the induced standards so that, given one question–answer pair and a target community, the evaluator generates the metrics, rubrics, scores, weights, and overall judgment. It does not require all candidate answers at inference. To evaluate rankings, independently score each answer and then sort the scores.

Separate transfer from exact ordering on familiar communities

Table 1 uses exact match, the fraction of examples on which the complete predicted ordering equals the reference ordering. Holding out questions within the training communities tests a different kind of generalization from holding out entire communities. The LFQA slice uses pairs, while the other columns below use triplets, so their percentages should be compared within columns.

EvaluatorHeld-out community questionsUnseen Stack Exchange communitiesLFQA-HP-1M sample
Qwen3-32B chain-of-thought prompt43.08%25.52%67.01%
Qwen3-32B structured prompt44.44%26.70%53.82%
Qwen3-32B rank-only fine-tuning25.81%10.99%49.10%
MERIT on Qwen3-32B52.14%27.31%72.36%
GPT-5.2 structured prompt61.54%32.56%68.75%

MERIT improves over same-backbone rank-only fine-tuning in each of these settings. The gain over same-backbone structured prompting is 7.70 points on the held-out set and 0.61 points on unseen Stack Exchange. GPT-5.2 structured prompting still scores higher on both of those Stack Exchange sets, while MERIT leads that particular comparison on the LFQA pairs. Structured supervision helps in this comparison, but its advantage does not transfer uniformly.

Identify which components help and what reviewers prefer

Figure 4 varies filtering, metric choice, and aggregation in separate experiments. They are different comparisons, so the changes should not be added together.

The manuscript separates three tests of MERIT's components: preference-consistency filtering, choice of metrics, and aggregation weights.
The manuscript separates three tests of MERIT's components: preference-consistency filtering, choice of metrics, and aggregation weights.
Component testExact-match results from Figure 4
Filtering, Qwen3-8B48.2% with filtering vs. 19.8% without it on held-out questions. On unseen communities, 29.7% vs. 20.6%
Metric choice, GPT-5.253.3% without listed metrics, 56.4% with static metrics, and 58.5% with induced community metrics
Aggregation, Qwen3-32B53.3% with inferred weights vs. 46.8% with uniform weights

In the 32B aggregation comparison, Kendall's τ\tau moves from .643 with uniform weights to .641 with inferred weights even as exact match rises. Exact match asks whether the full ordering is right. Kendall's τ\tau captures pairwise ordering agreement. Improving one does not guarantee improving the other.

The human study analyzed 24 people after exclusions. 21 of 24 preferred MERIT's reports to chain-of-thought reports, and 20 of 24 preferred them to structured prompting. Participants described shared scoring dimensions and clearer explanations of how particular answers could improve. Those judgments are evidence about the usefulness of the reports. The automatic experiment separately tests agreement with the reference rankings, which remains a measure of community preference.

The human study compares what each judging format exposes and shows how often the 24 analyzed participants preferred MERIT in each comparison.
The human study compares what each judging format exposes and shows how often the 24 analyzed participants preferred MERIT in each comparison.

Questions & Answers

Q: How is a metric different from a rubric or a weight?

A: A metric names a dimension of quality. A rubric describes what each score on that dimension means. A weight reflects how much that dimension contributes to the overall evaluation for a particular community.

Q: Does community preference establish objective quality?

A: The preference signal records how answers were ranked by participants on one platform. It can reflect exposure, popularity, or local norms. Examining individual criteria and testing the evaluator with intended users are especially useful when those preferences may not transfer.

Q: What does the comparison with rank-only training show?

A: On these backbones and datasets, fine-tuning on answer orderings alone performed worse than training on structured score-and-rationale targets. The ablations additionally test the contributions of filtering, community-specific metrics, and learned aggregation.

Future Ideas

Let an expert group inspect changes in its standards

Induce rubrics from two time periods or two groups within a professional community. Give reviewers a small set of matched answers and show where criteria or weights disagree. Ask them to identify which changes reflect a real shift in priorities and which reflect sampling or voting artifacts.

Evaluate the process against an expert-authored rubric and a frozen MERIT evaluator. Measure ranking agreement, the cost of revising a rubric, and whether reviewers find tradeoffs they would have missed in an overall score.

Sources

Original abstract

Evaluating LLM outputs in expert domains requires diagnostic standards that reflect the specialized knowledge and values shared within a community of domain experts. However, existing approaches either depend on the acquisition of expert-annotated rubrics, which are costly to scale across communities, or use LLM-as-a-judge with generic criteria that are invariant across all expert communities, which are not grounded. In this paper, we introduce MERIT (Metrics, Evaluation Rubrics Induced from Traces), a scalable yet grounded framework for inducing community-specific evaluation standards from naturally occurring preference traces, such as ranked answers on online QA platforms. MERIT transforms these noisy behavioral signals into structured supervision by inducing reusable community-specific metrics, constructing rank-aligned rubrics and metric-level scores, and learning preference-aligned aggregation weights. The resulting supervision is used to train evaluator LLMs that produce metric-level scores, rationales, and an overall score. On held-out, unseen-community, and external domain-oriented benchmarks, MERIT outperforms rank-only training and general-purpose judges, achieving gains of 3.98–14.90 pp over same-backbone prompting baselines. Human expert studies further confirm MERIT’s advantage in consistency, transparency, and diagnostic value.