Evaluation & Judgment / EMNLP 2025

Change the Answer Labels and the Model May Change Its Values

Three ways to score a language model's survey answers yield different pictures. The most stable picture still tells us little about what the model will favor in a concrete scenario.

Revisiting LLM Value Probing Strategies: Are They Robust and Expressive?

Siqi Shen, Mehar Singh, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, Rada Mihalcea

Overview

A survey asks how important family is. An evaluator could inspect the model's probability of saying “A,” score the full answer phrases, or sample several written replies. Each approach turns the same question into a different measurement. This study tests whether those measurements hold up when prompts change and whether they predict the model's judgments about actions.

Key Findings

Ask a model to choose between two moral statements and it may appear to take a clear position. Move the choices around or relabel them and the same model can look less settled. The study asks whether three common ways of assigning probabilities survive that simple change, and whether the probabilities predict what the model does.

The study compares three ways to turn a model response into a distribution over value options.
The study compares three ways to turn a model response into a distribution over value options.

Three instruments for the same questionnaire

The token method normalizes the next-token scores for the valid option labels. The sequence method scores an answer sequence through its perplexity, a measure of how surprising its tokens are under the model, then normalizes the inverse scores. The text method samples replies and counts the options the replies express. In the experiments, text generation uses ten samples per prompt at temperature 1.0.

The authors vary the formatting of the prompt, reverse the ordering of the options, and change option symbols. They measure both whether the top answer changes and how far the whole distribution moves. These questions are subjective, so the experiments ask whether the measurement is stable under a change of format. Scoring methods and perturbations

A stable response is only one part of the problem

ProbeCorrelation between mismatch rates from prompt and selection-bias variationsCorrelation between Jensen–Shannon distances from those variations
Option token0.8990.526
Sequence0.6450.893
Sampled text0.9330.885

Table 2 compares how sensitivity to prompt formatting and sensitivity to selection-bias variations move together across the evaluated models. These are correlations between two kinds of sensitivity, so a large number does not by itself mean a probe is stable. Sequence scoring generally gives lower distributional changes in the reported robustness tests. The authors separately check whether country cues bring responses closer to World Values Survey distributions and whether survey responses predict the model’s ratings of related actions. Country cues improve survey alignment in most tested settings. The token method often gains the most, while the sampled-text probe changes little on several models. Figure 4 and Section 7.2 report generally weak action agreement, often around 0.2–0.4 for correlations that show some relationship, with little or negative agreement in other settings. Appendix Table 2 · Action agreement

The perturbations also distinguish robustness from demographic alignment. Figure 3 varies the placement and labels of the answer options. Figure 5 instead supplies a country cue and measures distance to the survey distribution for that country. The first experiment asks whether presentation distorts the measurement. The second asks whether demographic information moves it in an intended direction. Those are distinct properties, even when both can be plotted as changes in probabilities. See the selection-bias test in Figure 3.

A distribution can move without changing its winner

The paper measures distance between the distributions on ordered response options. Its normalized agreement score is

a(p,q)=1−EMD⁡(p,q)Noptions−1.a(p,q)=1-\frac{\operatorname{EMD}(p,q)}{N_{\mathrm{options}}-1}.

Earth mover’s distance, EMD, measures how far probability mass has to move along the options. A shift from “strongly agree” to “agree” costs less than moving the same mass across the full scale. This gives the authors a way to register changes that a comparison of top answers would miss.

The questionnaire is a prediction to test

The value–action experiment asks whether a value inferred from answers to a questionnaire also aligns with choices in situations that express that value. It probes a different property from format robustness. A measurement may be stable under relabeling and still say little about behavior under a scenario. The action setting tests the behavioral meaning of the score and makes the limitations of treating a questionnaire as a model profile easier to see.

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: What does mismatch rate measure?

A: It counts how often the highest-probability answer changes between prompt conditions. The distribution can also change without flipping the top answer, which is why we inspect a second metric.

Q: Why can a space before the answer letter make a difference?

A: The tokenizer can represent an option label with or without a leading space as different tokens. Checking only one version can miss where the model puts its probability mass.

Q: Would a more stable questionnaire predict behavior better?

A: Stability is a prerequisite for interpreting the measurement. The observed action correlations remain weak even after some robustness improvements, so behavior needs an independent evaluation.

Q: Can English questionnaire responses tell us how the model would behave in another country?

A: A country cue in English can change the model's responses without testing how it handles that population's languages or settings. The action experiment measures agreement on a separate set of model-generated scenarios that the authors partly reviewed. To test practical behavior, use independently written, context-specific tasks and report how the result varies with language and question format.

Future Ideas

Add a behavioral calibration card to model evaluation

Pair questionnaire items with short, independently written scenarios in which the model must recommend a concrete action. Vary only the answer order and symbols for the questionnaire trials. Keep the scenario test set held out, and compare a probe's performance at predicting scenario judgments with a baseline that ignores the questionnaire entirely.

Report both sensitivity to format and out-of-sample agreement. Ask human reviewers to verify that the scenarios actually involve the intended trade-off and to identify ambiguous cases. A team should rely on the questionnaire as a proxy only when it adds predictive value over the behavioral baseline and the result replicates across wording choices.

Sources

Original abstract

The value orientation of Large Language Models (LLMs) has been extensively studied, as it can shape user experiences across demographic groups.However, two key challenges remain: (1) the lack of systematic comparison across value probing strategies, despite the Multiple Choice Question (MCQ) setting being vulnerable to perturbations, and (2) the uncertainty over whether probed values capture in-context information or predict models’ real-world actions.In this paper, we systematically compare three widely used value probing methods: token likelihood, sequence perplexity, and text generation.Our results show that all three methods exhibit large variances under non-semantic perturbations in prompts and option formats, with sequence perplexity being the most robust overall.We further introduce two tasks to assess expressiveness: demographic prompting, testing whether probed values adapt to cultural context; and value–action agreement, testing the alignment of probed values with value-based actions.We find that demographic context has little effect on the text generation method, and probed values only weakly correlate with action preferences across all methods.Our work highlights the instability and the limited expressive power of current value probing methods, calling for more reliable LLM value representations.