Overview
When a model gets a biology question wrong, a tempting fix is to tell it to answer as a biologist. The sentence may change how the answer sounds. Whether it improves accuracy is a separate, measurable question. This study runs the same factual questions with and without social roles and tests whether a system could pick the right role in advance.
Key Findings
“You are a helpful assistant” sounds like a direction. “You are an expert physicist” sounds more specific. This study treats those instructions as an experimental variable and asks what changes when the task stays exactly the same.
Test the label separately
The researchers sample 2,410 multiple-choice MMLU questions across 26 subjects. They collect 162 role labels, including occupations and interpersonal roles. They try prompts that assign the role to the model and prompts that describe the person the model is talking to. The control presents the question without a role.
Roles sometimes shift accuracy. Figure 2 shows the ten strongest and ten weakest estimated role effects, with the full ranking in the appendix, and reports no statistically better result for the leading roles overall. The per-model results in Figure 4 also include roles that reduce accuracy relative to the no-role control.
| Model in Figure 4 | No significant change | Significant decrease |
|---|---|---|
| FLAN-T5-XXL | 27.2% | 72.8% |
| Llama 3 8B | 88.3% | 11.7% |
| Llama 3 70B | 73.5% | 26.5% |
| Mistral 7B | 94.4% | 5.6% |
| Qwen2.5 7B | 100.0% | 0.0% |
| Qwen2.5 72B | 100.0% | 0.0% |
The figures summarize role effects under the paper’s experiment. A nonsignificant result leaves open effects smaller than the experiment could reliably detect.
Looking backward makes role choice look easier
For a particular question, some roles elicit a correct answer and others do not. If a system could examine the answer key and keep the best role for each question, its score could be much higher. That calculation is an upper bound. It does not tell us how to make the choice before we know the answer.
The authors try selection rules based on topic, semantic similarity, and learned role classification. Figure 9 shows that they fall well below the retrospective upper bound and offer at most a small gain over picking roles randomly and perform worse than random for Qwen2.5-72B. A higher domain-classification score does not automatically translate into choosing a role that changes the answer in the right direction.
Who is speaking matters less than the label suggests
The researchers fit models to compare families of roles and prompt formats. Audience-specific prompts fare worse than the other role templates. In a separate test restricted to 16 aligned interpersonal labels, gender-neutral roles have a small estimated advantage over gendered versions. Work and school roles also fare better than AI and occupational roles in the category comparison. These are aggregate differences within the particular labels and models tested. They offer little guidance about which role to choose for the next question. Figures 3 and 6–7
An accuracy result with a defined border
The study measures answers to factual, multiple-choice questions. Its design lets the authors count right answers under role changes at scale. It cannot measure whether a persona makes a conversation warmer or a specialized writing task easier. Within the factual test, the no-role control is a useful anchor precisely because the effects vary by model and by role.
Questions & Answers
The opening answers are simulated author responses based on the paper.
Q: Could a role help a single question?
A: Yes. A prompt can change an individual prediction. The hard part is choosing the role without access to the correct answer. Our retrospective result illustrates the potential, while the selector experiments measure what happened with usable selection rules.
Q: Does the paper say a role can never help a tutor?
A: Our outcome is factual multiple-choice accuracy on a sampled MMLU set. A tutoring persona may change clarity, style, or interaction. Those outcomes need their own evaluation.
Q: Does a domain-matched role make the model an expert?
A: A domain match is still text in a prompt. It can cue knowledge the model already has, while its small measured effect offers little reason to assume it reliably improves accuracy.
Q: Why can the best role in hindsight be difficult to use on a new question?
A: The hindsight comparison knows which role already led to the correct answer. An operational selector must choose before it sees that answer. The paper's automatic selection results show that the advantage can shrink sharply in that setting. A role prompt that helps after the fact may have little value as a choice you have to make up front.
Future Ideas
Keep an answer test beside a style test
For a university help desk, compare a no-role prompt, a role label, and concrete style instructions such as answering briefly and citing the relevant policy. Keep the model and source documents fixed. Score factual accuracy and citation support before collecting user ratings for clarity and tone.
Use held-out questions covering common and rare requests, then report the scores separately. Retain a style change only if its usability benefit survives without a material loss of accuracy. This makes it possible to choose a voice with evidence about what it actually changes.
Add blinded checks for unsupported claims of expertise and stereotyping to the style test. Power the accuracy comparison to detect the small changes that could matter at help-desk volume.
Sources
Original abstract
Prompting serves as the major way humans interact with Large Language Models (LLM). Commercial AI systems commonly define the role of the LLM in system prompts. For example, ChatGPT uses ”You are a helpful assistant” as part of its default system prompt. Despite current practices of adding personas to system prompts, it remains unclear how different personas affect a model’s performance on objective tasks. In this study, we present a systematic evaluation of personas in system prompts. We curate a list of 162 roles covering 6 types of interpersonal relationships and 8 domains of expertise. Through extensive analysis of 4 popular families of LLMs and 2,410 factual questions, we demonstrate that adding personas in system prompts does not improve model performance across a range of questions compared to the control setting where no persona is added. Nevertheless, further analysis suggests that the gender, type, and domain of the persona can all influence the resulting prediction accuracies. We further experimented with a list of persona search strategies and found that, while aggregating results from the best persona for each question significantly improves prediction accuracy, automatically identifying the best persona is challenging, with predictions often performing no better than random selection. Overall, our findings suggest that while adding a persona may lead to performance gains in certain settings, the effect of each persona can be largely random. Code and data are available at https://github.com/Jiaxin-Pei/Prompting-with-Social-Roles.