Learning & Reasoning / arXiv 2025

A System Prompt Has to Travel Across Languages

A multilingual study scores prompt quality by accuracy, consistency, and variability instead of optimizing only an English task score.

Cross-Lingual Prompt Steerability: Towards Accurate and Robust LLM Behavior across Languages

Lechen Zhang, Yusheng Zhou, Tolga Ergen, Lajanugen Logeswaran, Moontae Lee, David Jurgens

Overview

A system prompt that elicits careful reasoning in one language may change the model’s behavior in another. An assistant may even switch languages within a reply. This paper tests how prompt components steer performance across five languages and adapts the existing SPRIG optimizer with a four-metric objective to search English system prompts that hold up across those languages.

Key Findings

A system prompt can work well in English and still change what a model does when the user switches languages. The research team treats that gap as a feature of the prompt that can be measured and optimized. Their study follows the change from accuracy all the way to the structure of the answer.

Judge a prompt by several outcomes

The study measures average accuracy, variation in accuracy, cross-language consistency, and variation in output length. It examines components including chain-of-thought instructions, emotional phrasing, and scenario descriptions. Correlations from those comparisons motivate the search. They do not show that adding any one component will improve every language or task.

Table 4 reports optimized and random-prompt means across the three benchmarks. Consistency measures agreement across language versions in the study.

ModelAverage accuracy, random → optimizedConsistency, random → optimized
Qwen2.5-7B-Instruct0.521 → 0.6210.254 → 0.311
Llama-3.1-8B-Instruct0.376 → 0.4160.125 → 0.213
Gemma-3-12B-Instruct0.614 → 0.6980.416 → 0.462

These averages improve, while individual settings can move the other way. Llama’s MATH500 consistency falls from 0.027 to 0.011 in the table, and its accuracy rises from 0.249 to 0.285. The paper also analyzes reasoning traces and language choice.

Judge the average and the spread

Average performance tells us how well a prompt works across the tested languages. Its spread shows whether some languages benefit more than others. The paper separately measures agreement between answers and variation in response length. The metrics can move in different directions. A model can become more accurate overall while behaving less consistently in one task, as the MATH500 entry shows.

The study estimates how components such as reasoning instructions, role cues and cross-language directions relate to four multilingual metrics.
The study estimates how components such as reasoning instructions, role cues and cross-language directions relate to four multilingual metrics.

Read the reasoning units alongside the scores

The analysis breaks 1.6 million responses into roughly ten million reasoning units. That makes it possible to look at patterns associated with useful prompts instead of only comparing finished answers. The study adapts SPRIG to screen candidate prompts against its multilingual objectives. Figure 4 shows positive associations for reasoning instructions, emotional language and scenario descriptions in the aggregate analysis. Role, style and cross-language instructions are associated with lower accuracy and consistency in that setup. The screen therefore has to learn which combinations serve the full brief. The reasoning observations explain what the comparison found about model behavior. They do not establish that forcing the same pattern will cause an improvement on a new task.

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: Why track output length across languages?

A: Large differences can flag behavior changes that average accuracy misses. Length can also reflect language structure, so it needs interpretation.

Q: Does consistency guarantee the answer is right?

A: An answer can be consistently wrong. We report accuracy and agreement separately to expose that possibility.

Q: Can I infer the prompt search works equally well in every language?

A: The study optimizes English system prompts for multilingual tasks. A gain on average can coexist with a regression for a particular model, task, or language. A practical test should check semantic equivalence of translated questions and require a minimum result in each supported language before comparing the mean.

Future Ideas

A regression test for multilingual help pages

Have bilingual reviewers write equivalent requests for a knowledge base. Compare candidate prompts on claim correctness, accuracy variance across languages, requested-language adherence, and unintended switches at matched token budgets. Require each language to pass a minimum quality threshold and reserve a held-out product family for the final comparison.

Include a separate bilingual check of technical terms and conditions in the test questions. Compare the worst-language result and variation across languages before choosing a prompt on its average score.

Sources

Original abstract

System prompts provide a lightweight yet powerful mechanism for conditioning large language models (LLMs) at inference time. While prior work has focused on English-only settings, real-world deployments benefit from having a single prompt to operate reliably across languages. This paper presents a comprehensive study of how different system prompts steer models toward accurate and robust cross-lingual behavior. We propose a unified four-dimensional evaluation framework to assess system prompts in multilingual environments. Through large-scale experiments on five languages, three LLMs, and three benchmarks, we uncover that certain prompt components, such as CoT, emotion, and scenario, correlate with robust multilingual behavior. We develop a prompt optimization framework for multilingual settings and show it can automatically discover prompts that improve all metrics by 5-10%. Finally, we analyze over 10 million reasoning units and find that more performant system prompts induce more structured and consistent reasoning patterns, while reducing unnecessary language-switching. Together, we highlight system prompt optimization as a scalable path to accurate and robust multilingual LLM behavior.