Learning & Reasoning / ICLR 2026

What Should an AI Hear Before It Hears the Question?

SPRIG searches for a reusable system prompt across many tasks. The experiments compare that general instruction with prompts optimized for individual tasks.

SPRIG: Improving Large Language Model Performance by System Prompt Optimization

Lechen Zhang, Tolga Ergen, Lajanugen Logeswaran, Moontae Lee, David Jurgens

Overview

Before an assistant sees a user’s question, it may already have instructions about how to answer, how much reasoning to show, or what format to use. Teams often assemble those instructions by intuition. This paper treats their wording and order as measurable choices, using performance on varied tasks to search for combinations that travel beyond one benchmark.

Key Findings

Changing one sentence in a system prompt is easy. Determining whether it helps across dozens of tasks is expensive. The researchers turn prompt writing into a search problem, with a smaller model screening which candidate instructions deserve a full evaluation.

Search with sentences that can be inspected

The example separates system-wide instructions, task-specific directions, and the individual question. SPRIG searches the first layer.
The example separates system-wide instructions, task-specific directions, and the individual question. SPRIG searches the first layer.

SPRIG starts from a collection of short instruction components, typically sentences. The authors collect examples from existing work, group them by purpose, and use GPT-4o to expand the pool to 9,000 components across nine categories. Candidate system prompts combine those components.

Scoring every candidate on the 42 in-domain benchmarks would be expensive. The researchers train a ModernBERT reward model on prompt pairs to predict which one is likely to perform better. Its average Spearman rank correlation of 0.59 on unseen prompts in Figure 3 makes it a useful screen, while leaving enough uncertainty that the search still needs real benchmark results.

A general instruction can complement task-specific wording

Appendix Table 2 reports average scores and 95% bootstrap confidence intervals for the comparison in Figure 4. CoT refers to a chain-of-thought prompt. ProTeGi is the task-level optimization method used in the comparison.

Prompt conditionAverage score95% confidence interval half-width
Base prompt0.52320.0434
Base CoT prompt0.59720.0376
SPRIG system prompt0.62870.0367
ProTeGi task prompt0.65410.0358
System and task prompts optimized0.67140.0353

The general prompt improves on the CoT prompt under the paper’s paired test. The combined condition has a higher point estimate than task optimization alone, and the paired comparison reports p=0.053p=0.053. The combined condition runs ProTeGi on the SPRIG-optimized system prompt. Simply concatenating separately optimized parts performed worse. The English system prompt produces significant gains on four of five multilingual benchmarks. Transfer to other similar-size models preserves only part of the gain, and the larger models tested improve significantly only with combined optimization.

A search over editable instructions

Each search round uses the reward model to discard the bottom half of the prompt population. The strongest prompts can combine with others from the retained group or change through edits that add, remove, rephrase, reorder, or merge components. A sample of 100 updated prompts then runs against the 42 benchmarks, so actual scores feed back into the search. The authors trained the initial ranking model on 100,000 pairs built from 10,000 prompts. The search can therefore explore more wording combinations than a complete evaluation of every candidate would permit. See the search loop in Figure 2.

One good score can leave another problem

Figure 6 separates which questions the system and task optimizers miss. Both miss 18% of questions, while on a further 28% one succeeds and the other fails. Their changes therefore help different subsets of the benchmark. Figure 7 finds the largest gains in math and reasoning and smaller gains on knowledge-heavy tasks. These patterns help explain why the combined approach can help. A prompt tuned for an average score across a particular benchmark mix still needs evaluation on the work it will actually encounter.

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: What is the difference between a system prompt and a task prompt?

A: A system prompt supplies general instructions across interactions. A task prompt provides the wording and context for a particular task. Our experiment varies them separately and tests how they work together.

Q: Why train a prompt-ranking model?

A: The search evaluates many candidate prompts. A ranker can cheaply screen them before selected prompts undergo more expensive benchmark testing. It estimates quality from previous results and can misrank unfamiliar instruction styles.

Q: Should teams optimize every sentence?

A: Some instructions express nonnegotiable requirements such as access boundaries, privacy, or tone commitments. A team can hold those fixed and optimize wording that is legitimately open to empirical comparison.

Q: Will an optimized system prompt transfer to the next checkpoint?

A: The paper finds that moving a prompt to another model of similar size can lose much of the gain, and some task families respond less than others. The comparison with task prompt optimization also leaves less evidence for an additional gain than the comparison with an ordinary system prompt. Test edited prompts by task family and language on the intended checkpoint before reusing them.

Future Ideas

A change-review tool for shared assistant prompts

Keep a reusable system prompt in separate, named instruction components. Allow a search to change one or two editable components at a time while preserving fixed safety and format requirements. Attach to each candidate a test report that shows gains, regressions, and conflicts with existing instructions.

Test on customer-support, scheduling, and document-understanding tasks, including two languages and a newly released model checkpoint. Compare against manual edits and unconstrained search with the same test budget. Track average accuracy, the weakest task’s performance, instruction conflicts, and prompt length. Deploy a candidate only when it meets the fixed requirements and survives a held-out regression check.

Sources

Original abstract

Large Language Models (LLMs) have shown impressive capabilities in many scenarios, but their performance depends, in part, on the choice of prompt. Past research has focused on optimizing prompts specific to a task. However, much less attention has been given to optimizing the general instructions included in a prompt, known as a system prompt. To address this gap, we propose SPRIG, an edit-based genetic algorithm that iteratively constructs prompts from prespecified components to maximize the model's performance in general scenarios. We evaluate the performance of system prompts on a collection of 47 different types of tasks to ensure generalizability. Our study finds that a single optimized system prompt performs on par with task prompts optimized for each individual task. Moreover, combining system and task-level optimizations leads to further improvement, which showcases their complementary nature. Experiments also reveal that the optimized system prompts generalize effectively across model families, parameter sizes, and languages. This study provides insights into the role of system-level instructions in maximizing LLM potential.