Overview
When a language model learns a task from a handful of examples, every example usually goes into the same prompt. A weak example can compete for space and attention with a strong one. Demonstration ensembling runs the same model with different groups of examples, then combines their predictions. Each group can contribute evidence without sharing the same prompt.
Key Findings
A prompt can contain examples that compete with each other
An in-context classifier reads a few labeled examples and makes a prediction on a new input. A common prompt places every example in one long block. The examples closest to the query can end up beside unrelated ones, and changing their order can change a model’s prediction. The authors try letting smaller groups of examples speak independently. The same language model reads each group with the same test query, and the system combines the answers.
Give each group a vote and decide how to count it
The authors call the method DENSE. Each bucket yields a score for candidate answer on input . Three rules combine these scores. A product favors answers that keep support across groups. A weighted average combines support additively. The maximum lets one group’s strong score dominate. Equation 4 gives the last rule, where is the bucket’s importance weight.
The weights can be equal or depend on relevance to the current input. For the relevance weights, a sentence encoder turns texts into vectors. The method uses cosine similarity, which compares their directions, and averages it over the examples in each bucket. That gives the query a role in deciding which group to trust.
Put similar examples together or spread them out
The paper tests ordinary grouping by demonstration order and two clustering-based variants. The first puts similar examples together so a group can specialize. The second tries to give each group a mix, selecting examples across clusters. A small group with a narrow theme can make a strong prediction for one kind of question. A mixed group can help if the test input needs evidence across themes. The experiment asks which benefit matters at six or ten demonstrations.
A stronger prompt baseline changes the result
The authors compare GPT-J 6B across 12 datasets. One baseline concatenates the examples in their original order. A stronger baseline sorts the same examples by relevance so that similar ones are closest to the query. That distinction matters because the weighted ensemble also uses similarity. The authors report these gains from the weighted maximum, averaged across datasets, three demonstration seeds, and several bucket counts in the Figure 2 analysis.
| Demonstrations | Gain over ordinary concatenation | Gain over relevance-sorted concatenation |
|---|---|---|
| Six | 2.4 points | 1.2 points |
| Ten | 1.9 points | 0.75 points |
Equal-weight ensembles can improve on unsorted concatenation while trailing the relevance-sorted baseline. With six examples, keeping similar examples together helps weighted variants in the reported comparison. The best grouping depends on the task and number of examples. Smaller bucket counts tend to do worse in Figure 3. Some test sets are small, and the experiments use one language model. The practical choice also depends on the cost of scoring each query against several separate prompts.
Questions & Answers
The opening answers are simulated author responses based on the paper.
Q: Do the groups use different models?
A: We use the same GPT-J model for each group. Its input examples change, and we combine its scores afterward.
Q: Does a larger context window solve the same problem?
A: A larger window can fit more examples. An ensemble changes how the examples influence the final score, which is a separate question.
Q: Can we treat the margin from every ensemble as a probability?
A: No. Maximizing a score across demonstration groups can produce scores that do not sum to one. Compare the margin with error frequency on held-out data before using it to decide which cases need extra review. The paper uses one underlying 6B model, so the size of any gain from this rule could change with the model and workload.
Future Ideas
Send a hard support ticket to several example groups only when needed
Build a ticket-routing pilot where examples come from several closely related queues. Start with one relevance-sorted prompt. When its top two labels have a small score margin, run a small group of weighted prompts. Choose the threshold on a validation split and measure error by margin on the held-out set. Compare this conditional strategy with always running an ensemble and with spending the same average token budget on a larger single prompt. Report rare-queue recall, misrouting cost, and end-to-end latency on held-out tickets.
Sources
Original abstract
In-context learning (ICL) operates by showing language models (LMs) examples of input-output pairs for a given task, i.e., demonstrations. The standard approach for ICL is to prompt the LM with concatenated demonstrations followed by the test input. This approach suffers from some issues. First, concatenation offers almost no control over the contribution of each demo to the model prediction. This can be sub-optimal when some demonstrations are irrelevant to the test example. Second, due to the input length limit of some transformer models, it might be infeasible to fit many examples into the context, especially when dealing with long-input tasks. In this work, we explore Demonstration Ensembling (DENSE) as an alternative to simple concatenation. DENSE predicts outputs using subsets (i.e., buckets) of the demonstrations and then combines the output probabilities resulting from each subset to produce the final prediction. We study different ensembling methods using GPT-j and experiment on 12 language tasks. Our experiments show weighted max ensembling to outperform vanilla concatenation by as large as 2.4 average points. Code available at https://github.com/mukhal/icl-ensembling.