Learning & Reasoning / ICML 2023

A Library of Small Experts Can Rival One Broadly Tuned Model

The best single expert beats T0-3B on the reported unseen generation suite. An off-the-shelf similarity retriever falls short, leaving a clear gap between having the right expert and finding it.

Exploring the Benefits of Training Expert Language Models over Instruction Tuning

Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee, Minjoon Seo

Overview

A language model trained on many instructions has to fit many tasks into one set of weights. Moontae and his colleagues ask what happens if the tasks are trained separately. Their approach builds a library of small adapters and tries to retrieve one when a new task arrives. The experiments reveal strong transfer from individual experts, but they also show how easily the routing rule can miss the expert that works best.

Key Findings

Figure 1 from Exploring the Benefits of Training Expert Language Models over Instruction Tuning
Figure 1 from Exploring the Benefits of Training Expert Language Models over Instruction Tuning

Which specialist should answer the question?

A model that summarizes meetings may also be good at turning a long complaint into a short account of what went wrong. It may be a poor choice for deciding which of two events happened first. Instruction tuning usually puts many kinds of task into one training run. The authors try a different arrangement. They train specialists separately and build a library that can match the incoming work to a suitable specialist. The experiments ask when that separation preserves skills that would otherwise compete during training and when finding the right specialist becomes the harder problem.

Build a library, then find the shelf

The authors start with a common T5 base and train small, separately stored adapters for 296 prompts drawn from 36 training datasets. An adapter is a small trainable addition to the frozen base model. Two prompts applied to the same dataset can become two specialists. A second set of experiments trains whole models at the dataset level. This lets the authors test merging methods as well as switching among adapters. See the training hierarchy in Figure 3.

Each library entry pairs an embedded training example with the identity of its expert. At inference time the system embeds sampled examples of the new task, including the possible answer choices where applicable. It retrieves one similar training example for each of 32 queries and chooses the specialist that appears most often. The choice uses task examples rather than access to their correct target answers. An oracle benchmark instead chooses the expert with the best measured results on each evaluation task. The gap between the two measures room for improving retrieval.

A strong specialist can travel, and a weak router can waste one

One specialist trained on a Cosmos QA prompt reached an average of 54.63 across 11 unseen classification datasets, compared with 51.43 for T0-3B. Its mean on the 13 BIG-Bench tasks was also above T0-3B in the reported comparison. For eight generative tasks, a summarization specialist trained on SAMSum reached 44.81, against the same baseline’s 37.98. These are specific experts selected after evaluation. The library’s real test is whether retrieval can find a good expert before the answer is known.

SetupUnseen classification mean, Table 1Unseen generation mean, Table 3
T0-3B51.4337.98
T5-3B with retrieved expert53.4832.61
T5-3B with best expert for each task63.3751.67

Classification uses mean accuracy. The generative benchmark averages scores from task-specific metrics, including ROUGE, BLEU, and BERTScore. On these tasks, the simple retriever is enough to improve classification relative to T0-3B. Generation goes the other way, even though the oracle reveals that the library contains strong generative experts. The oracle is a diagnostic, since choosing it requires access to evaluation results.

What happens when specialists work together?

Switching between specialists gives one model the job. Merging asks whether two models’ changes to the base parameters can coexist in one. Equation 4 shows the authors’ merge, where θpre\theta_{\mathrm{pre}} is the common starting point, τi\tau_i is expert ii’s learned parameter change, and λi\lambda_i weights its contribution.

θnew=θpre+∑i=1Nλiτi\theta_{\mathrm{new}} = \theta_{\mathrm{pre}} + \sum_{i=1}^{N}\lambda_i\tau_i

In Table 4, averaging the two adapter specialists improves COPA and Story Cloze relative to the individual adapters, while the mean drops below both. A merge of fully tuned dataset experts has a small mean advantage over either individual expert in that test. In a separate multilingual trial, the authors combine summarization and translation experts to summarize English text in five other languages. The merged model has higher ROUGE-L in four of the five settings, and trails the multilingual baseline for Chinese. The mixed outcomes matter if the library grows. Every new combination still needs evaluation. See the multilingual comparison in Table 7.

Adding a specialist while keeping the rest of the library

A library can also accept a new skill by training a new expert. The authors compare this approach with continually tuning a shared model on new generative tasks while replaying examples from earlier tasks. In Table 6, the extended library has a mean generative score of 55.60 compared with 54.52 for the continually tuned model. Its score on the previously unseen classification datasets is 53.33, close to 53.48 before adding the experts. Table 5 separately shows strong results for the library on already seen datasets. With familiar instructions, the retriever often finds the training dataset that produced the matching expert. The trade-off is where the engineering work sits. A library simplifies adding independent training runs, and serving it requires storage and reliable task routing.

Questions & Answers

Q: Does each expert need its own copy of the language model?

A: Prompt experts share the frozen T5 base and train separate adapters. The authors also train full-model dataset experts in a separate part of the paper to investigate merging.

Q: Why can a summarization expert help another generation task?

A: Some skills can transfer across formats, such as finding salient information and expressing it concisely. The observed transfer varies by task, so the training dataset's name alone cannot predict which expert will work.

Q: Why is the best expert score so much higher than retrieval?

A: The oracle picks an expert after seeing which one performs best. The actual router only knows which stored examples are close in embedding space. Surface similarity may miss which adaptation changes the model in a useful way.

Q: What does the best-expert result reveal about the cost of routing?

A: It measures a hindsight upper comparison using the target suite. A live router must infer the task from examples and can select a weaker adapter. The possible gain from adding a specialist therefore depends on routing error and the cost of switching or batching adapters.

Future Ideas

Use a small calibration set to route a new document task

Give a support-document service a handful of newly labeled tasks each month. Compare nearest-example routing, a router calibrated from a few labeled samples per new task, and a general model. Keep the adapters and base model fixed. Add a fallback for tasks on which the calibrated router is uncertain.

Hold out whole document families and report quality at zero, three, and ten labels per task. Track adapter switches, latency, and the gap to the hindsight oracle. Use that gap to identify tasks that need new experts and tasks that already have a capable expert the router overlooks.

Train a separate adapter library for a second backbone if resources permit, then check whether the same calibration rule still improves routing there.

Sources

Original abstract

Recently, Language Models (LMs) instruction- tuned on multiple tasks, also known as multitask- prompted fine-tuning (MT), have shown the capa- bility to generalize to unseen tasks. Previous work has shown that scaling the number of training tasks is the key component in making stronger MT LMs. In this work, we report an unexpected find- ing that an expert LM fine-tuned on just a single task can outperform an MT LM trained with 300+ different tasks on 11 different unseen datasets and on 13 datasets of the BIG-bench benchmark by a mean accuracy of 3.20% and 1.29%, respec- tively. This finding casts doubt on the previously held belief that simply scaling the number of tasks makes stronger MT LMs. Leveraging this finding, we further show that this distributed approach of training a separate expert LM per training task in- stead of a single MT LM for zero-shot inference possesses many benefits including (1) avoiding negative task transfer that often occurs during instruction tuning, (2) being able to continually learn new tasks without having to re-train on previ- ous tasks to avoid catastrophic forgetting, and (3) showing compositional capabilities when merging individual experts together. The code is available at https://github.com/joeljang/ELM.