Overview
An assistant can answer from memory, carry out a quick search, or investigate more deeply. Before it sees the result, it has to weigh the chance of being right against the cost of looking. LLM-PriorCB uses descriptions of those tools and questions to set expectations in advance, then lets the observed rewards revise them.
Key Findings
One recommendation reveals one outcome
A movie service sees a user and a list of titles. It can learn from the rating attached to its recommendation, while the ratings it might have received for the other titles remain unknown. Kim and colleagues study settings where both sides of that choice are described in words. Film titles and genres, or a research tool’s description, can help before the first observed reward. The experiment asks whether a system can use those clues and still learn from outcomes that contradict them.
Teach the starting point offline
Before deployment, the authors ask a language model to estimate rewards for pairs of context and action descriptions. They average estimates across samples and train an encoder to turn each action’s description into the starting coefficients for its own linear reward model. No observed reward enters that stage. The same encoder can assign starting coefficients when a new action appears online, using its description.
Learn from the choice the system actually made
At deployment the bandit scores each action with its estimated reward and an uncertainty bonus. It updates the coefficients for the selected action once the actual reward arrives. The language model is only queried offline. In Section 5, the regret bound depends explicitly on how far the starting coefficients are from the true ones, under the paper’s linear reward assumptions. This helps explain why a weak initial preference can be useful while the system is still learning.
Follow the evidence from movies to search tools
In a semi-synthetic MovieLens experiment, the authors use a low-rank approximation of the ratings to define rewards. Figure 1 compares a regime that receives the underlying user features with one that receives only textual demographic descriptions. The proposed method accumulated less regret than the baselines in both settings. Figure 2 starts with priors trained on ten movies and increases the catalogue to thirty during online learning. The advantage persists in that comparison. Figure 3 shows that assigning more weight to the prior raised regret over the tested range.
The text-only GAIA experiment gives an assistant three options: answer using its own knowledge, issue one search, or issue three. A correct answer earns credit and the selected tool incurs a fixed cost. Figure 4 reports lower cumulative regret than LinUCB and CBLI when the prior was constructed from the evaluation questions or a separate set, using no answer labels in either case. The MovieLens rewards are simulated from ratings and the GAIA test reuses a limited validation set. Both results measure learning in controlled settings.
Questions & Answers
Q: What is the context in the help-center example?
A: It can be the visitor’s problem description and other available information about the current request. The actions are the links the system could show.
Q: Why can a prior be risky?
A: A persuasive action description can make a poor option look promising. Exploration and subsequent observations give the system opportunities to correct its initial estimate.
Q: How can a test distinguish a useful prior from one that recovers slowly from a mistake?
A: Measure the first decisions separately from how quickly the system improves after feedback arrives. A language-based prior can help when descriptions are reliable and still slow adaptation when they are misleading. In the paper’s MovieLens sensitivity test, giving more weight to the prior raises regret. A deployment can probe the same failure mode by comparing outcomes across plausible but inaccurate descriptions.
Future Ideas
Start in a simulated help-center catalogue with vetted actions and deliberately paraphrased requests. Compare a neutral starting distribution, text similarity, and a language-model prior under the same exploration policy and feedback budget. Measure wrong-link exposure, reward, and regret against the best action in the simulation, including how many observations it takes to recover when the initial ranking is intentionally wrong.