Overview
A household simulator asks an AI agent to put a mug in a coffee machine. A plausible command like “take the mug from the cabinet” is useless if the cabinet is empty. Prospector gives the acting model examples of stopping to ask whether it has found the right object. It can also try multiple sequences of actions and have a second model choose the sequence most likely to succeed. The experiments measure what those two changes add to agents navigating a virtual home or shopping site.
Key Findings
An agent buying a product online can make a plausible decision at each step and still end with the wrong item. The paper tries to catch trouble at two points. It makes the model ask a practical question during each attempt, then gives a separate critic a chance to compare completed attempts.
Ask, act, then compare routes
In a text-based environment, an agent reads a description of the current state and issues an action. ReAct prompts it to alternate reasoning and action. Prospector’s AskAct extends the examples in that prompt with a concrete question and answer about the goal or progress. If the model has seen no mug, the next action can be to look elsewhere. The examples are written for the task. The paper describes them in Figure 3 and gives full prompts in the appendix.
Prospector’s second change is trajectory ranking. A trajectory is the history of observations, reasoning, and actions from one attempt. The actor samples several trajectories. A critic reads them and predicts the reward each would earn. In ALFWorld, a prediction of “success” maps to 1 and “fail” maps to 0. If repeated critic samples yield success, success, and fail, their mean is two-thirds. The method orders candidate trajectories by their predicted mean reward. On WebShop, the agent can compare candidate paths before pressing “buy” to obtain the final reward.
What improves with another attempt?
| Llama-2-70B actor configuration | ALFWorld success | WebShop success |
|---|---|---|
| ReAct | 41.0% | 37.6% |
| AskAct | 56.7% | 42.2% |
| ReAct with trajectory ranking | 77.6% | 42.2% |
| AskAct with trajectory ranking | 86.6% | 43.6% |
Selected success rates from Tables 1 and 4 of the EMNLP paper. The ranking conditions use five sampled trajectories on ALFWorld and eight on WebShop, with a fine-tuned FLAN-T5 critic in these rows. The gain varies across tasks. With a text-davinci-002 actor and critic, Table 1 reports 91.0% on ALFWorld for both ReAct with ranking and AskAct with ranking. Table 4 also includes human WebShop reference scores of 50.0% for the average and 59.6% for the expert.
Table 2 changes the number of candidates with the ALFWorld actor and critic held fixed. Generating more attempts helps both prompts, and the first extra AskAct attempt gains 19.4 percentage points. By the fifth attempt, adding the last candidate gains 2.3 points. The experiment reports success at each candidate count, which exposes the trade-off with extra generations.
| Actor prompt | 1 attempt | 2 attempts | 3 attempts | 4 attempts | 5 attempts |
|---|---|---|---|---|---|
| ReAct | 41.0% | 59.0% | 69.4% | 73.1% | 77.6% |
| AskAct | 56.7% | 76.1% | 80.6% | 84.3% | 86.6% |
A comparison across attempts
A critic can learn from the reward the environment provides for completed attempts. If is the probability that the critic assigns reward label to trajectory , Equation 4 trains it to maximize the probability of the labels actually earned:
In ALFWorld the labels are success and failure. In WebShop they divide the score into high, medium, and low reward. Once trained, the critic can sample more than one judgment of the same attempt. If is the numeric value assigned to critic sample , the sample estimate of Equation 2 is:
The ranking chooses from the attempts the actor actually made. The critic sees how a decision changed what the actor could do next. If the actor never proposes a valid outcome, even a perfect ranking model has nothing useful to select. Table 3 · Table 5
The same intervention has different room to help
Trajectory ranking raises the ALFWorld success rate far more than its WebShop counterpart for the same actor. One difference appears in the critic's test, where few-shot reward classification is more accurate on ALFWorld. Tables 3 and 5 measure different label sets, so the rows describe how much fine-tuning helped within each task:
| Environment and label set | Two-shot text-davinci-002 | Fine-tuned FLAN-T5 3B |
|---|---|---|
| ALFWorld · success or fail | 97.0% | 98.5% |
| WebShop · high, medium, or low | 47.0% | 78.0% |
The authors fine-tuned on 3,000 ALFWorld and 12,000 WebShop trajectories. With the text-davinci-002 actor on WebShop, the paper reports 43.0% success using a fine-tuned Llama-2 critic and 47.0% with access to the candidate rewards. The difference between those two systems points to a ranking opportunity. The gap between even that oracle and human evaluation also leaves room to find better candidates. Section 3.2.1 and Tables 3–5
Questions & Answers
The opening answers are simulated author responses based on the paper.
Q: What is the difference between the actor and critic?
A: The actor proposes actions and follows the resulting observations. The critic estimates the outcome of a whole trajectory. The actor can use a few examples in its prompt, while the critic can be trained on labeled trajectories from the target environment.
Q: Why predict reward if the environment already provides it?
A: A final reward may become available only after the action has taken effect. In the shopping simulation, the final reward comes when the agent buys an item. Our ranking method uses a prediction so the agent can compare candidates before that point.
Q: Does a more accurate critic always produce a better agent?
A: Critic accuracy measures predictions on a set of trajectories. Agent success also depends on which candidates the actor generates, how often the critic ranks the right one first, and how errors change when the actor or environment changes. Measuring the full agent catches these additional failure modes.
Q: When are more attempts worth the cost?
A: The answer depends on how much quality rises with an additional candidate and what it costs to run. The ALFWorld results in Table 2 show improvements as the number of trajectories increases. A production system would need a budget and evidence about where the extra attempts help enough to justify that cost.
Q: How do you separate a weak critic from a weak set of candidate attempts?
A: First ask whether any generated trajectory would have succeeded. If the candidate set contains a success but the critic ranks another trajectory first, the ranking is the bottleneck. When no candidate would succeed, the actor needs a better set of attempts. Keep the generated set fixed when comparing critics, and charge for every attempt when comparing whole agents.
Future Ideas
Compare spreadsheet repairs before touching the original
For a shared spreadsheet with broken formulas, an actor could propose several repairs inside temporary copies. It would check which cells are allowed to change, ask which totals should remain invariant, and describe each candidate as a diff. A critic could rank the candidates using prior accepted repairs. Deterministic checks would reject changes that break protected cells or stated invariants. A person would approve the remaining proposal before any write to the original.
A feasible pilot can use spreadsheets with known synthetic faults and a small set of human-reviewed repairs. Compare single-attempt repair, several attempts selected by deterministic checks alone, and the same attempts ranked with the critic. Hold the number of attempts and token budget fixed. Measure successful repair, unintended changes, and reviewer time. Keep the critic only if it improves selection beyond the deterministic checks without increasing the rate of unrelated edits.
Include real, human-reviewed formula faults in the holdout. On that slice, report candidate-set repair coverage separately from whether the critic picks a successful repair when one is available.
Sources
Original abstract
Large language models (LLMs) have shown the ability to solve complex decision-making tasks beyond natural language processing tasks. LLM agents based on few-shot in-context learning (ICL) achieve surprisingly high performance without training. Despite their simplicity and generalizability, ICL-based agents are limited in their ability to incorporate feedback from an environment. In this paper, we introduce Prospector, an LLM agent that consists of two complementary LLMs, an Actor and a Critic. To elicit better instruction-aligned actions from the LLM agent, we propose AskAct prompting that performs an additional self-asking step such as goal and progress checking before generating an action. Furthermore, to implicitly incorporate the environment feedback, we propose Trajectory Ranking that orders generated trajectories by predicting the expected total reward. Prospector encourages the LLM Actor to generate diverse (creative) trajectories, and harnesses the LLM Critic to select the most rewarding trajectory. On representative decision-making benchmark environments such as ALFWorld and WebShop, we empirically demonstrate that Prospector can considerably increase the success rate of given tasks, while outperforming recent advancements such as ReAct and Reflexion.