Overview
An agent keeps trying to take a mug it hasn’t seen. An example in its prompt teaches it to pause and ask whether the mug is there. Prospector pairs this simple check with a second mechanism: it generates multiple attempts and ranks them by predicted reward. The 2023 workshop experiments show where each mechanism helps and how much an imperfect critic leaves on the table.
Key Findings
Check the evidence before the next action
The actor receives a text observation , then issues an action . Its context includes the observations and actions so far, as written in Section 2.1 of the workshop manuscript:
ReAct adds an intermediate thought to the context. AskAct’s few-shot demonstrations add a concrete question and answer about the goal or progress. The ALFWorld example in Figure 2 asks whether the mug has been found, and the actor changes where it looks based on the answer. The WebShop example asks which product matches the requested color, size, and price before selecting it. These are illustrative traces, with tested success rates reported separately.
Score completed attempts
AskAct can improve a single attempt. Prospector also varies its trajectories by sampling the actor at a higher temperature, then uses a separate critic to predict the task reward. Figure 1 shows the trajectory as the sequence of observations, reasoning, and actions. In explanatory notation, if is the candidate set and is the critic’s prediction, the final choice is:
A few-shot critic learns what to look for from labeled examples in its prompt. ALFWorld uses success and failure labels. WebShop’s critic can predict low, middle, or high reward, and the authors also fine-tuned critics on trajectories labeled with environmental rewards. At ranking time the ordinary critic estimates those rewards from the trajectory. The workshop’s WebShop oracle comparison shows what ranking could achieve with the actual rewards for the candidate set.
Measure success in the same benchmark
The workshop evaluates 134 unseen, text-only ALFWorld tasks and 500 text-based WebShop test instructions. These are selected rows of workshop Tables 2 and 7, with Llama-2-70B as the actor throughout. Ranking uses a fine-tuned FLAN-T5 3B critic with five ALFWorld candidates or eight WebShop candidates.
| Actor and selection | ALFWorld success | WebShop success |
|---|---|---|
| ReAct | 41.0% | 37.6% |
| AskAct | 56.7% | 42.2% |
| ReAct plus trajectory ranking | 77.6% | 42.2% |
| AskAct plus trajectory ranking | 86.6% | 43.6% |
With a text-davinci-002 actor on ALFWorld, the five-candidate ReAct ranking configuration reaches 91.0%, compared with 78.4% for ReAct alone and 86.0% for five Reflexion refinements. On WebShop, that actor paired with AskAct and a fine-tuned Llama-2-7B-Chat critic reaches 43.0%. An oracle ranking the evaluated candidates reaches 47.0%, and the table lists 50.0% as the average human reference.
Keep candidate count and critic accuracy separate
In the candidate-count sweep in Table 3, AskAct with ranking goes from 53.7% success at one candidate to 76.1% at two and 86.6% at five. The same sweep with ReAct yields 33.6%, 59.0%, and 77.6%, respectively. The one-candidate sweep entries are separate from the standalone baselines in Table 2. The paper reports considerable gains from the early additional attempts and smaller gains near five candidates in these settings.
The reward-label tests also depend on the environment. In Tables 6 and 9, two-shot text-davinci-002 classification scores 97.0% in ALFWorld and 47.0% on WebShop. Fine-tuned FLAN-T5 3B scores 98.5% and 78.0%, respectively, in those tests. ALFWorld’s test has two classes and WebShop’s has three. For WebShop, the workshop also reports in Table 10 that increasing the Llama-2-7B-Chat critic’s training set from 3,000 to 9,000 trajectories raises this label accuracy from 70.0% to 76.2%, with 76.2% again at 12,000.
The agent’s ultimate success still depends on both stages. At least one candidate needs to satisfy the instruction, and the critic needs to rank that candidate high enough to select it. In the task breakdown in Table 4, AskAct plus ranking improves the Llama-2 actor’s aggregate over ReAct plus ranking but drops from 9 of 17 to 8 of 17 on the two-object task. This is a useful example of why a single aggregate cannot tell an implementer which tasks will benefit.
Questions & Answers
The answers below are editorial explanations based on the workshop manuscript.
Q: What does the critic get to choose?
A: It ranks the complete trajectories already generated by the actor. It can choose a stronger route from that pool. If no candidate passes, the actor needs a better route.
Q: Why do two mechanisms help the same agent?
A: AskAct improves decisions within individual attempts by giving the actor examples of checking the task state before acting. Ranking compares those attempts afterward. The workshop’s separate actor settings and candidate-count sweep show what changes under each intervention.
Q: Does this use a trained actor?
A: AskAct is a prompt change. Some ranking experiments use a few-shot critic, and others fine-tune a separate critic on labeled trajectories. The actor can remain fixed in both cases.
Q: How can we tell whether the critic is the bottleneck?
A: For a fixed candidate pool, compare the actual reward of its best candidate with the reward earned by the selected candidate. The workshop reports both a learned-critic result and an oracle result in WebShop. The gap indicates room for improved selection in that experiment.
Future Ideas
In a shopping sandbox, let a critic rank several candidate carts and show its reasons next to the selected items. Compare with choosing the actor’s first result and with a rule-based constraint checker. Give each system the same candidate pool and measure completed requests, missed constraints, critic error against the sandbox’s actual reward, and the time needed for a reviewer to correct the cart.
Sources
- Foundation Models for Decision Making workshop paper list
- NeurIPS 2023 workshop manuscript
- LG AI Research publication record
- Later EMNLP Findings edition
Original abstract
Large language models (LLMs) have shown the ability to solve complex decision-making tasks beyond the natural language processing tasks. Current LLM agents such as ReAct can solve interactive decision-making tasks by imitating the few-shot demonstrations given in the prompt. The LLM agents based on few-shot in-context learning (ICL) achieve surprisingly high performance without training. Despite the simplicity and generalizability, the ICL-based approaches lack optimizing trajectories based on the reward from an environment. In this paper, we introduce Prospector, a LLM agent that consists of two complementary LLMs such as the LLM Actor and LLM Critic. To elicit more proper actions from the LLM Actor, we provide AskAct prompting that interleaves additional self-asking steps in the few-shot demonstrations. Furthermore, to take advantages of the stochasticity of LLMs, we provide Trajectory Ranking in which the LLM Actor generates diverse (creative) trajectories at high temperature and the LLM Critic selects the most rewarding trajectory by predicting the expected total reward of each trajectory. On the representative decision-making benchmark environments such as ALFWorld and WebShop, we empirically demonstrate that Prospector can considerably increase the success rate of given tasks, while outperforming recent advancements such as ReAct and Reflexion.