Overview
“Put a clean mug on the table” compresses a sequence of decisions into a few words. The mug has to be found, cleaned, and moved. A robot can plan at that level before deciding which individual action to take. This paper asks whether a pretrained language model can infer those intermediate goals from a handful of worked examples.
Key Findings
Planning a Heated Slice of Apple
Take the instruction "Move a heated slice of apple to the white table with shelving below." A household robot has to slice the apple, heat it, and carry it to the right surface. Which surface is that? A language model, shown only a few worked examples, may guess "side table" or "shelf," and the room may contain neither. Moontae and his colleagues built their NAACL 2022 study around this problem. They ask whether a pretrained language model can turn instructions into plans of intermediate goals, called subgoals, and whether a little feedback from the environment can fix the mistakes.
Subgoals as short sentences
The authors work with ALFRED, a benchmark of household tasks with more than 20k human-written descriptions. They keep only the interaction subgoals, such as picking up, heating, cooling, cleaning, slicing and putting, and drop navigation steps because those often depend on the room layout. Each subgoal pairs one of 7 actions with one of 80 object types and becomes plain text, so "(Pickup, apple)" reads "pick up apple" (Table 6 in the appendix).
The model receives a handful of instruction and plan pairs, then the query instruction, with no fine-tuning. Constrained beam search of width 10 allows only tokens that keep a valid subgoal sequence possible, so every candidate is a legal plan. The probability of a candidate plan given instruction follows Equation 1 in the paper, written here in compact form:
Each token is scored given the earlier tokens, the instruction, and the worked examples in the prompt.
Good candidates, poor ranking
The authors used 22 prompt examples, two per fine-grained task type, and evaluated on the 800-instance valid-seen split. GPT-J (6B parameters) found the reference plan within its ten candidates 71% of the time. GPT2-XL (1.5B) reached 59%. Tasks with three object arguments proved hardest, and the "pick place movable" type reached only 35.65% top-10 recall for GPT-J (Table 2).
The top-ranked candidate was a weaker guide. The authors confirmed a known bias in which models repeat prompt content. In their example, a martini glass instruction drew a wrong plan ending "put in sink" because the prompt mentioned sinks. Removing the sink examples flipped the ranking.
Scoring the instruction too
The fix borrows from mutual information. The authors also score how well the plan predicts the original instruction, , and combine the two directions. Their derivation (Equations 2 to 4, Appendix A) shows that a weighted mutual information objective reduces to
where sets the balance between the two directions. They use .
| GPT-J | N = 11 | N = 22 | N = 33 |
|---|---|---|---|
| Top-10 recall, | 64.02 | 71.10 | 72.07 |
| Top-1, | 30.12 | 36.10 | 35.85 |
| Top-1, | 34.15 | 34.39 | 35.00 |
| Top-1, combined | 44.02 | 47.80 | 49.63 |
Table 1 reports these recall percentages for N training examples. Reverse scoring beat forward scoring only with the fewest examples. The combination won in every column, for both models.
Letting the room vote
Some ambiguity cannot be resolved from text. A "look at object in light" task may involve a floor lamp or a desk lamp, depending on the scene, and Table 4 lists such confusions. So the authors trained a ranking model that combines a BERT encoding of the instruction and candidate plan with a state embedding of the room. They ran plans through the pretrained HLSM low-level policy and used task success as the training signal. They collected this feedback for 1000 instructions. At inference, the agent executes one subgoal at a time, drops plans that disagree with what it has done, and re-ranks the rest. Appendix D gives both algorithms.
Table 5 reports task success on ALFRED validation-seen, executed in the AI2-Thor simulator. The top language model plan succeeded on 19.8% of tasks. Re-ranking raised this to 23.9%, and oracle subgoals reached 37.2%. HLSM trained with full subgoal supervision on 20k instances scored 29.6%. The authors note their approach falls short of HLSM but competes with other baselines that use subgoal supervision, including MOCA at 19.2%.
How far few examples go
Against a GPT2-Medium model fine-tuned by Jansen (2020), GPT-J with 11 examples reached 44.02% top-1 recall. The fine-tuned model scored 5.07% with 77 instances and 41.92% with 779 (Table 3). With the full 7793 instances it reached 61.00%, so fine-tuning wins when data is plentiful.
In-context performance rose with more examples and then flattened, and beyond 44 examples the prompt no longer fit GPT's 1024-token limit. Across five random seeds, in-context top-1 predictions varied less than fine-tuned ones. The authors caution that the T5-large comparison is not head to head, since the 770M-parameter model was the largest their compute allowed. They also did not test other tuning methods, and they leave joint training of planner and low-level policy to future work.
Questions & Answers
Q: Why split a task into subgoals?
A: Subgoals summarize what the agent is trying to accomplish. A low-level controller can choose how to navigate or manipulate an object. That division lets a researcher ask whether a failure began with the plan or with carrying it out.
Q: Why score the instruction given the plan?
A: A likely plan can reflect which tasks were common in the demonstrations. Asking whether the plan also explains this particular instruction provides another ranking signal. The ranking combines the two signals because neither consistently selects the right plan by itself.
Q: Can the reported recall tell us whether a physical robot would succeed?
A: The recall test measures reference-sequence matching in a benchmark. A robot also needs reliable perception, a controller that can complete the subgoals, and a way to recover when the environment changes. End-to-end tests are needed for those capabilities.
Q: How much does a one-reference benchmark reveal about alternate plans that work?
A: A plan can finish the task through a sequence that differs from the single annotated plan. The top-one versus top-ten gap shows how often the reference is present but ranked lower on the tested examples. Simulator execution answers a different question by checking whether an action sequence actually reaches the goal.
Future Ideas
Replan from the first failed checkpoint
In a simulated household assistant, attach a cheap, observable check to each proposed subgoal. If the controller cannot find the named object, keep the earlier completed steps, discard candidate plans that require the unavailable object, and regenerate only the remainder. Record which check caused a new plan so a person can distinguish a changed environment from a planning mistake.
Test on held-out rooms with a removed or moved object. Include the paper’s execution-based candidate pruning as a comparator to isolate the effect of regenerating the remainder. Compare with restarting the full plan and with a scripted recovery policy using the same controller. Count completion, wasted actions, and the frequency of abandoning a feasible task. Start with checks that can be evaluated reliably in the simulator.
Score alternate valid routes separately from the single reference and use regeneration logs to distinguish failures of object detection, candidate ranking, and generation.
Sources
Original abstract
Pre-trained language models have shown successful progress in many text understanding benchmarks. This work explores the capability of these models to predict actionable plans in real-world environments. Given a text instruction, we show that language priors encoded in pre-trained models allow us to infer fine-grained subgoal sequences. In contrast to recent methods which make strong assumptions about subgoal supervision, our experiments show that language models can infer detailed subgoal sequences from few training sequences without any fine-tuning. We further propose a simple strategy to re-rank language model predictions based on interaction and feedback from the environment. Combined with pre-trained navigation and visual reasoning components, our approach demonstrates competitive performance on subgoal prediction and task completion in the ALFRED benchmark compared to prior methods that assume more subgoal supervision.