Retrieval & Search / ACL Workshop 2022

A Robot Can Plan the Next Goal Before Choosing the Next Motion

A few examples teach a language model to propose subgoals for household tasks. Reranking then helps it choose which plan to try.

Few-shot Subgoal Planning with Language Models

Lajanugen Logeswaran, Yao Fu, Moontae Lee, Honglak Lee

Overview

Moving two cups into a cabinet requires several actions. A plan can state the goals at the level of picking up a cup and putting it away. The robot's controller handles the movements. This version of the few-shot subgoal planning work explores how to produce those short plans from natural language and improve them as the robot observes its environment.

Key Findings

The last step hides in the room

The workshop paper asks a household agent to heat a slice of apple and carry it to a white table with shelving below. A language model can read the instruction and still name a shelf or a side table. To find the destination, the agent needs the words and the scene. The authors build a planner in two parts. A pretrained model proposes sequences of short goals, then a second model can revise their ranking as the agent learns what is in the room.

Figure 1 follows the apple instruction through a few-shot prompt, candidate plans, and re-ranking.
Figure 1 follows the apple instruction through a few-shot prompt, candidate plans, and re-ranking.

Let a few examples describe the shape of a plan

The authors use ALFRED, a simulator benchmark with human-written instructions and expert subgoal sequences. They represent interactions such as heating, picking up, and putting down as short phrases. Navigation moves depend heavily on the layout and are handled by the low-level policy. The language model reads worked instruction-and-plan examples in the prompt. A beam search keeps several candidates alive while allowing only tokens that can extend a valid subgoal sequence.

The model can still prefer actions that appeared often in the prompt. The authors give the example of a martini glass plan where unrelated sink demonstrations pull the answer toward the wrong destination. Their scoring change asks how well a plan predicts its own instruction as well as how likely the instruction makes the plan. In the score below, hh is a candidate plan, τ\tau is the instruction, and λ\lambda controls the balance.

(1−λ)log⁡p(h∣τ)+λlog⁡p(τ∣h)(1-\lambda)\log p(h\mid\tau)+\lambda\log p(\tau\mid h)

A plausible candidate still has to win

Table 1 compares rankings on the ALFRED seen validation split. The selected GPT-J rows all use 22 examples. Recall checks for the reference sequence. Several plans can solve a simulated task, so matching that sequence and finishing the task are different tests.

Search and rankingExact subgoal-sequence recall
First choice, forward score36.10%
First choice, forward and reverse scores47.80%
Reference appears in ten forward candidates71.10%

The largest remaining gap is between the best available candidate and the one that reaches the top. Tasks with a movable receptacle were particularly hard. Comparing GPT-J prompted on a few examples with a fine-tuned system also reveals a trade-off. The prompted model is competitive in the low-data comparisons. A separately reported fine-tuned model reaches higher top-one recall with access to its full training set.

Let the room change the ranking

A learned ranker gets the instruction, candidate plan, and the agent’s current representation of the room. It is trained from simulated execution outcomes for 1,000 instructions. During a run, the existing low-level policy attempts the next subgoal, the system updates the state and retains plans compatible with the completed prefix, and the ranker scores them again. This allows the planner to correct a destination after earlier steps have been carried out.

Figure 2 shows feedback from the agent’s observations and completed subgoals changing the order of the remaining plans.
Figure 2 shows feedback from the agent’s observations and completed subgoals changing the order of the remaining plans.

With the same low-level policy, Table 5 reports task success of 19.8% for the top language-model prediction, 23.9% for the state-based re-ranking, and 37.2% for oracle subgoals. The full HLSM system reaches 29.6% in the table. The authors’ improvement isolates a change to high-level planning inside this simulator setup. Room perception and execution still matter, including when the planner gets the reference subgoals.

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: What does a reverse score add?

A: It measures how well the proposed plan accounts for the original instruction. That can help when a likely plan does not explain one detail of the request.

Q: Does the language model control the robot directly?

A: Our planning model predicts symbolic subgoals. A separate pretrained policy executes them. HLSM's state estimation modules build the room map that supplies features to the ranker.

Q: Can a different plan succeed even if it misses the reference?

A: Yes. The constrained decoder restricts proposals to the allowed subgoal language. Several plans in that language can still lead to the same outcome. Exact agreement with a reference and success in the simulator answer different questions. The seen validation results do not determine how either metric changes in unfamiliar environments.

Future Ideas

Keep useful alternatives when a plan becomes infeasible

Give a simulated household robot several plans whose feasibility depends on which objects or receptacles are present. Remove one required receptacle in environments where the instruction permits an alternative. Compare performance across familiar and held-out rooms. Compare state-aware reranking with replanning from scratch and keeping the initial plan. Measure successful recovery, unnecessary movements, and the time spent deciding.

Vary the presence of objects and receptacles as well as room layout. Score eventual completion, recovery after an infeasible step, and exact subgoal match separately.

Sources

Original abstract

Pre-trained language models have shown suc-001 cessful progress in many text understanding002 benchmarks. This work explores the capability003 of these models to predict actionable plans in004 real-world environments. Given a text instruc-005 tion, we show that language priors encoded006 in pre-trained models allow us to infer fine-007 grained subgoal sequences. In contrast to re-008 cent methods which make strong assumptions009 about subgoal supervision, our experiments010 show that language models can infer detailed011 subgoal sequences from few training sequences012 without any fine-tuning. We further propose013 a simple strategy to re-rank language model014 predictions based on interaction and feedback015 from the environment. Combined with pre-016 trained navigation and visual reasoning com-017 ponents, our approach demonstrates compet-018 itive performance on subgoal prediction and019 task completion in the ALFRED benchmark020 compared to prior methods that assume more021 subgoal supervision.022