Learning & Reasoning / NeurIPS Workshop 2023

Before an Agent Acts, Ask What Has to Be True

A workshop paper represents actions as code so a model can infer and check the conditions that make an action plausible.

Reading note · Workshop results were checked against the indexed OpenReview manuscript. The illustrations are from the same authors’ November 2023 arXiv revision.

Reasoning about Action Preconditions with Programs

Lajanugen Logeswaran, Sungryull Sohn, Yiwei Lyu, Anthony Zhe Liu, Dong-Ki Kim, Dongsub Shim, Moontae Lee, Honglak Lee

Overview

Before a robot can close a cabinet, the cabinet has to be open. Before a support agent can confirm a booking, it needs enough information to identify it. The workshop paper investigates whether the rules behind actions can be inferred by pretrained code models through code completion.

Key Findings

A plan can be sensible and still fail at its next step

Imagine a robot reaching for a drawer it needs to close. If the drawer is already shut, the action is superfluous. If a task needs the contents, opening it has to come first. A dialogue agent can face a similar choice when it needs a reservation identifier before proceeding. The workshop abstract studies these preconditions, the facts that must hold before an action is plausible. It asks whether a model trained on code can use the procedural structure behind an action.

Put the condition where the agent can check it

The authors write demonstrations as sequences of function calls. Observation functions update state, such as which objects the agent can see or which details a customer has provided. Action functions are the places to put assertions. In the restaurant example, the predicted rule requires both a city and a cuisine before calling the search function.

Figure 1, from the November 2023 arXiv revision. Demonstration calls supply candidate preconditions, and the resulting assertions constrain predictions on a new trajectory.
Figure 1, from the November 2023 arXiv revision. Demonstration calls supply candidate preconditions, and the resulting assertions constrain predictions on a new trajectory.

A code model proposes assertions. Running the demonstration with each assertion filters out programs that fail and rules that reject actions the demonstrator took. To rank the survivors, let CahC_a^h be the set of demonstrated contexts in which assertion hh permits action aa. The workshop uses strict containment to keep the most discriminating candidates:

Haopt={h∈Havalid∣∄ h′∈Havalid: Cah′⊊Cah}.H_a^{\mathrm{opt}} = \{h\in H_a^{\mathrm{valid}}\mid \nexists\,h'\in H_a^{\mathrm{valid}}:\ C_a^{h'}\subsetneq C_a^h\}.

If one valid assertion allows a strict subset of the observed contexts another permits, it is stronger on these demonstrations. That reasoning can remove redundant checks. It can also reject a legitimate action when the demonstrations did not cover the relevant state. Figure 1 and Equation 1, workshop manuscript

Figure 10, from the November 2023 arXiv revision. A code model proposes candidates, execution tests them on demonstrations, and ranking keeps the strongest surviving rules.
Figure 10, from the November 2023 arXiv revision. A code model proposes candidates, execution tests them on demonstrations, and ranking keeps the strongest surviving rules.

Check the suggestion against the current state

The model sees a partial trajectory and the assertion code, then proposes the next action. The method executes the candidate to check whether it passes. It tries the model’s first choice and then samples alternatives up to an attempt limit. If none pass, the workshop algorithm returns the first choice. The check improves the chance of choosing a plausible action while retaining a fallback when an inferred rule is wrong or the model cannot find a match. Section 3.3, workshop manuscript

Measure the rules before measuring the whole agent

Table 2 checks an inferred rule against known conditions on held-out trajectories. Precision answers how often a rule permits an action in a context the reference also permits. Recall answers how much of the reference set the rule covers. The values below are the reported macro-averaged F1 scores.

Precondition inference setupDialogue F1ALFWorld F1
Direct prompting, CodeGen 2B0.740.71
Validate and rank, CodeGen 2B0.780.92
Direct prompting, StarCoder 16B0.750.68
Validate and rank, StarCoder 16B0.840.90

On ALFWorld, the validation and ranking pipeline reached precision 1.00 for both code models, while their recalls were 0.85 and 0.81. These averages reflect the choice to favor stricter assertions. Table 2, workshop manuscript

Two settings put the same idea to work

The dialogue benchmark compares actions with held-out demonstrations. ALFWorld can run a proposal end to end and score whether the agent finished its task. Table 3 shows how the same StarCoder 16B backbone performed in ALFWorld with and without the method.

ALFWorld policyTask successActions compatible with reference conditions
Few-shot prompting6%56%
Inferred rules and execution checks23%96%
Few-shot prompting with ReAct-style comments44%81%
Inferred rules and checks with ReAct-style comments48%96%

The final row passes reference checks much more often than it completes the task. An agent can take one plausible step after another and still fail to make enough progress. The action check addresses a local decision. The simulator measures the whole trajectory. Table 3, workshop manuscript

Inspect the rule before trusting the score

The later manuscript compares the learned assertions with reference rules action by action. Its ALFWorld example shows the model recovering a visibility check for moving to a receptacle. The learned assertion for closing a receptacle checks whether it is open but omits the reference visibility check. Two rules can pass the same demonstrations even if they disagree in a state the demonstrations never visited. This is a concrete reason to test preconditions on held-out situations as well as on successful examples.

Figure 9, from the November 2023 arXiv revision. The ALFWorld reference assertions and model-predicted assertions appear side by side.
Figure 9, from the November 2023 arXiv revision. The ALFWorld reference assertions and model-predicted assertions appear side by side.

Questions & Answers

The opening answers are simulated author responses based on the workshop manuscript.

Q: Why represent the rule in code?

A: An executable condition can be checked against the current state. That allows the agent to evaluate whether an action is plausible before attempting it.

Q: Could a demonstration teach a rule that is too strict?

A: Yes. If every example shows the same incidental feature, a model can mistake that feature for a requirement. Examples that violate incidental patterns help expose that error.

Q: How can an agent distinguish a bad rule from a bad observation?

A: A correct rule can block a valid action when the input state is stale, and a bad rule can pass when the state omits the relevant fact. When auditing a runbook, record the proposed rule and the observed state separately, then compare both with verified event logs. The workshop abstract supports the broad program-based approach. The later NAACL paper provides the fuller evaluation.

Future Ideas

For an internal runbook, show an operator the proposed action and the inferred condition that controls it. Pilot the tool in shadow mode using historic cases with missing, stale, and contradictory state. Have operators label the actual state and action validity from incident logs. Compare to a static checklist and measure both invalid actions flagged and valid actions wrongly blocked.

Sources

Original abstract

One of the fundamental skills required for an agent acting in an environment to complete tasks is the ability to understand what actions are plausible at any given point. This work explores a novel use of code representations to reason about action preconditions for sequential decision making tasks. Code representations offer the flexibility to model procedural activities and associated constraints as well as the ability to execute and verify constraint satisfaction. Leveraging code representations, we decompose the problem of learning an agent policy for sequential decision making tasks into the sub problems of precondition inference and action prediction. We show that these sub-problems can be formulated as code-completion problems and exploit pre-trained code understanding models to tackle them. We demonstrate that the proposed code representation coupled with our novel precondition-aware action prediction strategy outperforms prior policy learning approaches in a few-shot learning setting across task-oriented dialog and embodied textworld benchmarks.