Learning & Reasoning / NAACL 2024

Before an AI Agent Acts, a Line of Code Can Check Its Next Move

A method for inferring executable action rules improved the share of admissible moves in a simulated household and modestly raised the task success rate of a ReAct agent.

Code Models are Zero-shot Precondition Reasoners

Lajanugen Logeswaran, Sungryull Sohn, Yiwei Lyu, Anthony Zhe Liu, Dong-Ki Kim, Dongsub Shim, Moontae Lee, Honglak Lee

Overview

An assistant asked to move a set of keys can produce a perfectly grammatical command to put them on a table. The command is feasible only if the assistant has the keys. This paper asks whether a pretrained code model can infer conditions like that from examples, express them as executable assertions, and use them to filter the next action an agent proposes.

Key Findings

A game agent may know the verb “open” and still spend its turns trying to open something it cannot reach. The useful knowledge is the rule that connects the verb to the situation. This study makes that rule a piece of software the agent can try out before it acts.

The framework turns a demonstration into candidate preconditions that an agent can test before choosing an action.
The framework turns a demonstration into candidate preconditions that an agent can test before choosing an action.

Turning examples into executable checks

The researchers encode demonstrations as Python programs. State variables describe facts such as what the agent is holding, what it can see, or which pieces of information a user has already given in a dialogue. Functions represent actions, while assertions say when each action is plausible. The program specification and its state variables are designed by hand for each domain.

Figure 2 shows how the model generates candidate assertions for one action at a time. The method runs demonstrations with each candidate in place and discards assertions that break them. A statement such as assert True will survive every test, so validation alone cannot find a useful condition. The ranking step compares which states satisfy each surviving assertion. If one assertion is satisfied on a strict subset of the states where another holds, the method treats the stricter assertion as more discriminating on those observed states.

The restaurant dialogue makes the trade-off visible. The reference rule allows the agent to inform a user about a slot once the user requests it. The learned rule also requires a successful database query, a condition that happened to hold in the demonstrations. The narrower rule never admitted an invalid instance in this example, but its recall was 0.57. Across the ALFWorld rule-inference test, CodeGen 2B with execution and ranking reaches 0.92 F1, while prompting CodeGen 2B alone reaches 0.71. Those numbers measure the rules themselves. The next test asks what happens when an agent uses them. Table 1 and restaurant example

Compatibility and completion

The inferred rules are used to check proposed actions during sampling. If the process fails to find a compatible sample, it falls back to the greedy action. In ALFWorld, a text-based simulator for household tasks, the study measures both whether an action satisfies the ground-truth conditions and whether the task gets completed.

ALFWorld policyTask success ratePrecondition compatibility
Behavioral cloning11%53%
Behavioral cloning with inferred rules14%74%
Act6%56%
Act with inferred rules23%96%
ReAct44%81%
ReAct with inferred rules48%96%

The values are from Table 3 of the published paper. The policies and inferred rules use StarCoder 16B in this comparison. A plausible action can still be unhelpful to the overall plan, which explains why compatibility and task success measure different things. In the dialogue benchmark, the reported gains in matching demonstrated actions are smaller.

The rule helps most when it can say no

Section 4.4 compares putting a rule in the prompt with enforcing it while sampling. In the one-shot setting in Table 4, the action F1 score goes from 0.64 without preconditions to 0.68 with predicted rules used by the sampling procedure, and to 0.73 with ground-truth rules. F1 balances missed actions and incorrect proposals. The ground-truth row gives an indication of how much better the policy might do with more reliable preconditions. Figure 3 varies demonstration count and model size as well, testing whether execution can help a smaller policy or one with little demonstration data.

A programmer can inspect the resulting checks and revise one that rejects a valid move. That visibility also exposes the limits of the setup. If the program has no variable for whether the cabinet is within reach, checking the program cannot recover the missing fact.

Questions & Answers

The opening answers are simulated author responses based on the paper.

Q: What is zero-shot about this?

A: The pretrained code model proposes the assertions without task-specific training on labeled preconditions. It still sees demonstration programs in its prompt. The policy experiments use their own supervision or prompts depending on the baseline.

Q: Can passing all the demonstration tests prove that a rule is right?

A: A rule can fit every observed example and fail on a state absent from those examples. The ranking step uses a proxy for logical implication based on observed states. Additional tests with different valid and invalid configurations are needed to find gaps.

Q: If the next move is valid, why doesn’t the agent always finish?

A: Preconditions describe when an action is feasible. The agent must still choose a sequence of actions that achieves the goal. Picking up a different visible object can be legal and still move the plan in the wrong direction.

Q: Can the learned checks take the place of a permission gate?

A: The experiments show what execution-guided filtering can do when the domain already supplies structured state and actions. Inferred assertions can still reject valid actions rarely seen in demonstrations, and a greedy fallback can proceed without a learned check. Track both errors before deciding whether to use the checks as warnings. Verified permission rules remain the authority for who may act.

Future Ideas

A shadow checker for IT runbooks

Represent a limited class of support actions, such as restarting a service in a test environment, using a typed state with service health, deployment status, and permissions. Infer candidate checks from successful runbooks and evaluate them in a simulator with counterexamples assembled by operators. Translate the checks into a small declarative rule language for evaluation.

Run the checker alongside an existing assistant, without letting learned rules block or authorize operations. Compare its alerts with the current permission gate and with operator decisions. Measure missed invalid actions, valid actions flagged, and time saved during triage. A learned check could surface a warning once it improves that trade-off under simulated state changes.

Log whether each proposed action passed an inferred check or came from the greedy fallback. Split missed invalid actions by that decision so the pilot can tell whether to improve the assertions or change when the fallback is allowed.

Sources

Original abstract

One of the fundamental skills required for an agent acting in an environment to complete tasks is the ability to understand what actions are plausible at any given point. This work explores a novel use of code representations to reason about action preconditions for sequential decision making tasks. Code representations offer the flexibility to model procedural activities and associated constraints as well as the ability to execute and verify constraint satisfaction. Leveraging code representations, we extract action preconditions from demonstration trajectories in a zero-shot manner using pre-trained code models. Given these extracted preconditions, we propose a precondition-aware action sampling strategy that ensures actions predicted by a policy are consistent with preconditions. We demonstrate that the proposed approach enhances the performance of few-shot policy learning approaches across task-oriented dialog and embodied textworld benchmarks.