Overview
A simulated robot learns to prepare food in a kitchen where the salmon is always prepared on a cutting board. In another kitchen the salmon is in a closed refrigerator. The instruction is the same. The sequence that makes it possible has changed. SemGro asks a planner to propose a useful skill and a critic to check whether that skill fits the scene before the system carries it out.
Key Findings
The instruction may still say to make lunch after a robot moves to a different kitchen. What has changed is the space in which the instruction needs to become a sequence of actions. The researchers treat that change as a test of whether a skill can be grounded in the scene.
Try a skill at the right level
The method organizes skills into a hierarchy. At the bottom are short actions with objects, such as grasping something. Higher levels chain those actions into routines such as cooking a meal. A retrieval step finds example skills using both the wording of the instruction and the objects observed in the scene. The planner proposes a skill using those examples.
A vision-language component extracts object names and states from the current view. A language model checks feasibility using that description and returns feedback when the skill appears unsuitable. The planner can then break the skill into smaller steps, such as reaching the refrigerator and opening it before trying to retrieve an ingredient. One appendix trace records a proposed move that requires access to a closed cabinet. The critic identifies the obstacle and the planner introduces an opening step. In another room the same check can pass immediately. Figure 2 · Algorithm 1
A new room tests both language and action
The experiments use VirtualHome with shifts in object location, physical attributes, and room structure. Table 1 reports task completion rates over three seeds. These are the rows for instructions containing abstract nouns.
| Changed environment | LLM-Planner | SayCan | ProgPrompt | SemGro |
|---|---|---|---|---|
| Object locations | 20.51% | 30.77% | 17.95% | 53.85% |
| Physical attributes | 20.69% | 17.24% | 27.58% | 52.17% |
| Room structures | 16.22% | 18.93% | 13.51% | 56.52% |
The paper separately measures completed goal conditions and alignment with a reference skill sequence. One reason to keep those measurements separate is that a task can have more than one feasible action sequence. Exact agreement with the reference can penalize a plan that finishes the job by another route.
Let one model describe the room and another assess the plan
The skill critic divides up two jobs that can look similar in a demonstration. One model reads the image to identify relevant objects and states. The other checks whether those facts permit a proposed skill. Table 6 compares the split with single vision-language critics. For changed object locations the reported execution scores are 79.67% for the InstructBLIP critic and 89.06% for the two-model critic. These are skill execution results, separate from full task success in the earlier table.
When familiar examples become misleading
The retrieval ablation varies how many skill examples the planner receives. Extra demonstrations from the source environment can carry assumptions about the old room. A critic that sees the new scene and a planner that can fall back to smaller actions are both useful when those assumptions fail. VirtualHome gives the researchers reproducible changes to location, object attributes, and room layout. A physical robot would have additional failures to handle, including perception and control errors.
Questions & Answers
The opening answers are simulated author responses based on the paper.
Q: Why use high-level skills if small actions transfer more easily?
A: A high-level skill can express how several steps serve the same goal. Smaller actions give the planner more flexibility when the scene changes. The hierarchy lets us keep the useful abstraction and refine the parts that need adapting.
Q: What happens if the critic cannot see the relevant object?
A: The current design checks feasibility from the observation it receives. An object behind a door or out of view can leave the critic with incomplete evidence. An extension could gather another observation before deciding whether to execute or decompose.
Q: Why compare completion and reference alignment?
A: Completion measures whether the desired state is reached. Reference alignment asks whether the plan begins with the expected skill sequence. Alternative sequences can complete the task, so the two measures can disagree.
Q: Can the simulator results predict performance on a physical robot?
A: The tests vary layouts and instruction wording in simulation and measure planning and execution separately. They do not include every source of uncertainty in a physical setting. The critic can be wrong when its observation misses an object state relevant to the action. Physical tests need independent checks around which actions are safe to execute.
Future Ideas
Ask for a better view before breaking the plan apart
In a simulator with configurable occlusion, let the critic distinguish a visible obstruction from missing visual evidence. When evidence is missing, give the system a small budget for moving the camera or inspecting a safe nearby viewpoint before decomposing the skill. Log which additional observation changed the decision.
Compare active inspection, immediate decomposition, and a rule that always takes another view at matched action budgets. Include closed containers and objects outside the initial view. Measure completion, unnecessary decompositions, invalid actions, and additional steps. Separate gains due to visibility from gains on scenes where the original skill is genuinely unsuitable.
Repeat the comparison with and without the hierarchy and the feasibility critic, holding the skill library fixed. That can separate the effect of choosing a better observation from the effect of finer decomposition.
Sources
Original abstract
In embodied instruction-following (EIF), the integration of pretrained language models (LMs) as task planners emerges as a significant branch, where tasks are planned at the skill level by prompting LMs with pretrained skills and user instructions. However, grounding these pretrained skills in different domains remains challenging due to their intricate entanglement with the domain-specific knowledge. To address this challenge, we present a semantic skill grounding (SemGro) framework that leverages the hierarchical nature of semantic skills. SemGro recognizes the broad spectrum of these skills, ranging from short-horizon low-semantic skills that are universally applicable across domains to long-horizon rich-semantic skills that are highly specialized and tailored for particular domains. The framework employs an iterative skill decomposition approach, starting from the higher levels of semantic skill hierarchy and then moving downwards, so as to ground each planned skill to an executable level within the target domain. To do so, we use the reasoning capabilities of LMs for composing and decomposing semantic skills, as well as their multi-modal extension for assessing the skill feasibility in the target domain. Our experiments in the VirtualHome benchmark show the efficacy of SemGro in 300 cross-domain EIF scenarios.