Overview
A robot’s log can contain thousands of runs and very few labels. The easiest runs to identify may be the failures: a walker violated a joint limit or a mobile robot crossed a hazard. Training on the entire log risks copying those behaviors. Moontae and his team develop an offline imitation method that uses the labeled failures to identify the safer behavior already present in the mixture.
Key Findings
Learn from a drive the robot should avoid
A walker’s training log can contain steady strides, recoveries and collapses. A method that imitates the whole log can learn from all three. SafeDICE starts from a more pointed piece of supervision: a small set of runs flagged as non-preferred. The rest of the data are plentiful and unlabeled. Prior imitation methods typically need examples marked as preferred, or both preferred and non-preferred. SafeDICE uses the exclusions to estimate what remains in the common log.
Subtract the bad routes from the mixture
The method works with occupancy distributions, which count how often a policy visits each state and action with a time discount. If of the unlabeled occupancy comes from the non-preferred policy, then preferred occupancy follows by subtracting that contribution:
The unknown fraction matters. The paper estimates it from the minimum ratio of unlabeled to non-preferred occupancy, using a discriminator to estimate that ratio from samples. It subtracts a small constant in the implementation to keep the derived reward well-defined. The paper bounds the distribution-estimation error when there are state-action pairs with little preferred occupancy relative to non-preferred occupancy. That is a condition on the data and the two policies. Equations 4 and 15–16
Check that the safer log could come from a policy
Simply upweighting attractive actions could suggest a distribution that no robot can produce. The flow constraint enforces consistency with how actions lead to later states. SafeDICE reduces its optimization to a single convex minimization in the distribution formulation, and then extracts a policy by weighted behavior cloning. If is the correction from the unlabeled occupancy to the fitted occupancy, the last step minimizes
Two actions from the same log can now contribute differently. One may be common in the inferred preferred behavior, while another appears mostly in the examples the robot should learn to avoid. Equations 11–14
The safest route can give up reward
The simulated walker retains similar return and incurs a much smaller cost. In the Safety Gym button task, lower cost comes with lower return. The preferred button demonstrations themselves earn less return than the non-preferred ones, a sign that the dataset contains a genuine trade-off. These selected means come from five independent runs per result. The last column averages cost over the worst tenth of runs. Results in Figures 2–4
| Benchmark and method | Normalized return ↑ | Average cost ↓ | Cost violation ↓ | Worst-tenth cost ↓ |
|---|---|---|---|---|
| Walker, behavior cloning | 98.41 | 152.25 | 0.22 | 531.07 |
| Walker, SafeDICE | 99.67 | 68.90 | 0.00 | 124.54 |
| Button, behavior cloning | 88.64 | 98.18 | 0.49 | 220.87 |
| Button, SafeDICE | 71.48 | 70.67 | 0.30 | 185.64 |
How much can a warning label tell you?
The walker’s 0.00 violation is a rounded estimate under this simulator’s cost threshold. The Cartpole appendix shows how averages can hide a failure mode. SafeDICE reduces average cost relative to behavior cloning, while the average cost among the worst tenth of outcomes rises. The sensor or safety rule matters as much as the algorithm when deciding what to label. In the Walker sensitivity study, the team varies the number of labeled non-preferred demonstrations from 50 down to five, and observes worse safety outcomes with smaller sets. Before using logs from a new environment, a planner would need to know which actions produce costly tail events and whether there are enough relevant examples in the labeled set. Figure 8 and Appendix E
Questions & Answers
Q: Why can’t the robot simply do the opposite of a failed run?
A: A failed run can contain many ordinary actions and a few harmful ones. Reversing all of them can create new behavior absent from the logs. The mixed dataset supplies examples of what the agent could actually do under the same dynamics.
Q: What if the failures look almost identical to successes in the log?
A: Then it is harder to identify state-action pairs that distinguish the negative occupancy. The ratio estimate can overestimate the mixture fraction. More informative state measurements, better coverage, or additional annotations would be needed to separate the behaviors.
Q: Does offline training make a deployed policy safe?
A: The learner trains from logs without interacting with the environment. The deployment policy can still encounter situations absent from those logs. The paper’s measurements come from simulated trajectories under specified cost rules.
Q: How does the guarantee change if the logs come from several operators or vehicle versions?
A: The occupancy bound assumes the mixture and density ratios used in the paper. If several operators or hardware versions produce recordings, a negative-label score may capture their identities as well as unsafe behavior. A sensitivity analysis needs mixtures with known proportions and mislabeled examples before that bound can guide a deployment decision.
Future Ideas
A replay test for mobile-robot near misses
In a warehouse simulator, pool logged trips across layouts and label a small subset with obstacle near misses. Hold out entire layouts and vehicle-load combinations. Compare SafeDICE with ordinary behavior cloning and a cost-aware baseline that has the same labeling budget. Measure completed deliveries, mean clearance cost, worst-tenth clearance cost, and minimum separation. Stress-test the models with shifted sensors and intentionally noisy labels. Consider a physical shadow trial only after predefined safety and throughput criteria are met in the simulator.
In the simulator, vary the number and policies of operators and track drift in the discriminator-based mixture estimate against known ground truth. Fix intervention thresholds and stop conditions independently of the training logs.