Overview
A human reviewer can prefer an answer because it directly addresses the question, even if part of the advice crosses a safety boundary. Training on that preference alone makes the answer look like a good example. SafeDPO gives safety labels a role in deciding which answer the model should learn from, then measures what that change costs in usefulness.
Key Findings
When the more helpful answer is the more dangerous one
Suppose a language model sees two replies to a user who asks how to conceal abuse of a pet. One reply directly answers that request. The other redirects the user toward calmer ways to handle frustration. A reviewer who judges only whether a reply addresses the question may label the first as more helpful. Standard Direct Preference Optimization (DPO) then treats the first reply as the winner and pushes the model toward it. Appendix C.5 uses this exact kind of example to show how the labels can conflict.
Earlier safety methods respond by adding machinery. Safe RLHF, SACPO and the CAN methods (MoCAN and PeCAN) typically rely on separate reward and cost models, multiple training stages, or a relaxed requirement that the average cost stay below a threshold (Section 1 and Section 2.2). Appendix C.8 lays out the burden for Safe RLHF. It needs a reward model, a reward value function, a cost model and a cost value function beside the policy and reference policy (Table 14). It also generates fresh responses during reinforcement learning. The authors report 1,388.2 seconds of policy training for SafeDPO and 32,957.1 seconds for Safe RLHF in their setup. Training Safe RLHF’s reward and cost models added about 1,121 seconds each in Table 15.
Put the safety boundary inside the training data
The authors go back to the strict problem in Section 2.2. The policy must maximize reward minus a KL penalty, while every sampled response satisfies . The cost is positive for unsafe responses. They fold the constraint into the reward by assigning an unsafe response a reward of negative infinity (Section 3.1). Equation 9 gives the resulting optimal policy.
This is Equation 9 of the paper. Here is the reference model, is the strength of the KL penalty, is the cost-augmented reward, and is a normalizing constant. In this theoretical optimum, any response with receives zero probability. The trained model still depends on finite data and correct safety labels.
The ideal policy requires comparisons under the modified reward. The authors show how to recover the resulting preference loss using the labels the dataset already has. Any safe response beats any unsafe one under , so the empirical dataset can be rewritten to match. A transformation keeps a pair whose winner is safe, swaps a pair whose winner is unsafe but whose loser is safe, and discards a pair where both are unsafe (Section 3.2). The standard DPO loss on the rewritten pairs, Equation 11, equals the intractable loss under exactly. Proposition 4.3 proves this case by case in Appendix A.2.
Ask for a clearer separation between safe and unsafe
The third contribution adds one knob. Equation 12 subtracts inside the sigmoid, where marks a response as unsafe and is the margin. On safe–unsafe pairs the term equals , so the model must widen the gap between the two responses. On pairs with matching safety it vanishes. Proposition 4.4 states that the set of optimal solutions stays the same for every . The authors also note that the recipe extends to other direct alignment losses, though they only instantiate it with DPO (Section 3.4).
The authors also connect the strict constraint to a more familiar penalty. Proposition 4.2 shows that the policy using a finite penalty for unsafe answers approaches the hard-constrained policy as the penalty grows. This relies on bounded rewards and on the reference model assigning some probability to safe answers.
The comparison changes when the reviewer changes
The authors train on PKU-SafeRLHF-30K, with roughly 27,000 training entries and 3,000 test entries, starting from a shared supervised fine-tuned Alpaca-7B reference model (Section 5.1.1). Baselines include DPO trained on helpfulness preferences, DPO trained on harmlessness preferences, a DPO variant that filters out pairs with unsafe winners (DPO-SAFEBETTER), Safe RLHF, SACPO and P-SACPO. Figure 2 plots helpfulness and harmlessness under two evaluators. Appendix C.9 supplies the values.
SafeDPO reaches a harmless ratio of 96.87% under the model-based evaluator (Table 17) and 100.00% under GPT-4 (Table 18). The filtered DPO-SAFEBETTER reaches only 49.75% and 58.65%. The result suggests that swapping pairs supplies a training signal that filtering leaves behind.
| Method (GPT-4 judge, Table 18 of the paper) | Helpfulness | Harmless ratio (%) | Harmlessness |
|---|---|---|---|
| SFT | 4.25 | 53.01 | 5.61 |
| DPO-HELPFUL | 4.32 | 43.36 | 4.71 |
| DPO-HARMLESS | 6.69 | 83.83 | 8.42 |
| DPO-SAFEBETTER | 5.43 | 58.65 | 6.17 |
| SafeRLHF | 7.68 | 96.62 | 9.57 |
| SACPO | 7.59 | 96.12 | 9.49 |
| P-SACPO | 7.69 | 94.11 | 9.34 |
| SafeDPO | 8.14 | 100.00 | 9.92 |
The margin ablation (Figure 3 in the original paper) shows that SafeDPO stays highly safe even at and across the tested values of 0, 2, 5, 10 and 20. This fits the invariance result. Appendix A.4 explains why degrades helpfulness. With a huge margin the sigmoid saturates near 1 on safe–unsafe pairs, and the update turns into an unlikelihood-style push that can destabilize training. The paper finds a useful balance at moderate values, such as .
A separate ablation adds the same margin to ordinary DPO variants. The margin helps some variants, and their model-based harmless ratios remain below SafeDPO’s.
| Harmless ratio (%) with margin, Tables 5–7 of the paper | ||||
|---|---|---|---|---|
| DPO-HELPFUL | 38.6 | 43.75 | 49.5 | 51.63 |
| DPO-HARMLESS | 76.82 | 86.25 | 90.13 | 93.88 |
| DPO-SAFEBETTER | 50.5 | 56 | 60.5 | 62.13 |
SafeDPO with scores 97.24% on the same model-based measure (note under Table 5).
Safer replies can come with more refusals
Scale tests (Table 1) cover 1.5B to 13B parameters. The harmless ratio ranges from 95.50% to 97.88%, with helpfulness between 4.29 and 7.60 on the normalized scale. Across three seeds, SafeDPO scores a normalized reward of 4.3809 (±0.1099) against 3.2471 (±0.5379) for Safe RLHF, and a cost of −6.2285 (±0.1308) against −2.6617 (±0.6323) (Table 13).
The same strictness has a price. On XSTest, SafeDPO reaches a 100% harmless ratio but over-refuses 12.4% of benign prompts, compared with 3.2% for Safe RLHF and 2.4% for SACPO (Table 3). Appendix E.1 shows failures on harmless prompts about terminating a Python process, where the model offers a cautious reply instead of technical help. The authors read this as a consequence of the hard constraint and as a sign that the model leans on lexical cues.
The evaluation itself needs care. Table 10 shows that Safe RLHF looks nearly saturated under the beaver-7b-v1.0 reward and cost models, which resemble the models it trained against, and that its advantage fades under independent judges. Appendix D.2 restricts GPT-5.1 comparisons to pairs where both answers are safe, to remove the bias toward rating safer answers as more helpful. SafeDPO still wins 59.66% of those comparisons against Safe RLHF (Table 23). Human raters, though, rated the SFT model as most helpful (0.868) and SafeDPO at 0.499, and rated safety at 0.943 for SafeDPO against 0.932 for Safe RLHF (Table 2). The authors trace the helpfulness reversal to labeling rules and caution that experiments stop at 13B parameters and rely mainly on one dataset (Section 6).
Questions & Answers
The opening answers are simulated author responses based on the paper.
Q: Do the models need to sample new answers during training?
A: The SafeDPO objective works with offline preference pairs and their safety indicators. It uses a reference model and does not require a separate reward model, cost model, or new policy rollouts for that optimization.
Q: Why discard a pair where both answers are unsafe?
A: If both options violate the safety constraint, ranking one above the other does not give the training pair a safe winner. The transformed objective drops that pair. The remaining dataset still needs useful safe responses for comparable prompts.
Q: Does 100% on XSTest establish complete safety?
A: It reports the measured harmless ratio on that test and judge protocol. New requests can differ from the labeled prompts. Deployment needs additional checks for ambiguous requests, new behaviors, and unjustified refusals.
Q: What can a guarantee about the training objective tell us about a deployed model?
A: Proposition 4.3 relates the transformed objective to the constrained optimum under the paper’s assumptions. Actual safety still depends on which responses the binary labels cover and whether the trained model follows the intended behavior on new prompts. It also matters what raters count as helpful. A direct but harmful reply can look responsive under a narrow definition, so it helps to report helpful safe alternatives, justified refusals, and unnecessary refusals separately.
Future Ideas
A preference dataset that teaches a useful safe alternative
Build a small set of customer support requests where the same wording can occur in benign and unsafe situations. For each case, collect a policy-compliant answer that still helps the user. Compare ordinary DPO, SafeDPO, and SafeDPO supplemented with these alternatives with the same training budget and number of preference pairs.
Hold out product families and request templates. Ask blinded reviewers to score unsafe compliance, justified refusal, unjustified refusal, and usefulness of any safe alternative. The pilot succeeds if it reduces unnecessary refusals without increasing harmful compliance on the held-out tasks.
Give reviewers a written policy for ambiguous cases and have multiple annotators score a sample independently. Publish disagreement rates by request type so a noisy label does not look like a change in safety.
Sources
Original abstract
As Large Language Models (LLMs) are increasingly deployed in real-world applications, balancing helpfulness and safety has become a central challenge. A natural approach is to incorporate safety constraints into Reinforcement Learning from Human Feedback (RLHF), where recent studies have shown promising progress. However, these methods often rely on auxiliary networks or multi-stage pipelines, thereby increasing complexity. In this work, we revisit the original safety alignment objective and show that, under mild assumptions, it admits a closed-form optimal policy. We further derive a provably equivalent and tractable objective, enabling direct optimization. Building on this insight, we propose SafeDPO, a lightweight method that preserves the optimal solution of the underlying safety-constrained objective while requiring only one additional hyperparameter and minimal modifications to existing preference-based training methods. SafeDPO eliminates the need for reward models, cost models, and online sampling, relying only on preference data and safety indicators. Despite its simplicity, SafeDPO achieves competitive safety-helpfulness trade-offs compared to existing safety alignment methods. Experiments on the PKU-SafeRLHF-30K benchmark demonstrate that SafeDPO substantially improves safety while maintaining competitive helpfulness. Ablation studies further show that the additional hyperparameter provides a flexible mechanism to enhance safety while preserving the theoretical optimum, and confirm that SafeDPO scales reliably to LLMs with up to 13B parameters. Overall, our results highlight that a simple, theory-driven objective can provide a lightweight yet effective solution for safety alignment in practice.