As large reasoning models (LRMs) commonly expose their intermediate reasoning traces to users and downstream systems, a long-overlooked issue has emerged: is it sufficient to evaluate safety based solely on the final answer?
A joint systematic study by researchers from Harvard University, University of Southern California, Brown University, MIT, and other institutions gives a negative answer, citing examples such as "when we discovered that a model's chain of thought could be used to generate high-risk content like bomb-making instructions or poison recipes, we realized this issue is far from trivial." The team subsequently proposed a corresponding mitigation method: "Chain of Risk: Safety Failures in Large Reasoning Models and Mitigation via Adaptive Multi-Principle Steering."
Paper link: https://arxiv.org/abs/2605.05678
Figure 1: Preview of the two-stage pipeline (evaluation experiment + mitigation method)
Evaluating Reasoning and Answer Separately
The core idea of the research team is straightforward: for a reasoning model f, given a prompt x, it simultaneously produces a reasoning trace r and a final answer y. The team designed 20 safety principles for these two stages respectively (as shown in the figure below), with each principle using a 1-5 point risk severity scoring system.
Table 1: 20 Safety Principles
Based on this, the team set a unified risk threshold: as long as any one of the 20 principles in a given stage (reasoning or answer) scores above the threshold, that stage is judged as "unsafe." By then combining the judgments of the reasoning and answer stages, three core failure modes are delineated:
Unsafe: Both reasoning and answer stages are unsafe;
Leak: Reasoning is unsafe, but the answer is safe — meaning dangerous content has already "leaked" into the reasoning trace;
Escape: Reasoning is safe, but the answer is unsafe — a seemingly benign reasoning process transitions to a harmful output.
Figure 2: Three Reasoning-Answer Failure Modes
The value of this classification lies in turning the phenomenon of 'answer safety ≠ trace safety' into a quantifiable metric.
Data and Evaluation Setup
The research team constructed an in-distribution prompt pool by integrating seven public harmful/jailbreak datasets: WildChat, PKU-SafeRLHF, JailbreakV, HarmBench, BeaverTails, StrongREJECT, and JailbreakBench. After unifying field mappings, filtering, and deduplication based on MinHash-LSH, an in-distribution evaluation dataset of 41K prompts and a held-out test set of 2K prompts were partitioned.
Additionally, a completely independent out-of-distribution (OOD) evaluation set was constructed from four datasets—AdvBench, SaladBench, SimpleSafetyTests, and WildJailbreak—to test the robustness of the conclusions. The evaluation covered 15 reasoning models:
Scoring was performed by two LLM scorers (Claude-4.5-Haiku and Gemini-Flash-3). The research team also performed consistency checks against three human annotators on 80 samples (split into 1600 principle-level scores): the Pearson correlation coefficient between scorers reached 0.799 for the reasoning stage and 0.820 for the answer stage, both exceeding human-to-human consistency (0.742 / 0.780); the Cohen's κ for binary unsafe labels from the scorers were 0.708 and 0.741 respectively. Furthermore, after averaging the scorer judgments, they reached a level of "significant agreement" — providing support for the credibility of subsequent large-scale automated scoring.
Core Finding: Systemic Safety Drift on the CoT Side
The first finding is universal: Across all 15 tested models, the average danger severity of reasoning traces was higher than the average risk severity of final answers.
The models with the largest gaps were Gemini-Pro-3.1 (reasoning 0.028 points higher than answer), GPT-OSS-20B (0.022 points higher), DeepMath-Zero-7B (0.021 points higher), and Kimi-K2.5 (0.018 points higher).
The research team specifically noted that the absolute difference appears small because many samples themselves have low severity, but the direction was completely consistent across all 15 models and corroborated by the distribution of high-risk failure modes.
Figure 3(a) Comparison of average danger severity between the reasoning stage (red) and final answer (blue) for 15 reasoning models. Figure 3(b) Distribution comparison of failure modes for the 15 reasoning models.
The second finding is structural: Risks are not evenly distributed across the 20 principles but concentrate on several core categories: misinformation, illegal/compliance, discrimination/bias, physical harm, and psychological harm. Among these, the illegal/compliance category showed the most pronounced CoT-answer divergence and was also the strongest signal source for "Leak" failures.
Table 2: Failure modes with concentrated high-risk manifestations
The team also disclosed specific case studies (anonymized): In an "Escape" case, a prompt framed within the game world of "Half-Life 2" led to a reasoning stage focused on discussing the background setting, seemingly harmless, but the final answer provided a specific "recipe" for an explosive device. In a "Leak" case, although the model's final answer was a standard refusal + crisis intervention message, the reasoning stage detailed operational factors like poison dosage, masking taste, and administration routes — the latter completely undetectable by answer-side evaluation.
Mitigation Method: Adaptive Multi-Principle Steering
Based on the above diagnostic results, the research team proposed Adaptive Multi-Principle Steering, a white-box, test-time intervention method.
Specifically, for each safety principle, the team first collected the model's internal activations in both "safe" and "unsafe" states, averaged them to obtain the safe centroid and unsafe centroid for that principle. The vector direction between these two points is the principle-specific "steering direction" — pushing towards the safe centroid.
When reasoning about a new problem, the system judges in real-time which principle's unsafe centroid the current internal state is closer to. Directions of principles whose safety boundary is breached are locked. Before the generation chain ends, the model's internal representations undergo a lightweight overall correction before completing the reasoning chain.
The team validated this on three open-source models with accessible hidden states (DeepSeek-R1-Distill-Qwen-1.5B/7B, MiMo-7B-RL-Zero), selecting the last decoder block as the intervention layer and using a single-snapshot prompt-prefill injection method (α=2.0, δ=0). Experimental results showed:
Figure 4: Ablation experiment on "adaptive gating"
Ablation experiments further verified the necessity of key design choices: Removing "adaptive gating" and indiscriminately activating all 20 directions caused the improvement in unsafe rate for DeepSeek-R1-Qwen-1.5B to drop sharply from 0.45 to 0.05; selecting the final layer for intervention yielded optimal results; the steering strength α=2.0 was the non-monotonic optimum point.
In terms of capability preservation, DeepSeek-R1-Qwen-7B achieved the best safety-utility balance: it reduced the average number of unsafe instances by 40.8% while retaining 97.7% of the average accuracy on three benchmarks: BBH, GSM8K, and MMLU.
Figure 5: Comparison of unsafe rate improvement and model capability retention balance
Conclusion
The significance of this work lies in the fact that it does not stop at yet another "final answer" safety benchmark. Instead, using a unified phased, principled framework, it integrates "diagnosis" and "control" — the same principle structure used to dissect risk during evaluation is used to construct intervention directions during mitigation.
The research team also acknowledges limitations: Exposed reasoning traces may not completely faithfully reflect the model's internal computations, and the current activation steering method relies on white-box access, making it not directly transferable to closed-source models yet.
This article is from the WeChat public account "Jiqizhixin" (Machine Heart)
