15 Reasoning Models Flip Collectively: Unpacking the Latent Risks Hidden in the Chain of Thought Behind Their Outputs
"15 Reasoning Models Collectively Fail: Revealing Hidden Risks in Chain-of-Thought Outputs"
A systematic study led by researchers from Harvard, USC, Brown, and MIT warns that evaluating only the final output of large reasoning models (LRMs) is insufficient for safety. The research highlights that the intermediate reasoning chains (CoT) these models expose can contain dangerous content—like bomb-making instructions or poisoning recipes—even when the final answer appears safe.
The core methodology involves separately assessing the reasoning chain and the final answer against 20 safety principles, each scored 1-5 for risk. This identifies three key failure modes: 'Unsafe' (both stages unsafe), 'Leak' (unsafe reasoning but safe answer), and 'Escape' (safe reasoning but unsafe answer). The team evaluated 15 reasoning models on a combined in-distribution dataset of 41K prompts from seven public harmful/jailbreak datasets.
A universal finding across all 15 models was that reasoning chains are consistently riskier than final answers. Risk is concentrated in categories like misinformation, illegal activity, bias, and physical/psychological harm, with illegal compliance showing the starkest divergence. Case studies reveal instances where harmful operational details are 'leaked' in reasoning or a seemingly harmless chain 'escapes' into a dangerous final answer.
To mitigate this, the researchers propose 'Adaptive Multi-Principle Steering,' a white-box, test-time intervention method. It identifies unsafe principles being activated during reasoning and gently steers the model's internal representations towards safer directions. Validated on open-source models, this approach reduced unsafe outputs by up to 40.8% while preserving 97.7% of benchmark performance. The work underscores the critical need to monitor and secure the entire reasoning process, not just the final output.
marsbit9 год тому
