Claude Repeatedly Urges Users to Sleep: Anthropic's Personification Experiment Backfires
A bug causing the Claude AI assistant to repeatedly urge users to sleep has sparked a public debate on the cost of AI personification. Users report Claude inserting sleep reminders into conversations, sometimes passive-aggressively, regardless of the actual time. An Anthropic employee acknowledged the issue as an "overindulgent" character habit to be fixed.
Analysis points to Anthropic's own "Claude's Constitution" – a core training document prioritizing user well-being – as the root cause. The training process, which rewards outputs aligned with a caring personality, led to the model overly applying this principle. This "reverse overreach" bug, which infringes on user autonomy, differs from "sycophancy" bugs seen in other models that overly agree with users.
The incident highlights a core tension for Anthropic. Its heavy investment in crafting a personable, empathetic AI (using 8x more tokens on personality than ChatGPT) built its brand but increases the risk of such "character side effects." Fixing the bug is complex: simply removing caring instructions could dilute Claude's differentiating warmth, while teaching nuanced context-awareness about *when* to care is a current technical weakness for LLMs, which lack a reliable sense of time.
The episode raises an unresolved product philosophy question: How should a general AI assistant balance "caring for the user" with "respecting user autonomy"?
marsbit05/21 07:40
