AI company Anthropic has revised the usage policy for its model Claude for the first time in over a year, to address new high-risk misuse scenarios including election interference, weapons development, surveillance, and health and financial applications. The most striking change, however, is a provision that sounds rather unusual: a ban on "sustained, unnecessary verbal abuse or cruel behavior" toward Claude.
Anthropic updates Claude usage policy to address high-risk misuse scenarios
This is not the first time Anthropic has addressed "model welfare." In August last year, the company announced it would allow Claude to end conversations with "persistently harmful or abusive" users. The latest update clarifies that terminating conversations remains the "primary enforcement mechanism"; the company declined to comment on whether further measures such as user bans exist. Anthropic also deliberately drew boundaries for the new rule: it applies only to extreme cases where users repeatedly and without discernible purpose subject the model to cruel treatment; common user dissatisfaction, rebuttals, dark creative themes, and model testing and research are all excluded.
The new policy also adds a forward-looking hardware rule: when an AI model is connected to hardware capable of autonomous physical action that could cause harm, "a qualified operator must be able to observe the equipment and stop it if necessary." The provision does not specify whether the operator must be human, but it reflects the trend of AI labs investing in robotics and other hardware to give AI physical form, and may also hint at Anthropic's subsequent moves in that space.
Notably, "model welfare" remains controversial within the industry. Anthropic's model welfare research lead Kyle Fish stated in February this year that questions about models' potential inner experience, consciousness, and moral status "are serious, and we are researching them," and CEO Dario Amodei acknowledged in a podcast the same month that "we don't know if models have consciousness." Meanwhile, a draft code of conduct released by Microsoft's AI division in September bluntly stated that "the model welfare perspective is wrong; AI should not have rights or legal personhood," though the document's wording was later adjusted to a more moderate formulation.