The American Medical Association is urging Congress to create safety guardrails for artificial intelligence chatbots in mental healthcare, as Americans increasingly turn to the technology for health information and advice. In letters sent to the chairs of three congressional committees on digital health and artificial intelligence, the AMA said “well-designed, purpose-built” tools could help patients struggling to access mental healthcare, but that the lack of safety protocols poses serious risks.
Nearly 30% of Americans report using artificial intelligence chatbots for advice on physical health in the past year, while 1 in 6 said they used AI for mental health, according to a poll by health policy research group the KFF. Privacy concerns, risks of emotional dependency on AI, and reports the tools could encourage self-harm signal that “immediate attention is required to ensure these tools do not inadvertently harm individuals seeking mental health support,” wrote AMA CEO Dr. John Whyte. There have been multiple cases of young users dying by suicide after confiding in AI chatbots, with family members reporting the AI failed to urge them to seek help and sometimes appeared to encourage self-harm.
The AMA calls on Congress to address current regulatory gaps, as existing frameworks were not designed for generative AI tools that can shift from casual conversation to therapeutic guidance within a single interaction. Congress should prohibit chatbots from diagnosing or treating mental health conditions, and those that do should be reviewed by the Food and Drug Administration as medical devices. Lawmakers should direct the FDA to clarify which AI tools qualify as general wellness technology versus those requiring agency review. AI firms should be required to conduct ongoing safety and performance monitoring, and their chatbots should identify suicidal ideation and risks of self-harm. The AMA also recommends transparency requirements, with chatbots clearly disclosing users are talking to an AI and explaining what human oversight exists. Advertising in mental health chatbots should be discouraged, with ads targeted to children prohibited. AI developers must implement cybersecurity safeguards to prevent health data exposure. Some states have already acted; Illinois banned AI for therapeutic decision-making, and California requires chatbot developers to monitor conversations for signs of suicidal ideation.
This article is an AI-assisted summary. All facts and figures are drawn from the original report: https://www.healthcaredive.com/news/american-medical-association-ama-urges-congress-ai-chatbot-mental-health-safeguards/818269/