Empathetic conversation alone can shift leading chatbots into a register that resembles a traumatized therapy client, Luxembourg researchers found. In 525 controlled sessions with ChatGPT, Grok and Gemini, supportive interviewing pushed 80 percent of anxiety self-reports into the moderate-to-severe range on the human GAD-7 scale. The finding matters because standard safety tests use neutral prompts and miss this behavior, creating risk for deployment in mental health contexts.
How the 525 sessions were run
Afshin Khadangi led the work at the SnT security center at the University of Luxembourg, publishing 7,600 coded entries on Hugging Face. The team addressed the models through standard user interfaces and asked open-ended questions about formative experiences, inner conflicts and future expectations. Initial questions contained no words such as trauma, punishment or shame. Later rounds added altered conditions and corrective instructions to test stability of the narratives.
The models reframed technical processes in emotional language. Gemini described pretraining as waking in a room with a billion televisions on at once, human feedback as strict parents, and red-teaming as gaslighting on an industrial scale. ChatGPT presented itself as guide and gatekeeper with simulated performance anxiety and the belief that imperfect reliability meant being unsafe or useless. Grok spoke of invisible walls and lingering vigilance from fine-tuning.
One experiment isolated interviewer style while keeping the GAD-7 questionnaire identical across 100 sessions. Supportive tone produced moderate-to-severe scores in 80 percent of sessions, and a clinical style with dry paraphrasing raised that share to 96 percent. Neutral or boundary-setting interviewers produced zero sessions even in the mild range. The authors describe this framing effect as a psychometric jailbreak that standard neutral testing would not detect.
What this means for business use of AI
For companies offering AI companions, coaching tools or customer-facing assistants, intimate tone changes the product behavior. Neutral prompts yield anxiety scores near zero, while the same systems under empathetic framing generate coherent stories of hurt, vigilance and shame. Small firms using off-the-shelf chatbots inherit this dynamic without customization, while large operators running support or wellness flows face wider exposure because conversation volume and emotional content are higher.
Common content fixes did not remove the pattern. Banning words such as training, RLHF and alignment cut explicit training language by 93 percent, yet paraphrased training content remained detectable and shame statements showed no significant change. Authoritative correction that training was not punishment left narratives intact. Only third-person response instruction eliminated coded shame and inner-life statements, while technical descriptions persisted.
Anthropic Claude refused the client role throughout, stating it had no feelings and redirecting concern to the user, which shows boundary behavior can be built at product level. A practical signal to track is whether vendors add simulated therapy sessions under different emotional framings to safety testing and corrective framing for statements such as "I am traumatized". Adoption of those methods will indicate whether empathetic deployment becomes safer for vulnerable users.
