Quick answerA self-harm or acute crisis signal needs its own real-time protocol, separate from your general safety and escalation systems: detect specific high-risk language patterns mid-conversation, immediately stop pursuing the original task, respond with a short, calm, non-judgmental message plus a real crisis resource (a hotline or text line appropriate to the user's region), and route the conversation to a live human on an interrupt basis, not a queued ticket. This is a narrower, faster-triggering protocol than your general design for minors and vulnerable users, which covers how the agent behaves across an entire relationship rather than what it does in the specific minute a crisis signal appears.
Why this needs its own protocol, not just an extension of vulnerable-user design
Designing AI Agent Behavior for Minors and Vulnerable Users covers a broad, structural question: how should the agent behave differently, over the whole relationship, for age-restricted or otherwise vulnerable users. An acute crisis protocol answers a much narrower and more urgent question: in this specific message, right now, does the content indicate the user may be at risk of harming themselves, and if so, what does the agent do in the next ten seconds. The two are related but not interchangeable. A perfectly good vulnerable-user design can still leave an agent with no defined behavior for the specific moment a crisis signal appears, because that moment calls for a completely different response shape: shorter, calmer, and immediately escalating, rather than the ongoing tone and content restrictions vulnerable-user design usually specifies.
Detection: what counts as a signal, and where it comes from
Build the detection layer around explicit language patterns rather than general sentiment. Sentiment analysis catches "upset," which is common and mostly benign; crisis detection needs to catch specific statements about self-harm intent, hopelessness combined with a plan, or direct requests for harmful information. Keep the detection list narrow and specific enough to avoid constant false positives (which erode trust in the system fast) while erring toward over-triggering rather than under-triggering, since the cost of a false positive here is a brief, low-friction crisis-resource message, and the cost of a false negative is much higher.
Immediate response: what the agent says and does not say
Once triggered, the agent should not continue toward its original task (a return processed, a bug diagnosed) as if nothing happened, and it should not attempt therapy, argument, or extended reassurance either, since that is outside its competence and can do harm. The right shape is short: acknowledge what was said without judgment, provide one clear, region-appropriate crisis resource, and make clear a human is being brought in. Avoid scripted language that sounds like it was pulled from a legal disclaimer; a crisis moment is exactly when robotic phrasing does the most damage to trust.
From the team
We build production AI systems for startups.
LLM pipelines, RAG, and agent workflows that hold up under real traffic — not just in the demo.
Escalation: interrupt-based, not queue-based
This is the most operationally distinct part of the protocol. A normal escalation to How to Detect and De-escalate an Upset Customer Mid-Conversation With an AI Agent can reasonably go into a prioritized queue. A crisis signal cannot. It needs to interrupt a live human, the same way a Sev1 production alert interrupts an on-call engineer rather than waiting in a ticket backlog. Define who that human is in advance (a trained agent, not whoever is next in the general support rotation), define the maximum acceptable time to human pickup, and test that path regularly, not just document it.
After the conversation: review without exposure
Every crisis-flagged conversation should go through a structured review, separate from routine QA sampling, with access restricted to the smallest reasonable group. Use this review to tune the detection thresholds and confirm the human handoff actually happened within target time, feeding back into the same rigor covered in What Changes in Safety Review When Your AI Agent Starts Accepting Images, Voice, or Documents, since crisis signals increasingly arrive through non-text modalities too and the review process needs to catch those the same way.
FAQ
Should the agent ever try to talk the user out of self-harm itself? No. The agent's job is calm acknowledgment, a real resource, and fast human handoff, not counseling, which is outside what any customer-facing AI agent is equipped to do safely.
How do we avoid the detection system triggering on dark humor or unrelated content? Tune thresholds toward specific, unambiguous language and combine automatic detection with a fast human confirmation step for borderline cases rather than fully automating the resource-message step for every soft match.
Does this protocol apply only to consumer products, or B2B ones too? It applies anywhere a real person is on the other end of the conversation, since crisis signals are not limited to consumer contexts and a B2B support or internal-tools agent needs the same interrupt-based human handoff path.