How to Write an AI Agent System Prompt That Survives Production, Not Just a Demo

Quick answer

A demo system prompt only has to work for the three inputs you rehearsed. A production system prompt has to be general, unambiguous, and robust to adversarial or just-plain-weird real user input, which means it needs an explicit role and scope, concrete (not vague) behavioral rules, explicit handling for the "I don't know" and "this is out of scope" cases, and a tool-use contract that doesn't leave the model guessing. Treat it as a versioned artifact reviewed and evaluated like code, not a paragraph you tune once and forget.

Why demo prompts and production prompts are different documents

A demo prompt gets read and rehearsed a handful of times before the pitch. A production prompt gets executed thousands of times against inputs nobody wrote or rehearsed: typos, off-topic questions, adversarial attempts, multi-turn conversations that drift from where they started. The prompt that "worked" in the demo often fails the moment real traffic hits it, not because the model got worse, but because the prompt was never written to be general.

Start with role and scope, not a list of instructions

The most common mistake is opening with a list of dos and don'ts before establishing who the agent is and what it's for. Lead with a clear role definition (what the agent is, who it's talking to, and what it's explicitly not) before any behavioral instructions. This single ordering choice makes every rule that follows easier for the model to apply consistently, because the rules now have a frame to attach to.

Be concrete, not vague

"Be helpful and friendly" is a description, not an instruction. The model has no way to know what that means for a specific reply. "Professional, concise, never uses emojis, answers in under 150 words unless the user asks for detail" is an instruction it can actually follow. Every behavioral rule in a production prompt should be concrete enough that two different reviewers would judge a given output the same way against it.

Design for the cases the demo never showed you

A demo covers the happy path. Production needs explicit instructions for: what to say when the agent doesn't know the answer (this should never be silently fabricated), what to do when a request is out of scope, how to handle a user who's frustrated or hostile, and what to do when a tool call fails. If the prompt doesn't cover these, the model improvises. And improvised behavior in front of a real customer is where trust breaks.

Give the model an explicit tool-use contract

If the agent has tools, the prompt needs to say not just what each tool does but when to use it, what to do if it returns nothing or an error, and how to decide between two tools that could both plausibly apply. Ambiguous tool-use instructions are one of the most common causes of an agent that behaves inconsistently between otherwise-similar sessions.

Keep it short: length is usually padding, not precision

System prompts well past 1,500 tokens are usually padded with repeated instructions, unused examples, or safety language that doesn't change behavior. Every added paragraph is competing for the model's attention with every other paragraph; a shorter, sharper prompt often outperforms a longer, more "thorough" one because there's less to dilute.

Version it and evaluate every change

The prompt is the single longest-lived piece of "code" in most AI products, so treat it like code: store it outside the application, assign it a version, and run an eval set against every change before it ships. Backed by the observability and eval tooling that tells you whether a prompt change actually helped or quietly made things worse. The same behavioral rules that keep a prompt safe also need to survive contact with a hostile user, which is why security guardrails and graceful error and hallucination handling are written into the prompt, not bolted on after launch.

FAQ

How long should a production system prompt be? As short as it can be while still being unambiguous. Usually well under 1,500 tokens. If it's longer, look for repeated instructions or examples that aren't earning their space.

Should the system prompt handle security, or is that a separate layer? Both. The prompt should set behavioral boundaries (what the agent will and won't do), but it shouldn't be the only line of defense. Input filtering and output validation belong in the surrounding system, not solely in prompt text.

How often should a production system prompt change? Whenever an eval or real incident shows a gap. Not on a fixed schedule. Every change should go through the same versioned, evaluated process as the original, since a "small" wording tweak can shift behavior in ways that aren't obvious without testing.

Can the same system prompt work across multiple channels (chat, voice, email)? Rarely without adjustment: tone, response length, and error-handling expectations differ enough between channels that most teams maintain channel-specific variants of a shared core prompt rather than one prompt for everything.

Accelate writes and version-controls production system prompts as part of every agent build, evaluated against real conversation data rather than a handful of rehearsed demo inputs.