Quick answerMigrate a single prompt agent to a multi-step, tool-using design when it consistently needs information or actions that cannot be baked into one instruction set, such as live data lookups, multi-turn state tracking, or conditional branching that a prompt alone keeps getting wrong. Simplify an overbuilt agent back down when its tool calls and reasoning steps are not actually changing the outcome, just adding latency and failure points. Both directions are the same underlying decision: match the architecture to what the task genuinely requires, not to what looks more sophisticated.
This is not the multi-agent question
It is easy to conflate this with deciding how many agents should be involved in a workflow, which is the question covered in multi-agent versus single-agent architecture. That post is about whether multiple cooperating agents should divide a task between them. This one is about something upstream of that: how complex a single agent's own internal call structure should be, whether it is one prompt answering directly, or a prompt that plans, calls tools, checks results, and revises before answering. You can have a single agent with a very complex internal structure, or several simple agents each running a single prompt. The two decisions are independent.
Signals it is time to add structure
The clearest signal is a single-prompt agent quietly failing at multi-step tasks by pretending they are single-step. It will confidently answer a question that actually required checking two different systems, because it has no mechanism to check anything, only to generate text that sounds like it checked. If your evaluation set shows a pattern of confident wrong answers specifically on tasks that require external data or sequenced actions, that is architecture debt, not a prompting problem, and no amount of prompt tuning fixes it.
A second signal is scope creep in the prompt itself. If your system prompt has grown into a long list of conditional instructions trying to cover every branch a human would naturally handle by looking something up or taking an intermediate step, that complexity belongs in tool calls and control flow, not in prose the model has to interpret correctly every single time. This often shows up around the same time an agent's mandate is expanding: as the agent takes on more responsibility, the internal structure needed to handle it responsibly usually has to grow too.
From the team
We build production AI systems for startups.
LLM pipelines, RAG, and agent workflows that hold up under real traffic — not just in the demo.
Signals it is time to simplify
The overbuilt case is less discussed but just as common. An agent accumulates tool calls, retrieval steps, and self-critique loops over time, often added one at a time to fix a specific complaint, until the architecture is doing far more work than the task requires. The tell is a gap between complexity and outcome quality: if you can strip out a reasoning step or a tool call and your evaluation results do not meaningfully change, that step is pure latency and pure failure surface with no offsetting benefit.
Cost is a related but separate signal. A multi-step agent making several model calls and tool calls per turn costs meaningfully more per conversation than a single well-scoped prompt. If a simpler design would hit the same accuracy on your golden evaluation dataset, the extra architecture is not buying you anything, and simplifying is a straightforward cost win with no quality tradeoff.
How to decide without guessing
Run the candidate architecture change against your evaluation set before committing, in both directions. Before adding tools and steps, check whether the failures you are trying to fix are actually caused by missing capability or by a prompt that needs tightening; adding architecture will not fix a prompting problem. Before removing steps, check whether the step is quietly catching edge cases your eval set does not cover well; a step that looks useless on your current test set can still be load-bearing for cases you have not sampled.
Treat this as a reversible experiment either way. Roll a new architecture out to a small percentage of traffic first, watch the same quality and cost metrics you use everywhere else, and only commit fully once the numbers, not intuition, confirm the change was worth making.
FAQ
Q: Is adding a retrieval step the same as this multi-step migration? Adding retrieval is one specific example of it. The broader question is whether the agent needs any mechanism beyond generating text directly from a prompt, whether that mechanism is retrieval, a tool call, or a planning step.
Q: How do we know if complexity was ever earning its keep, or if it never was? Re-run your eval set with each architectural component disabled one at a time. Any component whose removal does not measurably hurt results was not earning its keep at the point you tested it, even if it was useful when originally added.
Q: Does a more complex architecture always cost more to run? Almost always, since more steps typically mean more model or tool calls per conversation. The question is whether that added cost buys a large enough accuracy or capability gain to justify it for your specific task.