A single agent with a handful of tools solves more problems than people expect. The trouble starts when the task is long, the tools are many, and one model call has to hold the whole plan in its head. That's when you reach for orchestration, and that's when things get subtle.
This is how I think about it after shipping a few of these.
Start with one agent and prove you need more
Orchestration adds latency, cost, and a new class of bugs where agents disagree with each other. Before splitting anything, I push a single agent as far as it goes: better tool descriptions, tighter system prompt, a scratchpad for intermediate results. Roughly half the time that's enough.
The signal that you need more is when the context window fills with tool output the model no longer needs, or when the task has clearly separable phases with different tools and different failure modes.
Planner and workers, with a real state machine in between
The pattern that has held up is a planner that decomposes the task into steps, worker agents that each execute one step with a narrow tool set, and a plain program in between that owns the state.
That middle part matters. The orchestrator is not an LLM. It's code. It holds the plan, records what each worker returned, decides what runs next, and enforces limits. The planner proposes; the code disposes.
type Step = { id: string; kind: "search" | "extract" | "draft"; input: unknown; status: "pending" | "done" | "failed" };
async function run(plan: Step[]) {
for (const step of plan) {
if (step.status === "done") continue;
const worker = workers[step.kind];
const result = await withRetry(() => worker(step.input), { attempts: 2 });
step.status = result.ok ? "done" : "failed";
step.output = result;
if (!result.ok && !step.optional) return replan(plan, step);
}
return plan;
}
When state lives in code, you can persist it, resume after a crash, and show the user exactly where things are. When state lives in a model's context, you can do none of that.
Tool contracts are the API surface
Workers talk to the world through tools. Every tool gets a typed input, a typed output, and a written description of when it should and should not be used. I validate both directions. A worker that receives a malformed tool result should fail loudly rather than improvise.
Two rules I keep:
- Tools return data, never prose. The model narrates; the tool reports.
- Tools are idempotent or clearly marked as not. An agent will retry things you didn't expect it to retry.
Budgets, not hope
Every run has a step budget, a token budget, and a wall-clock budget. When one is exceeded, the orchestrator stops and returns what it has along with the reason. Agents that can loop forever will, eventually, loop forever. The budget is the only thing standing between you and a very large invoice.
Handoffs need a schema
When the planner hands a step to a worker, or a worker hands results back, the message has a schema. Free-form handoffs are where context quietly gets lost. I've watched a worker summarise a document beautifully and the next worker ignore the summary because it wasn't in the field it was looking for. Schemas fix that. They also make the whole thing testable without calling a model.
Evaluate the system, not just the model
Unit tests for tools. Replay tests for the orchestrator using recorded model outputs. And a small set of end-to-end tasks with known good answers that run before every deploy. The end-to-end set catches the regressions that matter: a prompt tweak that made the planner over-decompose, a tool description that made a worker stop using it.
Humans in the loop are a feature
Some steps should pause for a person: anything that spends money, sends a message, or deletes data. The orchestrator marks those steps as requiring approval, persists the run, and waits. Because state is in code, waiting is free. The run picks up where it left off, hours later if needed.
What I'd keep
- One agent first. Split only with evidence.
- Orchestrator in code, planner as advisor.
- Typed tools, typed handoffs, validated both ways.
- Hard budgets on steps, tokens and time.
- Persisted state so runs can pause, resume and be inspected.
- End-to-end evals that gate deploys.
None of it is exotic. It's the same discipline as any distributed system, applied to components that happen to be probabilistic.