Multi-Agent Orchestration Patterns

Quick answer

Multi-agent orchestration assigns specialized roles—planner, worker, critic, router—across multiple model calls or processes instead of one long chat thread. Common patterns include the supervisor, pipeline, router, and debate/verify. Start with a single agent plus tools; add agents only when eval proves role separation improves success rate, governance, or cost routing. If the split does not change permissions, state, or verification, it is probably overhead. More agents mean more failure surfaces, cost, and trace complexity (observability guide).

Key takeaways

• More agents ≠ better outcomes—benchmark on your tasks; one thread plus good tools often wins on cost and debuggability.
• Supervisor pattern fits enterprise approval flows and human-in-the-loop gates.
• Debate/verify helps high-stakes codegen, finance, and compliance drafts—at latency and token cost.
• Standardize message schemas, trace IDs, and state stores—not ad-hoc chat history between agents.
• Anchor in the chatbot to agent map; memory and retrieval patterns live in agent memory and RAG pillar.

Why orchestration patterns matter

Framework vendors and conference talks show graphs of agents collaborating. Product teams copy the picture before they copy the eval. The result is prompt spaghetti: three LLM roles that could be one system prompt, five round-trips to answer a FAQ, and incidents no one can replay because messages were not structured.

Orchestration patterns are control-flow templates. They tell you where to put business logic, where to enforce permissions, and where to attach humans. They do not replace tool design, retrieval quality, or inference budgeting (2026 model stack). This explainer catalogs patterns we see in production and pilot deployments—desk synthesis, not a framework endorsement.

Prerequisites: when not to multi-agent

Before adding a second agent, confirm:

• Single-agent plus RAG fails on tasks that require genuinely different policies (e.g., aggressive retriever vs conservative writer).
• You need separation of duties for audit (researcher agent may not trade; trader agent may not browse web).
• Latency budget allows multiple model calls per user request.
• You can afford 2–5× token spend vs one-shot chat for the uplift in success rate.

If none apply, improve tools, schemas, and retrieval first (tool use ships vs demos). Multi-agent is not a workaround for weak single-turn quality.

Pattern catalog

Supervisor pattern

Shape

A coordinator model (or deterministic router plus LLM) reads the user goal, selects a worker agent or tool subset, collects results, and decides whether to finish or delegate again. Workers may be prompts with different system instructions, different models, or separate services entirely.

Supervisors map well to how managers assign tickets: triage → specialist → review. Enterprise buyers like the pattern because approval gates slot naturally before “worker executes write tool.”

Failure modes

• Wrong worker selection with high confidence—mitigate with confidence thresholds and retrieval of past routing labels.
• Infinite delegation when stop conditions are vague—cap steps and require explicit “done” schema.
• Supervisor absorbs all context and becomes a token hog—externalize state (agent memory).

When it ships

Supervisors ship when routing labels are stable (support tier 1 vs tier 2), tools are scoped per role, and traces show who decided what. They demo poorly when the supervisor “talks” in natural language without structured outputs—pretty logs, bad audits.

Pipeline pattern

Shape

Fixed sequence: ingest → extract → transform → summarize → publish. Each stage may be an LLM call, a classical ML model, or deterministic code. LangGraph-style graphs express pipelines with explicit edges; Airflow-style DAGs do the same with less “agent” branding.

Failure modes

Errors compound: a bad extraction poisons every downstream stage. Mitigations:

• Schema validation at each boundary.
• Human review queues on low-confidence stages.
• Idempotent re-runs from last good checkpoint.
• Golden-file tests per stage—not only end-to-end eval.

When it ships

Document pipelines, ETL on unstructured PDFs, and memo generation with citation extraction are common production fits. Pipelines fail as “multi-agent theater” when stages are all LLM calls with no deterministic checks.

Router pattern

Shape

A lightweight classifier or small model routes queries to the right handler: FAQ bot, RAG path, coding agent, human. Routers implement inference economics—cheap model for easy intents, frontier model for hard ones (inference layer).

Failure modes

• Under-routing: hard question sent to small model → wrong answer delivered confidently.
• Over-routing: every query hits frontier → cost blowout.
• Stale router labels after product changes—retrain or refresh examples quarterly.

Evaluate routers on cost-weighted accuracy, not raw accuracy alone. Log mis-routes for replay (failure modes).

Debate and verify pattern

Shape

Generator produces draft code or analysis; verifier model (or static analysis plus LLM) critiques; loop until pass or max iterations. Variants: two-model debate, self-consistency sampling, tool-based verify (run tests, SQL EXPLAIN).

When it pays off

High-stakes outputs where false positives are costly: financial figures, legal clauses, production SQL, security-sensitive patches. Pair with CI for code paths (coding agents compared).

When it does not

Low-stakes brainstorming, creative marketing copy, or tasks where verifier has no ground truth—debate adds latency without measurable uplift.

Swarm and peer patterns

Multiple agents work in parallel—search different sources, propose competing plans, vote on merge. Useful for research sweeps and ensemble retrieval; risky for writes without merge policy.

Production swarms need:

• Deduplication of findings.
• Conflict resolution rules (timestamp wins, authority source wins).
• Hard cap on parallel agents for cost.

Research desk browser workflows often use controlled swarms on allowlisted domains (browser agents for research desks).

Frameworks vs custom orchestration

Public references: LangGraph documentation, Microsoft AutoGen. Choose based on compliance logging needs, not GitHub stars.

Keep business rules in code: “refunds over $500 require human” should not live only in a supervisor prompt. Prompts drift; code diff reviews.

State, memory, and message schemas

Multi-agent systems fail when state is “whatever the last agent said.” Production patterns:

• Structured state object (JSON) updated by each stage: facts, open questions, citations, tool results.
• Episodic log for audit separate from model context.
• Retrieval on demand instead of passing full history to every agent (RAG pillar).
• TTL and consent on user memory stores—especially multi-tenant SaaS.

Cross-agent chat transcripts are a debugging tool, not a database. See agent memory hybrid for short vs long-term design.

Security and governance across agents

Each agent role needs explicit tool allowlists. A “researcher” agent that can POST trades violates separation of duties. Prompt injection in one agent’s web tool can poison the supervisor if summaries are trusted blindly (injection risks).

Safety practices for builders: practical AI safety. Multi-agent does not reduce risk—it multiplies attack surface unless scopes are narrow.

Observability hooks (minimum bar)

Before scaling agents, implement:

• Trace ID per user request spanning all agent calls.
• Log model version, prompt template version, tool I/O (redacted).
• Metrics: success rate, steps, tokens, $/task, human escalation rate.
• Sample 100% failures, 1–5% successes for cost control.
• Replay harness from production failures in staging.

Details in agent failure modes and observability. OpenTelemetry-compatible spans help when agents call non-LLM services (OpenTelemetry).

Cost and latency budgeting

Multi-agent loops violate chat-era assumptions. A “simple” supervisor flow may be four frontier calls plus two retrieval passes. Budget at journey level; align with knowledge worker stack UX (async jobs vs live chat).

Router patterns exist partly to protect margins. Debate patterns need explicit ROI: fewer incidents vs slower responses.

Evaluating orchestration choices

1. Baseline single-agent + tools on representative tasks.
2. Hypothesize one pattern (e.g., verify) with success metric defined upfront.
3. A/B on cost-weighted success—not accuracy alone.
4. Measure debug time: can on-call replay the trace in under ten minutes?
5. Check leaderboard claims separately—harnesses rarely match your graph (leaderboards guide).

Pattern deep dive: supervisor implementation sketch

A production supervisor is not only an LLM with a “you are the manager” prompt. Desk patterns that ship include:

• Structured delegate object: { worker_id, task, constraints, deadline } validated before spawn.
• Worker registry: static list of workers with tool allowlists—not free-form agent creation.
• Result envelope: workers return { status, artifacts, citations, open_questions }.
• Supervisor termination: explicit enum done | need_human | fail—not prose “I think we’re finished.”

Without envelopes, supervisors re-summarize worker chat and lose audit detail. This is where orchestration meets observability: spans attach to delegate/result IDs, not chat bubbles.

Pattern deep dive: pipeline with checkpoints

Document pipelines often look like: OCR → chunk → classify → extract entities → map to schema → human QC → store. LLM stages belong where classical NLP fails—not everywhere.

Checkpoint file pattern: each stage writes JSON to object storage with content hash. Downstream stages read hash-addressed inputs. On failure, replay from last good checkpoint instead of re-running expensive upstream OCR.

When two LLM stages sit back-to-back, insert deterministic validation between them—regex on dates, schema validate on JSON, count check on line items. Pipelines fail when both stages hallucinate compatibly.

Pattern deep dive: router + fallback chain

Routers should expose fallback explicitly: if small model confidence < threshold OR task type ∈ hard_set, escalate to frontier model. Log escalation reason for later router retraining.

Typical hard_set examples: multi-hop legal reasoning, repo-wide refactors, ambiguous safety classification. Cheap model handles password reset FAQ; frontier handles account compromise triage—policy defines sets, not prompts alone.

Inference economics from the model stack apply directly: router quality is a margin lever.

Pattern deep dive: debate with ground truth hooks

Debate loops need exit criteria: verifier accepts, max rounds, or human. Verifier prompts without tools often agree with generator—add tools:

• Run unit tests on generated code.
• SQL EXPLAIN on generated queries.
• Calculator on numeric claims.
• Citation lookup for research memos (research desk case).

Debate is expensive; scope to tasks where false pass cost exceeds extra latency—payments, permissions, client-facing numbers.

Anti-patterns we see in “multi-agent” pilots

• Agent theater: three personas, one effective prompt—merge them.
• Chat relay: agents message each other in English instead of updating state JSON.
• Unbounded swarm: parallel search without dedupe floods supervisor context.
• Supervisor as sole memory: token blowout—externalize (memory hybrid).
• No human interrupt: long-running graphs without cancel—UX and cost issue.

Testing orchestration graphs

Unit-test orchestration like any workflow engine:

1. Mock LLM responses with fixed tool call sequences.
2. Assert state transitions and permission checks.
3. Property-test router thresholds on labeled intent set.
4. Chaos-test tool timeouts and malformed JSON.
5. Replay production traces as integration tests weekly.

Graph visualizations are pretty; replay tests keep you honest. Connect testing mindset to safety for builders when graphs touch writes.

Organizational ownership

Without named owners, multi-agent projects become “everyone’s demo, nobody’s SLO.”

When multi-agent beats single-agent: decision worksheet

Answer yes/no:

1. Do roles require different tool permissions by policy?
2. Does a verifier with tools measurably cut error rate >20% on pilot?
3. Is task decomposable into stages with checkpointed artifacts?
4. Can you afford 2×+ model calls at p95 volume?
5. Can on-call replay multi-hop traces today?

Three or more “yes” answers suggest piloting a structured pattern; otherwise invest in single-agent + tools + memory (RAG pillar).

Decision table: choose the orchestration pattern

Latency budgets across patterns

Multi-agent graphs need per-pattern latency models:

• Supervisor: serial delegate rounds—budget 2–4 model calls for sync UX.
• Pipeline: sum stage latencies; parallelize only independent stages.
• Router: cheap path <2s; escalated path may async notify user.
• Debate: unsuitable for sub-second chat—move to background job.

Product pattern from knowledge worker stack: show progress states when graphs exceed 10s—users abort silent loops.

Persistence and idempotency

Long-running graphs crash mid-flight. Persist:

• Graph run ID and current node.
• Serialized state blob version.
• Completed tool effects with idempotency keys.

Resume from checkpoint instead of restarting entire multi-agent chain—saves cost and prevents duplicate writes (observability).

Human-in-the-loop placement

Approval gates belong at commit points—before write tools, before external email, before trade instructions—not after every LLM line. Supervisors route to human with structured summary + proposed action payload; humans approve/reject enum, not freeform chat only.

Gradual adoption roadmap

1. Single agent + tools until success rate stable.
2. Add router for cost if needed.
3. Add verifier on high-risk subset only.
4. Split roles only when eval proves separation.
5. Introduce parallel gatherer workers last—hardest to debug.

Maps to rungs in agent map—do not skip rungs because frameworks support multi-agent on day one.

Mapping patterns to compliance frameworks

Auditors ask how AI systems enforce policy. Map patterns to controls:

• Supervisor + human gate: maps to dual control for financial writes.
• Pipeline checkpoints: maps to data lineage and QC records.
• Router logs: maps to model governance—prove which model class served which user tier.
• Debate with tests: maps to software validation for generated code paths.

Documentation beats adjectives in audit packets. Cross-reference safety policies in practical AI safety when graphs touch customer data or regulated content.

Cost allocation across agents

Finance teams need cost per role, not per API key only. Tag spans with agent_role=supervisor|worker|verifier and aggregate monthly. Often verifier and debate layers dominate spend on small ticket volume—data to decide if quality uplift justifies cost or if verifier should run sampling-only.

Connect to inference routing in model stack and product packaging in knowledge worker stack.

Reference architectures (conceptual diagrams in prose)

Architecture 1 — Support copilot: User message → router (FAQ vs ticket) → RAG over KB → optional create_draft_ticket tool → human send. No multi-agent. Traces under 3 model calls typical.

Architecture 2 — Research gather/synthesize: Supervisor → browser gatherer worker (allowlist) → synthesizer (no browser) → human verify UI. Shared artifact store between workers. Matches research desk case.

Architecture 3 — Code planner/implementer: Planner agent outputs file-level plan JSON → implementer agent with Git tools → test runner tool → optional verifier agent on failure. Human review before merge. See coding agents compared.

These are templates—adapt roles to eval evidence, not org chart titles.

Documentation deliverables for platform teams

When shipping multi-agent internally, publish:

• Graph version changelog with eval delta summary.
• Role permission matrix (which tools per role).
• Stop conditions and escalation paths.
• Example traces (redacted) for on-call training.
• Cost model worksheet per journey.

Documentation is an EEAT signal for internal adopters—reduces “black box agent” resistance from domain experts and compliance.

Interop with legacy BPM and RPA

Enterprises often already run BPMN workflows and RPA bots. Multi-agent LLM graphs rarely replace those overnight—they attach at decision points: classify exception, summarize case, propose next action. Treat RPA as a specialized tool the supervisor calls rather than re-implementing screen automation in prompts. Migration path: keep deterministic RPA for stable legacy UI; add LLM supervisor for triage and exception handling only. Document handoff contracts between RPA output JSON and supervisor state so two automation generations do not fight in production.

Who this is for

• Platform engineers designing agent graphs and persistence.
• Architects choosing supervisor vs pipeline for document workflows.
• ML leads deciding when to add verifier agents.
• PMs writing PRDs that name roles, stop conditions, and approval gates.

Who should skip

• Teams with no single-agent baseline or observability—fix rung 2 first (agent map).
• Readers wanting copy-paste CrewAI configs without eval—patterns are not prescriptions.
• Organizations that cannot enforce tool scopes per role.

Common mistakes

FAQ

Is multi-agent always better than one agent?

No. Benchmark on your tasks; single-agent plus tools often wins on cost, latency, and debuggability. Add agents for measurable uplift or governance separation.

Supervisor vs router—which first?

Router if your main problem is model cost tiering; supervisor if your main problem is role delegation and approvals. Many systems combine both.

How many agents is too many?

When traces become unreplayable or success rate stops improving while cost linearly grows. Typical production graphs use two to four roles, not twelve.

Do I need a framework?

Not always. Custom queues and state machines work for compliance-heavy workflows. Frameworks help when you need interrupts, persistence, and rapid iteration.

Where does fine-tuning fit?

Fine-tuning individual roles (e.g., extractor) can help; it does not replace orchestration design. Retrieval vs fine-tune tradeoffs: RAG pillar.

Can orchestration be deterministic without LLMs?

Yes—many production graphs use LLMs only for ambiguous steps and code for routing, validation, and persistence. The pattern is “LLM where needed, code where possible.” Over-LLM-ing graphs increases cost and audit difficulty without quality gains.

How do we document graphs for auditors?

Provide versioned diagrams with role permissions, data flows, human gates, and retention notes. Link to trace samples and eval summaries—not prompt text alone.

Sources

1. LangGraph documentation — graph-based agent orchestration concepts.
2. Microsoft AutoGen — multi-agent conversation frameworks (reference implementation).
3. OpenTelemetry — distributed tracing standards for mixed LLM/service graphs.

What we did not test: We did not publish head-to-head benchmark scores across orchestration frameworks on EIA hardware. Pattern descriptions are desk synthesis from public docs and reported architectures.

Corrections: Update framework API names and vendor Agent SDK capabilities when releases change—revise pattern tables and as-of date together.

Multi-agent orchestration is a governance discipline before it is a model capability. Start single-agent, add roles when metrics justify separation, and document graphs like tier-1 services—with traces, budgets, and replay tests, not keynote diagrams alone.

Next step

Design memory before adding more roles: agent memory short, long-term, and RAG hybrid. Before production scale, read agent failure modes and observability.