orchestration

In AI, orchestration is the coordination of model calls, tools or agents, by code or by a lead model, so that together they complete one task.

A single request to a large language model returns a single answer. Many AI products need several steps instead. A support system might sort a ticket, look up the order, draft a reply and check it against company policy. Each step can be a separate model call or a tool. Orchestration is the layer that runs those steps, hands each result to the next and decides when the job is finished.

The word became the name of that layer in 2023, when the venture firm Andreessen Horowitz listed LangChain and LlamaIndex as orchestration frameworks. Such frameworks handle chains of prompts, calls to outside services, document retrieval and memory across model calls.¹

There are two broad ways to orchestrate. Code can fix the path in advance, which Anthropic calls a workflow. Or a model can choose the steps as it goes. Anthropic’s orchestrator-workers pattern is a workflow in which a model picks the subtasks. A central model splits each task, hands the parts to worker models and combines the results.² OpenAI’s documentation draws the same line, and calls orchestration by code more predictable in speed, cost and performance.³

Since late 2024, labs have used the word for teams of AI agents. Anthropic’s Research system has a lead agent direct subagents that work in parallel. It beat a single agent by 90.2% on Anthropic’s internal research evaluation, but multi-agent systems used about 15 times as many tokens as a chat.⁴ Walden Yan of Cognition argues that such agents make conflicting decisions because they cannot share enough context.⁵

The AI sense differs from container orchestration, as in Kubernetes, which deploys, scales and manages software packaged in containers. It also differs from the Model Context Protocol, which connects tools but does not decide which one runs next.

From French orchestration, the art of arranging music for an orchestra, which entered English in the 1840s. By 1883 the verb orchestrate also meant to coordinate parts so they combine harmoniously. Computing took up the word for automated coordination. A 2004 W3C glossary used it for one web service calling others in a set order. Kubernetes, released as open source in 2014, automates container orchestration. AI use followed in February 2023, when Microsoft described the new Bing working through a Bing Orchestrator. By June, Andreessen Horowitz named orchestration as a layer of LLM applications. From late 2024, OpenAI and Anthropic applied it to systems of several models or agents.