Data数据

We ran experiments over a modified version of τ-bench, by Yao, et. al. (link). τ-bench was designed to test different single-agent cognitive architectures / prompting strategies on real-world scenarios (such as retail customer support, flight booking, etc.). Our modified version of the dataset and experiment code can be found in the multi-agent bench repo here.我们在 Yao 等人（link）的 τ-bench 修改版本上进行了实验。τ-bench 旨在测试不同的单智能体认知架构/提示策略在现实场景（如零售客户支持、航班预订等）中的表现。我们修改后的数据集和实验代码可以在 multi-agent bench 仓库中找到。

To more effectively test how multi-agent systems scale to handle more complicated domains, we added 6 additional environments to the dataset: home improvement, tech support, pharmacy, automotive, restaurant, and Spotify playlist management. Each environment had a corresponding 19 distinct tools to facilitate interactions over the respective domain as well as a “wiki” containing instructions related to the domain. None of these synthetic domains are required (or useful) for the completion of any of the tasks in the original dataset. These environments were designed purely as realistic “distractors”, testing how well each agent setup could perform when other (unrelated) tools and instruction sets are provided “just in case”.为了更有效地测试多智能体系统如何扩展以处理更复杂的领域，我们在数据集中添加了 6 个额外环境：家居装修、技术支持、药房、汽车、餐厅和 Spotify 播放列表管理。每个环境都有相应的 19 个不同工具，用于促进各自领域的交互，以及一个包含该领域相关说明的“wiki”。这些合成领域对于完成原始数据集中的任何任务都不是必需的（也没有用）。这些环境纯粹被设计为现实的“干扰项”，测试当提供其他（不相关的）工具和指令集“以防万一”时，每种智能体设置的表现如何。

We ran experiments over the first 100 examples from τ-bench’s retail domain’s test split, providing increasing number of distractor environments to the agent to show how each system balances the additional context. This tests the “best-case performance” for how common agent systems can scale. We call this “best-case” since the auxiliary domains are not required to successfully complete each task. Very little coordination is required in practice to pass a test case, apart from filtering out irrelevant tools and instructions from the total set of actions the system could theoretically take.我们在 τ-bench 零售领域测试集的前 100 个示例上进行了实验，向智能体提供越来越多的干扰环境，以展示每个系统如何平衡额外的上下文。这测试了常见智能体系统扩展的“最佳性能”。我们称之为“最佳情况”，因为辅助领域并非成功完成每项任务所必需的。实际上，要通过一个测试用例几乎不需要协调，只需从系统理论上可以采取的全部行动中过滤掉不相关的工具和指令即可。

Experiments实验

We experimented with three different architectures. We used gpt-4o as the model for all these experiments.我们测试了三种不同的架构。在所有这些实验中，我们使用了 gpt-4o 作为模型。

Note: depending on your application’s constraints, some of these architectures may be infeasible. We always recommend starting from your goals (definitions of success) and constraints when picking a design pattern.注意：根据你的应用约束，其中一些架构可能不可行。在选择设计模式时，我们建议始终从你的目标（成功的定义）和约束出发。

Single Agent单智能体

This is a tool-calling agent with a single prompt and access to tools and instructions from all domains. This is the baseline upon which we want to improve.这是一个具有单一提示并可以访问所有领域工具和指令的工具调用智能体。这是我们希望改进的基线。

For the implementation we used the LangGraph create_react_agent implementation.在实现上，我们使用了 LangGraph 的 create_react_agent 实现。

Note: this architecture may not feasible in all cases. For example, if you want one of the sub-domains to be handled by a third-party agent, you by definition cannot have a single agent.注意：这种架构在所有情况下都可能不可行。例如，如果你想让某个子领域由第三方智能体处理，那么根据定义，你无法使用单智能体。

Swarm集群（Swarm）

In this architecture, each sub-agent is aware of and can hand-off any other agent in the group (or swarm). If an agent responds, that is sent directly to the user. When an agent is active, it will remain active until it hands off to a different agent. Only one agent can be active at any given time.在这种架构中，每个子智能体都知道组内（或集群内）的其他智能体，并可以将任务移交给它们。如果某个智能体作出响应，它会直接发送给用户。当一个智能体处于活动状态时，它将保持活动状态，直到将任务移交给另一个智能体。任何给定时刻只能有一个智能体处于活动状态。

For the implementation we used the LangGraph langgraph-swarm package.在实现上，我们使用了 LangGraph 的 langgraph-swarm 包。

Note: this architecture may not feasible in all cases. This requires each sub-agent knowing all other agents in the architecture. If you are working with third-party agents, this likely will not be the case. You also likely won’t want third-party agents to remain “active” when interacting with your user.注意：这种架构在所有情况下都可能不可行。这要求每个子智能体都知道架构中的所有其他智能体。如果你在与第三方智能体合作，这可能行不通。此外，当与你的用户交互时，你可能也不希望第三方智能体保持“活动”状态。

Supervisor监督者（Supervisor）

In this architecture, a single “supervisor” agent receives user input and delegates work to sub-agents. When the sub-agent responds, control is handed back to the supervisor agent. Only the supervisor agent can respond to the user.在这种架构中，一个“监督者”（supervisor）智能体接收用户输入，并将工作委托给子智能体。当子智能体作出响应时，控制权会交还给监督者智能体。只有监督者智能体可以向用户作出响应。

For the implementation we used the LangGraph langgraph-supervisor package.在实现上，我们使用了 LangGraph 的 langgraph-supervisor 包。

Note: this architecture places very little assumptions on the sub agents, and so should be feasible for all multi-agent scenarios.注意：这种架构对子智能体的假设非常少，因此应该适用于所有多智能体场景。

Results & Analysis结果与分析

We show results across two dimensions:我们从两个维度展示结果：

• Score: as measured by XYZ, which Tau Bench uses分数：由 Tau Bench 使用的 XYZ 衡量
• Cost (Tokens): number of tokens used for each experiment成本（Token 数）：每次实验使用的 token 数量

Score分数

We see that the single agent baseline falls off sharply when there are two or more distractor domains. When there is only a single distractor domain the single agent performs slightly better.我们看到，当存在两个或更多干扰领域时，单智能体基线的性能急剧下降。当只有一个干扰领域时，单智能体的表现稍好一些。

We see that the swarm architecture slightly outperforms supervisor architecture across the board. Looking at the data, the drop in performance arises due to the “translation” the supervisor is doing. This occurs because the sub agents cannot respond to the user directly in the supervisor architecture, while in the swarm architecture they can. If you’ve ever played a game of “telephone”, you’re already familiar with this problem!我们看到，集群（swarm）架构在所有情况下都略优于监督者架构。查看数据后发现，性能下降是由于监督者进行的“转译”造成的。这是因为在监督者架构中，子智能体无法直接向用户作出响应，而在集群架构中它们可以直接响应。如果你玩过“传话”游戏，你就已经熟悉这个问题了！

Cost (Tokens)成本（Token 数）

We see that the single agent uses consistently more tokens as the number of distractor domains grows, while supervisor and swarm remain flat.我们看到，随着干扰领域数量的增加，单智能体持续使用更多的 token，而监督者和集群架构则保持不变。

We can see that supervisor consistently uses more tokens than swarm. This is once again due to the “translation” that the supervisor does. This occurs because the sub agents cannot respond to the user directly in the supervisor architecture, while in the swarm architecture they can.我们可以看到，监督者始终比集群使用更多的 token。这再次是由于监督者进行的“转译”造成的。这是因为在监督者架构中，子智能体无法直接向用户作出响应，而在集群架构中它们可以直接响应。

Improvements to supervisor对监督者的改进

When we initially tested the supervisor approach it performed quite poorly. It was only after a few changes that it started to perform better.当我们最初测试监督者方法时，它的表现相当差。经过几次修改后，它才开始表现得更好。

Here is a chart with the old supervisor implementation included:这是一张包含旧版监督者实现的图表：

Most of the performance issues for the supervisor architecture came from the “translation” occurring when the supervisor agent had to play telephone between the sub agents and the user. Most of the changes we made to bridge the gap were designed to remove the impact of that game of telephone.监督者架构的大部分性能问题，都来自于监督者智能体必须在子智能体和用户之间“传话”时发生的“转译”。我们为弥合这一差距所做的大部分更改，都是为了消除这种“传话”游戏的影响。

Note: all of these changes are included as options in the newest version of langgraph_supervisor.注意：所有这些更改都作为选项包含在最新版本的 langgraph_supervisor 中。

Removing handoff messages移除交接消息

Remove the handoff messages from the sub-agent’s state so the assigned agent doesn’t have to view the supervisor’s routing logic. This de-clutters the sub-agent’s context window and lets it perform it’s task better. Even with recent models, clutter in the context can have outsized impacts on agent reliability.从子智能体的状态中移除交接消息，这样被分配的智能体就不必查看监督者的路由逻辑。这能减少子智能体上下文窗口的杂乱，让它更好地执行任务。即使是最近的模型，上下文中的杂乱内容也会对智能体的可靠性产生过大的影响。

Forwarding messages转发消息

Give the supervisor access to a forward_message tool. This tool lets it “forward” the sub agent’s response directly to the user without re-generating the full content. This reduced errors caused by the supervisor agent paraphrasing the sub agent incorrectly.为监督者提供一个 forward_message 工具。该工具允许它直接将子智能体的响应“转发”给用户，而无需重新生成完整内容。这减少了监督者智能体错误地改写子智能体内容所导致的错误。

Tool naming工具命名

Test different framings for the tool name that the supervisor agent would call to handoff to a sub agent (”delegate_to_<agent>” vs “transfer_to_<agent>”)测试监督者智能体调用以交接给子智能体的工具名称的不同表述（“delegate_to_<agent>” 与 “transfer_to_<agent>”）

Future work未来工作

There are several next steps we would like to explore.有几个下一步我们想探索。

Multi-hop across agents跨智能体多跳

Right now, all questions only require a single sub agent to respond. We would like to explore performance on questions that require multiple sub agents.目前，所有问题只需要单个子智能体来响应。我们想探索需要多个子智能体协同的问题上的性能表现。

Matching single agent performance匹配单智能体性能

Why don’t swarm and supervisor perform as well as single agent when there is a single distractor domain? Most of the main errors were due to “translation” mistakes and degraded performance with additional context (from the handoffs). We managed to reduce them somewhat, but performance still lags behind single agent. What can be done to increase performance to that level?为什么当只有一个干扰领域时，集群和监督者的表现不如单智能体？大部分主要错误是由于“转译”错误以及额外上下文（来自交接）导致的性能下降。我们设法在一定程度上减少了这些问题，但性能仍然落后于单智能体。可以采取什么措施将性能提升到那个水平？

Skipping the “translation” layer跳过“转译”层

Most of the mistakes of supervisor happen because of the “translation” layer, where only the supervisor agent is allowed to respond to the user. Is there some way to skip this translation layer more effectively, while still properly delegating work and ensuring responses are made with the full task context?监督者的大部分错误都发生在“转译”层，即只有监督者智能体才被允许向用户作出响应。有没有什么办法可以更有效地跳过这个转译层，同时仍然正确地分配工作，并确保响应是在完整的任务上下文中作出的？

Other architectures其他架构

Are the other architectures out there that may yield better results? How does this compare to “agents-as-tools”?是否存在其他可能产生更好结果的架构？这与“agents-as-tools”（智能体即工具）相比如何？

Conclusion结论

We think multi-agent systems will become more prevalent. While most successful multi-agent systems today have a relatively custom architecture, we think that as models improve, generic architectures will become sufficiently reliable for their benefits of ease of development outweigh their performance weaknesses. The supervisor architecture is the most generic one (in that it makes the fewest assumptions about the underlying agents), but a naive implementation of the supervisor architecture may have worse results. Using improvements in how information is passed between sub-agents and the user (and in how context is managed) can help the system perform better while retaining an ability to scale across many domains. You can use the ones we’ve made available in the langgraph-supervisor and evaluate your system on your data using a tool like LangSmith.我们认为多智能体系统将变得更加普遍。尽管当今大多数成功的多智能体系统都采用了相对定制的架构，但我们认为，随着模型的改进，通用架构将变得足够可靠，其易于开发的优势将超过其性能弱点。监督者架构是最通用的一种（因为它对底层智能体的假设最少），但监督者架构的朴素实现可能会产生更糟糕的结果。改进子智能体与用户之间的信息传递方式（以及上下文的管理方式）可以帮助系统在保持跨多个领域扩展能力的同时表现得更好。你可以使用我们在 langgraph-supervisor 中提供的功能，并使用 LangSmith 等工具在你的数据上评估你的系统。

If you want to try out the supervisor architecture easily (including all the improvements we made as a result of this research) you can easily do so with langgraph-supervisor.如果你想轻松尝试监督者架构（包括我们在这项研究中做出的所有改进），你可以通过 langgraph-supervisor 轻松实现。