Evaluating Deep Agents: Our Learnings评估深度智能体（Deep Agents）：我们的经验总结

Glossary术语表

Before diving in, we’ll define a few terms we use throughout this post.在深入探讨之前，我们先定义一下本文中用到的一些术语。

Ways to run an agent:运行智能体的方式：

• Single step: Constrain the core agent loop to run for only one turn, determining the next action the agent will take.单步（Single step）：限制核心智能体循环仅运行一轮，确定智能体将采取的下一个动作。
• Full turn: Run the agent in its entirety on a single input, which can consist of multiple tool-calling iterations.完整轮次（Full turn）：在单个输入上完整运行智能体，这可能包含多次工具调用迭代。
• Multiple turns: Run the agent multiple times in its entirety. Often used to simulate a “multi-turn” conversation between an agent and a user with several back-and-forth interactions.多轮（Multiple turns）：多次完整运行智能体。通常用于模拟智能体与用户之间包含多次往返互动的“多轮”对话。

Things we can test:我们可以测试的内容：

• Trajectory: The sequence of tools that are called by the agent, and the specific tool arguments the agent generates.轨迹（Trajectory）：智能体调用的工具序列，以及智能体生成的特定工具参数。
• Final response: The final returned response from the agent to the user.最终响应（Final response）：智能体返回给用户的最终响应。
• Other state: Other values that the agent generated while running (e.g. files, other artifacts)其他状态（Other state）：智能体在运行过程中生成的其他值（例如文件、其他工件）。

#1: Deep Agents require more bespoke test logic (code) for each datapoint#1：深度智能体需要针对每个数据点提供更定制化的测试逻辑（代码）

Traditional LLM evaluation is straightforward:传统的 LLM 评估非常简单直接：

1) Build a dataset of examples1) 构建一个示例数据集

2) Write an evaluator2) 编写一个评估器

3) Run your application over the dataset to produce outputs, and score those outputs with your evaluator3) 在数据集上运行应用程序以产生输出，并使用评估器对这些输出进行评分

Every data point is treated identically — run through the same application logic, scored by the same evaluator.每个数据点都被同等对待——通过相同的应用程序逻辑运行，并由相同的评估器评分。

Deep Agents breaks this assumption. You’ll want to test more than just the final message. The “success criteria” may be also more specific to each datapoint, and may involve specific assertions against the agent’s trajectory and state.深度智能体打破了这一假设。你不仅需要测试最终消息，还需要测试更多内容。“成功标准”可能对每个数据点更为具体，并且可能涉及针对智能体轨迹和状态的特定断言。

Consider this example:考虑这个例子：

We have a calendar scheduling deep agent that has the ability to remember user preferences. A user asks their agent to "remember to never schedule meetings before 9am". We want to assert that the calendar scheduling agent updates its own memories in its filesystem to remember this information.我们有一个能够记住用户偏好的日历调度深度智能体。用户要求智能体“记住永远不要在上午 9 点之前安排会议”。我们想要断言日历调度智能体在文件系统中更新了自己的记忆以记住此信息。

In order to test this, we might want to write assertions to verify that:为了测试这一点，我们可能需要编写断言来验证：

1) The agent called edit_file on the memories.md file path1) 智能体在 memories.md 文件路径上调用了 edit_file

2) The agent communicated the memory update to the user in its final message2) 智能体在最终消息中向用户传达了记忆更新

3) The memories.md file actually contains information about not scheduling early meetings. You could:3) memories.md 文件确实包含了关于不安排早间会议的信息。你可以：

• Use regex to look for a mention of “9am”使用正则表达式查找对“9am”的提及
• Or use an LLM-as-judge with specific success criteria for a more holistic analysis of the file update或者使用 LLM-as-judge（以 LLM 作为评判者）并设定特定的成功标准，对文件更新进行更全面的分析

LangSmith’s Pytest and Vitest integrations support this type of bespoke testing. You can make different assertions about the agent’s trajectory, final message, and state for each test case.LangSmith 的 Pytest 和 Vitest 集成支持这种定制化测试。你可以针对每个测试用例对智能体的轨迹、最终消息和状态做出不同的断言。

# Mark as a LangSmith test case
@pytest.mark.langsmith
def test_remember_no_early_meetings() -> None:
    user_input = "I don't want any meetings scheduled before 9 AM ET"
    # We can log the input to the agent to LangSmith
    t.log_inputs({"question": user_input})
 
    response = run_agent(user_input)
    # We can log the output of the agent to LangSmith
    t.log_outputs({"outputs": response})
 
    agent_tool_calls = get_agent_tool_calls(response)
 
    # We assert that the agent called the edit_file tool to update its memories
    assert any([tc["name"] == "edit_file" and tc["args"]["path"] == "memories.md" for tc in agent_tool_calls])
 
		# We log feedback from an llm-as-judge that the final message confirmed the memory update
		communicated_to_user = llm_as_judge_A(response)
    t.log_feedback(key="communicated_to_user", score=communicated_to_user)
 
    # We log feedback from an llm-as-judge that the memories file now contains the right info
    memory_updated = llm_as_judge_B(response)
    t.log_feedback(key="memory_updated", score=memory_updated)

For a general code snippet of how to use Pytest, check out these docs:有关如何使用 Pytest 的通用代码片段，请查看这些文档：

This LangSmith integration automatically logs all test cases to an experiment, so you can view traces for a failed test case (to debug what went wrong) and track results over time.此 LangSmith 集成会自动将所有测试用例记录到实验中，因此你可以查看失败测试用例的追踪记录（以调试出错原因）并随时间跟踪结果。

#2: Single step evals are valuable and efficient#2：单步评估既有价值又高效

When running our evals for Deep Agents, about half of our test cases looked like single step evals, i.e. what did the LLM decide to do immediately after a specific series of input messages?在运行深度智能体的评估时，我们大约一半的测试用例看起来像是单步评估，即 LLM 在特定的一系列输入消息之后立即决定做什么？

This is especially useful for validating that the agent called the correct tool with the correct arguments in a specific scenario. Common test cases include:这对于验证智能体在特定场景下是否调用了正确的工具并使用了正确的参数特别有用。常见的测试用例包括：

• Did it call the right tool to search for meeting times?它是否调用了正确的工具来搜索会议时间？
• Did it inspect the right directory contents?它是否检查了正确的目录内容？
• Did it update its memories?它是否更新了记忆？

Regressions often occur at individual decision points rather than across full execution sequences. If using LangGraph, its streaming capabilities allow you to interrupt the agent after a single tool call to inspect the output — so you can catch issues early without the overhead of a complete agent sequence.回归问题通常发生在单个决策点，而不是整个执行序列中。如果使用 LangGraph，其流式处理能力允许你在单次工具调用后中断智能体以检查输出——这样你就可以在不产生完整智能体序列开销的情况下尽早发现问题。

In the code snippet below, we manually introduce a break point before the tools node, allowing us to easily run the agent for a single step. We can then inspect and make assertions about the state after that single step.在下面的代码片段中，我们在工具节点之前手动引入了一个断点，使我们能够轻松地运行单步智能体。然后，我们可以检查并在该单步之后对状态做出断言。

@pytest.mark.langsmith
def test_single_step() -> None:
	state_before_tool_execution = await agent.ainvoke(
	    inputs,
	    # interrupt_before specifies nodes to stop before
	    # interrupting before the tool node allows us to inspect the tool call args
	    interrupt_before=["tools"]
	)
	# We can see the message history of the agent, including the latest tool call
	print(state_before_tool_execution["messages"])

#3: Full agent turns give you a complete picture#3：完整的智能体轮次为你提供完整的图景

Think of single-step evals as your “unit tests” that ensure the agent takes the expected action in a specific scenario. Meanwhile, full agent turns are also valuable — they show you a complete picture of the end-to-end actions that your agent takes.将单步评估视为你的“单元测试”，确保智能体在特定场景下采取预期的行动。同时，完整的智能体轮次也很有价值——它们向你展示了智能体所采取的端到端行动的完整图景。

Full agent turns let you test agent behavior in multiple ways:完整的智能体轮次让你能以多种方式测试智能体行为：

1) Trajectory: A very common way to evaluate a full trajectory is to ensure that a particular tool was called at some point during action, but it doesn’t matter exactly when. In our calendar scheduler example, the scheduler might need multiple tool calls to find a suitable time slot that works for all parties.1) 轨迹：评估完整轨迹的一种非常常见的方法是确保在行动过程中的某个时刻调用了特定工具，但具体何时调用并不重要。在我们的日历调度示例中，调度程序可能需要多次工具调用才能找到适合所有相关方的合适时间段。

2) Final Response: In some cases, the quality of the final output matters more than the specific path taken by the agent. We found this to be true for more open-ended tasks like coding and research.2) 最终响应：在某些情况下，最终输出的质量比智能体采取的具体路径更重要。我们发现对于编程和研究等更开放的任务，情况确实如此。

3) Other State: Evaluating other state is very similar to evaluating an agent’s final response. Some agents will create artifacts instead of responding to the user in a chat format. Examining and testing these artifacts is easy by examining an agent’s state in LangGraph.3) 其他状态：评估其他状态与评估智能体的最终响应非常相似。一些智能体会创建工件，而不是以聊天格式响应用户。通过检查 LangGraph 中的智能体状态，可以轻松检查和测试这些工件。

1. For a coding agent → read and then test the files that the agent wrote.对于编程智能体 → 读取并测试智能体编写的文件。
2. For a research agent → assert the agent found the right links or sources.对于研究智能体 → 断言智能体找到了正确的链接或来源。

Full agent turns give you a complete picture of your agent execution. LangSmith makes it really easy to view your full agent turns as traces, where you can see high level metrics like latency and token use, while also analyzing specific steps down to each model call or tool invocation.完整的智能体轮次为你提供智能体执行的完整图景。LangSmith 使你可以非常轻松地将完整的智能体轮次视为追踪记录，在那里你可以查看延迟和 Token 使用量等高级指标，同时还可以分析到每个模型调用或工具调用的具体步骤。

#4: Running an agent across multiple turns simulates full user interactions#4：跨多轮运行智能体可模拟完整的用户交互

Some scenarios require testing agents across multi-turn conversations that have multiple sequential user inputs. The challenge is that if you naively hardcode a sequence of inputs and the agent deviates from the expected path, the subsequent hardcoded user input may not make sense.有些场景需要测试跨越包含多个连续用户输入的多轮对话的智能体。挑战在于，如果你天真地硬编码了一系列输入，而智能体偏离了预期路径，那么后续硬编码的用户输入可能就说不通了。

We addressed this by adding conditional logic in our Pytest and Vitest tests. For example, we would:我们通过在 Pytest 和 Vitest 测试中添加条件逻辑解决了这个问题。例如，我们会：

• Run the first turn, and then check the agent output.
  • If the output was expected, run the next turn.如果输出符合预期，则运行下一轮。
  • If it was not expected, fail the test early. (This was possible because we had the flexibility to add checks after each step.)如果输出不符合预期，则提前终止测试。（这是可能的，因为我们有灵活性在每一步之后添加检查。）

This approach let us run multi-turn evals without having to model every possible agent branch. If we wanted to test the second or third turn in isolation, we simply set up a test starting from that point with appropriate initial state.这种方法让我们能够运行多轮评估，而无需对所有可能的智能体分支进行建模。如果我们想单独测试第二轮或第三轮，我们只需从该点开始设置一个具有适当初始状态的测试即可。

#5: Setting up the right eval environment is important#5：设置正确的评估环境很重要

Deep Agents are stateful and designed to tackle complex, long-running tasks — often requiring more complex environments to evaluate in.深度智能体是有状态的，旨在处理复杂、长时间运行的任务——通常需要更复杂的环境来进行评估。

Unlike simpler LLM evals where the environment is limited to a few usually stateless tools, Deep Agents need a fresh, clean environment for each eval run in order to ensure reproducible results.与环境仅限于少数通常无状态工具的简单 LLM 评估不同，深度智能体需要为每次评估运行提供一个新鲜、干净的环境，以确保结果的可复现性。

Coding agents illustrate this clearly. Harbor provides an evaluation environment for TerminalBench that runs inside a dedicated Docker container or sandbox. For DeepAgents CLI, we use a more lightweight approach: we create a temporary directory and run the agent inside it for each test case.编程智能体清楚地说明了这一点。Harbor 为 TerminalBench 提供了一个在专用 Docker 容器或沙箱中运行的评估环境。对于 DeepAgents CLI，我们使用了一种更轻量级的方法：我们创建一个临时目录，并在每次测试用例中在该目录内运行智能体。

The broader point: Deep Agent evals require environments that resets per test -- otherwise your evals become flaky and difficult to reproduce.更广泛的观点是：深度智能体评估需要能够按测试重置的环境——否则你的评估会变得不稳定且难以复现。

Tip: Mock out your API requests提示：模拟你的 API 请求

LangSmith Assist requires connecting to real LangSmith APIs. Running evals against live services can be slow and expensive. Instead, record HTTP requests into a filesystem and replay them during test execution. For Python, vcr works well; for JS, we proxy fetch requests through a Hono app works.LangSmith Assist 需要连接到真实的 LangSmith API。针对实时服务运行评估可能会很慢且昂贵。相反，将 HTTP 请求记录到文件系统中，并在测试执行期间重放它们。对于 Python，vcr 效果很好；对于 JS，我们通过 Hono 应用代理 fetch 请求。

Mocking or replaying API requests makes Deep Agent evals faster and easier to debug, especially when the agent depends heavily on external system state.模拟或重放 API 请求使深度智能体评估更快且更容易调试，特别是当智能体严重依赖外部系统状态时。

Evaluate Deep Agents with LangSmith使用 LangSmith 评估深度智能体

The above techniques are common patterns we saw when writing our own test suites for deep agents powered applications. You likely only need a subset of the above patterns for your specific application — and as such, it’s important for your evaluation framework to be flexible. If you’re building a deep agent and getting started with evals, check out LangSmith’s testing integrations!以上技术是我们为深度智能体驱动的应用程序编写自己的测试套件时看到的常见模式。你可能只需要针对你的特定应用程序使用上述模式的一个子集——因此，你的评估框架保持灵活性非常重要。如果你正在构建深度智能体并开始进行评估，请查看 LangSmith 的测试集成！