Key Takeaways关键要点
You can't build reliable agents without understanding how they reason, and you can't validate improvements without systematic evaluation. This article explains the primitives for agent observability, how to evaluate agents at different granularities, and how production traces become the foundation for continuous improvement.不了解智能体的推理过程，就无法构建可靠的智能体；没有系统化的评估，就无法验证改进效果。本文解释了智能体可观测性的基本要素、如何在不同粒度上评估智能体，以及生产追踪如何成为持续改进的基础。
TL;DR摘要
- You don't know what your agents will do until you actually run them — which means agent observability is different and more important than software observability在真正运行智能体之前，你无法预知其行为——这意味着智能体可观测性与软件可观测性不同，且更为重要
- Agents often do complex, open-ended tasks, which means evaluating them is different than evaluating software智能体通常执行复杂、开放式的任务，因此评估它们与评估软件有所不同
- Because traces document where agent behavior emerges, they power evaluation in a multitude of ways由于追踪记录了智能体行为涌现的过程，它们以多种方式为评估提供支持
When something goes wrong in traditional software, you know what to do: check the error logs, look at the stack trace, find the line of code that failed. But AI agents have changed what we're debugging. When an agent takes 200 steps over two minutes to complete a task and makes a mistake somewhere along the way, that’s a different type of error. There’s no stack trace - because there’s no code that failed. What failed was the agent’s reasoning.在传统软件中，当出现问题时，你知道该怎么做：检查错误日志、查看堆栈跟踪、找到出错的代码行。但AI智能体改变了我们调试的对象。当一个智能体花费两分钟、执行200步来完成一项任务，并在过程中某处出错时，这是一种不同类型的错误。没有堆栈跟踪——因为没有代码出错。出错的是智能体的推理。
From debugging code to debugging reasoning从调试代码到调试推理
You still write code to define your agent, e.g. which tools exist, what data is available. You write prompts and tool descriptions to guide the agent's behavior, but you won't know how the LLM will interpret these instructions until you run it. The source of truth thus shifts from code to traces that show what the agent actually did.你仍然需要编写代码来定义智能体，例如有哪些工具、哪些数据可用。你编写提示词和工具描述来引导智能体的行为，但在运行之前，你无法知道LLM将如何解释这些指令。因此，真相来源从代码转移到显示智能体实际行为的追踪记录。
Agent engineering is an iterative process, and tracing + evaluation are how you close the loop. In this post, we'll explore why agent observability and evaluation are fundamentally different from traditional software, what new primitives and practices you need, and how observability powers evaluation in ways that make them inseparable.智能体工程是一个迭代过程，而追踪+评估是闭环的关键。在这篇文章中，我们将探讨为什么智能体可观测性和评估与传统软件根本不同，你需要哪些新的基本要素和实践，以及可观测性如何以不可分割的方式为评估提供支持。
Agent observability ≠ software observability智能体可观测性 ≠ 软件可观测性
Pre-LLMs, software was largely deterministic — given the same input, you'd get the same output. Logic was codified. You could read the code and know exactly how the system behaved. When something went wrong, logs pointed you to which service or function failed, then you'd go back to the code to understand why it happened and fix it.在LLM出现之前，软件在很大程度上是确定性的——给定相同的输入，你会得到相同的输出。逻辑被编码化。你可以阅读代码并确切知道系统如何运行。当出现问题时，日志会指向哪个服务或函数失败，然后你回到代码中理解原因并修复。
AI agents break the assumptions of determinism and code as a source of truth. As we're moving from traditional software to LLM applications to agents, each step introduces more uncertainty. LLM apps make a single call to an LLM with context, introducing natural language's inherent "fuzziness" but remain constrained to one LLM call.AI智能体打破了确定性和代码作为真相来源的假设。随着我们从传统软件转向LLM应用再到智能体，每一步都引入了更多的不确定性。LLM应用通过上下文向LLM发出单次调用，引入了自然语言固有的“模糊性”，但仍局限于一次LLM调用。
However, agents call LLMs and tools in a loop until they determine a task is done — and can reason across dozens or hundreds of steps, calling tools, maintaining state, and adapting behavior based on context. When building an agent, you attempt to recommend the application logic in code and prompts. But you don't know what this logic will do until actually running the LLM.然而，智能体在循环中调用LLM和工具，直到确定任务完成——并且可以在数十或数百个步骤中进行推理，调用工具，维护状态，并根据上下文调整行为。在构建智能体时，你试图在代码和提示词中推荐应用逻辑。但直到实际运行LLM，你才知道这个逻辑会做什么。
When something goes wrong, you're not finding a single line of code that failed. Instead, you're asking:当出现问题时，你不是在寻找出错的单行代码。相反，你在问：
- Why did the agent decide to call
edit_fileinstead ofread_fileat step 23 of 200?为什么智能体在第200步的第23步决定调用edit_file而不是read_file？ - What context and prompt instructions informed that decision?哪些上下文和提示指令影响了那个决定？
- Where in this two-minute, 200-step trajectory did the agent go off track?在这两分钟、200步的轨迹中，智能体在哪里偏离了轨道？
Traditional tracing tools can't answer these questions. A 200-step trace is too large for a human to parse, and traditional traces don't capture the reasoning context behind each decision; they only capture which services were called and how long each took.传统的追踪工具无法回答这些问题。200步的追踪对于人类来说太大而无法解析，并且传统追踪不捕获每个决策背后的推理上下文；它们只捕获调用了哪些服务以及每个服务花费了多长时间。
Agent evaluation ≠ software evaluation智能体评估 ≠ 软件评估
Traditional software testing relies on deterministic assertions: write tests that check output == expected_output, verify they pass, then ship. Online evaluation (A/B tests, product analytics) measures business impact separately. Evaluating agents differ from evaluating software in a few key ways:传统软件测试依赖于确定性断言：编写测试检查输出 == 预期输出，验证通过，然后发布。在线评估（A/B测试、产品分析）单独衡量业务影响。评估智能体与评估软件在几个关键方面有所不同：
1. You're testing reasoning, not code paths1. 你测试的是推理，而不是代码路径
Traditional software has tests at different levels of granularity (unit, integration, e2e), testing deterministic code paths you can read and modify. Agents also need testing at different levels, but you're no longer testing code paths — you're testing reasoning:传统软件在不同粒度（单元、集成、端到端）上有测试，测试你可以阅读和修改的确定性代码路径。智能体也需要在不同级别进行测试，但你不再测试代码路径——你测试的是推理：
- Single-step: Did the agent make the right decision at this moment?单步：智能体在这一刻是否做出了正确的决定？
- Full-turn: Did the agent perform well in an end-to-end execution?全轮次：智能体在端到端执行中表现良好吗？
- Multi-turn: Did the agent maintain context across a conversation?多轮次：智能体在对话中是否保持了上下文？
2. Production becomes your primary teacher2. 生产环境成为你的主要老师
In traditional software, you can catch most correctness issues with offline tests (unit tests, integration tests, staging). You still test in production through canary deployments and feature flags, but the goal is to catch edge cases and integration issues you missed.在传统软件中，你可以通过离线测试（单元测试、集成测试、预发布环境）捕获大多数正确性问题。你仍然通过金丝雀部署和功能标志在生产环境中进行测试，但目标是捕获你遗漏的边缘情况和集成问题。
With agents, production plays a different role. Because every natural language input is unique, you can't anticipate how users will phrase requests or what edge cases exist. Production traces reveal failure modes you couldn't have predicted and help you understand what "correct behavior" actually looks like for real user interactions.对于智能体，生产环境扮演着不同的角色。因为每个自然语言输入都是独特的，你无法预测用户会如何表述请求或存在哪些边缘情况。生产追踪揭示了你无法预见的失败模式，并帮助你理解真实用户交互中“正确行为”的样子。
This shifts how you think about evaluation: production isn't just where you catch missed bugs. It's where you discover what to test for offline. Production traces become test cases, and your evaluation suite grows continuously from real-world examples, not just engineered scenarios.这改变了你对评估的看法：生产环境不仅仅是捕获遗漏错误的地方。它是你发现离线测试内容的地方。生产追踪成为测试用例，你的评估套件从真实世界的例子中持续增长，而不仅仅是工程化的场景。
The primitives of agent observability智能体可观测性的基本要素
Agent observability uses three core primitives to capture non-deterministic reasoning:智能体可观测性使用三个核心基本要素来捕获非确定性推理：
- Runs: A single execution step (one LLM call with its input/output)运行：单个执行步骤（一次LLM调用及其输入/输出）
- Traces: A complete agent execution showing all runs and their relationships追踪：完整的智能体执行，显示所有运行及其关系
- Threads: Multi-turn conversations grouping multiple traces over time线程：多轮对话，随时间分组多个追踪
These use the same concepts as traditional observability (e.g. traces, spans) but capture reasoning context rather than service calls and timing这些使用了与传统可观测性相同的概念（例如追踪、跨度），但捕获的是推理上下文，而不是服务调用和计时
Runs: capturing what the LLM did at a single step运行：捕获LLM在单步中的行为
A run captures a single execution step. This is most useful for capturing how the LLM behaved at a particular point in time. This captures the complete prompt for an LLM call, including all instructions, tools used, and context.运行捕获单个执行步骤。这对于捕获LLM在特定时间点的行为最为有用。它捕获了LLM调用的完整提示词，包括所有指令、使用的工具和上下文。
These runs serve dual purposes:这些运行具有双重目的：
- For debugging: See exactly what the agent was thinking at any step. What was in the prompt? What tools were available? Why did it choose this action?用于调试：在任何步骤中准确查看智能体的想法。提示词中有什么？哪些工具可用？为什么选择这个动作？
- For evaluation: Write assertions against this run. Did the agent call the right tool? With the right arguments?用于评估：针对此运行编写断言。智能体是否调用了正确的工具？使用了正确的参数？
Traces: capturing trajectories追踪：捕获轨迹
A trace captures a complete agent execution by linking together all the runs that occurred. A reasoning trace captures:追踪通过链接所有发生的运行来捕获完整的智能体执行。推理追踪捕获：
- All information about what goes into the model at each step, captured as runs that make up the trace每一步输入模型的所有信息，作为构成追踪的运行捕获
- All tool calls with their arguments and results所有工具调用及其参数和结果
- The nested structure showing how steps relate to each other显示步骤之间关系的嵌套结构
Agent traces are massive. While a typical distributed trace might be a few hundred bytes, agent traces can be orders of magnitude larger. For complex, long-running agents, traces can reach hundreds of megabytes. This context is necessary for debugging and evaluating the agent's reasoning.智能体追踪非常庞大。虽然典型的分布式追踪可能只有几百字节，但智能体追踪可能大几个数量级。对于复杂、长时间运行的智能体，追踪可能达到数百兆字节。这些上下文对于调试和评估智能体的推理是必要的。
Threads: multi-turn conversation context线程：多轮对话上下文
A single trace captures one agent execution, but agents often operate across sessions involving multiple interactions with a user or system. A thread groups multiple agent executions (traces) into a single conversational session, preserving:单个追踪捕获一次智能体执行，但智能体通常跨会话操作，涉及与用户或系统的多次交互。线程将多个智能体执行（追踪）分组到单个对话会话中，保留：
- Multi-turn context: All interactions between user and agent in chronological order多轮上下文：用户与智能体之间的所有交互，按时间顺序排列
- State evolution: How the agent's memory, files, or other artifacts changed across turns状态演变：智能体的记忆、文件或其他工件在轮次之间的变化
- Time span: Conversations can last minutes, hours, or days时间跨度：对话可以持续几分钟、几小时或几天
Consider debugging a coding agent that worked fine for 10 turns but suddenly started making mistakes in turn 11. The turn 11 trace in isolation might show the agent calling a reasonable tool. But when you examine the full thread, you discover that in turn 6 the agent updated its memory with an incorrect assumption, and by turn 11 that bad context had compounded into buggy behavior.考虑调试一个编码智能体，它在10轮中运行良好，但在第11轮突然开始出错。单独看第11轮的追踪可能显示智能体调用了合理的工具。但当你检查整个线程时，你发现在第6轮智能体用错误的假设更新了记忆，到第11轮时，那个错误的上下文已经累积成了有问题的行为。
Threads are essential for understanding how agent behavior evolves over time and how context accumulates (or degrades) across interactions.线程对于理解智能体行为如何随时间演变以及上下文如何在交互中累积（或退化）至关重要。
How this influences agent evaluation这对智能体评估的影响
Agent behavior only emerges at runtime, and is only captured by observability (runs, traces, and threads). This means that to evaluate behavior you need to evaluate your observability data. This raises two key questions:智能体行为只在运行时涌现，并且只能通过可观测性（运行、追踪和线程）捕获。这意味着要评估行为，你需要评估你的可观测性数据。这引出了两个关键问题：
- At what granularity do you evaluate agents? At the run, trace, or thread level?你在什么粒度上评估智能体？在运行、追踪还是线程级别？
- When do you evaluate agents? If behavior only emerges when you run the agents, can you evaluate them offline in the same way you do software?你何时评估智能体？如果行为只在运行智能体时涌现，你能像评估软件一样离线评估它们吗？
Evaluations agents at different levels of granularity在不同粒度上评估智能体
You can evaluate agents at different levels of granularity, which map 1:1 to the observability primitives. What you're evaluating determines which primitive you need:你可以在不同粒度上评估智能体，这与可观测性基本要素一一对应。你评估的内容决定了你需要哪个基本要素：
- Single-step evaluation validates individual runs → Did the agent make the right decision at a specific step?单步评估验证单个运行 → 智能体在特定步骤是否做出了正确的决定？
- Full-turn evaluation validates complete traces → Did the agent execute the full task correctly?全轮次评估验证完整追踪 → 智能体是否正确执行了完整任务？
- Multi-turn evaluation validates threads → Did the agent maintain context across a conversation?多轮次评估验证线程 → 智能体在对话中是否保持了上下文？
- Single-step evaluation: unit tests for decisions单步评估：决策的单元测试
Sometimes, you need to validate a specific decision point without running the entire agent. You may want to see if the agent choose the right tool in a specific scenario, or whether it used the correct arguments.有时，你需要验证特定的决策点，而无需运行整个智能体。你可能想看看智能体在特定场景中是否选择了正确的工具，或者是否使用了正确的参数。
This is like a unit test for agent reasoning: set up a specific state (conversation history, available tools, current task), run the agent for one step, and assert that it made the right decision. Single-step evaluation validates runs, i.e. individual LLM calls.这就像智能体推理的单元测试：设置特定状态（对话历史、可用工具、当前任务），运行智能体一步，并断言它做出了正确的决定。单步评估验证运行，即单个LLM调用。
Example: Testing a calendar agent's tool selection示例：测试日历智能体的工具选择
A scheduling agent needs to find available meeting times before scheduling. You want to verify it checks availability first rather than immediately trying to create the meeting:一个日程安排智能体需要在安排会议之前找到可用的会议时间。你想验证它首先检查可用性，而不是立即尝试创建会议：
Your single-step test:你的单步测试：
- Setup state: Conversation history = user said "Schedule a meeting with Harrison tomorrow morning", available tools = [
find_meeting_times,schedule_meeting,send_email]设置状态：对话历史 = 用户说“明天早上与Harrison安排会议”，可用工具 = [find_meeting_times, schedule_meeting, send_email] - Run one step: Agent generates next action运行一步：智能体生成下一个动作
- Assert: Agent chose
find_meeting_times(notschedule_meeting)断言：智能体选择了find_meeting_times（而不是schedule_meeting）
Why you need runs: Single step tests often come from real production cases that error. In order to recreate these, you need the exact state of the agent before that step. Detailed run captures are the only way to get this!为什么需要运行：单步测试通常来自出错的真实生产案例。为了重现这些，你需要智能体在该步骤之前的确切状态。详细的运行捕获是获得这一点的唯一方法！
Single-step evaluations are efficient and catch regressions at individual decision points. In practice, about half of agent test suites use these single-step tests to isolate and validate specific reasoning behaviors without the overhead of full agent execution.单步评估效率高，并在单个决策点捕获回归。在实践中，大约一半的智能体测试套件使用这些单步测试来隔离和验证特定的推理行为，而无需完整智能体执行的开销。
2. Full-turn evaluation: end-to-end trajectory assessment2. 全轮次评估：端到端轨迹评估
Other times, you need to see a complete agent execution. Full-turn evaluation validates traces, i.e. complete agent executions with all their runs, and let you test multiple dimensions:其他时候，你需要查看完整的智能体执行。全轮次评估验证追踪，即包含所有运行的完整智能体执行，并让你测试多个维度：
Trajectory: Did the agent call the necessary tools? For a coding agent fixing a bug, you might assert: "The agent should have called read_file, then edit_file, then run_tests." The exact sequence might vary, but certain tools must be called.轨迹：智能体是否调用了必要的工具？对于修复错误的编码智能体，你可能断言：“智能体应该调用read_file，然后edit_file，然后run_tests。”确切的顺序可能不同，但必须调用某些工具。
Final response: Was the output correct and helpful? For open-ended tasks like research or coding, the quality of the final answer often matters more than the specific path taken.最终响应：输出是否正确且有用？对于研究或编码等开放式任务，最终答案的质量通常比具体路径更重要。
State changes: Did the agent create the right artifacts? For a coding agent, you'd inspect the files it wrote and verify they contain the correct code. For an agent with memory, you'd check that it stored the right information.状态变化：智能体是否创建了正确的工件？对于编码智能体，你会检查它写入的文件并验证它们包含正确的代码。对于有记忆的智能体，你会检查它存储了正确的信息。
Testing that an agent remembers user preferences also requires validating three things:测试智能体是否记住用户偏好也需要验证三件事：
- Trajectory: Did the agent call
edit_fileon its memory file?轨迹：智能体是否在其记忆文件上调用了edit_file？ - Final response: Did the agent confirm the update to the user?最终响应：智能体是否向用户确认了更新？
- State: Does the memory file actually contain the preference?状态：记忆文件是否实际包含该偏好？
Each assertion requires different parts of the trace. You can't evaluate these dimensions without capturing the full trajectory and state changes.每个断言需要追踪的不同部分。如果不捕获完整的轨迹和状态变化，你就无法评估这些维度。
3. Multi-turn evaluation: realistic conversation flows3. 多轮次评估：真实的对话流程
Some agent behaviors only emerge over multiple turns. The agent might maintain context correctly for 5 turns but fail on turn 6, or handle individual requests fine but struggle when requests build on each other.某些智能体行为只在多轮次中涌现。智能体可能在前5轮正确维护上下文，但在第6轮失败，或者单独处理请求时表现良好，但在请求相互依赖时遇到困难。
Multi-turn evaluation validates threads, i.e. conversational sessions with multiple agent executions. You test whether the agent accumulates context correctly, maintains state across turns, and handles conversational flows that build on previous exchanges.多轮次评估验证线程，即具有多个智能体执行的对话会话。你测试智能体是否正确累积上下文、跨轮次维护状态，以及处理建立在先前交换基础上的对话流程。
For example, testing context persistence:例如，测试上下文持久性：
- Turn 1: User shares a preference ("I prefer Python over JavaScript")第1轮：用户分享偏好（“我更喜欢Python而不是JavaScript”）
- Turn 2: User asks a question building on that ("Show me an example")第2轮：用户基于此提问（“给我看一个例子”）
- Turn 3: Test that preference persists ("Write a script for this")第3轮：测试偏好是否持续（“为此编写一个脚本”）
The agent should provide Python examples in turns 2 and 3, not JavaScript. This requires maintaining context across turns.智能体应在第2轮和第3轮提供Python示例，而不是JavaScript。这需要跨轮次维护上下文。
The challenge is keeping multi-turn tests on rails. If the agent deviates from the expected path in turn 1, your hardcoded turn 2 input might not make sense. Use conditional logic to check the agent's output after each turn, and fail early if it goes off track.挑战在于让多轮次测试保持在轨道上。如果智能体在第1轮偏离预期路径，你硬编码的第2轮输入可能没有意义。使用条件逻辑在每轮后检查智能体的输出，如果偏离轨道则提前失败。
Multi-turn evaluation requires threads to group multiple agent executions (traces) into a single conversation. When a multi-turn test fails, you need the thread showing all turns to understand what went wrong and where.多轮次评估需要线程将多个智能体执行（追踪）分组到单个对话中。当多轮次测试失败时，你需要显示所有轮次的线程来理解哪里出了问题以及在哪里。
How to choose what granularity to evaluate your agent at如何选择评估智能体的粒度
There is no single right way to choose which granularity to evaluate agents at. Some heuristics we’ve seen:没有单一的正确方法来选择评估智能体的粒度。我们观察到的一些启发式方法：
- It’s often easiest to come up with inputs for trace-level evals (full-turn). These are inputs to your agent, so it’s pretty easy (and necessary) to come up with expected inputs. That being said, it can be harder to come up with expected outputs and/or a way to validate those programmatically. This means there may be some period time of where you automate the running of the agent over these datapoints, but not the scoring.通常最容易为追踪级评估（全轮次）提出输入。这些是智能体的输入，所以提出预期输入相当容易（且必要）。话虽如此，提出预期输出和/或编程验证的方法可能更难。这意味着可能有一段时间你自动化了智能体在这些数据点上的运行，但没有自动化评分。
- It’s easiest to fully automate the scoring of run-level evals (single-step). This is just a single model call, and often times can be evaluated by checking with tools were called. One word of caution: depending on how frequently you are changing the internals of your agent (which tools are available, the right sequence to call tools) these evaluations may get out of date quickly and require updating. For this reason, we generally see teams building these only after the general agent architecture is relatively stable.最容易完全自动化运行级评估（单步）的评分。这只是一个模型调用，通常可以通过检查调用了哪些工具来评估。一个警告：根据你更改智能体内部结构（哪些工具可用、调用工具的正确顺序）的频率，这些评估可能很快过时并需要更新。因此，我们通常看到团队只在通用智能体架构相对稳定后才构建这些评估。
- Thread-level evals are hard to implement effectively (multi-turn). They involve coming up with a sequence of inputs, but often times that sequence only makes sense if the agent behaves a certain way between inputs. They are also hard to evaluate automatically. This is least common type of evaluation that we see.线程级评估难以有效实施（多轮次）。它们涉及提出一系列输入，但通常该序列只有在智能体在输入之间以某种方式表现时才合理。它们也难以自动评估。这是我们看到的最不常见的评估类型。
Most production agents use a combination: full-turn tests for core workflows, single-step tests for known failure modes discovered in production, and multi-turn tests for stateful interactions.大多数生产智能体使用组合：全轮次测试用于核心工作流，单步测试用于生产中发现的已知失败模式，多轮次测试用于有状态交互。
When to evaluate an agent何时评估智能体
Agent behavior doesn’t fully emerge until you run it in production, which means that when you evaluate agents also differs from traditional software.智能体行为直到你在生产中运行它才完全涌现，这意味着评估智能体的时间也与传统软件不同。
- Offline evaluation: This is the equivalent of running unit tests before shipping. To run these tests, you’ll want to collect a dataset of inputs and, optionally, ground truth outputs to compare against. Depending on the cost to run and evaluate the agent over this dataset, you may run these evals on every commit, or just before you push to prod. If you are running offline evals frequently, you will want to set up some sort of caching so that you’re not calling the model unnecessarily. Note: when most people talk about “evaluation”, offline evals are the primary type of evaluation they are likely referring to.离线评估：这相当于在发布前运行单元测试。要运行这些测试，你需要收集输入数据集，并可选择收集真实输出进行比较。根据在该数据集上运行和评估智能体的成本，你可能在每次提交时运行这些评估，或者只在推送到生产环境之前运行。如果你频繁运行离线评估，你会希望设置某种缓存，以免不必要地调用模型。注意：当大多数人谈论“评估”时，离线评估是他们可能主要指的评估类型。
- Online evaluation: Since you don’t know how the agent will perform until you run it, you may want to run evaluations “online”, as the agent runs on production data. When doing this, these evaluators definitionally need to be “reference free”. Online evaluators typically run on ingestion of production data.在线评估：由于你不知道智能体将如何表现直到运行它，你可能希望在智能体在生产数据上运行时进行“在线”评估。这样做时，这些评估器在定义上必须是“无参考”的。在线评估器通常在摄入生产数据时运行。
- Ad-hoc evaluation: Agents are very unbounded in their inputs and behavior, so you don’t always know ahead of time what you want to test for. If you have a lot of traces in production, you may want to test them after they have already been ingested. This exploratory data analysis can be crucial for understanding your agents. Systems like Insights Agent in LangSmith can help you do this.临时评估：智能体在其输入和行为上非常无界，因此你并不总是提前知道要测试什么。如果你在生产中有大量追踪，你可能希望在它们已被摄入后对其进行测试。这种探索性数据分析对于理解你的智能体至关重要。像LangSmith中的Insights Agent这样的系统可以帮助你做到这一点。
The key shift: offline evaluation is necessary but not sufficient. Evaluating your agents in production is important because you can't anticipate all the ways users will interact with your agent.关键转变：离线评估是必要的，但不够充分。在生产中评估你的智能体很重要，因为你无法预见用户与智能体交互的所有方式。
How agent observability powers agent evaluation智能体可观测性如何驱动智能体评估
The traces you generate for observability are the same traces that power your evaluations, forming a unified foundation.你为可观测性生成的追踪与驱动评估的追踪相同，形成了统一的基础。
Traces → manual debugging追踪 → 手动调试
When you are running an agent locally on ad hoc queries and manually inspecting the results - that is still a form of (manual) evaluation! Traces power this workflow as they allow you to step into every step of the process and figure out exactly what when wrong.当你在本地对临时查询运行智能体并手动检查结果时——这仍然是一种（手动）评估形式！追踪支持此工作流，因为它们允许你进入过程的每一步并准确找出哪里出了问题。
Traces → offline evaluation datasets追踪 → 离线评估数据集
Production traces become your evaluation dataset automatically. For example, when a user reports a bug, you can see in the trace: the exact conversation history and context, what the agent decided at each step, and where specifically it went wrong.生产追踪自动成为你的评估数据集。例如，当用户报告错误时，你可以在追踪中看到：确切的对话历史和上下文、智能体在每一步的决定，以及具体在哪里出错。
An example workflow:一个示例工作流：
- User reports incorrect behavior用户报告不正确行为
- Find the production trace找到生产追踪
- Extract the state at the failure point提取失败点的状态
- Create a test case from that exact state从该确切状态创建测试用例
- Fix and validate修复并验证
Thus, your test suite for offline evaluation can be formed from real data points.因此，离线评估的测试套件可以由真实数据点形成。
Traces → online evaluation追踪 → 在线评估
The same traces generated for debugging power continuous production validation. Online evaluations run on traces you're already capturing. You can run checks on every trace or sample strategically:用于调试的相同追踪支持持续的生产验证。在线评估在你已经捕获的追踪上运行。你可以对每个追踪运行检查或策略性地抽样：
- Trajectory checks: Flag unusual tool call patterns轨迹检查：标记异常的工具调用模式
- Efficiency monitoring: Detect performance degradation trends效率监控：检测性能退化趋势
- Quality scoring: Run LLM-as-judge on production outputs质量评分：在生产输出上运行LLM作为评判者
- Failure alerts: Surface errors before user reports失败警报：在用户报告之前发现错误
This surfaces issues in real-time, validating that development behavior holds in production.这实时发现问题，验证开发行为在生产中是否成立。
Traces → ad-hoc insights追踪 → 临时洞察
When a trace contains 100,000+ lines of data or a thread spans dozens of turns, manual inspection becomes impossible. This is where AI-assisted analysis helps you query traces and threads to:当追踪包含100,000+行数据或线程跨越数十轮时，手动检查变得不可能。这时AI辅助分析可以帮助你查询追踪和线程以：
- Surface usage patterns across many agent executions揭示跨多个智能体执行的使用模式
- Identify common failure modes and inefficiencies识别常见的失败模式和低效
- Explain specific decisions: "Why did the agent call this tool at this step?"解释特定决策：“为什么智能体在这一步调用了这个工具？”
- Compare successful vs. failed executions to find patterns比较成功与失败的执行以发现模式
For example, recently we were investigating why an agent was taking inefficient paths. Instead of manually reading 150-step traces, we used an AI assistant which identified that the agent was calling read_file multiple times on the same file instead of storing content in context. The fix was a simple prompt adjustment (whereas spotting this pattern manually would have taken hours).例如，最近我们调查为什么一个智能体采取了低效路径。我们没有手动阅读150步的追踪，而是使用AI助手，它识别出智能体多次对同一文件调用read_file，而不是将内容存储在上下文中。修复是一个简单的提示调整（而手动发现这种模式需要数小时）。
What this means for teams building agents这对构建智能体的团队意味着什么
The teams shipping reliable agents have embraced the shift from debugging code to debugging reasoning. Traditional software separated tracing (for debugging) and testing (for validation). Now that we're debugging non-deterministic reasoning across long-running, stateful processes, these practices converge. You need reasoning traces to evaluate agent behavior, and you need systematic evaluation to make sense of traces.交付可靠智能体的团队已经接受了从调试代码到调试推理的转变。传统软件将追踪（用于调试）和测试（用于验证）分开。现在，我们在调试跨长时间运行、有状态过程的非确定性推理，这些实践趋于融合。你需要推理追踪来评估智能体行为，你需要系统化评估来理解追踪。
The teams that adopt both practices together, from day one, will be the ones shipping agents that actually work.从第一天起就同时采用这两种实践的团队，将是交付真正有效智能体的团队。
