Agent Evaluation Readiness Checklist智能体评估准备清单

Manually review 20-50 real agent traces before building any eval infrastructure在构建任何评估基础设施之前，手动审查 20-50 个真实的智能体追踪记录

Use LangSmith to go from traces to the annotation queue to datasets & experiments.使用 LangSmith 从追踪记录到标注队列，再到数据集与实验。

Before building any infrastructure, spend 30 minutes reading through real agent traces. You'll learn more about failure patterns from this than from any automated system. LangSmith's traces and annotation queues are excellent for this.在构建任何基础设施之前，花 30 分钟阅读真实的智能体追踪记录。相比任何自动化系统，你能从中了解更多关于失败模式的信息。LangSmith 的追踪和标注队列非常适合这一工作。

Define unambiguous success criteria for a single task为单个任务定义明确无误的成功标准

If two experts can't agree on pass/fail, the task needs refinement:如果两位专家无法就是否通过达成一致，则需要对任务进行细化：

• Unclear success: “Summarize this document well.”模糊的成功标准：“很好地总结这份文档。”
• Clear success: “Extract the 3 main action items from this meeting transcript. Each should be < 20 words and include an owner if mentioned.”明确的成功标准：“从这份会议记录中提取 3 个主要行动项。每个行动项应少于 20 个字，并包含被提及的负责人。”

Separate capability evals from regression evals将能力评估与回归评估分开

You need both because they serve different purposes. Capability evals push your agent forward by measuring progress on hard tasks, while regression evals protect what already works. Without the separation, you'll either stop improving because you're only guarding existing behavior, or you'll ship regressions because you're only chasing new capabilities.两者都需要，因为它们目的不同。能力评估通过在困难任务上衡量进展来推动智能体进步，而回归评估则保护已有功能。如果不加以区分，你要么会因为只守护现有行为而停止改进，要么会因为只追求新能力而引入回归问题。

• Capability evals answer “what can it do?”
  • Start with a low pass rate and give you a hill to climb.初始通过率较低，给你提供提升空间。
• Regression evals answer “does it still work?”
  • Should have ~100% pass rate and catch backsliding.应保持约 100% 的通过率，以发现倒退现象。

Ensure you can identify and articulate why each failure occurs确保你能够识别并阐明每次失败的原因

If you can't articulate why something failed, you need more error analysis before building automated evals. This is where you should spend 60-80% of your eval effort. Follow this process:如果你无法阐明某件事为何失败，那么在构建自动化评估之前，你需要进行更多的错误分析。评估工作中的 60-80% 应花在这里。请遵循以下流程：

1. Gather traces: Collect representative failures from production or testing收集追踪：从生产环境或测试中收集具有代表性的失败案例
2. Open coding: Review traces with a domain expert, noting every issue you see without pre-categorizing (or use our annotation queue to have subject matter experts review traces on their own)开放编码：与领域专家一起审查追踪记录，在不预先分类的情况下记录你发现的每一个问题（或者使用我们的标注队列，让主题专家自行审查追踪记录）
3. Categorize: Group issues into a failure taxonomy (prompt problems, tool design problems, model limitations, tool failures, data gaps, etc.)分类：将问题归入失败分类体系（提示词问题、工具设计问题、模型局限性、工具故障、数据缺失等）
4. Iterate: Keep reviewing until you stop discovering new failure categories迭代：持续审查，直到不再发现新的失败类别

Once you've categorized, the fix depends on the root cause:完成分类后，修复方案取决于根本原因：

• Prompt problem: The agent misunderstood because your instructions were unclear → fix the prompt提示词问题：智能体因指令不清而产生误解 → 修复提示词
• Tool design problem: The tool interface made it easy for the agent to make mistakes → redesign parameters, add examples, clarify boundaries工具设计问题：工具界面导致智能体容易出错 → 重新设计参数、添加示例、明确边界
• Model limitation: Instructions were clear but the LLM doesn't generalize to edge cases → add examples, try a different architecture, or use a different model模型局限性：指令清晰，但大语言模型无法泛化到边缘案例 → 添加示例、尝试不同的架构，或使用不同的模型
• Don't know yet: You haven't looked at enough failures to see the pattern → do more error analysis first尚不清楚：你尚未查看足够的失败案例以发现规律 → 先进行更多错误分析

Assign eval ownership to a single domain expert将评估职责分配给单一领域专家

Someone needs to own the eval process: maintaining datasets, recalibrating judges, triaging new failure modes, and deciding what “good enough” means. Ideally one domain expert acts as the quality arbiter for ambiguous cases rather than designing by committee.需要有人负责评估流程：维护数据集、重新校准评判器、对新失败模式进行分类，并决定什么是“足够好”。理想情况下，应由一位领域专家担任模糊案例的质量仲裁者，而不是通过委员会设计。

Rule out infrastructure and data pipeline issues before blaming the agent在归咎于智能体之前，先排除基础设施和数据流水线问题

The Witan Labs team found that a single extraction bug moved their benchmark from 50% to 73%. Infrastructure issues (timeouts, malformed API responses, stale caches) frequently masquerade as reasoning failures. Check the data pipeline first.Witan Labs 团队发现，仅一个抽取错误就将他们的基准测试从 50% 提升至 73%。基础设施问题（超时、格式错误的 API 响应、过期缓存）常常伪装成推理失败。请先检查数据流水线。

Single-step evals单步评估

These answer: “Did the agent choose the right tool?” “Did it generate a valid API call?” They're the easiest to automate but require stable agent architecture; if you're still changing your tool definitions, run-level evals may break.这些回答的问题是：“智能体是否选择了正确的工具？”“它是否生成了有效的 API 调用？”它们最容易自动化，但要求智能体架构稳定；如果你仍在修改工具定义，运行层级评估可能会失效。

Full-turn evals完整回合评估

This is where most teams should start. Grade a full trace across three dimensions:这是大多数团队应该开始的环节。从三个维度对完整追踪进行评分：

• Final response: Is the output correct and useful?最终响应：输出是否正确且有用？
• Trajectory: Did the agent take a reasonable path? (Not necessarily the exact path you expected, just a valid one)轨迹：智能体是否采取了合理的路径？（不一定是你预期的确切路径，只要是有效的即可）
• State changes: Did the agent create the right artifacts? (files written, database updated, meeting scheduled, etc.)状态变更：智能体是否创建了正确的产物？（写入的文件、更新的数据库、安排的会议等）

State change evaluation is often overlooked but critical for agents that do things, not just say things. For example, if your agent schedules meetings, don't just check that it said “Meeting scheduled!” Verify the calendar event actually exists with the right time, attendees, and description. If it writes code, run the code. If it updates a database, query the rows. The final response can say “Done!” while the actual state is wrong.状态变更评估常常被忽视，但对于做事而不仅仅是说话的智能体来说至关重要。例如，如果你的智能体安排会议，不要只检查它说了“会议已安排！”要验证日历事件是否确实存在，且时间、参与者和描述均正确。如果它编写代码，就运行这些代码。如果它更新数据库，就查询相关行。最终响应可能说“完成！”而实际状态却是错误的。

Multi-turn evals多轮评估

The hardest level to implement, layer them in after your trace-level evals are solid.这是最难实现的层级，请在追踪层级评估稳固后再逐步引入。

💡

Practical tip: Use N-1 testing. Take real conversation prefixes from production (the first N-1 turns) and let the agent generate only the final turn. This avoids the compounding error problem of fully synthetic multi-turn simulations.实用技巧：使用 N-1 测试。从生产环境中获取真实对话前缀（前 N-1 轮），然后让智能体仅生成最后一轮。这样可以避免完全合成多轮模拟中的误差累积问题。

Start with trace-level (full-turn) evals, then layer in run-level and thread-level as needed先从追踪层级（完整回合）评估开始，然后根据需要叠加运行层级和多轮层级评估

Trace-level gives you the most signal per eval. Run-level is useful for debugging specific steps. Thread-level matters when your agent has multi-turn conversations.追踪层级每次评估提供的信息最为丰富。运行层级有助于调试特定步骤。多轮层级在智能体进行多轮对话时很重要。

Ensure every task is unambiguous, with a reference solution that proves it's solvable确保每个任务都明确无误，并附有参考解决方案以证明其可解性

• Ambiguous: “Find me good flights to NYC.”模糊的示例：“帮我找去纽约的好航班。”
• Unambiguous: “Find roundtrip flights from SFO to JFK, departing Dec 15-17, returning Dec 22, under $400, economy class.”明确的示例：“查找从 SFO 到 JFK 的往返航班，12 月 15-17 日出发，12 月 22 日返回，400 美元以下，经济舱。”

If the agent can't possibly succeed (missing info, impossible constraints), the task is broken, not the agent. Include a reference solution for every task so you can prove it's solvable and have a baseline to grade against.如果智能体根本不可能成功（信息缺失、约束条件不可能满足），那么是任务本身有问题，而不是智能体。为每个任务包含一个参考解决方案，这样你可以证明它是可解的，并有一个用于评分的基线。

Test both positive cases (behavior should occur) and negative cases (behavior should not occur)同时测试正例（行为应该发生）和反例（行为不应该发生）

If you only test “did it search when it should?”, you'll optimize for an agent that searches everything. Test the negative cases too. Include examples designed to falsify your assumptions, not just confirm expected behavior.如果你只测试“它在应该搜索的时候是否搜索了？”，你会优化出一个对所有内容都进行搜索的智能体。也要测试反例。包含旨在证伪你假设的示例，而不仅仅是确认预期行为。

Ensure dataset structure matches your chosen evaluation level确保数据集结构与你选择的评估层级相匹配

• Run-level (single-step) evals need reference tool calls or decisions运行层级（单步）评估需要参考工具调用或决策
• Trace-level (full-turn) evals need expected final outputs and/or state changes追踪层级（完整回合）评估需要预期的最终输出和/或状态变更
• Thread-level (multi-turn) evals need multi-turn conversation sequences with expected context retention多轮层级（多轮）评估需要多轮对话序列，并具备预期的上下文保留能力

Tailor datasets to your agent type (coding, conversational, research)根据智能体类型（编码、对话、研究）定制数据集

• Coding agents: Include deterministic test suites (unit tests that pass/fail) alongside quality rubrics编码智能体：包含确定性测试套件（通过/失败的单元测试）以及质量评分标准
• Conversational agents: Include multi-dimensional criteria, task completion and interaction quality (empathy, clarity)对话智能体：包含多维标准，如任务完成度和交互质量（同理心、清晰度）
• Research agents: Include groundedness checks (are claims supported by sources?) and coverage checks (are key facts included?)研究智能体：包含真实性检查（主张是否有来源支持？）和覆盖范围检查（是否包含关键事实？）

Generate seed examples if you lack production data如果缺少生产数据，生成种子示例

Define the key dimensions of variation for your task (query complexity, topic, edge case type). Manually create ~20 example inputs covering those dimensions, run them through your existing agent, review and modify them to store as reliable ground truths.定义任务的关键变化维度（查询复杂度、主题、边缘案例类型）。手动创建约 20 个涵盖这些维度的示例输入，在现有智能体上运行，审查并修改它们，以存储为可靠的基准真相。

💡

Practical tip: 20-50 hand-reviewed examples you're confident in will outperform hundreds of synthetic examples you haven't verified. Quality beats quantity here!实用技巧：20-50 个你确信可靠的人工审查示例，将胜过数百个你未验证过的合成示例。这里质量胜于数量！

Source from dogfooding errors, adapted external benchmarks, and hand-written behavior tests从内部试用（dogfooding）错误、改编的外部基准测试和手写行为测试中获取来源

Once you're past the cold start, you need an ongoing pipeline for discovering new evals. Three strategies work well together:度过冷启动阶段后，你需要一个持续发现新评估的流水线。以下三种策略配合效果很好：

1. Dogfood your agent daily and turn every error into an eval. This is different from production monitoring; it's your team intentionally stress-testing the agent across real workflows.每天内部试用（dogfood）你的智能体，并将每个错误转化为一个评估。这与生产监控不同；这是你的团队有意在真实工作流中对智能体进行压力测试。
2. Pull and adapt tasks from external benchmarks like Terminal Bench or BFCL. Don't run full benchmarks in aggregate; cherry-pick tasks that test capabilities you care about and adapt them for your agent.从 Terminal Bench 或 BFCL 等外部基准测试中挑选并改编任务。不要整体运行完整的基准测试；挑选测试你所关心能力的任务，并针对你的智能体进行改编。
3. Write focused tests by hand for specific behaviors you think are important, like “does the agent parallelize tool calls?” or “does it ask clarifying questions for vague requests?”针对你认为重要的特定行为手写针对性测试，例如“智能体会并行化工具调用吗？”或“它会对模糊请求提出澄清问题吗？”

See “How we build evals for Deep Agents” for a concrete example of this approach.有关这种方法的具体示例，请参阅《How we build evals for Deep Agents》。

Select specialized graders per evaluation dimension针对每个评估维度选择专业评分器

Default to code-based evaluators when there's an objectively correct answer. LLM-as-judge grading for objective tasks can be unreliable, inconsistent judgments can mask real regressions. Switching to deterministic comparison can often eliminate inconsistency and provide better signal. Reserve LLM-as-judge for genuinely subjective assessments.当存在客观正确答案时，默认使用基于代码的评估器。对于客观任务，使用 LLM-as-judge 评分可能不可靠，不一致的判断会掩盖真正的回归问题。改用确定性比较通常可以消除不一致性并提供更好的信号。将 LLM-as-judge 保留给真正主观的评估。

💡

Practical Tip: Rather than trying to create a correctness evaluator, decompose evaluation into specialized graders per dimension rather than one monolithic grader.For example: the Witan Labs team built 5 specialized evaluators (content accuracy, structure, visual formatting, formula scenarios, text quality), each with dimension-appropriate thresholds. This gives you clearer signal about what's actually failing!实用技巧：与其试图创建一个正确性评估器，不如将评估分解为每个维度的专业评分器，而不是一个单一的整体评分器。例如：Witan Labs 团队构建了 5 个专业评估器（内容准确性、结构、视觉格式、公式场景、文本质量），每个都有适合该维度的阈值。这能让你更清楚地了解到底是什么在失败！

Distinguish guardrails from evaluators区分护栏与评估器

-judge scoring, trajectory analysis-judge 评分、轨迹分析

Safety checks and format validation are guardrails, they should run inline. Quality assessment and regression testing are evaluators, they run async. Don't confuse the two.安全检查与格式验证属于护栏，它们应该以内联方式运行。质量评估与回归测试属于评估器，它们异步运行。不要混淆这两者。

Prefer binary pass/fail over numeric scales优先使用二元通过/失败，而非数字量表

A 1-5 scale introduces subjective differences between adjacent scores and requires larger sample sizes for statistical significance. Binary forces clearer thinking: either the agent succeeded or it didn't. You can always decompose a complex task into multiple binary checks.1-5 分制会在相邻分数之间引入主观差异，并需要更大的样本量才能达到统计显著性。二元制促使更清晰的思考：智能体要么成功，要么失败。你总可以将复杂任务分解为多个二元检查。

Note: recent research suggests short scales (0-5) may yield stronger human-LLM alignment when using LLM-as-judge specifically, but binary remains simpler for human reviewers and faster iteration.注意：近期研究表明，在使用 LLM-as-judge 时，短量表（0-5）可能产生更强的人类-LLM 一致性，但对于人工审查者来说，二元制仍然更简单，迭代也更快。

Calibrate LLM-as-a-Judge graders to human preferences将 LLM-as-a-Judge 评分器校准到人类偏好

• Start with 20+ labeled examples using LangSmith's Align Evaluator feature, then grow toward ~100 for production-grade confidence从 20 多个使用 LangSmith Align Evaluator 功能的标注示例开始，然后逐步增加到约 100 个，以获得生产级的置信度
• Include reasoning in the judge's output; this improves accuracy and lets you audit why it scored something (Anthropic's Demystifying Evals emphasizes this as well)在评判器的输出中包含推理过程；这能提高准确性，并让你可以审计它为何给出某项分数（Anthropic 的《Demystifying Evals》也强调了这一点）
• Recalibrate regularly; judges drift over time and no single judge is uniformly reliable across all benchmarks定期重新校准；评判器会随时间漂移，且没有任何单一评判器在所有基准测试中都具有统一可靠性
• Use few-shot examples to improve evaluator consistency; corrections can auto-populate as few-shot examples in LangSmith使用少样本示例来提高评估器的一致性；在 LangSmith 中，修正可以自动填充为少样本示例

Grade the outcome, not the exact path, and build in partial credit for incremental progress评估结果而非确切路径，并为渐进式进步设置部分得分

Agents find creative solutions. As Anthropic puts it in Demystifying Evals: “Don't grade the path the agent took, grade what it produced.” If you require “must call tool A → B → C in that order,” you'll fail agents that found a smarter route. Better: “Did the meeting get scheduled correctly?” not “Did it call check_availability before create_event?”智能体会找到创造性的解决方案。正如 Anthropic 在《Demystifying Evals》中所说：“不要评估智能体走过的路径，评估它产出的结果。”如果你要求“必须按顺序调用工具 A → B → C”，你会让找到更聪明路线的智能体失败。更好的做法是：“会议是否正确安排了？”而不是“它是否在 create_event 之前调用了 check_availability？”

An agent that correctly identifies the problem but fails at the final step is better than one that fails immediately. Build in partial credit so your metrics reflect incremental progress.一个正确识别了问题但在最后一步失败的智能体，比立即失败的智能体更好。设置部分得分，让你的指标反映渐进式进步。

Use custom evaluators derived from your error analysis, not generic off-the-shelf metrics使用源自错误分析的自定义评估器，而非通用的现成指标

Off-the-shelf metrics like “helpfulness” or “coherence” create false confidence. The evaluators that matter are the ones that catch your specific failure modes, discovered through the error analysis process above.“helpfulness”或“coherence”等现成指标会造成虚假的信心。真正重要的评估器是那些能够捕捉你特定失败模式的评估器，而这些模式是通过上述错误分析过程发现的。

Distinguish between offline, online, and ad-hoc evaluation and use all three区分离线评估、在线评估和临时评估，并三者结合使用

Most of this checklist focuses on offline evaluation, and that's intentional. Offline evals are where you improve with: curated datasets, controlled experiments, iterating before you ship. You'll also need online and ad-hoc evaluation once your agent hits production.本清单的大部分内容侧重于离线评估，这是有意为之。离线评估是你通过以下方式进行改进的地方：精心策划的数据集、受控实验、在发布前迭代。一旦智能体投入生产，你还需要在线评估和临时评估。

‍
The Production readiness section below covers setting up online evaluations and scheduling ad-hoc trace exploration in detail.下方的“生产就绪”部分详细介绍了如何设置在线评估以及安排临时追踪探索。

Run multiple trials per task to account for non-determinism为每个任务运行多次试验，以考虑非确定性因素

Model outputs vary between runs. Use multiple repetitions if not cost prohibitive. When running multiple trials, compute confidence intervals before declaring improvement—single-run benchmarks are noisy. For non-deterministic agents, consider using pass@k (at least one of k attempts succeeds) or pass^k (all k attempts succeed) metrics depending on your product requirements.模型输出在每次运行之间会有所不同。如果成本允许，请使用多次重复。运行多次试验时，在宣称有改进之前先计算置信区间——单次运行的基准测试噪音很大。对于非确定性智能体，请根据产品需求考虑使用 pass@k（k 次尝试中至少一次成功）或 pass^k（k 次尝试全部成功）指标。

Track operational metrics alongside quality: turns taken, token usage, latency, cost per task. An agent that's 95% accurate but 10x slower might not be an improvement.在关注质量的同时跟踪运营指标：轮数、token 使用量、延迟、每项任务成本。一个准确率达到 95% 但速度慢了 10 倍的智能体可能算不上改进。

Tag evals by capability category, document what each measures, and track efficiency metrics alongside quality按能力类别为评估打标签，记录每个评估衡量的内容，并在关注质量的同时跟踪效率指标

Group evals by what they test, not where they come from. Categories like file_operations, retrieval, tool_use, memory, and conversation give you a “middle view” of performance between a single aggregate score and individual test results. Add a docstring to each eval explaining how it measures an agent capability. This keeps intent clear as the suite grows and lets you run targeted subsets (e.g., only tool_use evals after changing a tool definition).按它们测试的内容而非来源对评估进行分组。file_operations、retrieval、tool_use、memory 和 conversation 等类别能让你在单一总分和单个测试结果之间获得性能的“中间视角”。为每个评估添加文档字符串，说明它如何衡量智能体能力。这能在套件扩展时保持意图清晰，并允许你运行针对性的子集（例如，在更改工具定义后仅运行 tool_use 评估）。

Attach metadata to every experiment so you can filter, group, and compare runs across dimensions that matter. This makes it easy to answer questions like “did switching from GPT-4.1 to Claude Sonnet improve accuracy?” or “which prompt version regressed on this dataset?” without digging through logs. LangSmith automatically captures git info when available, but explicitly tagging model and prompt metadata pays off quickly as your experiment volume grows.为每个实验附加元数据，以便你可以按重要维度筛选、分组和比较运行结果。这样你就能轻松回答诸如“从 GPT-4.1 换到 Claude Sonnet 是否提高了准确率？”或“哪个提示词版本在这个数据集上出现了回归？”等问题，而无需翻查日志。LangSmith 会在可用时自动捕获 git 信息，但随着实验量的增长，显式标注模型和提示词元数据会很快带来回报。

Once quality is established, compare models on efficiency. An agent that's 95% accurate but 10x slower might not be an improvement. Track ratios like observed steps / ideal steps, observed tool calls / ideal tool calls, and observed latency / ideal latency. This doesn't conflict with “grade the outcome, not the exact path”: ideal trajectories measure efficiency, not correctness. You still pass an agent that found a creative route, but you can see if it took longer to get there. See the metrics framework in How we build evals for Deep Agents for a worked example.在质量确立后，比较模型的效率。一个准确率达到 95% 但速度慢了 10 倍的智能体可能算不上改进。跟踪诸如实际步骤数/理想步骤数、实际工具调用次数/理想工具调用次数、实际延迟/理想延迟等比率。这与“评估结果而非确切路径”并不冲突：理想轨迹衡量的是效率，而非正确性。你仍然可以通过找到创造性路径的智能体，但你可以看出它是否花了更长时间才到达目标。有关具体示例，请参阅《How we build evals for Deep Agents》中的指标框架。

Manually review traces for failed evaluations to verify grader fairness手动审查失败评估的追踪记录，以验证评分器的公平性

A “failed” task might actually be a creative valid solution your grader didn't anticipate. Reading traces is how you know if your graders are being fair.一项“失败”的任务实际上可能是你的评分器没有预料到的创造性有效解决方案。阅读追踪记录是判断评分器是否公平的方法。

Ensure each trial runs in a clean, isolated environment with no shared state确保每次试验都在干净、隔离且无共享状态的环境中运行

If trial 2 can see artifacts from trial 1, your results aren't independent. What this means in practice:如果试验 2 能看到试验 1 的产物，你的结果就不是独立的。在实践中这意味着：

• Coding agents: Fresh containers or VMs per trial编码智能体：每次试验使用全新的容器或虚拟机
• API-calling agents: Staging environments or mock services调用 API 的智能体：使用预发布环境或模拟服务
• Database agents: Snapshot and restore between trials数据库智能体：在试验之间进行快照和恢复

Recognize when pass rates plateau and evolve your test suite accordingly识别通过率何时进入平台期，并相应地演进你的测试套件

When your pass rate plateaus and adding more tasks of the same type stops revealing new failure modes, it's time to evolve: add harder tasks, test new capabilities, or shift to different dimensions. Grinding on a saturated eval set wastes effort.当你的通过率进入平台期，且添加更多同类型任务不再能发现新的失败模式时，就该进化了：增加更难的任务、测试新能力，或转向不同维度。在一个饱和的评估集上磨工夫是浪费精力。

Only keep evals that directly measure a production behavior you care about仅保留直接衡量你所关心的生产行为的评估

Every eval applies pressure on your system over time. It's tempting to blindly add hundreds of tests, but this creates an illusion of progress. You end up optimizing for an eval suite that doesn't reflect what matters in production. More evals does not equal better agents. Build targeted evals, and periodically prune the ones that no longer give you signal. For a concrete example of this approach, see How we build evals for Deep Agents.每项评估都会随着时间对你的系统施加压力。盲目添加数百个测试很诱人，但这会造成进步的假象。你最终会为一个无法反映生产中重要事项的评估套件进行优化。更多的评估不等于更好的智能体。构建有针对性的评估，并定期修剪那些不再提供信号的评估。有关这种方法的具体示例，请参阅《How we build evals for Deep Agents》。

Invest in tool interface design and testing, not just prompt optimization投入于工具接口设计与测试，而不仅仅是提示词优化

Tool design eliminates entire classes of agent errors. Anthropic's team noted they spent more time optimizing tools than prompts when building their SWE-bench agent. Test how the model actually uses your tools: try different parameter formats (diffs vs full rewrites, JSON vs. markdown), redesign interfaces to make mistakes harder, and invest in clear documentation with examples. The goal is to make mistakes structurally impossible, not just unlikely. For example, requiring absolute file paths eliminates an entire class of navigation errors.工具设计可以消除整类智能体错误。Anthropic 团队指出，在构建他们的 SWE-bench 智能体时，他们在优化工具上花的时间比提示词还多。测试模型实际使用你工具的方式：尝试不同的参数格式（diff 与完整重写、JSON 与 markdown）、重新设计接口以降低犯错的可能性，并投入于包含示例的清晰文档。目标是让错误在结构上就不可能发生，而不仅仅是降低发生概率。例如，要求使用绝对文件路径可以消除整类导航错误。

Distinguish between task failures (agent got it wrong) and evaluation failures (grader got it wrong)区分任务失败（智能体出错了）与评估失败（评分器出错了）

Track run status explicitly (complete, error, timeout). A grader that marks a timeout as “incorrect reasoning” pollutes your signal. Separate task failures from evaluation failures to keep your metrics clean.显式跟踪运行状态（完成、错误、超时）。将超时标记为“推理错误”的评分器会污染你的信号。将任务失败与评估失败分开，以保持指标干净。

Promote capability evals with consistently high pass rates into your regression suite将通过率持续较高的能力评估提升为回归套件的一部分

Once you've climbed the hill, protect it. Tasks that used to test “can we do this?” become “can we still do this?”当你已攻克难关，就要守护它。曾经用于测试“我们能做到吗？”的任务，变成了“我们还能做到吗？”

Integrate regression evals into your CI/CD pipeline with automated quality gates将回归评估集成到 CI/CD 流水线中，并设置自动化质量门禁

A typical flow:典型流程：

1. Code or prompt change triggers the pipeline (via git push, PromptHub update, or manual trigger)代码或提示词的变更触发流水线（通过 git push、PromptHub 更新或手动触发）
2. Offline evals run unit tests, integration tests, and evaluation against curated datasets using cheap, fast graders离线评估运行单元测试、集成测试，并使用廉价、快速的评分器对策划好的数据集进行评估
3. Preview deployment goes up if offline evals pass如果离线评估通过，则部署预览版本
4. Online evals run against the preview with live data using LLM-as-judge graders在线评估使用 LLM-as-judge 评分器，在预览环境中对实时数据运行
5. Promote to production only if all quality gates pass, otherwise route failing traces to annotation queues and alert the team仅当所有质量门禁通过时才提升为生产环境，否则将失败的追踪路由到标注队列并提醒团队

Use cheap code-based graders in CI for every commit. Reserve expensive LLM-as-judge evaluations for preview/production evaluation. See LangSmith's CI/CD pipeline guide for a full implementation example with GitHub Actions.在 CI 中为每次提交使用廉价的基于代码的评分器。将昂贵的 LLM-as-judge 评估保留给预览/生产环境评估。有关使用 GitHub Actions 的完整实现示例，请参阅 LangSmith 的 CI/CD 流水线指南。

Set up online evaluations for production traffic为生产流量设置在线评估

Safety checks, format validation, quality heuristics. You'll find failure modes in production you never anticipated (see You don't know what your agent will do until it's in production)安全检查、格式验证、质量启发式规则。你会在生产中发现从未预料到的失败模式（请参阅《You don't know what your agent will do until it's in production》）

Capture user feedback收集用户反馈

Once your agent is in production, user feedback becomes one of your most valuable signals. Automated evals can only catch the failure modes you already know about. Users will surface the ones you don't: edge cases your dataset missed, outputs that are technically correct but unhelpful, and workflows that break in ways you never anticipated.当你的智能体投入生产后，用户反馈将成为你最宝贵的信号之一。自动化评估只能捕捉你已知的失败模式。用户会浮现出你未知的那些：你的数据集遗漏的边缘案例、技术上正确但无用的输出，以及以你从未预料到的方式中断的工作流。

Capturing this feedback in a structured way lets you feed it back into your datasets, calibrate your graders against real-world expectations, and prioritize the improvements that actually matter to the people using your agent.以结构化方式收集这些反馈，可以让你将其反馈到数据集中，根据真实世界的预期校准评分器，并优先处理对实际用户真正重要的改进。

Schedule regular manual exploration of production traces beyond automated checks安排定期手动探索生产追踪记录，超出自动化检查的范畴

Don't rely solely on automated pass/fail. Periodically explore production traces for unexpected patterns or failure modes your graders don't cover, surprising user behaviors, or opportunities to improve. Our Insights Agent is a great way to do this!不要只依赖自动化的通过/失败结果。定期探索生产追踪记录，寻找意外模式或评分器未涵盖的失败模式、令人惊讶的用户行为，或改进机会。我们的 Insights Agent 是完成此任务的绝佳方式！

Version your prompts and tool definitions对提示词和工具定义进行版本控制

LangSmith makes it easy to version your prompts. Without this, you can't correlate eval results with specific changes or know which edit caused a regression.LangSmith 让你可以轻松对提示词进行版本控制。没有这一点，你就无法将评估结果与特定变更关联起来，也无法知道哪次编辑导致了回归。

Ensure production failures feed back into datasets, error analysis, and eval improvements确保生产失败能反馈到数据集、错误分析和评估改进中

Production successes and failures should feed back into your datasets, error analysis, and eval improvements. This is the flywheel that makes your agent better over time!生产的成功与失败都应反馈到你的数据集、错误分析和评估改进中。这是让你的智能体随时间变得更优秀的飞轮机制！

You don't need all of these items on day one. Pick the section that matches where you are right now, nail those items, and expand from there. The teams that ship reliable agents aren't the ones with the most sophisticated eval infrastructure - they're the ones who started evaluating early and never stopped iterating.你不需要在第一天就完成所有这些项目。选择与你当前阶段匹配的部分，把这些项目做好，然后从那里扩展。那些交付可靠智能体的团队，并不是拥有最复杂评估基础设施的团队——而是那些尽早开始评估并从未停止迭代的团队。