Agent Evaluation Readiness Checklist智能体评估就绪检查清单

Manually review 20-50 real agent traces before building any eval infrastructure在搭建任何评估基础设施前，先手动审阅 20-50 条真实的智能体追踪记录

Use LangSmith to go from traces to the annotation queue to datasets & experiments.利用 LangSmith，从追踪记录顺藤摸瓜，构建标注队列、数据集与实验。

Before building any infrastructure, spend 30 minutes reading through real agent traces. You'll learn more about failure patterns from this than from any automated system. LangSmith's traces and annotation queues are excellent for this.动工之前，不妨花半小时细读真实的智能体追踪记录。从中习得的失败模式，远胜任何自动化系统。LangSmith 的追踪功能与标注队列正是为此而生。

Define unambiguous success criteria for a single task为单一任务定义明确无误的成功标准

If two experts can't agree on pass/fail, the task needs refinement:若两位专家对成败意见不一，那任务本身便需打磨：

• Unclear success: “Summarize this document well.”模糊的成功标准：“把这份文档总结好。”
• Clear success: “Extract the 3 main action items from this meeting transcript. Each should be < 20 words and include an owner if mentioned.”清晰的成功标准：“从会议记录中提取 3 项主要行动事项。每项不得超过 20 字，且若提及负责人需一并包含。”

Separate capability evals from regression evals将能力评估与回归评估分离开来

You need both because they serve different purposes. Capability evals push your agent forward by measuring progress on hard tasks, while regression evals protect what already works. Without the separation, you'll either stop improving because you're only guarding existing behavior, or you'll ship regressions because you're only chasing new capabilities.两者缺一不可，用途各异。能力评估旨在攻坚克难，衡量智能体在复杂任务上的进境；回归评估则如守城之兵，护佑既有功能不致倒退。若不分明，要么因只守旧规而止步不前，要么因一味追新而导致旧疾复发。

• Capability evals answer “what can it do?”
  • Start with a low pass rate and give you a hill to climb.初期通过率较低，为你留下攀登的高山。
• Regression evals answer “does it still work?”
  • Should have ~100% pass rate and catch backsliding.应保持约 100% 的通过率，以防退步。

Ensure you can identify and articulate why each failure occurs确保你能识别并阐明每一次失败的缘由

If you can't articulate why something failed, you need more error analysis before building automated evals. This is where you should spend 60-80% of your eval effort. Follow this process:若你无法道明失败因何而起，便需在自动化评估前投入更多精力进行错误分析。此处应占据你评估工作 60-80% 的心力。请遵循此法：

1. Gather traces: Collect representative failures from production or testing搜集追踪：从生产环境或测试中汇总具有代表性的失败案例
2. Open coding: Review traces with a domain expert, noting every issue you see without pre-categorizing (or use our annotation queue to have subject matter experts review traces on their own)开放式编码：与领域专家一同审阅记录，记录所见每一处瑕疵，无需预先分类（或利用我们的标注队列，让专家自行审阅）
3. Categorize: Group issues into a failure taxonomy (prompt problems, tool design problems, model limitations, tool failures, data gaps, etc.)分类归纳：将问题归入失败分类法（如提示词问题、工具设计缺陷、模型局限、工具故障、数据缺失等）
4. Iterate: Keep reviewing until you stop discovering new failure categories迭代优化：反复审阅，直至不再发现新的失败类别

Once you've categorized, the fix depends on the root cause:一旦分类完成，修复之道便取决于病灶所在：

• Prompt problem: The agent misunderstood because your instructions were unclear → fix the prompt提示词问题：智能体因指令不明而误解 → 修正提示词
• Tool design problem: The tool interface made it easy for the agent to make mistakes → redesign parameters, add examples, clarify boundaries工具设计问题：工具接口易诱导智能体犯错 → 重新设计参数、增加示例、明确边界
• Model limitation: Instructions were clear but the LLM doesn't generalize to edge cases → add examples, try a different architecture, or use a different model模型局限：指令清晰但 LLM 无法泛化至边缘情况 → 增加示例、尝试不同架构或更换模型
• Don't know yet: You haven't looked at enough failures to see the pattern → do more error analysis first尚不明确：审阅案例不足，未见规律 → 先深入分析错误

Assign eval ownership to a single domain expert指定一名领域专家全权负责评估工作

Someone needs to own the eval process: maintaining datasets, recalibrating judges, triaging new failure modes, and deciding what “good enough” means. Ideally one domain expert acts as the quality arbiter for ambiguous cases rather than designing by committee.总得有人掌舵评估大局：维护数据集、校准裁判、梳理新出现的失败模式，并裁定何为“足够好”。理想状态下，由一名领域专家充当模糊案例的质量仲裁者，而非通过委员会来决策。

Rule out infrastructure and data pipeline issues before blaming the agent在怪罪智能体之前，先排除基础设施与数据流水线的问题

The Witan Labs team found that a single extraction bug moved their benchmark from 50% to 73%. Infrastructure issues (timeouts, malformed API responses, stale caches) frequently masquerade as reasoning failures. Check the data pipeline first.Witan Labs 团队曾发现，一个简单的提取错误竟将基准测试结果从 50% 拖累至 73%。基础设施问题（如超时、API 响应畸形、缓存滞后）常伪装成推理失败。务必先查数据流水线。

Single-step evals单步评估

These answer: “Did the agent choose the right tool?” “Did it generate a valid API call?” They're the easiest to automate but require stable agent architecture; if you're still changing your tool definitions, run-level evals may break.此类评估旨在回答：“智能体选对了工具吗？”“它生成的 API 调用有效吗？”它们最易自动化，但要求智能体架构稳定；若你仍在变动工具定义，运行级评估可能会失效。

Full-turn evals全程评估

This is where most teams should start. Grade a full trace across three dimensions:这是大多数团队的起步之选。从三个维度对完整追踪进行评分：

• Final response: Is the output correct and useful?最终响应：输出是否正确且有用？
• Trajectory: Did the agent take a reasonable path? (Not necessarily the exact path you expected, just a valid one)轨迹：智能体所走的路径是否合理？（不必是你预期的精确路径，只要有效即可）
• State changes: Did the agent create the right artifacts? (files written, database updated, meeting scheduled, etc.)状态变更：智能体是否创建了正确的工件？（如文件写入、数据库更新、会议预约等）

State change evaluation is often overlooked but critical for agents that do things, not just say things. For example, if your agent schedules meetings, don't just check that it said “Meeting scheduled!” Verify the calendar event actually exists with the right time, attendees, and description. If it writes code, run the code. If it updates a database, query the rows. The final response can say “Done!” while the actual state is wrong.状态变更评估常被忽视，但对于不仅是“说”还要“做”的智能体而言至关重要。例如，若智能体负责预约会议，别只检查它是否说了“会议已预约！”，务必核实日历事件是否确实存在，时间、参会者及描述是否无误。若它编写代码，便去运行代码；若它更新数据库，便去查询行记录。毕竟，最终响应可能写着“搞定！”，而实际状态却是一团糟。

Multi-turn evals多轮评估

The hardest level to implement, layer them in after your trace-level evals are solid.这是实现难度最高的一层，待你的追踪级评估稳固后再行叠加。

💡

Practical tip: Use N-1 testing. Take real conversation prefixes from production (the first N-1 turns) and let the agent generate only the final turn. This avoids the compounding error problem of fully synthetic multi-turn simulations.实操建议：使用 N-1 测试法。截取生产环境中的真实对话前缀（即前 N-1 轮），仅让智能体生成最后一轮。这能避免全合成多轮模拟中常见的误差累积问题。

Start with trace-level (full-turn) evals, then layer in run-level and thread-level as needed先从追踪级（全程）评估起步，再按需叠加运行级与线程级评估

Trace-level gives you the most signal per eval. Run-level is useful for debugging specific steps. Thread-level matters when your agent has multi-turn conversations.追踪级评估能提供最高的单次评估信号。运行级有助于调试特定步骤。而当智能体涉及多轮对话时，线程级评估便显得尤为重要。

Ensure every task is unambiguous, with a reference solution that proves it's solvable确保每个任务明确无误，且配有可证明其可解的参考答案

• Ambiguous: “Find me good flights to NYC.”模糊任务：“帮我找去纽约的好航班。”
• Unambiguous: “Find roundtrip flights from SFO to JFK, departing Dec 15-17, returning Dec 22, under $400, economy class.”明确任务：“查找 12 月 15-17 日出发、12 月 22 日返回，从旧金山 (SFO) 到纽约 (JFK) 的往返经济舱航班，价格低于 400 美元。”

If the agent can't possibly succeed (missing info, impossible constraints), the task is broken, not the agent. Include a reference solution for every task so you can prove it's solvable and have a baseline to grade against.若智能体根本无法成功（信息缺失、约束条件矛盾），那是任务本身的问题，而非智能体的过错。为每个任务提供参考答案，既能证明其可解，又有了评分的基准。

Test both positive cases (behavior should occur) and negative cases (behavior should not occur)同时测试正向案例（应发生行为）与负向案例（不应发生行为）

If you only test “did it search when it should?”, you'll optimize for an agent that searches everything. Test the negative cases too. Include examples designed to falsify your assumptions, not just confirm expected behavior.若你只测试“该搜索时它搜了吗？”，你最终会优化出一个什么都搜的智能体。务必测试负向案例。加入旨在证伪你假设的示例，而不仅仅是确认预期行为。

Ensure dataset structure matches your chosen evaluation level确保数据集结构与所选评估层级匹配

• Run-level (single-step) evals need reference tool calls or decisions运行级（单步）评估需要参考工具调用或决策
• Trace-level (full-turn) evals need expected final outputs and/or state changes追踪级（全程）评估需要预期的最终输出和/或状态变更
• Thread-level (multi-turn) evals need multi-turn conversation sequences with expected context retention线程级（多轮）评估需要包含预期上下文留存的多轮对话序列

Tailor datasets to your agent type (coding, conversational, research)根据智能体类型（代码、对话、研究）定制数据集

• Coding agents: Include deterministic test suites (unit tests that pass/fail) alongside quality rubrics代码智能体：除质量准则外，需包含确定性测试套件（通过/失败的单元测试）
• Conversational agents: Include multi-dimensional criteria, task completion and interaction quality (empathy, clarity)对话智能体：包含多维度准则，如任务完成度及交互质量（共情、清晰度）
• Research agents: Include groundedness checks (are claims supported by sources?) and coverage checks (are key facts included?)研究智能体：包含可信度检查（观点是否有来源支撑？）及覆盖率检查（关键事实是否包含？）

Generate seed examples if you lack production data若缺乏生产数据，可先生成种子示例

Define the key dimensions of variation for your task (query complexity, topic, edge case type). Manually create ~20 example inputs covering those dimensions, run them through your existing agent, review and modify them to store as reliable ground truths.定义任务的关键变体维度（查询复杂度、主题、边缘情况类型）。手动创建约 20 个覆盖这些维度的示例输入，在现有智能体上运行，审阅并修改后作为可靠的基准事实存储。

💡

Practical tip: 20-50 hand-reviewed examples you're confident in will outperform hundreds of synthetic examples you haven't verified. Quality beats quantity here!实操建议：20-50 个经手动审阅、你对其信心十足的示例，效果远胜数百个未经核实的合成示例。此处质量重于数量！

Source from dogfooding errors, adapted external benchmarks, and hand-written behavior tests从“吃自己的狗粮”发现的错误、改编的外部基准及手写行为测试中获取来源

Once you're past the cold start, you need an ongoing pipeline for discovering new evals. Three strategies work well together:一旦度过冷启动期，你需要一个持续发现新评估点的流水线。三种策略相辅相成：

1. Dogfood your agent daily and turn every error into an eval. This is different from production monitoring; it's your team intentionally stress-testing the agent across real workflows.每日“吃自己的狗粮”，将每一次错误转化为评估点。这不同于生产监控，这是团队有意在真实工作流中对智能体进行压力测试。
2. Pull and adapt tasks from external benchmarks like Terminal Bench or BFCL. Don't run full benchmarks in aggregate; cherry-pick tasks that test capabilities you care about and adapt them for your agent.从 Terminal Bench 或 BFCL 等外部基准中提取并改编任务。不必整体运行完整基准，只需挑选测试你所关注能力的任务，并根据你的智能体进行适配。
3. Write focused tests by hand for specific behaviors you think are important, like “does the agent parallelize tool calls?” or “does it ask clarifying questions for vague requests?”针对你认为重要的特定行为手写专注测试，例如“智能体是否并行调用工具？”或“对于模糊请求，它是否会提出澄清问题？”

See “How we build evals for Deep Agents” for a concrete example of this approach.参阅《我们如何为深度智能体构建评估》以获取此方法的具体案例。

Select specialized graders per evaluation dimension针对不同评估维度选择专业评分器

Default to code-based evaluators when there's an objectively correct answer. LLM-as-judge grading for objective tasks can be unreliable, inconsistent judgments can mask real regressions. Switching to deterministic comparison can often eliminate inconsistency and provide better signal. Reserve LLM-as-judge for genuinely subjective assessments.当存在客观正确答案时，默认使用基于代码的评估器。对于客观任务，LLM-as-judge 评分可能不稳定，不一致的判断会掩盖真实的回归。切换至确定性对比往往能消除不一致，并提供更好的信号。仅针对真正主观的评估保留 LLM-as-judge。

💡

Practical Tip: Rather than trying to create a correctness evaluator, decompose evaluation into specialized graders per dimension rather than one monolithic grader.For example: the Witan Labs team built 5 specialized evaluators (content accuracy, structure, visual formatting, formula scenarios, text quality), each with dimension-appropriate thresholds. This gives you clearer signal about what's actually failing!实操建议：与其试图创建一个万能的正确性评估器，不如将评估分解为针对不同维度的专业评分器。例如：Witan Labs 团队构建了 5 个专业评估器（内容准确性、结构、视觉格式、公式场景、文本质量），每个都设有维度适配的阈值。这能让你更清晰地洞察到底哪里出了问题！

Distinguish guardrails from evaluators区分护栏与评估器

-judge scoring, trajectory analysis-judge 评分、轨迹分析

Safety checks and format validation are guardrails, they should run inline. Quality assessment and regression testing are evaluators, they run async. Don't confuse the two.安全检查与格式验证属于护栏，应内联运行。质量评估与回归测试属于评估器，应异步运行。切勿混淆两者。

Prefer binary pass/fail over numeric scales优先选择二元通过/失败，而非数值量表

A 1-5 scale introduces subjective differences between adjacent scores and requires larger sample sizes for statistical significance. Binary forces clearer thinking: either the agent succeeded or it didn't. You can always decompose a complex task into multiple binary checks.1-5 分的量表会引入相邻分数间的主观差异，并需要更大的样本量来确保统计显著性。二元制强制你进行更清晰的思考：智能体要么成功，要么失败。你总可以将复杂任务分解为多个二元检查。

Note: recent research suggests short scales (0-5) may yield stronger human-LLM alignment when using LLM-as-judge specifically, but binary remains simpler for human reviewers and faster iteration.注：近期研究表明，当专门使用 LLM-as-judge 时，短量表（0-5）可能产生更强的人机对齐，但对于人工审阅者而言，二元制依然更简单，迭代速度更快。

Calibrate LLM-as-a-Judge graders to human preferences将 LLM-as-a-Judge 评分器校准至人类偏好

• Start with 20+ labeled examples using LangSmith's Align Evaluator feature, then grow toward ~100 for production-grade confidence利用 LangSmith 的 Align Evaluator 功能，从 20+ 个标注示例起步，逐步增长至约 100 个，以获得生产级的信心。
• Include reasoning in the judge's output; this improves accuracy and lets you audit why it scored something (Anthropic's Demystifying Evals emphasizes this as well)在裁判的输出中包含推理过程；这能提高准确性，并让你能够审计其评分依据（Anthropic 的《揭秘评估》亦强调了这一点）。
• Recalibrate regularly; judges drift over time and no single judge is uniformly reliable across all benchmarks定期重新校准；裁判会随时间漂移，且没有任何单一裁判在所有基准上都表现均一。
• Use few-shot examples to improve evaluator consistency; corrections can auto-populate as few-shot examples in LangSmith使用少样本示例提升评估器一致性；在 LangSmith 中，修正内容可自动填充为少样本示例。

Grade the outcome, not the exact path, and build in partial credit for incremental progress评估结果而非路径，并为增量进展建立部分得分机制

Agents find creative solutions. As Anthropic puts it in Demystifying Evals: “Don't grade the path the agent took, grade what it produced.” If you require “must call tool A → B → C in that order,” you'll fail agents that found a smarter route. Better: “Did the meeting get scheduled correctly?” not “Did it call check_availability before create_event?”智能体往往能找到创造性的解决方案。正如 Anthropic 在《揭秘评估》中所言：“不要评估智能体所走的路径，评估它产出的成果。”若你要求“必须按 A → B → C 的顺序调用工具”，你将扼杀那些找到更聪明路径的智能体。更好的做法是：“会议预约正确了吗？”，而不是“它在 create_event 之前调用了 check_availability 吗？”

An agent that correctly identifies the problem but fails at the final step is better than one that fails immediately. Build in partial credit so your metrics reflect incremental progress.一个能正确识别问题但在最后一步失败的智能体，远胜于直接失败的智能体。建立部分得分机制，让你的指标能反映出增量进展。

Use custom evaluators derived from your error analysis, not generic off-the-shelf metrics使用源自错误分析的自定义评估器，而非通用的现成指标

Off-the-shelf metrics like “helpfulness” or “coherence” create false confidence. The evaluators that matter are the ones that catch your specific failure modes, discovered through the error analysis process above.像“有用性”或“连贯性”这类现成指标只会制造虚假的信心。真正重要的评估器，是那些能捕捉到你特定失败模式的工具，而这些模式正是通过上述错误分析过程发现的。

Distinguish between offline, online, and ad-hoc evaluation and use all three区分离线、在线与即席评估，并三者并用

Most of this checklist focuses on offline evaluation, and that's intentional. Offline evals are where you improve with: curated datasets, controlled experiments, iterating before you ship. You'll also need online and ad-hoc evaluation once your agent hits production.本清单大部分内容聚焦于离线评估，这是刻意为之。离线评估是你实现优化的关键：精心策划的数据集、受控的实验、交付前的反复迭代。一旦智能体上线，你同样需要在线与即席评估。

‍
The Production readiness section below covers setting up online evaluations and scheduling ad-hoc trace exploration in detail.下方的“生产就绪”部分详细涵盖了设置在线评估与调度即席追踪探索的内容。

Run multiple trials per task to account for non-determinism对每个任务运行多次试验，以应对不确定性

Model outputs vary between runs. Use multiple repetitions if not cost prohibitive. When running multiple trials, compute confidence intervals before declaring improvement—single-run benchmarks are noisy. For non-deterministic agents, consider using pass@k (at least one of k attempts succeeds) or pass^k (all k attempts succeed) metrics depending on your product requirements.模型输出在不同运行间存在差异。若成本允许，请多次重复。在运行多次试验时，请在宣布改进前计算置信区间——单次基准测试充满噪声。对于非确定性智能体，根据产品需求，可考虑使用 pass@k（k 次尝试中至少一次成功）或 pass^k（k 次尝试全部成功）指标。

Track operational metrics alongside quality: turns taken, token usage, latency, cost per task. An agent that's 95% accurate but 10x slower might not be an improvement.在质量之外跟踪运营指标：轮次、Token 使用量、延迟、任务成本。一个准确率 95% 但速度慢 10 倍的智能体，未必是改进。

Tag evals by capability category, document what each measures, and track efficiency metrics alongside quality按能力类别标记评估，记录每项衡量指标，并在质量之外跟踪效率指标

Group evals by what they test, not where they come from. Categories like file_operations, retrieval, tool_use, memory, and conversation give you a “middle view” of performance between a single aggregate score and individual test results. Add a docstring to each eval explaining how it measures an agent capability. This keeps intent clear as the suite grows and lets you run targeted subsets (e.g., only tool_use evals after changing a tool definition).按测试内容而非来源对评估进行分组。如 file_operations、retrieval、tool_use、memory 和 conversation 等类别，能为你提供介于总分与单项测试结果之间的“中间视图”。为每项评估添加 docstring，解释其如何衡量智能体能力。这能确保随着套件增长，意图依然清晰，并允许你运行有针对性的子集（例如在修改工具定义后，只运行 tool_use 评估）。

Attach metadata to every experiment so you can filter, group, and compare runs across dimensions that matter. This makes it easy to answer questions like “did switching from GPT-4.1 to Claude Sonnet improve accuracy?” or “which prompt version regressed on this dataset?” without digging through logs. LangSmith automatically captures git info when available, but explicitly tagging model and prompt metadata pays off quickly as your experiment volume grows.为每次实验附加元数据，以便你能在关键维度上过滤、分组和对比运行结果。这使得回答问题变得轻而易举，例如“从 GPT-4.1 切换到 Claude Sonnet 是否提高了准确率？”或“哪个提示词版本在数据集上出现了退步？”，而无需翻阅日志。LangSmith 会在可用时自动捕获 git 信息，但随着实验量增加，显式标记模型与提示词元数据将很快带来回报。

Once quality is established, compare models on efficiency. An agent that's 95% accurate but 10x slower might not be an improvement. Track ratios like observed steps / ideal steps, observed tool calls / ideal tool calls, and observed latency / ideal latency. This doesn't conflict with “grade the outcome, not the exact path”: ideal trajectories measure efficiency, not correctness. You still pass an agent that found a creative route, but you can see if it took longer to get there. See the metrics framework in How we build evals for Deep Agents for a worked example.质量确立后，对比模型的效率。跟踪比率，如观测步骤/理想步骤、观测工具调用/理想工具调用、观测延迟/理想延迟。这与“评估结果而非路径”并不冲突：理想轨迹衡量的是效率而非正确性。你依然可以给找到创造性路径的智能体通过，但能看出它是否花费了更多时间。参阅《我们如何为深度智能体构建评估》中的指标框架获取实操案例。

Manually review traces for failed evaluations to verify grader fairness手动审阅失败评估的追踪记录，核实评分器是否公正

A “failed” task might actually be a creative valid solution your grader didn't anticipate. Reading traces is how you know if your graders are being fair.一个“失败”的任务可能实际上是评分器未预料到的、具有创造性的有效方案。阅读追踪记录，是你了解评分器是否公正的唯一途径。

Ensure each trial runs in a clean, isolated environment with no shared state确保每次试验都在干净、隔离且无共享状态的环境中运行

If trial 2 can see artifacts from trial 1, your results aren't independent. What this means in practice:若试验 2 能看到试验 1 的工件，则结果便不再独立。在实践中这意味着：

• Coding agents: Fresh containers or VMs per trial代码智能体：每次试验使用全新的容器或虚拟机
• API-calling agents: Staging environments or mock servicesAPI 调用智能体：使用暂存环境或模拟服务
• Database agents: Snapshot and restore between trials数据库智能体：试验间进行快照与恢复

Recognize when pass rates plateau and evolve your test suite accordingly在通过率停滞时及时识别，并相应演进你的测试套件

When your pass rate plateaus and adding more tasks of the same type stops revealing new failure modes, it's time to evolve: add harder tasks, test new capabilities, or shift to different dimensions. Grinding on a saturated eval set wastes effort.当通过率停滞，且增加同类任务不再能揭示新的失败模式时，便是演进之时：增加更难的任务、测试新能力，或转向不同维度。在饱和的评估集上死磕只会浪费精力。

Only keep evals that directly measure a production behavior you care about仅保留直接衡量你所关注生产行为的评估

Every eval applies pressure on your system over time. It's tempting to blindly add hundreds of tests, but this creates an illusion of progress. You end up optimizing for an eval suite that doesn't reflect what matters in production. More evals does not equal better agents. Build targeted evals, and periodically prune the ones that no longer give you signal. For a concrete example of this approach, see How we build evals for Deep Agents.每一项评估都会随时间对系统施加压力。盲目增加数百个测试很诱人，但这只会制造进步的幻觉。你最终会为了一个并不反映生产重要性的评估套件而优化。评估越多并不等于智能体越好。构建有针对性的评估，并定期修剪那些不再提供有效信号的测试。参阅《我们如何为深度智能体构建评估》获取此类方法的具体案例。

Invest in tool interface design and testing, not just prompt optimization投资于工具接口设计与测试，而不只是提示词优化

Tool design eliminates entire classes of agent errors. Anthropic's team noted they spent more time optimizing tools than prompts when building their SWE-bench agent. Test how the model actually uses your tools: try different parameter formats (diffs vs full rewrites, JSON vs. markdown), redesign interfaces to make mistakes harder, and invest in clear documentation with examples. The goal is to make mistakes structurally impossible, not just unlikely. For example, requiring absolute file paths eliminates an entire class of navigation errors.工具设计能消除整类智能体错误。Anthropic 团队指出，在构建 SWE-bench 智能体时，他们花在优化工具上的时间远超提示词。测试模型如何实际使用你的工具：尝试不同的参数格式（差异对比 vs. 全量重写，JSON vs. Markdown），重新设计接口以增加犯错难度，并投资于清晰的文档与示例。目标是让错误在结构上变得不可能，而不仅仅是概率降低。例如，强制使用绝对文件路径能消除整类导航错误。

Distinguish between task failures (agent got it wrong) and evaluation failures (grader got it wrong)区分任务失败与评估失败

Track run status explicitly (complete, error, timeout). A grader that marks a timeout as “incorrect reasoning” pollutes your signal. Separate task failures from evaluation failures to keep your metrics clean.显式跟踪运行状态（完成、错误、超时）。将超时标记为“推理不正确”的评分器会污染你的信号。将任务失败与评估失败分开，以保持指标纯净。

Promote capability evals with consistently high pass rates into your regression suite将通过率持续高企的能力评估提升至回归测试套件

Once you've climbed the hill, protect it. Tasks that used to test “can we do this?” become “can we still do this?”一旦你攀上了高峰，便要守住它。那些曾经测试“我们能做到吗？”的任务，现在应变为“我们还能做到吗？”

Integrate regression evals into your CI/CD pipeline with automated quality gates利用自动化质量门禁，将回归评估集成至 CI/CD 流水线

A typical flow:典型流程：

1. Code or prompt change triggers the pipeline (via git push, PromptHub update, or manual trigger)代码或提示词变更触发流水线（通过 git push、PromptHub 更新或手动触发）
2. Offline evals run unit tests, integration tests, and evaluation against curated datasets using cheap, fast graders离线评估运行单元测试、集成测试，并使用廉价、快速的评分器对策划数据集进行评估
3. Preview deployment goes up if offline evals pass若离线评估通过，则部署预览环境
4. Online evals run against the preview with live data using LLM-as-judge graders在线评估使用 LLM-as-judge 评分器，基于实时数据在预览环境中运行
5. Promote to production only if all quality gates pass, otherwise route failing traces to annotation queues and alert the team仅在所有质量门禁通过后才发布至生产环境，否则将失败追踪路由至标注队列并提醒团队

Use cheap code-based graders in CI for every commit. Reserve expensive LLM-as-judge evaluations for preview/production evaluation. See LangSmith's CI/CD pipeline guide for a full implementation example with GitHub Actions.在 CI 中为每次提交使用廉价的代码评分器。将昂贵的 LLM-as-judge 评估留给预览/生产评估。参阅 LangSmith 的 CI/CD 流水线指南，获取 GitHub Actions 的完整实现案例。

Set up online evaluations for production traffic为生产流量设置在线评估

Safety checks, format validation, quality heuristics. You'll find failure modes in production you never anticipated (see You don't know what your agent will do until it's in production)安全检查、格式验证、质量启发式算法。你会在生产环境中发现从未预料到的失败模式（参阅《在上线前你永远不知道智能体会做什么》）。

Capture user feedback捕获用户反馈

Once your agent is in production, user feedback becomes one of your most valuable signals. Automated evals can only catch the failure modes you already know about. Users will surface the ones you don't: edge cases your dataset missed, outputs that are technically correct but unhelpful, and workflows that break in ways you never anticipated.一旦智能体上线，用户反馈便成为你最宝贵的信号之一。自动化评估只能捕捉你已知的问题。用户会揭示你未曾察觉的失败模式：数据集错失的边缘情况、技术上正确但无用的输出，以及以你从未预料到的方式崩溃的工作流。

Capturing this feedback in a structured way lets you feed it back into your datasets, calibrate your graders against real-world expectations, and prioritize the improvements that actually matter to the people using your agent.以结构化方式捕获这些反馈，能让你将其反哺至数据集，根据现实期望校准评分器，并优先改进对用户真正重要的问题。

Schedule regular manual exploration of production traces beyond automated checks除了自动化检查，定期安排手动探索生产追踪记录

Don't rely solely on automated pass/fail. Periodically explore production traces for unexpected patterns or failure modes your graders don't cover, surprising user behaviors, or opportunities to improve. Our Insights Agent is a great way to do this!不要仅仅依赖自动化通过/失败。定期探索生产追踪记录，以发现评分器未覆盖的意外模式或失败模式、令人惊讶的用户行为，或是改进机会。我们的 Insights Agent 是实现此目标的绝佳工具！

Version your prompts and tool definitions将提示词与工具定义版本化

LangSmith makes it easy to version your prompts. Without this, you can't correlate eval results with specific changes or know which edit caused a regression.LangSmith 让提示词版本化变得简单。若无此功能，你将无法将评估结果与特定变更相关联，也无法知晓是哪次编辑导致了回归。

Ensure production failures feed back into datasets, error analysis, and eval improvements确保生产失败反馈至数据集、错误分析及评估改进中

Production successes and failures should feed back into your datasets, error analysis, and eval improvements. This is the flywheel that makes your agent better over time!生产环境中的成功与失败应反馈至你的数据集、错误分析与评估改进中。这正是让智能体随时间不断进化的飞轮！

You don't need all of these items on day one. Pick the section that matches where you are right now, nail those items, and expand from there. The teams that ship reliable agents aren't the ones with the most sophisticated eval infrastructure - they're the ones who started evaluating early and never stopped iterating.你无需第一天就集齐所有条目。挑选与你当前状态匹配的章节，扎实完成，再行扩展。交付可靠智能体的团队，并非拥有最复杂的评估基础设施，而是那些尽早开始评估且从未停止迭代的团队。