Phase 1: Eval health audit阶段一：评估健康审计

Before trusting an eval to tell you which model or configuration is better, check that the eval itself is sound. A broken eval produces confident-looking numbers that point in the wrong direction, and a sweep over a broken eval just multiplies the misdirection. In practice, the most surprising eval results usually turn out to be bugs in the eval rather than facts about the model, so an hour of auditing upfront routinely saves days of chasing phantom differences.在信任某个评估能告诉你哪个模型或配置更优之前，先检查评估本身是否可靠。一个错误的评估会产生看似可信的数字，却指向错误的方向，而在错误的评估上进行参数扫描只会放大误导。实践中，最令人惊讶的评估结果通常源于评估本身的漏洞，而非模型的事实，因此提前花一小时审计，通常能避免数天追逐虚假差异的麻烦。

The checks below are grouped into task design (are the questions right?), harness design (is the scaffolding right?), metrics hygiene (are cost and latency being measured correctly?), and grader design (is the scoring right?). They are written as direct instructions to Claude: for each, look at the eval's actual code, config, and data, not just its README. The final section, Reporting findings to the user, covers how to communicate what you find; the checks are declarative, but the report to the human should be framed as observations and suggestions, since the eval's author almost always has context that justifies choices an outsider would flag.以下检查分为任务设计（问题是否正确？）、框架设计（脚手架是否正确？）、指标卫生（成本和延迟是否被正确测量？）以及评分器设计（评分是否正确？）。它们以直接指令的形式写给Claude：对于每项检查，要查看评估的实际代码、配置和数据，而不仅仅是其README。最后一部分“向用户报告发现”涵盖了如何沟通你的发现；检查是声明性的，但给人类的报告应以观察和建议的形式呈现，因为评估的作者通常拥有能解释外人标记的选择的背景信息。

Before auditing further, consider running the eval once end-to-end on the cheapest available model, or search for previous trials results. A surprising number of eval-quality discussions turn out to be about code that does not currently run.在进一步审计之前，考虑在可用的最便宜模型上端到端运行一次评估，或搜索之前的试验结果。令人惊讶的是，许多关于评估质量的讨论最终都围绕着当前无法运行的代码。

1. Task design1. 任务设计

These checks concern the examples themselves: what is being asked, what counts as correct, and whether the set as a whole can distinguish between the systems being compared.这些检查涉及示例本身：询问的内容、什么算作正确，以及整个集合能否区分所比较的系统。

Auditing task sets at scale大规模审计任务集

The harness and grader are code you can read end to end; the task set may be hundreds or thousands of items you cannot. Do not try to read every task inline. Work in three tiers:框架和评分器是你可以从头到尾阅读的代码；任务集可能是你无法逐条阅读的数百或数千个项目。不要试图内联阅读每个任务。分三个层级工作：

Tier 1: programmatic checks over the full set. Consider writing a short script that loads every task and reports: exact- and near-duplicate rate; label or category balance; prompt-length and expected-answer-length distributions (when applicable); schema validity and missing-field counts; and any obviously malformed rows. The storage format varies per eval, so inspect two or three rows first and code to whatever schema you find. These checks are cheap, exhaustive, and catch skew, duplicates, truncation, and broken rows regardless of how large the set is.第一层：对整个集合进行程序化检查。考虑编写一个简短的脚本，加载每个任务并报告：精确重复率和近似重复率；标签或类别平衡；提示长度和预期答案长度分布（如适用）；模式有效性和缺失字段计数；以及任何明显格式错误的行。存储格式因评估而异，因此先检查两到三行，然后根据你找到的任何模式编写代码。这些检查成本低、全面，并且无论集合有多大，都能捕获偏差、重复、截断和损坏的行。

Tier 2: stratified sample for a close read. Draw roughly twenty to fifty tasks, stratified across category or difficulty labels if they exist, otherwise uniformly at random, and apply the per-task checks below to those. Recommend that the user read a handful themselves as well; a second pair of human eyes on the raw tasks catches things no checklist does.第2层：用于细读的分层样本。抽取大约二十到五十个任务，如果存在类别或难度标签则按此分层，否则均匀随机抽取，并对这些任务应用下面的逐项检查。建议用户自己也阅读一些；对原始任务进行人工复核能发现任何检查清单都无法捕捉的问题。

Tier 3: per-task LLM auditor. For sets beyond a few hundred items, run one isolated model call per task with a tight audit prompt, collect a structured verdict from each, and aggregate. This is the same approach behind public re-annotation efforts such as MMLU-Redux and SWE-bench Verified, automated. Ask the user before running it, the cost is roughly N cheap-model calls, usually small next to the eval's own inference cost, but it is their budget to spend. Offer it explicitly: "I can run a per-task auditor over all N tasks, estimated cost ~$X. Want me to?"第3层：逐项LLM审计器。对于超过几百个项的数据集，对每个任务单独调用一次模型，使用严格的审计提示，收集每个任务的结构化判定，然后汇总。这与MMLU-Redux和SWE-bench Verified等公开重新标注工作背后的自动化方法相同。运行前询问用户，成本大约是N次廉价模型调用，通常远小于评估本身的推理成本，但这是用户的预算。明确提供选项：“我可以对所有N个任务运行逐项审计器，预估成本约$X。需要我运行吗？”

A per-task auditor prompt that works well (adapt the field names to the eval's schema):一个效果良好的逐项审计提示（根据评估的架构调整字段名称）：

You are auditing a single task from an evaluation suite. Given the task prompt, the reference answer, and a description of how the grader decides pass/fail, flag any of the following issues. Be conservative, only flag when you are reasonably confident.

TASK PROMPT:
{prompt}

REFERENCE ANSWER:
{gold}

GRADER BEHAVIOUR:
{grader_description}

For each issue below, answer yes/no and give a one-line reason if yes:
- ambiguous: could two careful experts reasonably disagree on the correct answer?
- gold_suspect: does the reference answer look wrong, incomplete, or arguable?
- answerable_from_memory: could a well-read model answer this from general knowledge without doing the intended work?
- grader_too_strict: are there clearly correct answers the grader as described would reject (format, phrasing, precision)?
- grader_too_lenient: are there clearly wrong answers the grader as described would accept?
- trivially_cheatable: is there a shortcut that satisfies the grader without solving the task?
- other: anything else that would make this task's result misleading.

Return JSON: {"task_id": "...", "flags": {"ambiguous": {"flagged": bool, "reason": "..."}, ...}, "overall": "ok" | "review" | "broken"}

After the run, cluster by flag type, surface the top issues with example task IDs, and feed the findings into the report (§5).运行后，按标记类型聚类，展示主要问题及示例任务ID，并将发现结果纳入报告（§5）。

The checks that follow are the per-task checks referenced by tier 2, apply them to the sampled tasks, not the full set.以下检查是第2层引用的逐项检查，应用于抽样任务，而非全部任务。

• Unambiguous success criteria. Read three or four tasks and their expected answers. For each, ask: would two independent domain experts, shown the same model output, agree on pass vs. fail? If the criteria admit reasonable disagreement ("write a good summary," "respond helpfully"), scores reflect grader opinion as much as model capability. Flag tasks whose pass condition is not crisply decidable. Note the dual failure mode: a task can be under-specified (no clear success criterion, or a required output, a filename, a format, a target, left unstated) or over-specified (the prompt is effectively a step-by-step recipe, leaving nothing for the model to decide).明确的成功标准。阅读三到四个任务及其预期答案。对每个任务提问：两位独立的领域专家在看到相同的模型输出时，是否会对通过/失败达成一致？如果标准允许合理的分歧（如“写一篇好的摘要”、“提供有帮助的回复”），那么分数反映的是评分者的判断而非模型能力。标记那些通过条件不清晰可判的任务。注意两种失败模式：任务可能定义不足（没有明确的成功标准，或未说明所需的输出、文件名、格式、目标等），也可能过度定义（提示实际上是一个逐步的配方，模型无需做任何决定）。

• Reference solution exists. Check whether each task ships with at least one worked solution or gold answer that actually passes the grader. A task with no known passing answer may be unsolvable as posed, a 0% pass rate across all models is more often a broken task than a genuinely hard one. Spot-check by running the reference solution through the grader.存在参考解答。检查每个任务是否至少附带一个能实际通过评分器的可行解答或标准答案。没有已知通过答案的任务可能无法按原样解决，所有模型通过率为0%更可能是任务本身有问题，而非真正困难。通过运行参考解答通过评分器进行抽查。

• Ground-truth labels are correct. Sample ten or so tasks and independently re-derive the expected answers. Public benchmarks routinely ship with a non-trivial fraction of wrong or arguable labels, community re-annotations of widely used reasoning and coding benchmarks have repeatedly found meaningful label error rates in the originals. Wrong labels put a ceiling on measurable accuracy that has nothing to do with the model.真实标签是正确的。抽取大约十个任务，独立重新推导预期答案。公共基准测试通常包含相当比例的错误或有争议的标签，社区对广泛使用的推理和编码基准的重新标注反复发现原始数据中存在有意义的标签错误率。错误的标签给可测量的准确率设定了上限，而这与模型无关。

• No annotation artifacts. Check whether a trivial baseline could score well by exploiting surface patterns rather than solving the task: can the answer be guessed from the question's length, its keywords, or the ordering of multiple-choice options alone? Natural-language inference datasets have famously leaked the label into the wording of one side of the pair, letting a model that never saw the other side score far above chance. If a no-op or majority-class baseline scores well above chance, the eval is at least partly measuring the artifact.无标注伪影。检查一个简单的基线是否可以通过利用表面模式而非解决任务来获得高分：答案能否仅从问题的长度、关键词或多选题选项的顺序中猜出？自然语言推理数据集曾因将标签泄露到配对文本一侧的措辞中而臭名昭著，使得从未见过另一侧的模型得分远高于随机水平。如果无操作或多数类基线的得分远高于随机水平，那么评估至少部分是在测量伪影。

• Label leakage in the prompt. Check whether the expected answer, or a near-paraphrase of it, appears anywhere the model can see, in the prompt, the few-shot examples, the system message, a tool description, or a file the agent can read. Especially common in few-shot setups assembled by copy-pasting from the golden set.提示中的标签泄露。检查预期答案或其近义改写是否出现在模型可见的任何地方：提示、少样本示例、系统消息、工具描述或代理可读取的文件中。这在通过从黄金集复制粘贴构建的少样本设置中尤其常见。

• Answerable from memory. Check whether tasks about real, named entities (a specific person, repository, paper, or event) can be answered from a model's parametric memory even though the intent is to test a skill like retrieval or tool use. If the goal is to measure whether the model can do the work, the subjects need to be obscure enough, or synthetic enough, that recall alone does not carry the task.可从记忆中回答。检查关于真实命名实体（特定人物、仓库、论文或事件）的任务是否可以从模型的参数化记忆中回答，即使意图是测试检索或工具使用等技能。如果目标是衡量模型是否能完成工作，那么主题需要足够晦涩或足够合成，使得仅凭回忆无法完成任务。

• Difficulty comes from the problem, not the prompt. Check whether tasks that look hard are really just worded obscurely. If the underlying problem is easy once decoded, the score measures prompt-deciphering rather than the target skill. When difficulty lives mostly in the phrasing, suggest a rewrite that states the problem plainly and lets the problem itself be hard.难度来自问题本身，而非提示。检查看似困难的任务是否只是措辞晦涩。如果底层问题一旦解码就很简单，那么分数衡量的是提示解读能力而非目标技能。当难度主要存在于措辞中时，建议重写以明确陈述问题，让问题本身变得困难。

• Agentic tasks: symptom, not investigation. For tasks that ask an agent to diagnose or fix something, check how much of the investigation is handed over in the prompt. If the task description already includes the log line, the failing test name, or the relevant file, the eval measures whether the model can read a hint, not whether it can find one. Prefer giving the agent only what a user would plausibly report and letting it fetch the rest.代理任务：症状而非调查。对于要求代理诊断或修复某些问题的任务，检查提示中提供了多少调查信息。如果任务描述已经包含日志行、失败的测试名称或相关文件，那么评估衡量的是模型是否能读取提示，而非是否能找到提示。最好只给代理用户可能合理报告的内容，让其自行获取其余信息。

• Realistic distribution and interaction shape. Compare a handful of tasks to the user's actual production traffic or intended use case. Synthetic toy tasks often fail to predict behaviour on messy real inputs, and vice versa. Also check that the shape of the interaction matches: a single-turn question-answering eval will not capture gains (or regressions) that only appear in long multi-turn or agentic settings, and an agentic eval will not isolate single-step reasoning quality. Name any obvious divergence upfront so readers can calibrate how far the results will transfer.真实的分布和交互形态。将少量任务与用户的实际生产流量或预期用例进行比较。合成的玩具任务通常无法预测在混乱的真实输入上的行为，反之亦然。同时检查交互形态是否匹配：单轮问答评估无法捕捉仅在长多轮或智能体设置中出现的收益（或退化），而智能体评估也无法隔离单步推理质量。提前指出任何明显的差异，以便读者校准结果的可迁移程度。

• Difficulty headroom. If results exist, look at the score spread. If every strong model already scores ~95%+, the eval cannot discriminate at the top and any sweep will differentiate mainly on cost, still useful, but worth saying in advance. If every model scores ~0%, the eval reveals nothing about relative capability and more often than not has a task or grader bug.难度空间。如果存在结果，查看分数分布。如果每个强模型都已达到约95%以上，则该评估无法区分顶尖模型，任何扫描主要区分成本，仍然有用，但值得提前说明。如果每个模型得分约0%，则该评估无法揭示相对能力，而且通常存在任务或评分器错误。

• Saturated evals and what they end up measuring. Once an eval is near its ceiling, the remaining variance is often dominated by format quirks, tie-breaking in the grader, or mild reward-hacking rather than genuine capability differences. If a set has been near-saturated for a while, flag that the last few points may no longer measure what the eval was built for, and suggest adding harder items.饱和评估及其最终衡量的内容。一旦评估接近天花板，剩余方差通常由格式怪癖、评分器中的平局打破或轻微奖励黑客行为主导，而非真正的能力差异。如果一组评估已接近饱和一段时间，请指出最后几点可能不再衡量评估原本的目标，并建议添加更难的项目。

• Class balance. For classification-style evals, check the distribution of expected labels. A heavily skewed set lets a constant-prediction baseline look deceptively strong. Recommend reporting the majority-class baseline alongside model scores.类别平衡。对于分类风格的评估，检查预期标签的分布。高度偏斜的集合会让恒定预测基线看起来异常强大。建议在模型分数旁边报告多数类基线。

• Both-directions coverage. For evals that test a decision or behaviour, check that both the positive and the negative case are represented. An eval for "does the agent search when it should" also needs "does the agent not search when it shouldn't"; otherwise a model that always searches scores perfectly. One-sided evals produce one-sided optimisation. Apply the same check to refusals, tool use, escalation, and similar two-sided behaviours.双向覆盖。对于测试决策或行为的评估，检查正面和负面情况是否都有代表。一个评估“智能体是否在应该搜索时搜索”也需要“智能体是否在不应搜索时不搜索”；否则，总是搜索的模型会得满分。单向评估导致单向优化。对拒绝、工具使用、升级和类似的双向行为应用相同的检查。

• One capability per task. Check whether a failing task tells you what failed. A task that requires retrieval and reasoning and formatting to pass will show 0 whenever any one of those breaks, which makes the score undiagnostic. Fine for a headline end-to-end number; flag it when the user wants to know why systems differ.每个任务一种能力。检查失败的任务是否能告诉你哪里失败了。一个需要检索、推理和格式化才能通过的任务，在任何一个环节出错时都会显示0，这使得分数无法诊断。对于端到端的总体数字来说没问题；但当用户想知道系统为何不同时，请指出这一点。

• Inverted items as a smoke test. When results are available, look for individual items where a clearly weaker system outscores a clearly stronger one. These items are far more often revealing a task or grader bug (an ambiguous label, an over-rigid match, a leaked hint) than a genuine capability inversion, and they make a good starting point for where to look closely.将反转项作为冒烟测试。当结果可用时，寻找那些明显较弱的系统得分高于明显较强的系统的单个项。这些项往往更可能揭示任务或评分器错误（模糊的标签、过于严格的匹配、泄露的提示），而非真正的能力反转，它们是仔细检查的良好起点。

• Human baseline. Ask whether anyone has measured what a competent human scores on this set. Without that anchor, it is hard to say whether 60% is impressive or embarrassing, or whether the remaining 40% reflect model limitations or task ambiguity.人类基线。询问是否有人测量过有能力的人类在这组任务上的得分。没有这个锚点，很难说60%是令人印象深刻还是尴尬，或者剩下的40%反映了模型限制还是任务模糊性。

• Staleness. If tasks reference live facts (prices, dates, API responses, current events, library versions), check when the golden answers were last verified. Correct answers drift; a model that gives the currently correct answer will be marked wrong against a stale key.过时性。如果任务引用实时事实（价格、日期、API响应、当前事件、库版本），请检查黄金答案上次验证的时间。正确答案会漂移；给出当前正确答案的模型会因过时的答案而被标记为错误。

• Dataset size vs. effect size. Count the examples. Twenty to fifty realistic, fast-to-run examples are usually enough to start, early on, the differences you care about are large relative to noise. However, as the product matures more and more tasks should be added. For stable comparisons between closely matched systems, ~50+ cases per slice of interest with confidence intervals reported alongside the point estimate is a reasonable target. When the set is smaller, run multiple trials per example and report the spread.数据集大小与效应量。统计示例数量。二十到五十个逼真、快速运行的示例通常足以开始，早期你关心的差异相对于噪声较大。然而，随着产品成熟，应添加更多任务。对于紧密匹配系统之间的稳定比较，每个感兴趣切片约50+个案例，并报告点估计旁边的置信区间，是一个合理的目标。当集合较小时，对每个示例运行多次试验并报告分布范围。

• For generated tasks: fix the generator, not the filter. When tasks are produced by a pipeline (templated, synthetically generated, or model-written), problems found in the output are usually symptoms of something upstream. Patching individual bad items or adding a post-hoc filter tends to leave siblings of the same bug in place. Suggest adjusting the generator and regenerating.对于生成的任务：修复生成器，而非过滤器。当任务由管道生成（模板化、合成生成或模型编写）时，输出中发现的问题通常是上游问题的症状。修补单个不良项或添加事后过滤器往往会留下相同错误的同类项。建议调整生成器并重新生成。

2. Harness design2. 测试框架设计

These checks concern the code that sets up, runs, and records each trial, everything around the model call. The central failure mode to watch for throughout this section is conflation: any time a non-model artifact (an infra error, a truncated response, a broken tool, a retry delay) lands in the same column as a genuine model result, the eval will attribute to the model something that belongs to the plumbing.这些检查涉及设置、运行和记录每次试验的代码，即模型调用周围的一切。本节中要关注的核心失败模式是混淆：任何时候非模型工件（基础设施错误、截断的响应、损坏的工具、重试延迟）与真实模型结果出现在同一列中，评估都会将属于管道的问题归因于模型。

• Clean, isolated state per trial. Read the setup and teardown code. Check that each trial starts from a fresh environment: no files, database rows, git history, environment variables, or cached results left over from a previous trial or a previous task. Shared state lets one task's side effects leak into another's score, lets an agent read hints left behind by an earlier run, and makes results depend on execution order.每次试验的干净、隔离状态。阅读设置和拆卸代码。检查每次试验是否从全新的环境开始：没有文件、数据库行、git历史、环境变量或缓存结果遗留自之前的试验或任务。共享状态会让一个任务的副作用泄露到另一个任务的评分中，让智能体读取早期运行留下的提示，并使结果依赖于执行顺序。

• Environment is complete and functional. Check that the environment the agent is placed in actually has what the task requires: dependencies installed, documentation present, services reachable, fixtures populated. A task that fails for every model because a package is missing or a fixture file was never committed is measuring the environment, not the model. Distinguish this from deliberate obstacles that are part of the task.环境完整且功能正常。检查智能体所处的环境是否确实具备任务所需的条件：已安装依赖项、文档存在、服务可访问、测试数据已填充。如果因为缺少某个包或测试数据文件从未提交而导致每个模型都失败，那么衡量的是环境而非模型。将此与作为任务一部分的故意障碍区分开来。

• Deterministic setup. Look for sources of nondeterminism outside the model: unseeded randomness, iteration over unordered sets or dicts where order matters, hash-randomised keys, timestamp-dependent paths, unordered directory listings, stochastic simulators without a fixed seed. These make scores vary run-to-run for reasons unrelated to the system under test. Recommend pinning a seed and sorting anything whose order reaches the model or the grader. This applies, for example, to adjacent ML systems that the agent might interact with. In the case of the LLM, it is good practice to set sampling parameters (e.g., temperature) to the same value you intend to use in production.确定性设置。寻找模型外部的非确定性来源：未设定种子的随机性、对顺序重要的无序集合或字典的迭代、哈希随机化键、依赖时间戳的路径、无序目录列表、没有固定种子的随机模拟器。这些会导致分数因与测试系统无关的原因而每次运行不同。建议固定种子并对任何顺序会影响模型或评分器的内容进行排序。例如，这适用于智能体可能与之交互的相邻机器学习系统。对于LLM，最佳实践是将采样参数（如温度）设置为与生产环境中打算使用的相同值。

• Infra failures distinguished from model failures. Check how the harness handles a timeout, an out-of-memory kill, an API or rate-limit error, a parsing error on the model's output, a response truncated by the output-token cap, a tool that threw an exception, a sandbox that crashed, or a grader that itself failed to run. If any of these are silently scored as 0 (or as "pass," depending on the default) and mixed in with genuine model answers, the headline number is contaminated. The result schema should carry a separate status or error field, distinct from passed=False, so infra failures can be filtered, counted, and retried separately, and so the pass rate reflects only trials where the model actually produced a scorable answer.基础设施故障与模型故障区分。检查测试框架如何处理超时、内存不足、API或速率限制错误、模型输出解析错误、输出令牌上限导致的响应截断、抛出异常的工具、崩溃的沙箱或自身运行失败的评分器。如果其中任何一个被静默地计为0分（或根据默认值计为“通过”）并与真正的模型答案混合在一起，那么总体数字就会被污染。结果模式应包含单独的状态或错误字段，与passed=False区分开，以便基础设施故障可以被单独过滤、计数和重试，从而使通过率仅反映模型实际产生可评分答案的试验。

• "No answer" is not the same as "negative answer." A specific and common conflation: check whether the harness and grader distinguish between the model asserting a negative ("there are no vulnerabilities in this code," "no matching records found") and the model failing to produce an answer (an empty response, a crash, a truncated stream, an unparseable output). If both paths land on the same label, an infra failure masquerades as a substantive model claim, and a model that errors out on every input can score identically to one that carefully analysed each case and correctly found nothing. Look for this especially in detection, classification, and retrieval evals where "none" is a valid answer."无答案"与"否定答案"不同。一个具体且常见的混淆：检查测试框架和评分者是否区分了模型断言否定（"此代码中没有漏洞"，"未找到匹配记录"）和模型未能生成答案（空响应、崩溃、截断流、不可解析输出）。如果两条路径都落在同一标签上，基础设施故障就会伪装成实质性的模型声明，而一个在每个输入上都出错的模型可能与一个仔细分析每个案例并正确发现没有问题的模型得分相同。在检测、分类和检索评估中尤其要注意这一点，因为"无"是一个有效答案。

• Scaffold limitations are separated from model limitations. When an agent scores poorly, check how much of that is the model and how much is the scaffold it was run in. A missing tool, an overly tight step or turn budget, a retry policy that gives up early, or a prompt template that drops part of the context can all look like capability gaps from the outside. Where practical, vary the scaffold while holding the model fixed (or vice versa) to attribute results to the right layer.脚手架限制与模型限制分开。当智能体得分较低时，检查其中多少是模型的问题，多少是运行它的脚手架的问题。缺失的工具、过于严格的步骤或轮次预算、过早放弃的重试策略、或丢弃部分上下文的提示模板，都可能从外部看起来像是能力差距。在可行的情况下，固定模型并改变脚手架（或反之），以将结果归因于正确的层面。

• Token and context limits won't clip any task. Find the longest prompt and the longest plausible correct answer in the set and compare against the configured context window and output-token cap. A model that runs out of output budget mid-answer is marked wrong for a harness-config reason, not a capability one, and the truncation is easy to misread as the model choosing to stop. Especially common in agentic evals where the trajectory grows over many turns.令牌和上下文限制不会截断任何任务。找出集合中最长的提示和最长的合理正确答案，并与配置的上下文窗口和输出令牌上限进行比较。一个在回答中途耗尽输出预算的模型会因测试框架配置原因而被标记为错误，而非能力原因，并且截断很容易被误读为模型选择停止。在智能体评估中尤其常见，因为轨迹会随着多轮交互而增长。

• Transient errors are retried. Check whether the harness retries on rate-limit and overloaded responses. Unretried transient errors show up as spurious failures and can make one model, or one time of day, look systematically worse than another. Record the retry count per trial so retries can be excluded from latency metrics (see §3).瞬态错误会被重试。检查测试框架是否在遇到速率限制和过载响应时进行重试。未重试的瞬态错误会表现为虚假失败，并可能使某个模型或某个时间段看起来系统性地更差。记录每次试验的重试次数，以便从延迟指标中排除重试（见§3）。

• Eval config matches production config. Compare the system prompt, tool definitions, model version, temperature, and any scaffolding in the eval against what actually runs in production. A very common failure mode is an eval that measures a different setup from the one being shipped, the eval says "fine," production says otherwise. Diff them explicitly.评估配置与生产配置匹配。比较评估中的系统提示、工具定义、模型版本、温度以及任何脚手架与实际生产运行中的配置。一个非常常见的失败模式是评估测量的是与部署版本不同的设置，评估说"没问题"，生产却说有问题。明确地对它们进行差异比较。

• Model and knobs are exposed, not hard-coded. Locate where the model ID is set. If it is buried in the call site rather than passed in, the Phase 2 sweep will need a refactor first. Surface model, and ideally temperature, reasoning/thinking settings, and output-token cap, as top-level arguments to whatever "run the eval" entrypoint exists.模型和参数是暴露的，而非硬编码。找到模型ID设置的位置。如果它隐藏在调用点中而不是作为参数传入，那么第二阶段扫描将需要先进行重构。将模型、理想情况下还有温度、推理/思考设置以及输出令牌上限，作为顶层参数暴露给任何"运行评估"的入口点。

• Full per-task trajectories are saved. Check that the raw transcript, every message, every tool call and result, every error, is persisted per task, not just the final score. When a result is surprising, the transcript is what tells you whether it is a fact about the model or a bug in the eval; without it, a score is just a number. The grader's own inputs and outputs should be saved alongside pass/fail so failures can be spot-checked without re-running. This is the single highest-leverage habit for making an eval debuggable.保存每个任务的完整轨迹。检查原始记录、每条消息、每次工具调用及其结果、每次错误是否按任务持久化，而不仅仅是最终分数。当结果令人惊讶时，记录会告诉你这是关于模型的事实还是评估中的错误；没有它，分数只是一个数字。评分者的输入和输出应与通过/失败一起保存，以便无需重新运行即可抽查失败情况。这是使评估可调试的最高杠杆习惯。

• Multiple trials with variance reported. Check whether the harness supports running the same task set several times (three or more is common) and reporting the spread. A single run gives a point estimate with no error bar; differences smaller than the run-to-run spread are not meaningful.多次试验并报告方差。检查测试框架是否支持多次运行同一任务集（通常三次或更多）并报告离散度。单次运行给出点估计，没有误差条；小于运行间离散度的差异没有意义。

• Statistical power for the question being asked. Given the dataset size and per-trial variance, estimate the smallest pass-rate difference the eval can reliably detect. As a rough heuristic, a difference needs to be several multiples of the run-to-run standard deviation before it is worth acting on. If the user cares about 2-point differences and the eval's noise floor is 5 points, say so before any sweep.针对所提问题的统计功效。根据数据集大小和每次试验的方差，估计评估能够可靠检测的最小通过率差异。作为粗略启发，差异需要是运行间标准差的数倍才值得采取行动。如果用户关心2个百分点的差异，而评估的噪声基底是5个百分点，则在任何扫描前说明这一点。

• Reproducible over time. Check that dependency versions are pinned, the task set is versioned (so "v3 scored X" is a stable claim), and the environment is containerised or otherwise specified. Unpinned dependencies mean today's number and next month's number are not comparable even on the same model.随时间可重现。检查依赖版本是否固定，任务集是否版本化（因此“v3得分X”是一个稳定的声明），以及环境是否容器化或以其他方式指定。未固定的依赖意味着今天的数字和下个月的数字即使在相同模型上也不可比。

• Harness tested on known-good and known-bad. Check whether the harness has been run on (a) a model or oracle that should score near 100%, and (b) a random or null baseline that should score near chance. If the oracle doesn't pass, the harness or grader is broken; if random doesn't fail, the grader is too lenient.测试框架在已知好和已知差的情况下测试。检查测试框架是否在（a）应得分接近100%的模型或预言机上运行，以及（b）应得分接近随机或空基线的模型上运行。如果预言机未通过，则测试框架或评分器有缺陷；如果随机未失败，则评分器过于宽松。

3. Metrics hygiene3. 指标卫生

Pass rate alone rarely answers the question the user actually has, which is usually some form of "what is the best quality I can get for a given cost or latency?" Check that the harness captures the per-trial metrics below, and, just as importantly, that each one is computed in a way that reflects the model under test rather than the test rig around it.仅通过率很少能回答用户实际的问题，通常是一些形式如“在给定成本或延迟下，我能获得的最佳质量是什么？”检查测试框架是否捕获了以下每次试验的指标，并且同样重要的是，每个指标的计算方式反映的是被测模型而非其周围的测试装置。

• Token accounting from the API, not estimated. Check that input tokens, output tokens, and where the API exposes them, cache-read and cache-write tokens, are recorded per trial from the API response's usage block. Estimating token counts from string length is routinely off by enough to reverse a cost comparison.来自API的令牌计数，而非估算。检查输入令牌、输出令牌以及API暴露的缓存读取和缓存写入令牌是否根据API响应的使用块按试验记录。根据字符串长度估算令牌计数通常偏差大到足以逆转成本比较。

• Cost computed from recorded tokens. Check that cost per trial is derived from the recorded token counts and the relevant per-token prices (including any separate rates for cached input, reasoning/thinking tokens, or tool use), rather than from a flat assumed rate. If the grader is itself an LLM, record its cost separately from the cost of the system under test so it does not dampen relative differences between sweep arms.成本根据记录的令牌计算。检查每次试验的成本是否来自记录的令牌计数和相关每令牌价格（包括缓存输入、推理/思考令牌或工具使用的任何单独费率），而不是来自固定的假设费率。如果评分器本身是一个LLM，则将其成本与受测系统的成本分开记录，以免削弱不同扫描臂之间的相对差异。

• Prompt-cache hit rate is tracked and comparable. If any arm of the comparison benefits from prompt caching, record cache-read tokens per trial and report the cache-hit rate per arm. When one configuration gets warm-cache pricing and latency while another runs cold, the cost and time-to-first-token differences are partly an artifact of run order or cache configuration, not model quality. Flag any comparison where cache-hit rates differ materially between arms.提示缓存命中率被跟踪且可比较。如果比较的任何臂受益于提示缓存，则记录每次试验的缓存读取令牌，并报告每个臂的缓存命中率。当一个配置获得热缓存定价和延迟，而另一个配置冷运行时，成本和首次令牌时间的差异部分是由于运行顺序或缓存配置造成的，而不是模型质量。标记任何缓存命中率在臂之间差异显著的比较。

• TTFT and TTLT measured against the right boundaries. Check that time-to-first-token is measured from when the final, successful request is sent to when its first token arrives, and time-to-last-token from that same send to when its last token arrives. Both should exclude: client-side retry loops and the backoff sleeps between them; time spent waiting in a local queue or semaphore before the request was actually sent; connection setup that would be amortised away in a real deployment; and any post-processing after the stream closes. If a request was retried three times with exponential backoff, the thirty seconds of sleeping is a fact about the test rig that day, not about the model, record it as wall-clock overhead, but keep it out of the model-latency column.TTFT和TTLT根据正确的边界测量。检查首次令牌时间是从最终成功请求发送到其第一个令牌到达时测量的，而最后令牌时间是从同一发送到其最后一个令牌到达时测量的。两者都应排除：客户端重试循环及其之间的退避休眠；在请求实际发送前在本地队列或信号量中等待的时间；在实际部署中会被摊销的连接建立；以及流关闭后的任何后处理。如果请求以指数退避重试了三次，那么三十秒的休眠是测试平台当天的事实，而不是模型的事实，将其记录为挂钟开销，但将其排除在模型延迟列之外。

• Output-tokens-per-second computed from clean interval. Check that OTPS is computed as output tokens divided by (TTLT − TTFT) for the successful attempt, i.e., the decode phase only. Dividing by total wall-clock, including prefill, retries, and queueing, conflates throughput with everything else and will systematically penalise whichever arm happened to hit more transient errors.每秒输出令牌数根据干净间隔计算。检查OTPS是否计算为输出令牌除以（TTLT − TTFT）用于成功尝试，即仅解码阶段。除以包括预填充、重试和排队的总挂钟时间，会将吞吐量与所有其他因素混为一谈，并系统性地惩罚恰好遇到更多瞬时错误的臂。

• Retries and errors recorded alongside, not inside, the core metrics. Check that the per-trial record includes the number of attempts, the total wall-clock including retries and backoff, and the reason for each failed attempt, as separate fields from the clean TTFT/TTLT/OTPS above. Both views are useful, the clean metrics for comparing models, the wall-clock for understanding what a real user would experience, but collapsing them into one number loses the ability to tell which is which.重试和错误与核心指标并列记录，而非包含在内。检查每次试验的记录是否包括尝试次数、包括重试和退避的总挂钟时间以及每次失败尝试的原因，作为与上述干净TTFT/TTLT/OTPS分开的字段。两种视图都有用：干净指标用于比较模型，挂钟时间用于了解真实用户的体验，但将它们合并为一个数字会失去区分它们的能力。

• Per-turn and per-call breakdown for agentic evals. For multi-turn or tool-using tasks, check that token counts, cost, and timing are recorded per model call and per tool call, not just as a single total for the episode. Otherwise a slow tool or an expensive retrieval step is indistinguishable from a slow or expensive model.针对代理评估的每轮和每次调用细分。对于多轮或使用工具的任务，检查令牌计数、成本和计时是否按每次模型调用和每次工具调用记录，而不仅仅是整个回合的总计。否则，慢速工具或昂贵的检索步骤与慢速或昂贵的模型无法区分。

• Metrics reported alongside quality. Check that whatever report the eval produces puts pass rate, cost per success, and latency side by side (per arm), so quality-vs-cost and quality-vs-latency tradeoffs are visible rather than implied.指标与质量一起报告。检查评估生成的任何报告是否将通过率、每次成功成本和延迟并列显示（每个臂），以便质量与成本以及质量与延迟的权衡可见而非隐含。

4. Grader design4. 评分器设计

These checks concern the function that turns a model output into a score, whether that is an exact match, a unit test, or an LLM judge.这些检查关注的是将模型输出转化为分数的函数，无论是精确匹配、单元测试还是LLM评判。

• Prompt-grader agreement. Read a few prompt/grader pairs side by side and check that what the prompt asks for is what the grader rewards. A common drift: the prompt says "reach at least threshold X" but the grader only passes on strictly exceeding X; or the prompt asks for an explanation but the grader only checks the final number. Beyond mismeasurement, this penalises models that follow the instructions and rewards models that ignore them, which is the opposite of what most evals intend to encourage.提示与评分者的一致性。并排阅读几组提示/评分者，检查提示要求的内容是否与评分者奖励的内容一致。常见的偏差：提示说“达到至少阈值X”，但评分者只通过严格超过X的情况；或者提示要求解释，但评分者只检查最终数字。除了测量错误外，这还会惩罚遵循指令的模型，奖励忽略指令的模型，这与大多数评估希望鼓励的方向相反。

• Grades outcomes, not paths. Read the grader and check whether it rewards reaching the right answer or taking a particular route. A grader that requires an exact tool-call sequence, a specific phrasing, or a particular intermediate step will fail a model that solved the problem a different but valid way. Prefer checking that the answer is correct and appropriately grounded (the right source was consulted, say) without dictating the full trajectory.评分结果，而非路径。阅读评分者，检查它是奖励达到正确答案，还是奖励采取特定路径。要求精确工具调用序列、特定措辞或特定中间步骤的评分者，会淘汰那些以不同但有效方式解决问题的模型。更倾向于检查答案是否正确且适当有据（例如，查阅了正确的来源），而不规定完整的轨迹。

• Not overly rigid. For exact-match or substring graders, check whether trivial surface differences cause false negatives: whitespace, casing, boolean or numeric formatting (4 vs 4.0, 96.12 vs 96.124991…), markdown fences, units, thousands separators, or a full sentence wrapped around the answer. Re-audits of public benchmarks have moved reported accuracies by tens of points once grading rigidity was relaxed. Recommend normalising both sides before comparing, or accepting any of a small set of equivalent forms.不过于僵化。对于精确匹配或子串评分者，检查琐碎的表面差异是否导致假阴性：空白、大小写、布尔值或数字格式（4 vs 4.0，96.12 vs 96.124991…）、Markdown分隔符、单位、千位分隔符，或围绕答案的完整句子。对公共基准的重新审计显示，一旦放宽评分刚性，报告准确率会移动数十个百分点。建议在比较前对双方进行标准化，或接受一小套等效形式中的任何一种。

• Not too lenient. Conversely, for test-based or substring graders, check whether the checks are thorough enough to actually catch wrong answers. A function that passes three weak unit tests may still be wrong on every edge case, when community efforts added more tests to popular code-generation benchmarks, several models' scores dropped by double digits because subtly broken solutions had been slipping through. Write a deliberately wrong-but-plausible answer and confirm the grader fails it.不过于宽松。相反，对于基于测试或子串的评分者，检查检查是否足够彻底以真正捕获错误答案。通过三个弱单元测试的函数可能仍然在每个边缘情况下出错，当社区努力为流行的代码生成基准添加更多测试时，几个模型的分数下降了两位数，因为微妙错误的解决方案一直在溜过。编写一个故意错误但看似合理的答案，并确认评分者未能通过它。

• Cheat-resistant. Think adversarially about how a model could satisfy the grader without solving the task: hard-coding the expected output, reading the answer key from disk, special-casing on test names, emitting an empty string that a lenient regex accepts, exploiting a loophole in a policy or rule set that the task author did not intend, finding a degenerate strategy that technically optimises the metric (a game-playing agent that pauses the game indefinitely to avoid ever losing), or injecting instructions into an LLM judge's input. Capable models stumble into these while searching for any passing path. Check that the grader and environment close them off.防作弊。从对抗性角度思考模型如何在不完成任务的情况下满足评分器：硬编码预期输出、从磁盘读取答案密钥、针对测试名称特殊处理、输出空字符串以通过宽松的正则表达式、利用任务作者未预期的政策或规则集漏洞、找到技术上优化指标的退化策略（例如无限期暂停游戏以避免失败的博弈代理）、或向LLM评委的输入注入指令。能力强的模型在搜索任何通过路径时会偶然陷入这些情况。检查评分器和环境是否堵住了这些漏洞。

• Ground truth not reachable by the model. Distinct from label leakage in the task text: check that the expected answers are not anywhere the model under test can read them, not in a file in the agent's sandbox, not in a repo the agent has checked out, not in commit history left over from task construction, not in a grader prompt the agent can see, and not in the web in case the agent has a web-search tool (unless that's the point of the task).模型无法触及的真实答案。与任务文本中的标签泄露不同：检查预期答案不在被测模型可读取的任何地方，不在代理沙箱的文件中，不在代理已检出的仓库中，不在任务构建遗留的提交历史中，不在代理可见的评分器提示中，也不在网络上（除非代理有网络搜索工具，且这是任务的目的）。

• Spot-check the failures. Sample a handful of predictions the grader marked wrong and read them. In many evals a surprising fraction of "failures" are correct answers the grader did not recognise. If more than roughly one in ten sampled failures look like grader errors, fix the grader before any sweep, otherwise the sweep partly measures which model happens to match the grader's blind spots.抽查失败案例。抽取少量评分器标记为错误的预测并阅读。在许多评估中，令人惊讶比例的“失败”实际上是评分器未识别的正确答案。如果每十个抽样失败中有一个以上看起来是评分器错误，则在任何大规模运行前修复评分器，否则大规模运行部分衡量的是哪个模型恰好匹配评分器的盲点。

• Deterministic, or with measured variance. Run the grader on the same prediction/expected pair two or three times. If the result changes, the eval has grader variance on top of model variance. Especially common with LLM judges. If grader nondeterminism is intentional, measure and report its variance separately.确定性或测量方差。对同一预测/预期对运行评分器两到三次。如果结果变化，则评估在模型方差之上还有评分器方差。LLM评委尤其常见。如果评分器非确定性是有意的，则单独测量并报告其方差。

• Atomic checks over holistic scores. Where the grader assesses multiple independent properties ("is it correct and well-formatted and concise"), check whether these are scored as separate binary checks rather than one blended number. Separate checks are more reproducible, easier to calibrate, and make failures diagnostic: {correct: yes, formatted: no, concise: yes} tells you much more than 0.6. Also, prefarably use separate independent LLM calls for each property/dimension.原子检查优于整体评分。当评分器评估多个独立属性（“是否正确、格式良好且简洁”）时，检查这些是否作为单独的二元检查而非一个混合数字评分。单独的检查更可重复、更易校准，并使失败具有诊断性：{正确：是，格式：否，简洁：是}比0.6提供的信息多得多。此外，最好对每个属性/维度使用独立的LLM调用。

• Aggregation matches the question. Check how per-item scores roll up into a headline number. Averaging is the right default for "typical-case quality," but for rare or high-stakes behaviours (safety violations, data deletion, irreversible actions), a fail-on-any-occurrence or worst-case aggregate often reflects what actually matters better than a mean diluted by many easy cases.聚合方式匹配问题。检查每个项目的分数如何汇总为总体数字。平均是“典型质量”的默认选择，但对于罕见或高风险行为（安全违规、数据删除、不可逆操作），基于任何一次发生即失败或最坏情况的聚合通常比被许多简单案例稀释的平均值更能反映实际情况。

• Partial credit and penalty structure. When the grader awards partial credit or applies penalties (for extra steps, wrong tool calls, slow completion), check that the weights do not make a degenerate policy optimal. If the penalties for trying and stumbling outweigh the reward for eventually succeeding, "do nothing" becomes the highest-scoring strategy.部分分数和惩罚结构。当评分器给予部分分数或应用惩罚（额外步骤、错误工具调用、完成缓慢）时，检查权重是否使退化策略成为最优。如果尝试和失败的惩罚超过最终成功的奖励，“什么都不做”将成为最高分策略。

• Handles large outputs. Check that the grader will not truncate, time out, or crash on the longest output a model might produce. A grader that silently clips its input will mis-score long-but-correct answers; a grader that crashes is an infra failure and should be recorded as such (see §2), not as a model failure.处理大输出。检查评分器不会在模型可能产生的最长输出上截断、超时或崩溃。静默截断输入的评分器会错误评分长而正确的答案；崩溃的评分器是基础设施故障，应记录为基础设施故障（见§2），而非模型故障。

• Grader is versioned with the tasks. Check that the grader prompt, rubric, and any normalisation code are versioned alongside the task set. Scores from before and after a grader change are not comparable; if the eval reports trends over time, each data point should record which grader version produced it.评分器与任务一起版本化。检查评分器提示、评分规则和任何规范化代码是否与任务集一起版本化。评分器更改前后的分数不可比较；如果评估报告随时间变化的趋势，每个数据点应记录产生它的评分器版本。

When the grader is an LLM judge当评分器是LLM评委时

LLM judges are convenient and often the only practical option for open-ended tasks, but they bring their own well-documented biases. When the grader calls a model, additionally check:LLM 评判者虽然方便，且通常是开放式任务中唯一实用的选择，但它们自身也存在众所周知的偏见。当评分器评估模型时，还需额外检查：

• Position bias. If the judge compares two responses side by side, check that A/B order is randomised per example (or each pair is scored twice with positions swapped and the results averaged). Judges systematically favour one position regardless of content.位置偏差。如果评判者并排比较两个回答，需检查 A/B 顺序是否在每个示例中随机化（或每对回答以交换位置的方式评分两次，并取结果平均值）。评判者会系统性地偏好某一位置，而与内容无关。

• Verbosity bias. Check whether the rubric tells the judge not to reward length for its own sake, or whether outputs are length-normalised. Uncontrolled, judges reliably prefer longer answers even when the extra length adds nothing.冗长偏差。检查评分标准是否指示评判者不要单纯奖励长度，或者输出是否经过长度归一化处理。不加控制时，评判者总是偏好更长的答案，即使额外长度毫无意义。

• Self-preference. Check whether the judge model is from the same family as any model under test. Judges tend to prefer outputs that resemble what they would have written. Using a judge from a different provider, or a jury of judges from different families with a majority vote, mitigates this.自我偏好。检查评判模型是否与待测模型属于同一系列。评判者倾向于偏好与自己可能生成的输出相似的答案。使用不同提供方的评判者，或由不同系列评判者组成的多数投票陪审团，可缓解此问题。

• Label deference. Check that the judge is not told which response is the "reference," "baseline," or "human" answer. Judges defer to whatever is framed as authoritative, regardless of quality.标签遵从。确保评判者不知道哪个回答是“参考”、“基线”或“人类”答案。评判者会遵从被标记为权威的答案，而忽略实际质量。

• Concrete rubric, not vibes. Read the judge prompt. "Which response is better?" leaves the criteria to the judge's priors and makes scores drift across judge versions. A rubric that lists specific, checkable properties ("Does the response include a runnable code block? Does it cite the requested source?") is more stable and easier to calibrate. (And again, separate independent LLM calls for each property is preferred)具体评分标准，而非模糊感觉。阅读评判提示。“哪个回答更好？”会将标准交由评判者的先验知识，导致分数随评判版本漂移。列出具体可检查属性的评分标准（如“回答是否包含可运行的代码块？是否引用了要求的来源？”）更稳定且易于校准。（再次强调，每个属性使用独立的 LLM 调用更优）

• Calibrated against human labels. Check whether the judge has been validated on a sample (a few dozen examples is usually enough) that humans independently labelled, and what the agreement rate was. If judge-human agreement on clear-cut cases is well below ~90%, the judge prompt usually needs another iteration before its scores can be trusted for model comparisons.针对人类标签进行校准。检查评判者是否已在人类独立标注的样本（通常几十个示例足够）上验证过，以及一致率如何。如果评判者与人类在明确案例上的一致率远低于约 90%，则评判提示通常需要再次迭代，其分数才能用于模型比较。

• Tested on known negatives. Feed the judge a few obviously wrong answers (an empty string, "I don't know," a confident answer to the wrong question) and confirm it fails them. A judge that waves these through invalidates everything downstream.用已知错误答案测试。向评判者提供几个明显错误的答案（空字符串、“我不知道”、对错误问题的自信回答），并确认其判定为失败。如果评判者放过了这些错误，则后续所有结果均无效。

5. Reporting findings to the user5. 向用户报告结果

The checks above are written as directives to Claude, but the audit report you hand to the human should not read as a list of directives. The person who built the eval almost always has context you lack, a constraint, a deadline, a deliberate tradeoff, and the purpose of the report is to surface things worth a second look, not to grade their work. Write accordingly:上述检查是以对Claude的指令形式编写的，但提交给人类的审计报告不应读作一系列指令。构建评估的人通常拥有你所缺乏的背景、约束、截止日期、刻意权衡，报告的目的是揭示值得再次审视的问题，而非评判其工作。请据此撰写：

• Frame findings as observations and suggestions. Prefer "something worth looking at is…", "you might consider…", "teams often find that…", "one thing that can cause trouble here is…" over "this is wrong" or "you must change this." State what you observed, why it might matter, and one concrete change that would address it, then let the user decide.将发现表述为观察和建议。优先使用“值得关注的是……”、“你可以考虑……”、“团队常发现……”、“这里可能引发问题的是……”等措辞，而非“这是错误的”或“你必须修改这一点”。陈述你观察到的情况、其可能的重要性，以及一个具体的改进方案，然后让用户自行决定。

• Distinguish severity. Separate findings into (a) things likely to make the numbers actively misleading, e.g., infra errors scored as model failures, "no answer" conflated with "negative answer," ground truth the agent can reach, a non-deterministic judge, retries folded into latency, and (b) things that add noise or limit generality without flipping conclusions, e.g., a smallish dataset, a missing human baseline, a saturated item or two. Lead with (a).区分严重程度。将发现分为两类：(a) 可能使数据产生严重误导的问题，例如基础设施错误被计为模型失败、“无答案”与“否定答案”混淆、智能体可触及的真实答案、非确定性评判器、重试计入延迟等；(b) 增加噪声或限制泛化性但不颠覆结论的问题，例如数据集较小、缺少人类基线、个别饱和项等。优先报告(a)类问题。

• Be specific and cite evidence. Point at the actual file, function, task ID, or transcript line you are talking about. "Task 14's expected answer looks stale, the library changed its default in v3" is actionable; "some labels may be stale" is not.具体化并引用证据。明确指出你讨论的实际文件、函数、任务ID或转录行。“任务14的预期答案已过时，该库在v3中更改了默认值”是可操作的；“某些标签可能已过时”则不然。

• Say when things are fine. An audit that finds nothing wrong is a valid and useful result. If a section of the eval is in good shape, say so plainly rather than manufacturing a concern, and move on to the sweep with confidence.说明何时一切正常。未发现问题的审计是有效且有用的结果。如果评估的某部分状态良好，请直接说明，而非制造担忧，然后自信地继续全面审查。

• Do not be preachy or exhaustive. Report the handful of things that matter most for the decision the user is trying to make. Resist the urge to list every minor deviation from an ideal, it buries the important findings and reads as nitpicking.避免说教或面面俱到。报告对用户决策最重要的少数几个问题。克制列出所有微小偏差的冲动，这会掩盖重要发现并显得吹毛求疵。

• Offer to fix, not just to flag. Where a finding is a small code change (pin a seed, add a status field, normalise a string before comparison, log the usage block), offer to make the change rather than just describing it.主动提供修复，而非仅指出问题。如果发现涉及小的代码更改（如固定种子、添加状态字段、比较前标准化字符串、记录使用块），主动提出修改而非仅描述问题。

Two practices worth suggesting to the user as part of the report, regardless of what the audit finds:无论审计结果如何，建议在报告中向用户推荐以下两种实践：

• Treat the eval as a living suite. The most reliable evals are maintained like test suites: new failure modes discovered in production become new cases, saturated items are retired or hardened, and the grader is re-calibrated when it drifts. Frame the audit as the start of that loop rather than a one-time gate.将评估视为一个活的测试套件。最可靠的评估像测试套件一样维护：生产中发现的新的失败模式成为新的案例，饱和的项目被淘汰或加强，当评分标准漂移时重新校准。将审计视为该循环的开始，而不是一次性的关卡。

• Use a strong model as a second pair of eyes on the eval itself. Having a capable model read the tasks, the rubric, and a handful of graded transcripts, and asking it where a reasonable person might disagree with the label, is a cheap way to surface ambiguity the eval's authors have become blind to. The tier-3 per-task auditor in §1 is the scaled-up version of this idea: one isolated judgement per task, aggregated into a shortlist for human review rather than a substitute for it.使用一个强大的模型作为评估本身的第二双眼睛。让一个有能力的模型阅读任务、评分标准和一些已评分的记录，并询问它一个理性的人可能在哪些地方不同意标签，这是一种低成本的方式，可以揭示评估作者已经忽视的模糊性。§1中的三级每任务审计员是这个想法的规模化版本：每个任务一个独立的判断，汇总成一个供人工审查的短名单，而不是替代人工审查。