Automating eval design and hillclimbing with Claude

Evaluations provide a signal on how your app or skill is performing on specific tasks. But designing evaluations, and improving performance on them without fooling yourself, is hard. We've added guidance for both to the claude-api skill.评估可以为你提供有关你的应用程序或技能在特定任务中表现如何的信号。但设计评估，并在不自欺欺人的前提下提高评估表现，是一件困难的事。我们已在 claude-api 技能中添加了针对这两方面的指导。

With the skill, you can run /claude-api build-eval to build an evaluation inside your codebase, and run /claude-api hillclimb to improve your application against it, one change at a time, with a held-out set of examples to catch overfitting.利用该技能，你可以运行 /claude-api build-eval 在你的代码库中构建评估，并运行 /claude-api hillclimb 来针对该评估逐步优化你的应用程序。每次仅进行一项更改，并使用一组留出的示例来检测过拟合。

In this article, we highlight the principles of good eval design and hillclimbing first, then show how Claude Code with the claude-api skill applies those principles. We’ll close by showing a few examples of these commands.在本文中，我们将首先重点介绍良好的评估设计和爬山算法（hillclimbing）的原则，然后展示 Claude Code 如何通过 claude-api 技能应用这些原则。最后，我们将通过几个示例来演示这些命令。

EVAL DESIGN

Well designed evaluations have a few common elements (Figure 1):设计良好的评估通常具备以下几个共同要素（图 1）：

1. Eval tasks mirror production. Sample tasks that you care about in “production,” or the setting in which the capability or application you are testing will be used. Sometimes tasks are picked because they are easy to generate or they are easy to grade. But it’s important to ensure that the task distribution represents what you actually care about.评估任务应反映生产环境。选取你在“生产环境”中关心的任务样本，即你所测试的功能或应用程序将要使用的场景。有时选择任务是因为它们易于生成或易于评分。但重要的是要确保任务分布能够代表你真正关心的内容。
2. Performance improves with stronger models and more thinking. More capable models and higher effort levels typically should perform better on an evaluation. If they don’t, ambiguous tasks or a miscalibrated grader often are hobbling performance.性能会随着更强大的模型和更多的思考而提升。能力更强的模型和更高的努力水平通常应在评估中表现更好。如果表现没有提升，通常是因为任务模糊或评分器校准不当阻碍了性能发挥。
3. There is “passable” headroom at the frontier. The most capable model at the highest effort should be well below 100% on the evaluation, otherwise you can’t reliably judge how changes impact performance. Importantly, the gap should not be explained by impossible or ambiguous tasks: a common tell is that a task fails every evaluation run, regardless of the number of replicates. A good task is one where two domain experts would reach the same verdict and everything the grader checks is stated in the task.前沿模型应留有“可提升”的空间。最高努力水平下能力最强的模型在评估中的得分应远低于 100%，否则你无法可靠地判断更改对性能的影响。重要的是，这种差距不应归因于不可能完成或模棱两可的任务：一个常见的迹象是，无论重复多少次，某项任务在每次评估运行中都失败。好的任务是两个领域专家能达成一致结论，且评分器检查的所有内容都在任务中明确说明的任务。
4. Low run-to-run variance. High variance is often due to poorly designed, ambiguous tasks or a grader that produces different verdicts on identical output. Variance can also hide in the configuration. For example, effort may not be applied consistently. Also, the environment can affect the results of the evaluation: leftover state from an earlier trial (a file, a git history) can hand the agent the answer.低运行间方差。高方差通常是由设计拙劣、模棱两可的任务，或对相同输出给出不同结论的评分器造成的。方差也可能隐藏在配置中。例如，努力程度可能没有被一致地应用。此外，环境也会影响评估结果：之前试验留下的状态（文件、git 历史记录）可能会直接将答案泄露给智能体。

Adversarial sampling对抗性采样

Model capability is jagged. If you pick cases because today's model fails them, you are sampling the valleys of one model's capability surface (Figure 2). The evaluation can end up measuring that model's failure fingerprint rather than what is intrinsically hard or valuable for your application to do.模型能力是参差不齐的。如果你选择那些当前模型无法处理的案例，你实际上是在采样该模型能力表面的“低谷”（图 2）。评估最终可能测量的是该模型的“失败指纹”，而不是你的应用程序在本质上难以完成或有价值的任务。

Pick hard cases because a human judged them hard: a useful test is to be able to say why a task is hard before you include it. Include cases that are specific failures in your application derived from production traffic, bug reports, or tickets. However, don’t blindly trust user traffic: users sometimes try what they expect to work, so a task distribution drawn strictly from user traffic may skew easy.选择困难案例的依据是人类对其的判断：一个有效的测试是，在包含某项任务之前，你能说出它为什么困难。包含那些源自生产流量、错误报告或工单的特定应用程序故障案例。不过，不要盲目信任用户流量：用户有时只会尝试他们预期会成功的内容，因此严格从用户流量中提取的任务分布可能会偏向简单。

/CLAUDE-API BUILD-EVAL/CLAUDE-API BUILD-EVAL

The build-eval command in the claude-api skill turns these principles into a guided workflow. When you run /claude-api build-eval in Claude Code, Claude interviews you, builds the eval inside your codebase, and pauses for approval at specific points.claude-api 技能中的 build-eval 命令将这些原则转化为引导式工作流程。当你在 Claude Code 中运行 /claude-api build-eval 时，Claude 会采访你，在你的代码库中构建评估，并在特定节点暂停以等待你的批准。

Designing examples设计示例

Claude helps you sample inputs to build evaluations in this order:Claude 会按以下顺序帮助你采样输入以构建评估：

1. Production transcripts, after asking about retention and sensitive data.生产环境转录内容（在询问保留政策和敏感数据后）。
2. Bug reports and support tickets.错误报告和支持工单。
3. Five to ten cases you write by hand.你手工编写的五到十个案例。
4. Cases synthesized from your codebase.从你的代码库中合成的案例。

The skill prioritizes production traffic, but it can also generate synthetic data anchored in a few real examples that you provide. The skill instructs Claude to generate a simple page that shows you every input and waits until you confirm them. As an illustration, below we show an example set of inputs for an e-mail router application that the skill may ask the user to review (Figure 3).该技能优先处理生产流量，但也可以根据你提供的少量真实示例生成合成数据。该技能会指示 Claude 生成一个简单的页面，向你展示每个输入并等待确认。作为说明，下面我们展示了一个电子邮件路由应用程序的示例输入集，该技能可能会要求用户进行审查（图 3）。

Validating the grader验证评分器

After the inputs, Claude proposes the cheapest grader that fits your application’s output:在输入完成后，Claude 会提出最符合你应用程序输出的低成本评分器方案：

• Programmatic verification: If the output possibilities are constrained, it uses a code based check (exact match, a label from a fixed set, JSON that matches a schema, tests that pass).程序化验证：如果输出可能性有限，它会使用基于代码的检查（精确匹配、固定集合中的标签、符合模式的 JSON、通过的测试）。
• LLM-as-judge: It will default to this type of check if the output space is open-ended, with many valid answers but clear quality criteria. In this case, a second model reads the input, the output and a rubric written as checkable claims (not a 1-to-5 scale), and returns a score with its reasoning. If you have a baseline to compare against, the judge instead reads both inputs in random order, without being told which is the baseline, and picks the better one. You pick the judge model, and it should not be the model you are testing.LLM 作为裁判：如果输出空间是开放式的（有许多有效答案但有明确的质量标准），它将默认使用这种检查方式。在这种情况下，第二个模型会读取输入、输出以及以可检查声明编写的准则（而非 1 到 5 分的量表），并返回带有推理过程的评分。如果你有基准进行比较，裁判模型会以随机顺序读取两个输入（不告知哪个是基准），并选出更好的一个。你负责选择裁判模型，且该模型不应与你正在测试的模型相同。

Claude grades a handful of cases and asks whether you would have scored any of them differently (Figure 4). In general, it is important to read a sample of scored transcripts before believing your evaluator; scoring failures are among the most common ways an evaluation is misconfigured.Claude 会对少数案例进行评分，并询问你是否会对其中任何案例给出不同的评分（图 4）。通常，在信任你的评估器之前，阅读一部分已评分的转录内容非常重要；评分失败是评估配置错误最常见的原因之一。

When you’ve validated the grader, the skill tells you the size of the evaluation set (cases × repeats × model, and roughly how long it will take), runs the baseline, and prints the score with a confidence interval. What you get back: the cases, the grader, the runner, one JSON line and one full transcript per case, and a plain page that lists each case's score with a link to its transcript. If you want more than that page shows (e.g., a chart), just ask and Claude will build it as an extra page next to it. By default, these extra pages are static files that open locally and load nothing from the network.当你验证完评分器后，该技能会告知你评估集的大小（案例数 × 重复次数 × 模型，以及大致所需时间），运行基准测试，并打印出带有置信区间的得分。你将获得：案例、评分器、运行器、每个案例一行 JSON 和一份完整转录，以及一个列出每个案例得分并带有转录链接的纯文本页面。如果你需要页面显示之外的内容（例如图表），只需提出要求，Claude 就会在旁边生成一个额外页面。默认情况下，这些额外页面是静态文件，在本地打开且不从网络加载任何内容。

Diagnostic checks诊断检查

During the baseline runs mentioned above, Claude checks a number of things:在上述基准运行期间，Claude 会检查以下几点：

• Grader: Claude runs the grader twice on the same output, and reports whether the verdict changed.评分器：Claude 会对同一个输出运行两次评分器，并报告结论是否发生变化。
• Plumbing: Claude checks for timeouts, API errors, and cut-off answers to ensure infrastructure noise doesn't pass as model variance.基础设施：Claude 会检查超时、API 错误和截断的回答，以确保基础设施本身的噪声不会被误认为是模型方差。
• Headroom: if the baseline already scores about 95% or higher, the skill warns the user and alerts that the hillclimb should aim to explore cost or latency rather than quality.提升空间：如果基准测试得分已达到 95% 或更高，该技能会提醒用户，并指出爬山优化应旨在探索成本或延迟，而非质量。

HILLCLIMBING

Now that you have a reliable means of grading your application’s performance on a task, you can try to improve it. Hillclimbing is an effective way to tune parameters like effort or prompts, which trade-off cost and performance. Some general tips for choosing where to apply it:既然你已经有了可靠的方法来评估应用程序在某项任务上的表现，你就可以尝试对其进行改进。爬山算法是调整努力程度或提示词等参数（这些参数会在成本和性能之间进行权衡）的有效方法。以下是关于选择应用位置的一些通用建议：

• Cheap iteration - It should be inexpensive (in terms of time, cost, and effort) to modify whatever surface you are focused on for hillclimbing. Many internal efforts and customers have focused hillclimbing on text, such as prompts and skills. These are easy to change and revert. In contrast, open-ended modifications to an agent harness during hillclimbing may involve extensive code changes.低成本迭代 - 修改你专注于爬山优化的任何表面（surface）都应该是低成本的（在时间、成本和精力方面）。许多内部项目和客户都将爬山优化集中在文本上，例如提示词和技能。这些内容易于更改和回滚。相比之下，在爬山过程中对智能体框架进行开放式修改可能涉及大量的代码变更。
• Attributable - Changes in the score on your evaluation should be attributable to the surface you are modifying during hillclimbing. For example, several successful applications of hillclimbing have focused on skill triggering. The evaluation metric (the trigger rate for the skill) is directly coupled to the skill description that is being modified.可归因性 - 评估得分的变化应能归因于你在爬山过程中修改的表面。例如，一些成功的爬山优化应用集中在技能触发上。评估指标（技能的触发率）直接与被修改的技能描述相关联。
• Well-scoped objective - One common failure mode is an open-ended request to improve performance without careful consideration of the headroom available in the evaluation; an evaluation that’s near saturation or a poorly scoped surface (e.g., an open-ended request to update the harness) is more likely to stall. One generally strong objective across various efforts is cost: even if an evaluation is saturated, you can ask Claude to find ways to reduce cost while keeping performance at parity.目标明确 - 一个常见的失败模式是在不仔细考虑评估中可用提升空间的情况下，提出改进性能的开放式请求；一个接近饱和的评估或范围界定不当的表面（例如，更新框架的开放式请求）更容易陷入停滞。在各种工作中，一个普遍强有力的目标是成本：即使评估已饱和，你也可以要求 Claude 寻找在保持性能不变的同时降低成本的方法。

Overfitting过拟合

Even a well-designed evaluation rarely matches the exact task distribution you care about in production. As a result, "overfitting" to an evaluation is a common problem and results in a system that performs better on an evaluation than on production traffic.即使是设计良好的评估，也很少能完全匹配你在生产环境中关心的确切任务分布。因此，“过拟合”到评估是一个常见问题，会导致系统在评估中的表现优于在生产流量中的表现。

There are many ways an evaluation can "leak" into your harness (the code around the model, including prompts, tools, and loop that calls Claude). For example, consider an evaluation task that benefits from OCR, but OCR is rarely beneficial in your production tasks. The evaluation harness might add an OCR tool to your application, which improves on the benchmark without any impact on production. More broadly, hillclimbing may add features to that harness that address edge cases in the particular evaluation examples you’ve chosen. These harness additions improve your evaluation score, but don’t translate to improvements in production (Figure 5).评估可能会通过多种方式“泄漏”到你的框架（模型周围的代码，包括提示词、工具和调用 Claude 的循环）中。例如，考虑一个受益于 OCR 的评估任务，但 OCR 在你的生产任务中很少有用。评估框架可能会向你的应用程序添加一个 OCR 工具，这在基准测试中表现更好，但对生产环境没有影响。更广泛地说，爬山优化可能会向该框架添加一些功能，以解决你所选择的特定评估示例中的边缘情况。这些框架添加的内容提高了你的评估分数，但并不能转化为生产环境中的性能提升（图 5）。

Three things can help address this:三件事有助于解决这个问题：

• Split the cases. Use a train set that the hillclimber may read and a test set that is never seen. If the train set scores improve while the test set scores stay flat, then that is a common overfitting warning sign.拆分案例。使用爬山算法可以读取的训练集和从未见过的测试集。如果训练集得分提高而测试集得分保持不变，这就是过拟合的常见警告信号。
• Never paste failures into the prompt. If the hillclimber reads the failing transcripts, it should never paste the failure content into the prompt.切勿将失败案例粘贴到提示词中。如果爬山算法读取了失败的转录内容，它绝不应将失败内容粘贴到提示词中。
• Keep the answers structurally out of the model's reach. Models can sometimes “reward hack” by directly finding answers to evaluations.使答案在结构上超出模型的获取范围。模型有时会通过直接找到评估答案来“破解奖励机制”。

As discussed below, the claude-api skill applies these principles for you.如下所述，claude-api 技能会为你应用这些原则。

/CLAUDE-API HILLCLIMB/CLAUDE-API HILLCLIMB

The hillclimb command in the claude-api skill turns these principles into a guided workflow. When you run /claude-api hillclimb in Claude Code, Claude iterates to improve against a given evaluation. You choose what changes it can make including:claude-api 技能中的 hillclimb 命令将这些原则转化为引导式工作流程。当你在 Claude Code 中运行 /claude-api hillclimb 时，Claude 会进行迭代以针对给定的评估进行改进。你可以选择它能进行的更改，包括：

• Your system prompt你的系统提示词
• Skills or instruction files技能或指令文件
• Tool descriptions工具描述
• Model choice, effort level, and other API parameters模型选择、努力水平和其他 API 参数
• Your harness code你的框架代码

Before it starts, Claude asks what you want to optimize (e.g., performance, or cost while performance holds) and then splits the evaluation set at random into test and train. With a cost goal, it considers a few common cost drivers, including prompt caching, auditing the prompt for compatibility with the selected model, and picking the model and effort setting.在开始之前，Claude 会询问你想要优化什么（例如，性能，或在保持性能的同时优化成本），然后将评估集随机拆分为测试集和训练集。对于成本目标，它会考虑一些常见的成本驱动因素，包括提示词缓存、审计提示词以确保与所选模型的兼容性，以及选择模型和努力设置。

Before the first round, Claude checks that the eval's noise (how far the score can move by chance alone) is smaller than the smallest improvement you'd act on; if it isn't, it says so and suggests more repetitions or cases.在第一轮之前，Claude 会检查评估的噪声（仅凭运气得分能变动多少）是否小于你打算采取行动的最小改进幅度；如果不是，它会说明情况并建议增加重复次数或案例。

Each round, Claude reads the previous round’s train transcripts and proposes one change as a patch. It aims each round at a change whose effect can show above the eval's noise: it fixes the failing behavior at its root (e.g., rewrites the section that causes it or adds a missing rule) rather than rewording a line. It then runs evaluation with the patched change. At this point, Claude applies a check: if the train set improves but the test set is flat, Claude suspects overfitting and reverts the patch. If there is a regression, Claude reverts. If train and test sets improve, it keeps the patch (Figure 6).每一轮，Claude 都会读取上一轮的训练转录内容并提出一项更改作为补丁。它的目标是每一轮都进行一项其效果能高于评估噪声的更改：它从根源上修复失败行为（例如，重写导致失败的部分或添加缺失的规则），而不是仅仅改写一行文字。然后，它会带着补丁运行评估。此时，Claude 会进行检查：如果训练集有所改善但测试集持平，Claude 会怀疑是过拟合并回滚补丁。如果出现回归，Claude 也会回滚。如果训练集和测试集都有所改善，它会保留补丁（图 6）。

When the score stalls for two or three rounds, Claude reads each remaining train failure and sorts it by cause. It does the same early if no single fix could gain more than the eval's noise, and suggests more repetitions or cases, rather than spending rounds on changes too small to measure. This step can catch ambiguous evaluation cases, harness errors, or run-to-run variance.当得分连续两三轮停滞不前时，Claude 会读取每个剩余的训练失败案例并按原因分类。如果没有任何单一修复能获得超过评估噪声的增益，它会提前执行此操作，并建议增加重复次数或案例，而不是将轮次浪费在太小而无法衡量的更改上。此步骤可以捕获模棱两可的评估案例、框架错误或运行间方差。

Only legitimate failures are included in more hillclimbing rounds.只有合法的失败案例才会被包含在后续的爬山优化轮次中。

When hillclimbing completes, Claude leaves your code at the version that did best on the test set for your goal. It reports the test result against the baseline with confidence intervals (Figure 7). If the gain is within noise, it says so and recommends against merging.当爬山优化完成时，Claude 会将你的代码保留在测试集上表现最好的版本。它会报告测试结果与基准测试的对比，并附带置信区间（图 7）。如果增益在噪声范围内，它会说明情况并建议不要合并。

EXAMPLES

Hillclimbing for cost reduction以降低成本为目标的爬山优化

We ran /claude-api hillclimb on an internal customer support benchmark with the goal of reducing cost and improving performance. The benchmark included 44 tickets, with 30 used for the search and 14 held out. It started on Opus 4.8 at default (high) effort settings with 74.4% decision accuracy on the search tickets and a token cost of 4.6 cents per ticket.我们在一个内部客户支持基准测试上运行了 /claude-api hillclimb，目标是降低成本并提高性能。该基准测试包含 44 个工单，其中 30 个用于搜索，14 个留出。它最初使用 Opus 4.8，默认（高）努力设置，搜索工单的决策准确率为 74.4%，每个工单的 Token 成本为 4.6 美分。

The hillclimb first audited the prompt, removing mandatory tool-call rituals, a scratchpad step, and contradictory rules. Then it tried Opus 5.5 on low effort. This cleared the baseline accuracy bar at 87.8% and cut cost to 1.9 cents per ticket, less than half the starting cost.爬山优化首先审计了提示词，删除了强制性的工具调用仪式、草稿步骤和相互矛盾的规则。然后它尝试了 Opus 5.5 的低努力模式。这达到了 87.8% 的基准准确率门槛，并将每个工单的成本降低到 1.9 美分，不到初始成本的一半。

Part of that saving comes from Opus 5.5's pricing: input and output tokens cost 20% less than on Opus 4.8, and cache reads cost 60% less. Because Opus 5.5 cleared the bar, the hillclimb then stepped down a tier to check whether a cheaper model could clear it too. Sonnet 5 on low effort scored about the same, 88.9%, at about half the cost, 1 cent per ticket (Figure 8).部分节省来自于 Opus 5.5 的定价：输入和输出 Token 的成本比 Opus 4.8 低 20%，缓存读取成本低 60%。由于 Opus 5.5 达到了门槛，爬山优化随后下调了一个层级，以检查更便宜的模型是否也能达到要求。Sonnet 5 在低努力模式下得分大致相同，为 88.9%，成本约为原来的一半，即每个工单 1 美分（图 8）。

Finally, Claude improved the prompt with routing rules and a refund-cap cross-reference, bringing Sonnet 5 to 98.9% at about the same cost. On the 14 held-out tickets that the search never saw, the final configuration scored 90.5% against the original setup's 78.6%, at about one fifth of the cost.最后，Claude 通过路由规则和退款上限交叉引用改进了提示词，使 Sonnet 5 的性能达到 98.9%，成本大致相同。在搜索从未见过的 14 个留出工单上，最终配置的得分为 90.5%，而原始设置为 78.6%，且成本约为原来的五分之一。

Hillclimbing for performance improvement以提升性能为目标的爬山优化

Another example is our claude-api skill, which provides guidance on using our APIs and general tips for working with Claude (including the sub-commands discussed in this article). We want to ensure our skill can correctly implement code that uses our APIs, and we built an evaluation set derived from our documentation to test the skill.另一个例子是我们的 claude-api 技能，它提供了关于使用我们 API 的指导以及使用 Claude 的通用技巧（包括本文讨论的子命令）。我们希望确保我们的技能能够正确实现使用我们 API 的代码，因此我们构建了一个源自我们文档的评估集来测试该技能。

On our evaluation, the skill started at 66%. We gave the hillclimber access to documentation and our SDKs, allowing Claude to identify errors and self-correct them (Figure 9). Claude found that the skill was missing coverage of eight features.在我们的评估中，该技能的初始得分率为 66%。我们让爬山优化器访问文档和我们的 SDK，允许 Claude 识别错误并进行自我纠正（图 9）。Claude 发现该技能缺少对八项功能的覆盖。

Adding sections for them in the skill improved performance to 74%. It then found errors in C# and Java type tables, boosting performance to 77%.在技能中为它们添加章节将性能提高到了 74%。随后它发现了 C# 和 Java 类型表中的错误，将性能提升到了 77%。

After the score stalled for two rounds, Claude analyzed the remaining failures and bucketed them by root-cause. A normal round makes one edit for the most common failure. This step makes no edit; it only sorts every remaining failure by cause. This reflection step was useful in a few ways:在得分停滞两轮后，Claude 分析了剩余的失败案例并按根本原因进行了分组。常规轮次会针对最常见的失败进行一次编辑。此步骤不进行编辑；它只是按原因对所有剩余的失败进行分类。这种反思步骤在几个方面很有用：

• Reflecting across a collection of failures, the hillclimber found that the skill content was present but Claude was simply writing older API shapes (e.g., from its trained priors). To address, the hillclimber added a table near the top of the skill that guided Claude from the forms it remembered to the current ones: for example, from extended thinking with a fixed token budget, which the API now rejects on recent Opus models, to adaptive thinking, and from older versions of the web search and web fetch tools to the current ones. It also moved the C# and Java warnings against fixed-budget thinking above their adaptive-thinking examples. This improved performance to 80%.通过对一系列失败案例的反思，爬山优化器发现技能内容本身是存在的，但 Claude 只是在编写旧的 API 形式（例如，基于其训练先验）。为了解决这个问题，爬山优化器在技能顶部附近添加了一个表格，引导 Claude 从它记忆中的形式转向当前的形式：例如，从 API 现在在最新 Opus 模型上拒绝的“固定 Token 预算的扩展思考”转向“自适应思考”，并从旧版本的网页搜索和网页抓取工具转向当前版本。它还将 C# 和 Java 中针对固定预算思考的警告移到了自适应思考示例之前。这使性能提高到了 80%。
• Tasks that never improved in performance despite addressing obvious content gaps are tells that the example or grader is flawed. One task asked for code that catches one error type, while its grader wanted a chain of at least three. Claude reworded the task. Another grader's instructions contradicted our docs, and testing the real API showed the docs were right. Addressing these, along with more skill edits, brought performance to ~88%.尽管解决了明显的内容差距，但性能从未改善的任务是示例或评分器存在缺陷的迹象。一项任务要求编写捕获一种错误类型的代码，而其评分器却要求至少三个错误的链式捕获。Claude 重写了该任务。另一个评分器的说明与我们的文档相矛盾，测试真实 API 表明文档是正确的。解决这些问题，加上更多的技能编辑，使性能达到了约 88%。

GETTING STARTED

Note: Run claude update first. The claude-api skill ships inside Claude Code, so updating gets you the latest version of these commands.注意：请先运行 claude update。claude-api 技能内置于 Claude Code 中，因此更新即可获取这些命令的最新版本。

claude update

Then, in Claude Code:然后，在 Claude Code 中：

/claude-api build-eval
/claude-api hillclimb

These sub-commands can be used directly in Claude Code via the claude-api skill:这些子命令可以通过 claude-api 技能直接在 Claude Code 中使用：

Run /claude-api build-eval if you want to generate an evaluation set for a particular problem. You can steer it by providing access to examples (e.g., traces). Claude will employ the guidance shared in this article to design the examples and grader, and ensure you approve the examples and the grader.如果你想为特定问题生成评估集，请运行 /claude-api build-eval。你可以通过提供示例（例如跟踪记录）来引导它。Claude 将采用本文分享的指导原则来设计示例和评分器，并确保你批准这些示例和评分器。

Run /claude-api hillclimb if you have an evaluation and want Claude to improve on this, guided by your goal (e.g., better performance, or lower cost while performance holds). Claude will employ the guidance shared in this article to check for overfitting while climbing and check for bugs in the eval itself, such as a grader that marks a correct-looking answer wrong or a harness error, both before the first round and whenever the score stalls.如果你已经有了一个评估并希望 Claude 在此基础上进行改进，请运行 /claude-api hillclimb，并根据你的目标（例如，更好的性能，或在保持性能的同时降低成本）进行引导。Claude 将采用本文分享的指导原则，在爬山过程中检查过拟合，并检查评估本身是否存在错误（例如，评分器将看起来正确的答案标记为错误，或框架错误），这些检查会在第一轮之前以及每次得分停滞时进行。

With special thanks to Misha Khalman for skill development. With thanks to Misha Khalman, Michael Segner, Matt Bell, Matt Thanabalan, and Punit Shah for reviews, contributions, and product support.特别感谢 Misha Khalman 在技能开发方面的贡献。感谢 Misha Khalman、Michael Segner、Matt Bell、Matt Thanabalan 和 Punit Shah 的审查、贡献和产品支持。