Quantifying infrastructure noise in agentic coding evals量化智能体编程评测中的基础设施噪声

Agentic coding benchmarks like SWE-bench and Terminal-Bench are commonly used to compare the software engineering capabilities of frontier models—with top spots on leaderboards often separated by just a few percentage points. These scores are often treated as precise measurements of relative model capability and increasingly inform decisions about which models to deploy. However, we’ve found that infrastructure configuration alone can produce differences that exceed those margins. In internal experiments, the gap between the most- and least-resourced setups on Terminal-Bench 2.0 was 6 percentage points (p < 0.01).像 SWE-bench 和 Terminal-Bench 这样的智能体编程基准测试常被用于比较前沿模型的软件工程能力，而排行榜上的领先位置往往仅相差几个百分点。这些分数通常被视为模型相对能力的精确衡量标准，并日益成为部署决策的参考依据。然而，我们发现仅基础设施配置一项就能产生超过这些差距的差异。在内部实验中，Terminal-Bench 2.0 在资源配置最丰富和最匮乏的环境下，测试结果差距达到了 6 个百分点（p < 0.01）。

Static benchmarks score a model's output directly—the runtime environment doesn’t factor into the result. Agentic coding evals are different: models are given a full environment where they write programs, run tests, install dependencies, and iterate over multiple turns. The runtime is no longer a passive container, but an integral component of the problem-solving process. Two agents with different resource budgets and time limits aren't taking the same test.静态基准测试直接对模型的输出进行评分，运行环境不会影响结果。但智能体编程评测则不同：模型被置于一个完整的环境中，需要在此编写程序、运行测试、安装依赖并进行多轮迭代。此时，运行环境不再是一个被动的容器，而是解决问题过程中的一个组成部分。两个资源预算和时间限制不同的智能体，实际上并没有参与同一场测试。

Eval developers have begun accounting for this. Terminal-Bench 2.0, for instance, specifies recommended CPU and RAM on a per-task basis in their latest 2.0 release. However, specifying resources isn't the same as enforcing them consistently. Moreover, we discovered that enforcement methodology can change what the benchmark ends up actually measuring.评测开发者已经开始关注这一点。例如，Terminal-Bench 2.0 在其最新的 2.0 版本中，按任务指定了推荐的 CPU 和内存配置。然而，指定资源并不等同于一致地执行资源限制。此外，我们发现执行方法本身可能会改变基准测试最终衡量的实际内容。

How we got here我们是如何发现这一点的

We run Terminal-Bench 2.0 on a Google Kubernetes Engine cluster. While calibrating the setup, we noticed our scores didn't match the benchmark’s official leaderboard, and infra error rates were surprisingly high: as many as 6% of tasks were failing because of pod errors, most of which were unrelated to the model’s ability to solve the tasks.我们在 Google Kubernetes Engine 集群上运行 Terminal-Bench 2.0。在校准设置时，我们注意到我们的得分与基准测试的官方排行榜不符，且基础设施错误率高得惊人：多达 6% 的任务因 Pod 错误而失败，其中大部分与模型解决任务的能力无关。

The discrepancy in scores came down to enforcement. Our Kubernetes implementation treated the per-task resource specs as both a floor and a hard ceiling: each container was guaranteed the specified resources but killed the moment it exceeded them. Container runtimes enforce resources via two separate parameters: a guaranteed allocation—the resources reserved up front—and a hard limit at which the container is killed. When these are set to the same value, there's zero headroom for transient spikes: a momentary memory fluctuation can OOM-kill a container that would otherwise have succeeded. To account for this, Terminal-Bench’s leaderboard uses a different sandboxing provider, whose implementation is more lenient, allowing temporary overallocation without terminating the container in order to favor infrastructural stability.得分差异归结于资源执行方式。我们的 Kubernetes 实现将按任务指定的资源配置既视为最低限额，也视为硬性上限：每个容器都能保证获得指定的资源，但一旦超出该限额就会被终止。容器运行时通过两个独立参数来执行资源限制：保证分配（即预留的资源）和容器被终止的硬性上限。当这两者设置为相同值时，瞬时峰值将毫无缓冲空间：微小的内存波动就可能导致原本能够成功的容器因内存不足（OOM）而被终止。为了解决这个问题，Terminal-Bench 的排行榜使用了另一家沙箱提供商，其实现方式更为宽松，允许在不终止容器的情况下进行临时的超额分配，从而优先保证基础设施的稳定性。

This finding raised a larger question: how much does resource configuration impact evaluation scores?这一发现引出了一个更大的问题：资源配置对评测得分的影响究竟有多大？

To quantify the effect of the scaffold, we ran Terminal-Bench 2.0 across six resource configurations, from strict enforcement of the per-task specs (1x), having them act as both floor and ceiling, to completely uncapped. Everything else stayed constant: same Claude model, same harness, same task set.为了量化这种支架效应，我们在六种资源配置下运行了 Terminal-Bench 2.0，从严格执行按任务指定的配置（1 倍，将其同时作为最低和最高限额），到完全不设上限。其他所有条件保持不变：相同的 Claude 模型、相同的测试工具和相同的任务集。

In our experiments, success rates increased with resource headroom. This was primarily driven by infra error rates dropping monotonically at each step, going from 5.8% at strict enforcement to 0.5% when uncapped. The drop between strict enforcement to 3x headroom (5.8% to 2.1%) was significant at p < 0.001. With more headroom, fewer containers get killed for exceeding their allocation.在我们的实验中，成功率随资源缓冲空间的增加而提高。这主要是因为基础设施错误率在每一步都单调下降，从严格执行时的 5.8% 降至不设上限时的 0.5%。从严格执行到 3 倍缓冲空间（从 5.8% 降至 2.1%）的下降在 p < 0.001 水平上具有显著性。有了更多的缓冲空间，因超出分配额而被终止的容器就更少了。

From 1x through 3x, success scores fluctuate within the margins of noise (p=0.40). Most of the tasks that were crashing at 1x would have failed regardless—which is something that we observed in the data. The agent explores, hits a resource wall, and gets preempted, but it was never on a path to a correct solution.在 1 倍到 3 倍配置区间内，成功得分在噪声范围内波动（p=0.40）。大多数在 1 倍配置下崩溃的任务，无论如何都会失败——我们在数据中也观察到了这一点。智能体在探索过程中触碰到了资源墙并被抢占，但它本身并未处于通往正确解决方案的路径上。

Starting around 3x, however, this trend changes: success rates climb faster than infra errors decline.然而，从 3 倍左右开始，这一趋势发生了变化：成功率的上升速度超过了基础设施错误率的下降速度。

Between 3x to uncapped, infra errors drop an additional 1.6 percentage points, while success jumps almost 4 percentage points. The extra resources enable the agent to try approaches that only work with generous allocations, such as pulling in large dependencies, spawning expensive subprocesses, and running memory-intensive test suites. At uncapped resources, the total lift over 1x is +6 percentage points (p < 0.01). At the margins, tasks like rstan-to-pystan and compile-compcert significantly improve their success rates when getting memory headroom.在 3 倍到不设上限之间，基础设施错误率额外下降了 1.6 个百分点，而成功率则跃升了近 4 个百分点。额外的资源使智能体能够尝试那些仅在资源充裕时才有效的方法，例如引入大型依赖项、生成占用大量资源的子进程以及运行内存密集型测试套件。在不设资源上限的情况下，相比 1 倍配置，总提升为 6 个百分点（p < 0.01）。在边缘情况下，像 rstan-to-pystan 和 compile-compcert 这样的任务，在获得内存缓冲空间后，其成功率有显著提高。

How this affects measurement这对评估有何影响

Up to roughly 3x Terminal-Bench specs, the additional resources fix infrastructure reliability problems, namely transient resource spikes. The sandboxing provider used by the Terminal-Bench maintainers is implicitly doing this behind the scenes; the eval gets more stable without getting easier.在达到约 3 倍 Terminal-Bench 推荐配置之前，额外的资源主要解决了基础设施可靠性问题，即瞬时资源峰值。Terminal-Bench 维护者所使用的沙箱提供商在后台隐式地执行了这一点；评测在没有变得更简单的情况下变得更加稳定了。

Above the 3x mark, however, additional resources start actively helping the agent solve problems it couldn't solve before, which shows that limits can actually change what the eval measures. Tight limits inadvertently reward very efficient strategies, while generous limits are more forgiving and reward agents that can better exploit all available resources.然而，超过 3 倍这一界限后，额外的资源开始主动帮助智能体解决之前无法解决的问题，这表明限制实际上会改变评测所衡量的对象。严格的限制无意中奖励了非常高效的策略，而宽松的限制则更具包容性，奖励了那些能够更好地利用所有可用资源的智能体。

An agent that writes lean, efficient code very fast will do well under tight constraints. An agent that brute-forces solutions with heavyweight tools will do well under generous ones. Both are legitimate things to test, but collapsing them into a single score without specifying the resource configuration makes the differences—and real-world generalizability—hard to interpret.编写精简、高效代码的智能体在严格约束下表现良好。使用重量级工具暴力破解解决方案的智能体在资源充裕时表现良好。这两者都是合理的测试对象，但若不说明资源配置就将它们合并为一个分数，就会导致差异——以及现实世界中的通用性——难以解读。

On bn-fit-modify, a Terminal-Bench task requiring Bayesian network fitting, some models’ first move is to install the standard Python data science stack: pandas, networkx, scikit-learn, and all their toolchain. Under generous limits, this works. Under tight ones, the pod runs out of memory during installation, before the agent writes a single line of solution code. A leaner strategy exists (implementing the math from scratch using only the standard library), and some models do default to it. Others don’t. Different models have different default approaches, and the resource configuration determines which of those approaches happen to succeed.We replicated the core finding across different Anthropic models. The direction of the effect was consistent, while the magnitude varied. The same trends seem to hold on models other than Claude, but we haven’t rigorously tested them.在 bn-fit-modify（一项需要贝叶斯网络拟合的 Terminal-Bench 任务）中，一些模型的第一步动作是安装标准的 Python 数据科学栈：pandas、networkx、scikit-learn 及其所有工具链。在资源充裕时，这可行。但在严格限制下，Pod 会在安装过程中内存耗尽，甚至在智能体写出一行解决方案代码之前就失败了。其实存在一种更精简的策略（仅使用标准库从头实现数学逻辑），一些模型确实默认采用了这种策略，但另一些则不然。不同的模型有不同的默认方法，而资源配置决定了其中哪些方法能够成功。我们在不同的 Anthropic 模型上重复了这一核心发现。效果的方向是一致的，但幅度有所不同。同样的趋势似乎也适用于 Claude 以外的模型，但我们尚未对其进行严格测试。

We also tested whether this pattern holds on evals outside Terminal-Bench by running a crossover experiment on SWE-bench. We varied the total available RAM up to 5x the baseline across 227 problems with 10 samples each. The same effect held, though the magnitude was smaller: Scores again increased monotonically with RAM, but were only 1.54 percentage points higher at 5x than 1x. SWE-bench tasks are less resource-intensive, so a smaller effect is expected, but it shows resource allocation isn't neutral there either.我们还通过在 SWE-bench 上进行交叉实验，测试了这种模式是否适用于 Terminal-Bench 之外的评测。我们在 227 个问题上（每个问题 10 个样本）将总可用内存调整为基准的 5 倍。同样的效果依然存在，尽管幅度较小：得分同样随内存增加而单调上升，但在 5 倍配置下仅比 1 倍配置高出 1.54 个百分点。SWE-bench 任务的资源密集度较低，因此预期效果较小，但这表明资源分配在那里同样不是中性的。

Other sources of variance其他变异来源

Resource allocation isn't the only hidden variable. In certain configurations, time limits too start playing a role.资源分配并不是唯一的隐变量。在某些配置下，时间限制也开始发挥作用。

In principle, every element of the evaluation setup can influence the final score, from the cluster health to the hardware specs, from the concurrency level to even egress bandwidth. Agentic evals are end-to-end system tests by construction, and any component of that system can act as a confounder. We have observed anecdotally, for instance, that pass rates fluctuate with time of day, likely because API latency varies with traffic patterns and incidents. We have not formally quantified this effect, but it illustrates a larger point: the boundary between "model capability" and "infrastructure behavior" is blurrier than a single benchmark score suggests. A model provider can shield its eval infrastructure from this by dedicating hardware, but external evaluators can't easily do the same.原则上，评测设置的每一个要素都可能影响最终得分，从集群健康状况到硬件规格，从并发级别到出口带宽。智能体评测本质上是端到端的系统测试，该系统的任何组件都可能成为混杂因素。例如，我们曾观察到通过率会随一天中的时间波动，这很可能是因为 API 延迟随流量模式和突发事件而变化。我们尚未正式量化这种影响，但它说明了一个更宏大的观点：在“模型能力”与“基础设施行为”之间的界限，比单一基准分数所暗示的要模糊得多。模型提供商可以通过专用硬件将其评测基础设施与此隔离开来，但外部评估者很难做到这一点。

Public benchmarks are typically meant to measure pure model capabilities, but in practice they risk conflating them with infrastructure quirks. Sometimes this may be desirable, as it enables end-to-end testing of the entire stack, but more often it's not. For coding evals meant to be shared publicly, running at multiple times and on multiple days would help average out the noise.公共基准测试通常旨在衡量纯粹的模型能力，但在实践中，它们存在将模型能力与基础设施特性混为一谈的风险。有时这可能是可取的，因为它能实现对整个技术栈的端到端测试，但更多时候并非如此。对于旨在公开分享的编程评测，在不同时间、不同日期多次运行将有助于平滑噪声。

What we recommend我们的建议

The ideal scenario is to run each eval under the exact same hardware conditions—both the scaffold running the eval and the inference stack—as it would ensure perfect reproducibility across the board. However, this may not always be practical.理想的情况是在完全相同的硬件条件下运行每次评测——包括运行评测的支架和推理栈——因为这能确保各方面的完美可复现性。然而，这在实践中并不总是可行的。

Given how container runtimes actually enforce resources—via a guaranteed allocation and a separate hard kill threshold—we recommend that evals specify both parameters per task, not a single pinned value. A single exact spec sets the guaranteed allocation equal to the kill threshold, leaving zero margin: the transient memory spikes we documented at 1x are enough to destabilize the eval. Separating the two parameters lets you give containers enough breathing room to avoid spurious OOM kills, while still enforcing a hard ceiling that prevents score inflation.鉴于容器运行时实际执行资源限制的方式（通过保证分配额和一个单独的硬性终止阈值），我们建议评测为每个任务指定这两个参数，而不是单个固定值。单一的精确规格将保证分配额等同于终止阈值，不留任何余地：我们记录的 1 倍配置下的瞬时内存峰值足以破坏评测的稳定性。将这两个参数分开，可以为容器提供足够的喘息空间以避免虚假的 OOM 终止，同时仍能执行硬性上限以防止分数虚高。

The band between them should be calibrated so that scores at the floor and ceiling fall within noise of each other. For instance, in Terminal-Bench 2.0, a 3x ceiling over the per-task specs cut infra error rates by roughly two-thirds (5.8% to 2.1%, p < 0.001) while keeping the score lift modest and well within noise (p = 0.40). That's a reasonable tradeoff: the infrastructure confounder is largely neutralized without removing meaningful resource pressure. The exact multiplier will vary by benchmark and task distribution, and should thus be reported, but the empirical calibration principle is general.两者之间的区间应进行校准，使得最低限额和最高限额下的得分处于彼此的噪声范围内。例如，在 Terminal-Bench 2.0 中，将上限设为按任务规格的 3 倍，使基础设施错误率降低了约三分之二（从 5.8% 降至 2.1%，p < 0.001），同时保持得分提升适中且完全在噪声范围内（p = 0.40）。这是一个合理的权衡：在不消除有意义的资源压力的情况下，基础设施混杂因素在很大程度上被中和了。具体的乘数会因基准测试和任务分布而异，因此应予以报告，但这种经验校准原则是通用的。

Why we care为什么我们关注这一点

These findings have practical consequences beyond eval infrastructure. Benchmark scores are increasingly used as decision-making inputs, but this increased attention (and reliance) hasn’t always come with corresponding rigor in how they’re run or reported. As things stand today, a 2-point lead on a leaderboard might reflect a genuine capability difference, or it might reflect that one eval ran on beefier hardware, or even at a luckier time of day, or both. Without published (or standardized) setup configurations, it’s hard to tell from the outside unless interested parties go the extra mile to reproduce objective results under identical conditions.这些发现具有超越评测基础设施的实际后果。基准测试分数日益被用作决策输入，但这种增加的关注（和依赖）并未伴随着在运行或报告方式上相应的严谨性。就目前的情况而言，排行榜上 2 分的领先可能反映了真正的能力差异，也可能反映了某次评测是在更强大的硬件上运行的，甚至是处于一天中更幸运的时段，或者两者兼而有之。如果没有公布（或标准化）的设置配置，除非相关方付出额外努力在相同条件下复现客观结果，否则从外部很难判断。

For labs like Anthropic, the implication is that resource configuration for agentic evals should be treated as a first-class experimental variable, documented and controlled with the same rigor as prompt format or sampling temperature. For benchmark maintainers, publishing recommended resource specs (as Terminal-Bench 2.0 does) can go a long way, while specifying enforcement methodology would close the gap we identified. And for anyone consuming benchmark results, the core takeaway is that small score differences on agentic evals carry more uncertainty than the precision of the reported numbers suggests—especially as some confounders are simply too hard to control for.对于像 Anthropic 这样的实验室来说，这意味着智能体评测的资源配置应被视为一等实验变量，并以与提示词格式或采样温度相同的严谨性进行记录和控制。对于基准测试维护者来说，发布推荐的资源规格（如 Terminal-Bench 2.0 所做的那样）大有裨益，而明确执行方法则能填补我们所发现的空白。对于任何使用基准测试结果的人来说，核心结论是：智能体评测中微小的分数差异所承载的不确定性，比所报告数字的精确度所暗示的要大——尤其是因为某些混杂因素实在太难控制了。

Until resource methodology is standardized, our data suggests that leaderboard differences below 3 percentage points deserve skepticism until the eval configuration is documented and matched. The observed spread across the moderate range of resource configurations in Terminal-Bench is just below 2 percentage points. Naive binomial confidence intervals already span 1-2 percentage points; the infrastructure confounders we document here stack on top of that, not within it. At the extremes of the allocation range, the spread reaches 6.在资源方法论标准化之前，我们的数据表明，在评测配置未被记录且未匹配的情况下，排行榜上低于 3 个百分点的差异值得怀疑。在 Terminal-Bench 的中等资源配置范围内观察到的差异略低于 2 个百分点。朴素的二项式置信区间已经跨越了 1-2 个百分点；我们在此记录的基础设施混杂因素是在此基础之上的叠加，而非包含在内。在资源分配范围的极端情况下，差异可达 6 个百分点。

A few-point lead might signal a real capability gap—or it might just be a bigger VM.几分的领先可能预示着真正的能力差距——但也可能仅仅是因为虚拟机性能更强。

Acknowledgements致谢

Written by Gian Segato. Special thanks to Nicholas Carlini, Jeremy Hadfield, Mike Merrill, and Alex Shaw for their contributions. This work reflects the collective efforts of several teams working on evaluations for coding agents. Interested candidates who would like to contribute are welcome to apply at anthropic.com/careers.作者：Gian Segato。特别感谢 Nicholas Carlini、Jeremy Hadfield、Mike Merrill 和 Alex Shaw 的贡献。这项工作反映了多个致力于编程智能体评估的团队的集体努力。欢迎有兴趣的候选人通过 anthropic.com/careers 申请加入我们。