Introducing FrontierHarness Eval介绍 FrontierHarness Eval

TL;DR摘要

Run the same tasks through different harnesses and you get very different bills, pass rates, and wall-clock times. FrontierHarness v1.0 covers software development and terminal-based tasks.在不同的评测框架中运行相同的任务，会产生截然不同的账单、通过率和实际耗时。FrontierHarness v1.0 涵盖了软件开发和基于终端的任务。

Claude Code and DSH (DeepSeek Harness) Creator both landed at a 63% pass rate. Claude Code paid 5.6x more per pass to get there.Claude Code 和 DSH (DeepSeek Harness) Creator 的通过率均为 63%。但 Claude Code 为此付出的单次通过成本高出 5.6 倍。

If you want a quick decision, pick based on what matters most to you:如果您想快速做出决定，请根据您最看重的指标进行选择：

Full leaderboard at frontierharness.org. Explore the source data and task definitions on GitHub.完整排行榜请见 frontierharness.org。您可以在 GitHub 上查看原始数据和任务定义。

---

Results结果

Across all 360 cells: 209 successes, 151 failures. Field-wide pass rate is 58.1%; field-wide token-weighted cache hit rate is 92.4%.在全部 360 个测试单元中：209 次成功，151 次失败。整体通过率为 58.1%；整体 token 加权缓存命中率为 92.4%。

The pass rates for all 12 configurations are close, ranging from 50.0% to 66.7%, just a 17-point difference. But the cost per completed task varies much more, from $1.05 to $18.34 on the same tasks. That means two harnesses can solve the same problem, but one might cost over ten times more than the other.全部 12 种配置的通过率非常接近，在 50.0% 到 66.7% 之间，仅有 17 个百分点的差距。但完成单项任务的成本差异巨大，在相同任务下从 $1.05 到 $18.34 不等。这意味着两个评测框架都能解决同一个问题，但其中一个的成本可能比另一个高出十倍以上。

Initially sorted by pass rate. Click any column header to sort by a different column.初始按通过率排序。点击任意列标题可按不同列进行排序。

Exo Harness and Pi solve tasks for a fraction of what the rest of the field spends. Codex and DSH Creator show that efficiency and quality are not opposed: both sit near the top of the pass-rate column while spending close to the field median. Claude Code marks the other end. It passes as many tasks as DSH Creator and pays 5.6x more per pass to do it. As discussed below, that result may reflect the interaction between the harness, model, and gateway rather than the harness alone.Exo Harness 和 Pi 解决任务的成本仅为其他框架的一小部分。Codex 和 DSH Creator 表明效率与质量并不冲突：两者均位于通过率榜单前列，且成本接近整体中位数。Claude Code 则处于另一端，它通过的任务数与 DSH Creator 相同，但单次通过成本高出 5.6 倍。如下文所述，这一结果可能反映了评测框架、模型和网关之间的交互作用，而不仅仅是评测框架本身的问题。

Three patterns are worth pulling out:值得关注的三种模式：

1. Finishing a task and finishing it cheaply are separate skills. Claude Code and DSH Creator both pass 19 tasks. One costs $3.28 per pass, the other $18.34.完成任务与低成本完成任务是两种独立的能力。Claude Code 和 DSH Creator 都通过了 19 项任务。但一个单次通过成本为 $3.28，另一个则为 $18.34。
2. Cache hit rate is not cost. A cached 300-turn failure can still burn more than a short cache miss. Steps and tokens are what you pay for; caching only changes the unit price.缓存命中率不等于成本。一次缓存了 300 轮的失败任务，其开销可能比一次短暂的缓存未命中更高。您支付的是步骤和 token 的费用；缓存只会改变单价。
3. Vendor lineage did not produce an advantage here. Kimi Code, made by the same lab as the model, tied for seventh on pass rate at 56.7% and sat mid-pack on cost.厂商血统在此并未带来优势。Kimi Code 由模型所属实验室开发，其通过率在榜单中并列第七（56.7%），成本处于中游水平。

---

Why we built FrontierHarness我们为何构建 FrontierHarness

Runta is an agent execution layer designed for token efficiency, secret protection, and governed access. What we care about is how much compute it takes an agent to finish a task it is already capable of finishing, and how much of that is the harness’s doing rather than the model’s.Runta 是一个智能体执行层，专为 token 效率、密钥保护和受控访问而设计。我们关心的是智能体完成其本身有能力完成的任务需要消耗多少算力，以及其中有多少消耗源于评测框架而非模型本身。

Model evaluations establish the baseline capability. The harness supplies the tools and environment, and it determines how much of that capability is spent on the problem versus on overhead.模型评估建立了基准能力。评测框架提供工具和环境，并决定了多少能力被用于解决问题，多少被用于系统开销。

Three questions shaped the evaluation:三个问题塑造了本次评估：

• What does a task actually cost? Tokens and steps, measured directly from each trajectory. This is the number we care most about.任务的实际成本是多少？通过直接从每个轨迹测量 token 和步骤数得出。这是我们最关心的数字。
• What does a failure cost? Agents spend the most money on the problems they cannot solve. A cost metric that only counts successes hides the majority of the bill.失败的成本是多少？智能体在无法解决的问题上花费了最多的资金。仅统计成功案例的成本指标会掩盖大部分账单。
• How efficiently does a harness use tools? Effective tool calls reduce unnecessary steps, improve accuracy, and save tokens.评测框架使用工具的效率如何？有效的工具调用可以减少不必要的步骤，提高准确性并节省 token。

---

How we built FrontierHarness我们如何构建 FrontierHarness

Benchmark scope基准测试范围

The benchmark covers 30 tasks (21 Terminal-Bench tasks and 9 DeepSWE tasks) across 9 harnesses. These run in 12 configurations for 360 total evaluations, with one canonical result for every task and harness-configuration pair. Outcomes use deterministic verifier-based pass/fail scoring, and first-turn cache reads are repriced consistently across harnesses to make costs comparable.该基准测试涵盖了 9 个评测框架下的 30 个任务（21 个 Terminal-Bench 任务和 9 个 DeepSWE 任务）。这些任务在 12 种配置下进行了总计 360 次评估，每个任务与评测框架配置对都有一个规范结果。结果采用基于确定性验证器的通过/失败评分，且首次缓存读取在不同评测框架间进行了统一重定价，以确保成本可比。

The execution model执行模型

Every harness and every task environment is installed once and captured as a golden Runta checkpoint. Each of the 360 trials is a fresh restore from that checkpoint: identical vCPU, memory, disk size, disk contents, and memory state. No trial inherits filesystem drift, a warmed page cache, or a partially-installed dependency from the trial before it.每个评测框架和任务环境均安装一次，并捕获为黄金 Runta 检查点。360 次试验中的每一次都是从该检查点进行的新鲜恢复：具有相同的 vCPU、内存、磁盘大小、磁盘内容和内存状态。没有任何试验会继承之前试验产生的文件系统漂移、预热的页面缓存或部分安装的依赖项。

A few rules make the numbers comparable:几条规则确保了数字的可比性：

• Benchmark tasks are never run before the formal evaluation. Implicit prefix caching means a debug run leaves the trajectory warm for hours. All debugging happens on a separate smoke task set.基准测试任务在正式评估前从不运行。隐式前缀缓存意味着调试运行会让轨迹预热数小时。所有调试都在单独的冒烟测试任务集上进行。
• All 12 configurations run the same task in the same job, so comparisons within a task use the same environment.所有 12 种配置在同一个作业中运行相同的任务，因此同一任务内的比较使用的是相同的环境。
• One attempt per cell. Retries are warmer by definition, and best-of-N selection would bias quality upward.每个单元一次尝试。重试在定义上属于预热，且“N选优”选择会人为提高质量指标。

---

Why Kimi K3为何选择 Kimi K3

We wanted a model that was not native to one of the leading harnesses in the comparison.我们希望选用一个对于比较中的主流评测框架而言不是“原生”的模型。

Using Claude would favor Claude Code’s native environment; using a GPT model would do the same for Codex. Kimi K3 gave us a more neutral starting point, although it does not remove every compatibility effect.使用 Claude 会偏向 Claude Code 的原生环境；使用 GPT 模型也会对 Codex 产生同样的效果。Kimi K3 为我们提供了一个更中立的起点，尽管它无法完全消除所有兼容性影响。

Kimi K3 is a frontier-class coding model with published token pricing and implicit prompt caching. We served it through Fireworks so every harness reached the same third-party inference backend.Kimi K3 是一款具有公开 token 定价和隐式 prompt 缓存的前沿级编码模型。我们通过 Fireworks 提供服务，确保每个评测框架都能连接到相同的第三方推理后端。

All nine harnesses connect through a single gateway that speaks both the OpenAI Responses API and the Anthropic Messages API, so each harness talks its native protocol to identical weights.所有 9 个评测框架都通过单一网关连接，该网关同时支持 OpenAI Responses API 和 Anthropic Messages API，因此每个评测框架都以其原生协议与相同的模型权重进行通信。

Why K3 gives Kimi Code no home-field advantage为什么 K3 没有给 Kimi Code 带来主场优势

Kimi Code is built by the same lab as Kimi K3, but it did not have a native-path advantage in this evaluation.Kimi Code 由与 Kimi K3 相同的实验室开发，但在本次评估中，它并没有享受到原生路径优势。

Kimi Code’s kimi provider type returned a 404 against our gateway, so we configured it as type = "anthropic" and routed it over the Messages API. Its stream output also omits usage data, which we recovered from usage.record events in the session wire log.Kimi Code 的 kimi 提供者类型在我们的网关上返回 404，因此我们将其配置为 type = "anthropic" 并通过 Messages API 进行路由。其流式输出也省略了使用数据，我们从会话日志的 usage.record 事件中恢复了这些数据。

Kimi Code tied for seventh at 56.7%, with a $3.65 median cost per pass and a 7m 56s median runtime. Its 88.0% median cache hit rate tied for the best in the field, but that did not translate into a higher pass rate or lower cost.Kimi Code 以 56.7% 的通过率并列第七，单次通过中位成本为 $3.65，中位运行时间为 7 分 56 秒。其 88.0% 的中位缓存命中率并列全场最高，但这并没有转化为更高的通过率或更低的成本。

---

Finding: Claude Code’s token-weighted cached hit rate is far low发现：Claude Code 的 token 加权缓存命中率极低

The leaderboard shows a median cache hit rate, but that number does not describe where the total token spend went.排行榜显示了中位缓存命中率，但该数字并不能说明总 token 开销的去向。

Claude Code’s cache hit rate is 25.0% token-weighted and 67.8% by median cell. Every other harness in the field sits between 91.6% and 99.1% weighted. The dashboard plots the median; the total token bill is driven by the weighted number. Nothing on a normal leaderboard reconciles the two, and for Claude Code the gap is 2.7x.Claude Code 的缓存命中率按 token 加权为 25.0%，按中位单元计算为 67.8%。全场其他所有评测框架的加权命中率均在 91.6% 到 99.1% 之间。仪表板绘制的是中位数；而总 token 账单是由加权数字驱动的。常规排行榜上没有任何指标能调和这两者，对于 Claude Code 而言，差距为 2.7 倍。

The median, 67.8%, describes a typical cell and can be compared with the $18.34 median cost per pass. On that measure, Claude Code’s cache rate was 1.3x lower than Codex’s 88.0%. The token-weighted figure, 25.0%, describes the total input-token volume. It is heavily affected by the most expensive task, where Claude Code cost $64.36 with a 15.7% cache hit rate.67.8% 的中位数描述了一个典型单元，可以与 $18.34 的单次通过中位成本进行比较。按该指标衡量，Claude Code 的缓存率比 Codex 的 88.0% 低 1.3 倍。25.0% 的 token 加权数字描述了总输入 token 量。它受到最昂贵任务的严重影响，在该任务中，Claude Code 耗资 $64.36，缓存命中率仅为 15.7%。

The weighted cache hit rate is heavily skewed: three of Claude Code’s nineteen successful cells account for 88% of its total token usage, and just one task (python-statemachine-state-data-scoping, 24.6M input tokens at a 15.7% hit rate) is responsible for 68% by itself. In effect, the 25.0% weighted cache rate mostly reflects the performance on this single, exceptionally large cell, so it doesn’t represent typical Claude Code behavior.加权缓存命中率严重偏斜：Claude Code 的 19 个成功单元中，有三个占了其总 token 使用量的 88%，仅一个任务（python-statemachine-state-data-scoping，24.6M 输入 token，15.7% 命中率）本身就占了 68%。实际上，25.0% 的加权缓存率主要反映了在该单一、异常庞大单元上的表现，因此它不能代表 Claude Code 的典型行为。

Why is the weighted rate so low?为什么加权率如此之低？

One likely contributor is harness-model fit. Claude Code is designed around Anthropic models and their explicit cache-control semantics. K3 exposes an implicit prefix cache with different behavior, so the same caching strategy may not transfer cleanly. This is a property of the complete harness-model configuration, not necessarily a defect in Claude Code.一个可能的因素是评测框架与模型的适配度。Claude Code 是围绕 Anthropic 模型及其显式缓存控制语义设计的。K3 暴露的是一种行为不同的隐式前缀缓存，因此相同的缓存策略可能无法完美迁移。这是完整的“评测框架-模型”配置的属性，未必是 Claude Code 的缺陷。

The trajectories also show that caching was inconsistent from call to call. In one archived Claude Code session, the cache read stayed near 18.4K tokens (the size of the system prompt) on most calls, while the growing conversation was billed at the fresh rate. On roughly one call in five, the same append-only prefix reached a 100% hit rate:轨迹还显示，缓存命中在不同调用之间是不一致的。在一个已归档的 Claude Code 会话中，大多数调用中的缓存读取保持在 18.4K token 左右（系统提示词的大小），而不断增长的对话则按新鲜速率计费。大约每五次调用中，就有一次相同的仅追加前缀达到了 100% 的命中率：

call   fresh   cacheRead   total_in   hit%
   1    1174       18432      19606   94.0
   4   19381       18432      37813   48.7
   5       0       38345      38345  100.0   <-- same conversation, full prefix hit
   8   23092       18688      41780   44.7
  13       0       45277      45277  100.0
  25   35474       18944      54418   34.8
  26       0       55514      55514  100.0

That variation within one append-only conversation is difficult to explain as a stable property of the harness alone. The same prefix was cached in full on some calls and not at all on others, which is the behavior of an implicit cache rather than a fixed caching strategy.这种在单次仅追加对话中的变化，很难解释为评测框架本身的稳定属性。相同的前缀在某些调用中被完整缓存，而在另一些调用中则完全没有，这是隐式缓存的行为，而非固定缓存策略的行为。

We cannot yet attribute the gap to the harness by itself. The $18.34 and $64.36 figures describe the exact configuration we tested, not Claude Code in isolation. Separating those effects is the first item on the v1.1 list.我们目前还不能将这种差距单独归因于评测框架。$18.34 和 $64.36 的数字描述的是我们测试的确切配置，而不是孤立的 Claude Code。分离这些影响是 v1.1 列表中的第一项。

---

An example task任务示例

datacurve/python-statemachine-state-data-scoping is a feature request against fgmacedo/python-statemachine at a pinned base commit. Official DeepSWE pass rate: 38%. Median historical trajectory: 117 agent steps.datacurve/python-statemachine-state-data-scoping 是针对 fgmacedo/python-statemachine 在固定基础提交上的功能请求。官方 DeepSWE 通过率为 38%。历史中位轨迹：117 个智能体步骤。

Solving this means touching the metaclass, the callback injection machinery, history restoration, pickling, the SCXML parser, and the diagram renderer, then keeping the existing suite green. Seven of the 12 configurations passed.解决此问题意味着要触及元类、回调注入机制、历史恢复、序列化、SCXML 解析器和图表渲染器，并保持现有测试套件通过。12 种配置中有 7 种通过了测试。

This task is the thesis in miniature. Seven configurations produce the same outcome, a passing verifier, and the receipts for that outcome are nothing alike.这个任务是本文核心论点的缩影。7 种配置产生了相同的结果（通过验证器），但实现该结果的账单却截然不同。

Pi and Claude Code both passed, with a 26x difference in price. Pi reached the fix in 90 turns; Claude Code took 381, and recorded a 15.7% cache hit rate against Pi’s 98.2%. More steps, each one more expensive. The pass/fail column cannot see either effect. The difference belongs to the complete harness-model configuration tested here; the current data cannot isolate each component’s contribution.Pi 和 Claude Code 都通过了测试，但价格相差 26 倍。Pi 在 90 轮内完成了修复；Claude Code 用了 381 轮，缓存命中率记录为 15.7%，而 Pi 为 98.2%。步骤越多，每一步就越昂贵。通过/失败列无法反映出这两种效应。这种差异属于此处测试的完整“评测框架-模型”配置；当前数据无法分离出每个组件的贡献。

DSH PTC spent $12.95 on a failed run, more than six of the seven configurations that passed. DSH Creator failed at $3.88 while DSH Minimal passed at $10.49. Both used the same harness, model, and runtime, but different presets. Creator reached the 60-minute limit, so prompt and tool configuration affected both pacing and outcome.DSH PTC 在一次失败的运行中花费了 $12.95，超过了 7 种通过配置中的 6 种。DSH Creator 在 $3.88 时失败，而 DSH Minimal 在 $10.49 时通过。两者使用相同的评测框架、模型和运行时，但预设不同。Creator 达到了 60 分钟的限制，因此提示词和工具配置影响了节奏和结果。

---

Limitations and future work局限性与未来工作

Limitations局限性

FrontierHarness v1.0 is currently limited to software development and terminal-based tasks. It does not yet evaluate other types of agentic or knowledge work. Broader coverage is needed before making general conclusions.FrontierHarness v1.0 目前仅限于软件开发和基于终端的任务。它尚未评估其他类型的智能体或知识工作。在得出一般性结论之前，需要更广泛的覆盖范围。

The benchmark uses each harness exactly as it is shipped, without modifying anything inside. For harnesses like Claude Code with strong internal opinions (such as around caching), we do not alter their implementation to improve cache hit rates or to standardize caching across gateways. As a result, the reported cost differences reflect the full harness-model setup as delivered, and we cannot say how much of the gap would remain if every harness were equally tailored to K3’s caching behavior.该基准测试完全按原样使用每个评测框架，不对内部进行任何修改。对于像 Claude Code 这样有强烈内部偏好（例如缓存方面）的评测框架，我们不会更改其实现以提高缓存命中率或在网关间标准化缓存。因此，报告的成本差异反映了所交付的完整“评测框架-模型”设置，我们无法说明如果每个评测框架都针对 K3 的缓存行为进行同等调整，差距会剩下多少。

Future work未来工作

Version 1.1 will add more harnesses and evaluate a broader set of models. Rather than holding the model fixed, we plan to evaluate the full harness × model matrix to reveal interaction effects (which harnesses work best with which models) and produce a harness-model compatibility matrix.版本 1.1 将增加更多的评测框架并评估更广泛的模型。我们计划评估完整的“评测框架 × 模型”矩阵，以揭示交互效应（哪些评测框架与哪些模型配合得最好），并生成评测框架-模型兼容性矩阵，而不是固定模型。

---

Conclusion结论

The harness makes a difference. Pass rates in this field cluster within 17 points; median cost per pass spans 17x.评测框架确实会产生影响。该领域的通过率集中在 17 个百分点以内；单次通过的中位成本跨度达 17 倍。

Codex is the safe default: highest pass rate, cost near the field median at $3.47.Codex 是稳妥的默认选择：通过率最高，成本接近全场中位数 $3.47。

Pi is the pick when the same job runs many times and token spend compounds. On our hardest task it landed the same fix as Codex in 90 turns against 187, for $2.50 against $5.97.当同一作业多次运行且 token 开销累积时，Pi 是首选。在我们最难的任务上，它在 90 轮内完成了与 Codex 相同的修复（Codex 用了 187 轮），成本为 $2.50（Codex 为 $5.97）。

Exo is the cheapest per completed task at $1.05, partly because it stops early: on the hardest task it hit its 51-step cap and failed at $1.46 while others kept spending. We did not test retries, but a harness that fails cheaply is the one you would want to retry.Exo 是单任务完成成本最低的，为 $1.05，部分原因是它提前停止：在最难的任务上，它达到了 51 步的上限并以 $1.46 的成本失败，而其他框架则继续消耗。我们没有测试重试，但一个低成本失败的评测框架正是您想要重试的对象。

DSH shows how much the preset matters. Its four presets differ by 6.6 points of pass rate and 1.4x in cost on the same model and runtime, which is about the spread between many of the harnesses themselves.DSH 展示了预设的重要性。在相同的模型和运行时下，其四个预设的通过率相差 6.6 个百分点，成本相差 1.4 倍，这大约是许多评测框架本身之间的差距。

The cost spread across the field comes mostly from the harness-model pairing. Cache strategy, tool looping, and prompt design belong to the pair, not to the harness alone, and a harness tuned for one family’s cache contract can look efficient there and expensive elsewhere. Holding the model, tasks, and runtime constant is what let v1.0 see those effects. Future runs will widen the harness × model matrix and tasks diversity.全场的成本差异主要来自“评测框架-模型”的配对。缓存策略、工具循环和提示词设计属于配对属性，而非评测框架本身。一个针对某系列缓存契约调整的评测框架可能在那里表现高效，而在其他地方却显得昂贵。保持模型、任务和运行时不变，使 v1.0 能够观察到这些效应。未来的运行将扩大“评测框架 × 模型”矩阵和任务的多样性。