Finding the Right Fit:
Model–Harness Interactions across Agent TasksThanks: Corresponding author: boan@ntu.edu.sg.寻找最佳契合点：智能体任务中模型与框架的交互 致谢：通讯作者：boan@ntu.edu.sg。

The configurable grid contains $4\times 5\times 3=60$ reported entries. OpenHands [16], DSH [4], PI [19], and openJiuwen are each evaluated with the five models on all three task collections. Codex is evaluated only with GPT, and Claude Code only with Claude, adding six native references. Dashes in Table 1 denote unevaluated pairings.可配置网格包含 4×5×3=60 个报告条目。OpenHands [16]、DSH [4]、PI [19] 和 openJiuwen 分别与五种模型在所有三个任务集上进行评估。Codex 仅与 GPT 一起评估，Claude Code 仅与 Claude 一起评估，增加了六个原生参考。表 1 中的破折号表示未评估的组合。

The intended task-set sizes are 120 for TUA-Bench (TUA), 99 for ALE-CLI (ALE), and 63 for Terminal-Bench 4 (TB4). ALE uses the campaign’s local-Docker selection, and Terminal-Bench uses the shared non-H100 subset. The collections differ in workflow composition rather than forming orthogonal capability tests. All configurations use the same task identifiers.预期的任务集大小为：TUA-Bench (TUA) 120 个，ALE-CLI (ALE) 99 个，Terminal-Bench 4 (TB4) 63 个。ALE 使用了该活动的本地 Docker 选择，Terminal-Bench 使用了共享的非 H100 子集。这些集合在工作流组成上有所不同，而非形成正交的能力测试。所有配置使用相同的任务标识符。

Tasks affected by infrastructure failures were rerun. Each task counts once, using its last run for both reward and cost; a task without a valid outcome in that run receives zero, and partial rewards are retained. For openJiuwen–DeepSeek on ALE, image inputs in four image and computer-use tasks are served by DeepSeek V4.1 Flash, the provider’s standard fallback for image requests (Appendix A).受基础设施故障影响的任务已重新运行。每个任务计一次，使用其最后一次运行的结果同时计算奖励和成本；在该运行中没有有效结果的任务得分为零，保留部分奖励。对于 ALE 上的 openJiuwen–DeepSeek，四个图像和计算机使用任务中的图像输入由 DeepSeek V4.1 Flash 提供服务，这是该供应商针对图像请求的标准回退方案（附录 A）。

We keep harness changes to a minimum, mainly what is needed to reach the models through a common endpoint. All models are served through OpenRouter and pinned to their first-party provider (Anthropic, OpenAI, Moonshot AI, DeepSeek, and Z.ai) with fallbacks disabled. Every harness requests high reasoning effort, although the same label need not imply the same thinking budget across harnesses. Sampling parameters are left at provider defaults, and context management, compaction, retries, and turn limits follow each harness’s native defaults (Appendix Table 5). Among the packaged harnesses, the one deliberate exception is OpenHands, whose context and output limits we set to 1M and 128K tokens because its defaults for a custom endpoint cap output at 16K. No configuration adds skills, MCP servers, memory, or custom prompts beyond its defaults. Harness tools are the native defaults; on ALE-CLI every harness also receives the benchmark’s fourteen computer-use tools. Time limits are the benchmarks’ own: 40 minutes per task for TUA-Bench, eight hours for Terminal-Bench, and the per-task limit of ALE-CLI (two hours for most tasks).我们将框架的更改保持在最低限度，主要是为了通过通用端点连接模型所需的更改。所有模型均通过 OpenRouter 提供服务，并锁定在其第一方供应商（Anthropic、OpenAI、Moonshot AI、DeepSeek 和 Z.ai），且禁用了回退功能。每个框架都要求高推理强度，尽管相同的标签在不同框架下并不一定意味着相同的思考预算。采样参数保留为供应商的默认值，上下文管理、压缩、重试和轮次限制遵循每个框架的原生默认值（附录表 5）。在打包的框架中，唯一刻意的例外是 OpenHands，我们将它的上下文和输出限制设置为 1M 和 128K Token，因为其自定义端点的默认输出上限为 16K。没有任何配置添加了超出其默认设置的技能、MCP 服务器、内存或自定义提示词。框架工具为原生默认值；在 ALE-CLI 上，每个框架还接收基准测试的十四个计算机使用工具。时间限制为基准测试自身的限制：TUA-Bench 每个任务 40 分钟，Terminal-Bench 为八小时，ALE-CLI 为每个任务的限制（大多数任务为两小时）。

openJiuwen [17] is an agent framework rather than a packaged coding agent and provides no official entry point for running these benchmarks. We therefore assembled a coding agent from its public harness API (v0.1.18), keeping the framework’s native components wherever it provides them: seven coding tools (file read, write, and edit; glob; directory listing; grep; and a shell), its task loop, its context compression, and its security and model-anomaly rails. Skills, persistent memory, task planning, and sub-agents are disabled. On TUA-Bench and Terminal-Bench, a small added rail reports the remaining time limit before each model call. The same composition is used for every model and benchmark, without model- or task-specific prompts or tuning.openJiuwen [17] 是一个智能体框架，而不是一个打包好的编码智能体，且没有提供运行这些基准测试的官方入口。因此，我们利用其公开的框架 API (v0.1.18) 组装了一个编码智能体，并在其提供原生组件的地方保留了这些组件：七个编码工具（文件读取、写入和编辑；glob；目录列表；grep；以及 Shell）、其任务循环、上下文压缩，以及安全和模型异常轨道。技能、持久内存、任务规划和子智能体均已禁用。在 TUA-Bench 和 Terminal-Bench 上，添加了一个小轨道，在每次模型调用前报告剩余时间限制。同样的组成用于每个模型和基准测试，没有特定于模型或任务的提示词或调整。

For benchmark $b$ with intended task set $\mathcal{T}_{b}$ and size $N_{b}$, we report对于具有预期任务集 𝒯b 和大小 Nb 的基准测试 bb，我们报告

TUA and ALE allow partial credit, while Terminal-Bench is binary; we do not combine the three rewards into an overall score.TUA 和 ALE 允许部分得分，而 Terminal-Bench 是二元的；我们没有将三个奖励组合成总分。

We use empirical fit to describe which available harness has the highest recorded reward for a model and task collection:我们使用经验契合度（empirical fit）来描述对于给定的模型和任务集，哪个可用框架具有最高的记录奖励：

where $\mathcal{H}_{m,b}$ contains only evaluated pairings. All score differences below are points on the 0–100 reward scale.其中 ℋm,b 仅包含已评估的组合。以下所有分数差异均为 0-100 奖励量表上的分值。

Cost is the agent-model API cost in USD at OpenRouter prices, including auxiliary model calls made by the harness and excluding evaluator calls. Costs, model-call counts, token counts, and agent time are taken from the same last run of each task that determines its reward. The trajectory analyses in Section 5 compare the final scored runs of the same task and model under different harnesses.成本是以 OpenRouter 价格计算的美元智能体模型 API 成本，包括框架进行的辅助模型调用，不包括评估器调用。成本、模型调用次数、Token 数量和智能体时间均取自决定其奖励的每个任务的最后一次运行。第 5 节中的轨迹分析比较了同一任务和模型在不同框架下的最终评分运行。

On Terminal-Bench 4, Claude records $36/63$ in OpenHands and GPT records $31/63$, a Claude advantage of 7.94 points. DSH reverses the ordering, with $22/63$ for Claude and $33/63$ for GPT. PI records $19/63$ and $38/63$, respectively, while openJiuwen records $26/63$ and $34/63$. The Claude-minus-GPT differences are $+7.94$, $-17.46$, $-30.16$, and $-12.70$ points. Thus the same models yield opposite rankings depending on the harness.在 Terminal-Bench 4 上，Claude 在 OpenHands 中记录为 36/63，GPT 记录为 31/63，Claude 领先 7.94 分。DSH 逆转了顺序，Claude 为 22/63，GPT 为 33/63。PI 分别记录为 19/63 和 38/63，而 openJiuwen 分别记录为 26/63 和 34/63。Claude 减去 GPT 的差异分别为 +7.94、−17.46、−30.16 和 −12.70 分。因此，相同的模型根据框架的不同会产生相反的排名。

The reversal is especially clear when comparing OpenHands with PI on the same Terminal-Bench 4 task set. Claude drops from 57.14% with OpenHands to 30.16% with PI, whereas GPT rises from 49.21% to 60.32%, shifting the Claude-minus-GPT gap by 38.09 percentage points. This contrast shows that the observed model ordering is not simply preserved across harnesses.这种逆转在同一 Terminal-Bench 4 任务集上比较 OpenHands 和 PI 时尤为明显。Claude 从 OpenHands 的 57.14% 下降到 PI 的 30.16%，而 GPT 从 49.21% 上升到 60.32%，使 Claude 减去 GPT 的差距变化了 38.09 个百分点。这种对比表明，观察到的模型排序并不能简单地在框架间保持一致。

Table 2 compares each native pairing with the strongest evaluated alternative using the same model. With Claude, Claude Code leads the best alternative on TUA and ALE, by 3.50 and 1.14 points, whereas OpenHands leads Claude Code on Terminal-Bench by 7.94 points. For GPT, openJiuwen exceeds Codex by 2.67 points on TUA, while PI exceeds Codex by 2.15 points on ALE and 4.76 points on Terminal-Bench. Native provenance therefore does not consistently identify the best harness.表 2 比较了每个原生组合与使用相同模型的最强评估替代方案。对于 Claude，Claude Code 在 TUA 和 ALE 上领先最佳替代方案 3.50 和 1.14 分，而在 Terminal-Bench 上 OpenHands 领先 Claude Code 7.94 分。对于 GPT，openJiuwen 在 TUA 上比 Codex 高出 2.67 分，而 PI 在 ALE 上比 Codex 高出 2.15 分，在 Terminal-Bench 上高出 4.76 分。因此，原生血统并不能始终确定最佳框架。

The highest-scoring harness changes across collections for four of five models (Table 3). GPT changes from openJiuwen on TUA-Bench to PI on ALE-CLI and Terminal-Bench; GLM and DeepSeek change from openJiuwen to OpenHands to DSH, while Claude changes from Claude Code on TUA-Bench and ALE-CLI to OpenHands on Terminal-Bench. Kimi is the persistent pairing: openJiuwen scores 64.39%, 54.94%, and 28.57% on the three collections, leading the strongest observed alternatives by 5.61, 6.91, and 11.11 percentage points. The changing winners and one stable favorable pairing both argue for evaluating fit in the intended task setting.五个模型中有四个模型的最高分框架会随集合而变（表 3）。GPT 从 TUA-Bench 上的 openJiuwen 变为 ALE-CLI 和 Terminal-Bench 上的 PI；GLM 和 DeepSeek 从 openJiuwen 变为 OpenHands 再变为 DSH，而 Claude 从 TUA-Bench 和 ALE-CLI 上的 Claude Code 变为 Terminal-Bench 上的 OpenHands。Kimi 是持续表现良好的组合：openJiuwen 在三个集合上的得分分别为 64.39%、54.94% 和 28.57%，领先最强观察替代方案 5.61、6.91 和 11.11 个百分点。不断变化的获胜者和一种稳定的有利组合都支持在预期的任务环境中评估契合度。

The Kimi advantage is not attributable to a single outlier task. Against the strongest alternative harness in each collection (PI on TUA-Bench and ALE-CLI, and DSH on Terminal-Bench), openJiuwen is higher, tied, and lower on 22/85/13, 25/61/13, and 8/54/1 paired tasks, respectively (Figure 2). Removing the three largest positive task gaps still leaves mean advantages of 3.11, 3.88, and 6.35 points. The ALE grouping further localizes the effect: openJiuwen leads PI by 24.83 points over 19 life-science tasks and by 11.99 points over 12 business/finance tasks, but trails by 2.32 points over 18 computing/math tasks. Thus the aggregate advantage is broad across tasks but not universal across domains.Kimi 的优势并非归因于单个离群任务。在每个集合的最强替代框架（TUA-Bench 和 ALE-CLI 上的 PI，以及 Terminal-Bench 上的 DSH）对比中，openJiuwen 在 22/85/13、25/61/13 和 8/54/1 对任务中分别表现更高、持平或更低（图 2）。剔除三个最大的正向任务差距后，平均优势依然分别为 3.11、3.88 和 6.35 分。ALE 分组进一步定位了这种效应：openJiuwen 在 19 个生命科学任务中比 PI 领先 24.83 分，在 12 个商业/金融任务中领先 11.99 分，但在 18 个计算/数学任务中落后 2.32 分。因此，这种总体优势在任务间广泛存在，但并非在所有领域都通用。

We coded every actionable failure signal and the agent’s response in ten same-task, same-model TB4 pairs (OpenHands vs. PI, all five models; 192 events in final scored attempts), plus six Kimi pairs contrasting openJiuwen with PI, and checked the resulting mechanisms against all final attempts on the three benchmarks. Almost all responses were model-initiated (180 of 192). Diagnosis or a targeted patch was the most common response (69%), and it resolved the signal in 116 of these 133 cases; strategy changes were rarer but always resolved the signal (16), and repetition was rare (11 events, 7 of them in one run). Runs that passed answered feedback with diagnosis or repair more often than runs that failed (81% vs. 56%) and almost never repeated themselves (1% vs. 11%). The decisive difference, however, was usually not whether the model could repair an error but whether the harness returned the failure as usable feedback. OpenHands ends a run after repeated identical failing actions: this ended 55 runs, 48 of them Kimi re-issuing an edit call without its required content argument, and only one of these runs scored above zero (on TB4 payments-pipeline-fix, Kimi completed its design under OpenHands, then re-issued one malformed edit four times and was stopped with no code written, while the same design passed under PI and openJiuwen). PI’s shell has no default timeout, so 34 PI runs on TB4 and TUA sat blocked in a final command that never returned and produced no signal at all; the other harnesses bound commands by default (see the matched case below). PI also ends a run when a turn is cut short by the output cap, whereas OpenHands continues and openJiuwen re-prompts the model to continue with its partial reasoning kept (100 runs resumed, 48 of which went on to score above zero). Of openJiuwen’s harness prompts, only this resumption was decisive in the pairs we read (2 of 6); its request to re-check all requirements before finishing triggered extra testing but did not decide any outcome. When the decisive signal came from the agent’s own test, failing runs in both harnesses tended to dismiss or skip it. The fit is model-specific rather than a uniform ranking: GPT-6, which sets explicit timeouts on 56–67% of its PI shell calls, obtains its best TB4 and ALE scores under PI’s minimal scaffold, while Kimi, penalized by the OpenHands editor and by PI’s unbounded and non-resuming behavior, scores highest under openJiuwen on all three benchmarks. Turns cut off by provider stream errors (25 of Kimi’s 62 PI runs on TB4) and sandbox stalls are scored as failures; we attribute them to harness resilience rather than model behavior.我们对十个相同任务、相同模型的 TB4 配对（OpenHands 对比 PI，所有五个模型；最终评分尝试中的 192 个事件）中的每一个可操作失败信号和智能体响应进行了编码，外加六个对比 openJiuwen 与 PI 的 Kimi 配对，并针对三个基准测试上的所有最终尝试检查了所得机制。几乎所有的响应都是模型启动的（192 个中的 180 个）。诊断或针对性补丁是最常见的响应（69%），并在这些 133 个案例中的 116 个中解决了信号；策略改变较少见但总是能解决信号（16 个），重复则很少见（11 个事件，其中 7 个在一次运行中）。通过的运行比失败的运行更频繁地以诊断或修复来响应反馈（81% 对 56%），并且几乎从不重复自己（1% 对 11%）。然而，决定性的差异通常不在于模型能否修复错误，而在于框架是否将失败作为可用反馈返回。OpenHands 在重复出现相同的失败动作后结束运行：这导致了 55 次运行终止，其中 48 次是 Kimi 在没有内容参数的情况下重复发出编辑调用，且这些运行中只有一次得分高于零（在 TB4 payments-pipeline-fix 上，Kimi 在 OpenHands 下完成了设计，然后重复了四次格式错误的编辑，在没有编写任何代码的情况下被停止，而相同的设计在 PI 和 openJiuwen 下通过了）。PI 的 Shell 没有默认超时，因此 TB4 和 TUA 上的 34 次 PI 运行在一个永远不会返回且没有产生任何信号的最终命令中被阻塞；其他框架默认绑定了命令（见下文匹配案例）。当轮次被输出上限截断时，PI 也会结束运行，而 OpenHands 会继续，openJiuwen 则提示模型在保留其部分推理的情况下继续（100 次运行恢复，其中 48 次最终得分高于零）。在 openJiuwen 的框架提示中，只有这种恢复在我们阅读的配对中是决定性的（6 次中的 2 次）；其在完成前重新检查所有要求的请求触发了额外的测试，但没有决定任何结果。当决定性信号来自智能体自身的测试时，两个框架中的失败运行往往会忽略或跳过它。这种契合度是特定于模型的，而非统一的排名：GPT-6 在其 56-67% 的 PI Shell 调用中设置了显式超时，在 PI 的精简脚手架下获得了最佳 TB4 和 ALE 分数，而 Kimi 因 OpenHands 编辑器和 PI 的无界及不恢复行为而受损，在所有三个基准测试中均在 openJiuwen 下得分最高。因供应商流错误（Kimi 在 TB4 上 62 次 PI 运行中的 25 次）和沙箱停滞而被截断的轮次被计为失败；我们将其归因于框架的弹性而非模型行为。

We inspected all 45 failed Terminal-Bench 4 tasks for openJiuwen with Kimi K3 and found that the bottleneck lies at the end of the trajectory rather than in intermediate progress. Intermediate state is largely preserved: openJiuwen’s context compression was triggered in only 1 of 63 Terminal-Bench 4 tasks, a rate we cross-checked on 120 TUA-Bench tasks and again observed only once. In the sole Terminal-Bench 4 case, layout-config-recreation, compression reduced 1,450 messages to three ($\sim$834k to $\sim$7.8k tokens) while preserving the deliverable path, measured scores, rejected alternatives, and an explicit resume point.我们检查了 openJiuwen 与 Kimi K3 下所有 45 个失败的 Terminal-Bench 4 任务，发现瓶颈在于轨迹的末尾，而不是中间进度。中间状态在很大程度上得到了保留：openJiuwen 的上下文压缩仅在 63 个 Terminal-Bench 4 任务中的 1 个中被触发，我们在 120 个 TUA-Bench 任务上交叉检查了这一比率，也仅观察到一次。在唯一的 Terminal-Bench 4 案例 layout-config-recreation 中，压缩将 1,450 条消息减少为 3 条（约 834k 到约 7.8k Token），同时保留了交付路径、测量分数、拒绝的替代方案和显式的恢复点。

What is missing instead is a reliable mechanism for verifying completion. Of the 45 failures, 35 ended with a final report, and our review of each one found that all 35 state that every specified requirement has been verified (Figure 4). Three did not lack a signal: the agent observed a discrepancy while self-checking but attributed it to a cause outside the deliverable, namely a defect in its own test harness, a fix already applied in the current run, or pre-existing upstream behavior, and stopped treating it as unresolved. This is a characteristic self-verification failure [5, 3]; in all three, the item dismissed is a test the grader subsequently failed.缺失的是一种可靠的完成验证机制。在 45 个失败中，35 个以最终报告结束，我们对每一份报告的审查发现，所有 35 份报告都声明每项指定要求已得到验证（图 4）。有三个任务并不缺乏信号：智能体在自检时观察到了差异，但将其归因于交付物之外的原因，即其自身测试框架中的缺陷、当前运行中已应用的修复或预先存在的上游行为，并停止将其视为未解决。这是一种典型的自验证失败 [5, 3]；在这三个案例中，被驳回的项目都是评分器随后判定失败的测试。

The failure is structural. No harness exposes the acceptance criterion to the agent during the run, so every configuration must construct its own, and the evidence a final check rests on is therefore produced by the agent itself. The 35 self-certified reports show that checking cannot substitute for supplying the criterion against which to check. Nor is the pattern unique to openJiuwen: on layout-config-recreation, PI with the same model failed the same two tests.这种失败是结构性的。没有任何框架在运行期间向智能体公开验收标准，因此每个配置都必须构建自己的标准，最终检查所依据的证据因此是由智能体自身产生的。35 份自认证报告表明，检查不能代替提供检查所依据的标准。这种模式并非 openJiuwen 所独有：在 layout-config-recreation 上，使用相同模型的 PI 也未能通过同样的两个测试。