Improving Deep Agents with harness engineering通过装备工程改进深度智能体

The Knobs we can Turn我们可以调节的旋钮

An agent harness has a lot of knobs: system prompts, tools, hooks/middleware, skills, sub-agent delegation, memory systems, and more. We deliberately compress the optimization space and focus on three: System Prompt, Tools, and Middleware (our term for hooks around model and tool calls).智能体装备有很多旋钮：系统提示、工具、钩子/中间件、技能、子智能体委派、记忆系统等。我们有意压缩优化空间，专注于三个：系统提示、工具和中间件（我们对模型和工具调用周围的钩子的术语）。

We start with a default prompt and standard tools+middleware. This scores 52.8% with GPT-5.2-Codex. A solid score, just outside the Top 30 of the leaderboard today, but room to grow.我们从默认提示和标准工具+中间件开始。使用 GPT-5.2-Codex 得分为 52.8%。这是一个不错的分数，刚好在排行榜 Top 30 之外，但有提升空间。

The Trace Analyzer Skill追踪分析技能

We wanted trace analysis to be repeatable so we made it into an Agent Skill. This serves as our recipe to analyze errors across runs and make improvements to the harness. The flow is:我们希望追踪分析可重复，因此将其制作成智能体技能。这作为我们的配方，用于分析运行中的错误并对装备进行改进。流程是：

1. Fetch experiment traces from LangSmith从 LangSmith 获取实验追踪
2. Spawn parallel error analysis agents → main agent synthesizes findings + suggestions生成并行错误分析智能体 → 主智能体综合发现和建议
3. Aggregate feedback and make targeted changes to the harness.汇总反馈并对装备进行有针对性的更改。

This works similarly to boosting which focuses on mistakes from previous runs. A human can be pretty helpful in Step 3 (though not required) to verify and discuss proposed changes. Changes that overfit to a task are bad for generalization and can lead to regressions in other Tasks.这类似于提升方法，专注于之前运行中的错误。人类在步骤 3 中可能很有帮助（尽管不是必需的），以验证和讨论提议的更改。过度拟合某个任务的更改不利于泛化，并可能导致其他任务中的回归。

Automated trace analysis saves hours of time and made it easy to quickly try experiments. We’ll be publishing this skill soon, we’re currently testing it for prompt optimization generally.自动化追踪分析节省了数小时的时间，并使得快速尝试实验变得容易。我们将很快发布这个技能，目前正在测试它用于一般的提示优化。

Build & Self-Verify构建与自我验证

Today’s models are exceptional self-improvement machines.今天的模型是卓越的自我改进机器。

Self-verification allows agents to self-improve via feedback within a run. However, they don’t have a natural tendency to enter this build-verify loop.自我验证允许智能体通过运行内的反馈进行自我改进。然而，它们没有自然倾向进入这种构建-验证循环。

The most common failure pattern was that the agent wrote a solution, re-read its own code, confirmed it looks ok, and stopped. Testing is a key part of autonomous agentic coding. It helps test for overall correctness and simultaneously gives agents signal to hill-climb against.最常见的失败模式是智能体编写了一个解决方案，重新阅读了自己的代码，确认看起来没问题，然后停止。测试是自主智能体编码的关键部分。它有助于测试整体正确性，同时为智能体提供爬山信号。

We added guidance to the system prompt on how to approach problem solving.我们在系统提示中添加了关于如何解决问题的指导。

1. Planning & Discovery: Read the task, scan the codebase, and build an initial plan based on the task specification and how to verify the solution.规划与发现：阅读任务，扫描代码库，并根据任务规范和如何验证解决方案制定初始计划。
2. Build: Implement the plan with verification in mind. Build tests, if they don’t exist and test both happy paths and edge cases.构建：在考虑验证的情况下实施计划。构建测试，如果不存在，则测试快乐路径和边缘情况。
3. Verify: Run tests, read the full output, compare against what was asked (not against your own code).验证：运行测试，读取完整输出，与要求进行比较（而不是与自己的代码比较）。
4. Fix: Analyze any errors, revisit the original spec, and fix issues.修复：分析任何错误，重新审视原始规范，并修复问题。

We really focus on testing because it powers the changes in every iteration. We found that alongside prompting, deterministic context injection helps agents verify their work. We use a PreCompletionChecklistMiddleware that intercepts the agent before it exits and reminds it to run a verification pass against the Task spec. This is similar to a Ralph Wiggum Loop where a hook forces the agent to continue executing on exit, we use this for verification.我们非常关注测试，因为它驱动每次迭代中的更改。我们发现，除了提示之外，确定性上下文注入有助于智能体验证其工作。我们使用 PreCompletionChecklistMiddleware，它在智能体退出前拦截并提醒它根据任务规范运行验证。这类似于 Ralph Wiggum 循环，其中钩子强制智能体在退出时继续执行，我们将其用于验证。

Giving Agents Context about their Environment为智能体提供环境上下文

Part of harness engineering is building a good delivery mechanism for context engineering. Terminal Bench tasks come with directory structures, built-in tooling, and strict timeouts.装备工程的一部分是为上下文工程构建良好的传递机制。Terminal Bench 任务带有目录结构、内置工具和严格超时。

1. Directory Context & Tooling: A LocalContextMiddleware runs on agent start to map the cwd and other parent+children directories. We run bash commands to find tools like Python installations. Context discovery and search are error prone, so injecting context reduces this error surface and helps onboard the agent into its environment.目录上下文与工具：LocalContextMiddleware 在智能体启动时运行，映射当前工作目录和其他父/子目录。我们运行 bash 命令来查找 Python 安装等工具。上下文发现和搜索容易出错，因此注入上下文减少了这个错误面，并帮助智能体适应其环境。
2. Teaching Agents to Write Testable Code: Agents don’t know how their code needs to be testable. We add prompting say their work will be measured against programatic tests, similar to when committing code. For example, Task specs that mention file paths should be followed exactly so the solutions works in an automated scoring step. Prompting that stresses edge-cases helps the agent avoid only checking “happy path” cases. Forcing models to conform to testing standards is a powerful strategy to avoid “slop buildup” over time.教导智能体编写可测试代码：智能体不知道它们的代码需要如何可测试。我们添加提示，说它们的工作将根据程序化测试进行衡量，类似于提交代码时。例如，提到文件路径的任务规范应严格遵循，以便解决方案在自动化评分步骤中有效。强调边缘情况的提示有助于智能体避免只检查“快乐路径”情况。强制模型符合测试标准是避免随时间“积累错误”的强大策略。
3. Time Budgeting: We inject time budget warnings to nudge the agent to finish work and shift to verification. Agents are famously bad at time estimation so this heuristic helps in this environment. Real world coding usually doesn’t have strict time limits, but without adding any knowledge of constraints, agents won’t work within time bounds.时间预算：我们注入时间预算警告，以促使智能体完成工作并转向验证。智能体在时间估计方面非常糟糕，因此这种启发式方法在这种环境中有所帮助。现实世界的编码通常没有严格的时间限制，但如果不添加任何约束知识，智能体将无法在时间范围内工作。

The more that agents know about their environment, constraints, and evaluation criteria, the better they can autonomously self-direct their work.智能体对其环境、约束和评估标准了解得越多，它们就能更好地自主指导自己的工作。

The purpose of the harness engineer: prepare and deliver context so agents can autonomously complete work.装备工程师的目的：准备和传递上下文，以便智能体能够自主完成工作。

Encouraging Agents to Step Back & Reconsider Plans鼓励智能体退后一步重新考虑计划

Agents can be myopic once they’ve decided on a plan which results in “doom loops” that make small variations to the same broken approach (10+ times in some traces).智能体一旦决定了计划，可能会变得短视，导致“厄运循环”，对同一错误方法进行微小变化（在某些追踪中超过 10 次）。

We use a LoopDetectionMiddleware that tracks per-file edit counts via tool call hooks. It adds context like “…consider reconsidering your approach” after N edits to the same file. This can help agents recover from doom loops, though the model can continue down the same path if it thinks it’s correct.我们使用 LoopDetectionMiddleware，通过工具调用钩子跟踪每个文件的编辑次数。在对同一文件进行 N 次编辑后，它会添加上下文，如“……考虑重新考虑你的方法”。这可以帮助智能体从厄运循环中恢复，尽管如果模型认为正确，它可能会继续沿着同一路径前进。

Important note. This is a design heuristic that engineers around today’s perceived model issues. As models improve, these guardrails will likely be unnecessary, but today helps agents execute correctly and autonomously.重要说明。这是一种设计启发式方法，围绕今天感知到的模型问题进行工程。随着模型的改进，这些护栏可能不再必要，但今天有助于智能体正确自主地执行。

Choosing How Much Compute to Spend on Reasoning选择在推理上花费多少计算资源

Reasoning models can run autonomously for hours so we have to decide how much compute to spend on every subtask. You can use the max reasoning budget on every task, but most work can benefit from optimizing reasoning compute spend.推理模型可以自主运行数小时，因此我们必须决定在每个子任务上花费多少计算资源。你可以在每个任务上使用最大推理预算，但大多数工作可以从优化推理计算花费中受益。

Terminal Bench timeout limits create a tradeoff. More reasoning helps agents evaluate each step, but can burn over 2x more tokens/time. gpt-5.2-codex has 4 reasoning modes, low, medium, high, and xhigh.Terminal Bench 超时限制造成了权衡。更多推理有助于智能体评估每一步，但可能消耗超过 2 倍的令牌/时间。gpt-5.2-codex 有 4 种推理模式：低、中、高和极高。

We found that reasoning helps with planning to fully understand the problem, some Terminal Bench tasks are very difficult. A good plan helps get to a working solution more quickly.我们发现推理有助于规划以充分理解问题，一些 Terminal Bench 任务非常困难。一个好的计划有助于更快地获得可行的解决方案。

Later stage verification also benefits from more reasoning to catch mistakes and get a solution submitted. As a heuristic, we choose a xhigh-high-xhigh "reasoning sandwich" as a baseline.后期验证也受益于更多推理，以捕捉错误并提交解决方案。作为一种启发式方法，我们选择极高-高-极高的“推理三明治”作为基线。

Running only at xhigh scored poorly at 53.9% due to agent timeouts compared to 63.6% at high. There weren’t large differences in trial runs across reasoning budget splits so we stuck with our approach which pushed the score to 66.5%.仅以极高模式运行得分较低，为 53.9%，因为智能体超时，而高模式为 63.6%。在推理预算分配的试验运行中没有太大差异，因此我们坚持我们的方法，将分数推至 66.5%。

The natural approach for models is Adaptive Reasoning, seen with Claude and Gemini models where the model decides how much compute to spend on reasoning.模型的自然方法是自适应推理，如 Claude 和 Gemini 模型所见，模型决定在推理上花费多少计算资源。

In a multi-model harness, balancing reasoning budgets could play out as using a large model for planning and handing off to a smaller model for implementation.在多模型装备中，平衡推理预算可能表现为使用大型模型进行规划，并将实现交给较小的模型。