/Pydantic AI/Pydantic AI

What makes a good harness什么构成了一个好的 harness

David Sanchez avatar
David SanchezDavid Sanchez
7 mins7 分钟

Three years ago, a model just predicted the next token. It could write you a story, but it could not look anything up, run anything, or check its own work. We wrapped it in a loop: call the model, run a tool, feed the result back, repeat, and called that an agent. We built a system around the agent to keep it reliable on real work, and called that a harness.三年前,一个模型只能预测下一个 token。它可以为你写故事,但它无法查找信息、运行任何东西或检查自己的工作。我们把它放进一个循环中:调用模型,运行工具,将结果反馈回去,重复,这就叫做代理。我们围绕代理构建了一个系统,使其在真实工作中保持可靠,这就叫做 harness。

The harness thesis made the case that long-running agents need that harness. This article is about what makes one good.harness 论点指出,长期运行的代理需要 harness。本文讨论了什么使得 harness 优秀。

A harness is everything around the agent loop that turns a model's output into verified, durable work instead of a transcript you have to read and trust. The familiar pieces:harness 是围绕代理循环的所有内容,它将模型的输出转化为经过验证、持久的工作,而不是需要你阅读并信任的文字记录。常见组成部分包括:

  • Tools the agent can call safely代理可以安全调用的工具
  • Memory that survives a single model call在单次模型调用后仍然保留的记忆
  • Guardrails that catch a bad action before it lands在错误行为发生前捕获它的防护栏
  • Context management that keeps the window full of what matters保持窗口中充满重要信息的上下文管理

What makes a harness good is timing: getting the right text to the model at the right moment. Ryan Lopopolo, who works on Codex at OpenAI, reduces it to one line: "all the harness should do is surface instructions to the model at the right time."1 That comes up twice over a run: what the model is told before it acts, and how it gets caught once it starts to drift.一个好的 harness 的关键在于时机:在正确的时刻把正确的文本送给模型。OpenAI 的 Codex 工作者 Ryan Lopopolo 将其简化为一句话:“harness 所要做的就是在恰当的时机向模型呈现指令。”1 这在一次运行中会出现两次:模型在行动前被告知的内容,以及它开始偏离时被捕获的方式。

The instinct, when you want an agent to behave, is to tell it everything up front. Every rule, every tool, loaded into the prompt before the first turn. This backfires two ways:当你希望代理表现良好时,本能的做法是一次性把所有信息都告诉它。把每条规则、每个工具都放进提示的开头。这会适得其反,表现为两种情况:

  • Front-load every instruction and you overwhelm the model before it has done any work.把所有指令前置,会在模型完成任何工作之前就把它压垮。
  • Front-load every tool and you spend the context window describing capabilities instead of using them.把所有工具前置,会在上下文窗口中花费大量空间描述能力,而不是实际使用它们。

The second cost is measurable. Cloudflare found an MCP server with thousands of endpoints would burn 1.17 million tokens on tool definitions before the user said a word.2 Anthropic hit the same wall and fixed it the same way: discover tools on demand, and one workload dropped from 150,000 tokens to 2,000.3第二种成本是可衡量的。Cloudflare 发现一个拥有数千端点的 MCP 服务器在用户说话前会消耗 1.17 百万 token 用于工具定义。2 Anthropic 遇到同样的问题并以相同方式解决:按需发现工具,单个工作负载的 token 从 150,000 降至 2,000。3

So a good harness defers. It shows the model what is available in one line, and hands over the full instructions and tools only when the model reaches for them.因此,一个好的 harness 会延迟加载。它用一行文字向模型展示可用内容,仅在模型需要时才交付完整的指令和工具。

Lopopolo's example is a coding agent. Don't load the rules for decomposing a React component at the start. Let the agent prototype, then surface the rule at lint time, when it becomes relevant.1 The agent meets the instruction at the moment it can act on it.Lopopolo 的例子是一个编码代理。不要在开始时加载分解 React 组件的规则。让代理先进行原型设计,然后在 lint 时呈现规则,当规则变得相关时再提供。1 代理在能够执行指令的瞬间收到指令。

In Pydantic AI v2, this is a flag on a capability. Mark it defer_loading=True and it stays out of the prompt until the model loads it: the model sees a one-line description in a catalog, then pulls in the whole bundle, instructions and tools together, in one step. ToolSearch does the same for tools, so an agent can carry hundreds and pay for the handful it uses.在 Pydantic AI v2 中,这通过能力上的一个标志实现。将其设为 defer_loading=True,直到模型加载时才出现在提示中:模型先在目录中看到一行描述,然后一次性拉取完整的捆绑包,包括指令和工具。ToolSearch 对工具也采用相同机制,使代理可以携带数百个工具,却只为使用的少数付费。

Thanks to Pydantic Monty, our sandboxed Python subset, we can take that further. With code mode the agent writes code that calls tools as functions and discovers them as it goes, instead of reading a schema for every one. We learned this building tools for Pydantic Logfire: after forty-plus of them, we realized "we didn't have an MCP server, we had an API with delusions of grandeur," and the model did better writing a few lines of Python than picking from a wall of unlabeled buttons.4多亏了 Pydantic Monty——我们的受限 Python 子集,我们可以更进一步。使用代码模式,代理编写调用工具的代码,并在运行时发现它们,而不是为每个工具读取模式。我们在为 Pydantic Logfire 构建工具时学到:在四十多个工具后,我们意识到“我们没有 MCP 服务器,而是拥有一个自负的 API”,模型写几行 Python 的效果要好于从一堆未标记的按钮中挑选。4

The rule under both: information should reach the model when it is needed, not before.两者的共同规则是:信息应在模型需要时送达,而不是提前。

Disclosure decides what the model sees. Steering catches the run when it goes wrong anyway.披露决定模型看到什么。引导则在运行出错时进行拦截。

Every long run drifts. The agent wanders down a dead end, misreads a result, or keeps going when it should have stopped to ask. What matters is how far it gets before anything notices. Steering shortens the distance between a mistake and its correction.每一次长时间运行都会漂移。代理可能走进死胡同、误读结果,或在应该停下来询问时继续前进。关键在于它在被发现之前走了多远。引导缩短了错误与纠正之间的距离。

Lopopolo's test for this is sharp:Lopopolo 对此的测试非常犀利:

Every time I have to type "continue" to the agent is a failure of the harness to provide enough context around what it means to continue to completion.1每当我必须对代理输入“continue”时,说明 harness 未能提供足够的上下文来说明继续到底意味着什么。1

If a human has to nudge the loop by hand, the harness left something out.如果需要人工手动推动循环,说明 harness 漏掉了某些内容。

Steering comes from two places:引导来源于两个方面:

  • A check the harness runs: verify the artifact before the agent declares victory, or force a different move after the same tool gets called fifty times.harness 自己运行的检查:在代理宣告成功前验证产物,或在同一工具被调用五十次后强制采取不同的动作。
  • A signal from outside the run: a production alert that redirects a live agent mid-task without stopping it.来自运行外部的信号:生产警报在任务进行中重定向实时代理,而不停止它。

Pydantic AI v2 ships the second case as a pending message queue: a message pushed into a live run is held until the current step finishes, then delivered on the next turn, so a steer never lands mid-thought. It is new enough that we are still growing what you can push into a running agent.Pydantic AI v2 将第二种情况实现为待处理消息队列:推送到实时运行的消息会等到当前步骤结束后才在下一轮交付,从而避免在思考中途插入引导。该功能仍在成长阶段,我们仍在探索可以向运行中的代理推送哪些内容。

Disclose and steer aren't two features you bolt on. In Pydantic AI v2 they are the same primitive seen twice: the capability.披露和引导并不是可以随意添加的两个特性。在 Pydantic AI v2 中,它们是同一种原语的两次呈现:能力。

A capability bundles an agent's instructions, tools, hooks, and model settings into one composable unit, and you attach it by adding it to a list. The powerful ones use hooks to read and rewrite what the model sees on every step: its tools, its instructions, its message history. That is how a capability defers its own tools, trims a result before it overflows the window, or slips in a reminder without poisoning the durable history.能力将代理的指令、工具、钩子和模型设置打包成一个可组合单元,你只需将其加入列表即可。强大的能力会使用钩子在每一步读取并重写模型看到的内容:工具、指令、消息历史。这就是能力能够延迟自身工具、在结果溢出窗口前裁剪、或在不污染持久历史的情况下插入提醒的方式。

The first-party capabilities live in the Pydantic AI Harness, the batteries for your agent: file system access, code mode, context management, guardrails. Core stays small, the harness moves fast, and a capability graduates into core once it proves essential.第一方能力位于 Pydantic AI Harness 中,提供你的代理所需的电池:文件系统访问、代码模式、上下文管理、防护栏。核心保持精简,harness 迭代快速,能力一旦被证明是必需的就会升级为核心。

Capabilities can also be built dynamically, per run. That is the first hint of something larger: a harness that assembles its own capabilities while it runs is a short step from a harness that assembles agents.能力也可以在运行时动态构建。这是更大事物的第一条线索:一个在运行时自行组装能力的 harness 与一个在运行时自行组装代理的系统只差一步。

A harness is mostly a list you compose. Here a coding agent gets scoped file access, code mode, and tools it can find on demand instead of all at once:harness 大多是你组合的列表。下面的编码代理获得了受限的文件访问、代码模式以及按需发现的工具,而不是一次性全部加载:

from pydantic_ai import Agent
from pydantic_ai.capabilities import ToolSearch
from pydantic_ai_harness import CodeMode, FileSystem

capabilities = [
    FileSystem(root_dir='.'),  # scoped reads and writes, no traversal above the root
    CodeMode(),                # write code that calls tools, sandboxed by Monty
    ToolSearch(),              # discover tools on demand, not all up front
]

agent = Agent(
    'anthropic:claude-opus-4-7',
    instructions='Fix the issue, then verify the fix.',
    capabilities=capabilities,
)

Our own coding agent, loopy (it runs headless or live in our chat UI), builds that list per run: it reads the repo it is pointed at, turns the skills it finds there into capabilities, and spins up new sub-agents as it works.我们自己的编码代理 loopy(可以无头运行或在聊天 UI 中实时运行),会在每次运行时构建该列表:读取指向的仓库,将发现的技能转化为能力,并在工作时启动新的子代理。

Instrumentation is now a capability too, so every model call, tool call, and hook lands on one OpenTelemetry timeline in Pydantic Logfire. A harness you cannot see is a harness you cannot fix, and those traces are what later let a harness reason about its own runs.仪表化现在也是一种能力,因此每一次模型调用、工具调用和钩子都会记录到 Pydantic Logfire 中的同一条 OpenTelemetry 时间线。看不见的 harness 是修复不了的 harness,而这些追踪正是后续让 harness 能够自我推理运行的依据。

The agent was already a loop: a model calling tools until it hits its goal. The next word borrows the term for something bigger, a loop of agents, and the gap between the two is real.代理本身已经是一个循环:模型调用工具直至达成目标。下一个词借用了更大的概念——代理的循环,而两者之间的差距是真实存在的。

A harness arms a single run and makes it reliable. You reach for a loop when one run is not enough. Two things set it apart from anything a single harnessed agent can do:harness 为单次运行提供武装并保证可靠。当一次运行不足以完成任务时,就需要循环。它与单一 harness 代理能做的事情有两点不同:

  • It chooses its own structure. Instead of following a plan you wrote, the loop decides how to split the work. It spawns sub-agents for the pieces, gives each only the tools its job needs, checks their results, and merges them back. The decomposition is the system's call, not yours.它自行决定结构。循环不遵循你写的计划,而是自行决定如何拆分工作。它为各个子任务生成子代理,只为每个子代理分配其工作所需的工具,检查结果并将其合并。拆解是系统的调用,而不是你的指令。
  • It outlives the run. The loop can go idle and wake on an outside event: a closed PR, a failed test, a teammate's reply. It survives restarts, carries state across long horizons, and takes new context mid-flight instead of starting over.它的寿命超出单次运行。循环可以闲置并在外部事件触发时唤醒:关闭的 PR、失败的测试、同事的回复。它能够跨重启存活,跨越长时间跨度携带状态,并在飞行中接受新上下文,而不是重新开始。

Loopy's live run-graph: a goal split into subgoals that each carry their own verification, moving through Triage, Plan, Implement, Verify, Review, and Done, with a running decisions log.

So a loop is not a bigger harness. It is a system that assembles its own agents and stays alive between answers.因此循环并不是更大的 harness,而是一个在答案之间组装自身代理并保持存活的系统。

That is what the Pydantic AI Harness is chasing, and where this series goes next.这正是 Pydantic AI Harness 所追求的目标,也是本系列接下来要探讨的方向。


This is part one of three on where harnesses are heading. Next: what changes when agents start building agents.这是关于 harness 未来走向的三篇系列的第一篇。下一篇:当代理开始构建代理时会发生什么变化。

To feel the ideas here first, build an agent with the Pydantic AI examples, add a capability or two from the Harness capability matrix, and watch it run on one timeline in Logfire.想先感受这些想法,请使用 Pydantic AI 示例构建一个代理,添加一两个来自 Harness 能力矩阵的能力,然后在 Logfire 的同一时间线中观察其运行。

Footnotes脚注

  1. Ryan Lopopolo, "Harness Engineering: How to Build Software When Humans Steer, Agents Execute", AI Engineer. 2 3Ryan Lopopolo, "Harness Engineering: How to Build Software When Humans Steer, Agents Execute", AI Engineer. ↩ ↩2 ↩3

  2. Cloudflare, "Code Mode: the better way to use MCP". Cloudflare, "Code Mode: the better way to use MCP". ↩

  3. Anthropic, "Code execution with MCP: building more efficient agents". Anthropic, "Code execution with MCP: building more efficient agents". ↩

  4. Jiri Kuncar, "Your agent would rather write code", Pydantic. Jiri Kuncar, "Your agent would rather write code", Pydantic. ↩