written on July 04, 2026写于 2026 年 7 月 4 日
A very strange Pi issue
sent me down a rabbit hole over the last two days. The short version is that
newer Claude models sometimes call Pi’s edit tool with extra, invented fields in
the nested edits[] array. And not Haiku or some small model: Opus 4.8. The
edit itself is usually correct but the arguments do not match the schema as
the model invents made-up keys and Pi thus rejects the tool call and asks to
try again.过去两天里，一个非常奇怪的 Pi 问题让我陷入了困境。简单来说，较新的 Claude 模型有时会在嵌套的 edits[] 数组中调用 Pi 的编辑工具时，加入额外且凭空捏造的字段。而且这并非 Haiku 或其他小模型的问题：Opus 4.8 也会这样。编辑内容本身通常是正确的，但参数与模式不匹配，因为模型生成了虚构的键，导致 Pi 拒绝了工具调用并要求重试。
That alone is not too surprising as models emit malformed tool calls sometimes. Particularly small ones. What surprised me is that this is getting worse with newer Anthropic models as both Opus 4.8 and Sonnet 5 show it but none of the older models. In other words, the SOTA models of the family are worse at this specific tool schema than their older siblings.单看这一点并不令人惊讶，因为模型有时确实会发出格式错误的工具调用，尤其是较小的模型。令我惊讶的是，随着 Anthropic 模型更新，这种情况反而变严重了：Opus 4.8 和 Sonnet 5 都会出现此问题，而旧版模型则不会。换句话说，该系列中性能最强的模型在处理这种特定工具模式时，表现反而不如它们的“前辈”。
In case you are curious about Fable: I intentionally did not test it because I was not sure if the classifiers they are running might downgrade me to Opus silently.如果你对 Fable 感兴趣：我特意没有测试它，因为我不确定他们运行的分类器是否会在后台悄悄地将我降级到 Opus。
If you have not spent too much time looking at LLM tool calling internals, the important thing to understand is that tool calls are not magic and use some rather crude in-band signalling. The model receives a transcript, a system prompt and a list of available tools. The server munches that into a large prompt with special marker tokens. Because the model was trained and reinforced on examples of that format, at some point during generation it emits something that the API or client interprets as “call this tool with these arguments”.如果你没有花太多时间研究 LLM 工具调用的内部机制，那么需要理解的关键一点是：工具调用并非魔法，而是使用了一些相当原始的带内信令（in-band signalling）。模型会接收到一个转录文本、一个系统提示词以及可用工具列表。服务器会将这些内容整合进一个带有特殊标记符的大型提示词中。由于模型是在该格式的示例上进行训练和强化学习的，因此在生成过程中的某个时刻，它会输出一些被 API 或客户端解释为“用这些参数调用此工具”的内容。
For a file edit tool, the intended invocation payload might say something like this:对于文件编辑工具，预期的调用载荷可能如下所示：
{
"path": "some/file.py",
"edits": [
{
"oldText": "text to replace",
"newText": "replacement text"
}
]
}
A harness then validates the arguments, performs the edit, and feeds the result back into the model. If validation fails, the model sees an error and usually tries again.随后，一个工具接口（harness）会验证参数、执行编辑并将结果反馈给模型。如果验证失败，模型会看到错误，通常会再次尝试。
How exactly that formatting happens is not known for the Anthropic models, but some people have gotten out “ANTML” markers and they at times do leak also into public communications. To the best of my knowledge, the call above would come out serialized like this from the model:Anthropic 模型具体是如何进行格式化的尚不明确，但有些人已经提取出了“ANTML”标记，这些标记有时也会泄露到公开通信中。据我所知，上述调用在模型输出时会序列化为：
<antml:function_calls>
<antml:invoke name="edit">
<antml:parameter name="path">some/file.py</antml:parameter>
<antml:parameter name="edits">
[
{
"oldText": "text to replace",
"newText": "replacement text"
}
]
</antml:parameter>
</antml:invoke>
</antml:function_calls>
An important thing to note here is that this thing, while looking like XML, is not really XML. It’s just a thing they found convenient to tokenize and train on. The other thing to note is that a basic top-level string parameter appears in-line whereas an array of objects is implemented via JSON serialization. While I’m not entirely sure that this is how it works, there are some indications that this is not too far off. This will become relevant later.需要注意的一点是，虽然它看起来像 XML，但实际上并不是 XML。它只是他们觉得方便进行分词和训练的一种格式。另一点需要注意的是，基本的顶级字符串参数是内联出现的，而对象数组则是通过 JSON 序列化实现的。虽然我不完全确定其运作机制，但有一些迹象表明情况大致如此。这一点稍后会变得很重要。
There are two very different ways to make the model produce a structure like this:有两种截然不同的方法可以让模型生成这种结构：
The second approach is what people usually refer to as grammar-aware or
constrained decoding. The sampler masks out tokens that would violate the
grammar. If the model is currently inside a JSON object and the schema says
only oldText and newText are allowed, the sampler can prevent it from
emitting "in_file" or "type". Grammar-aware decoding can be used both to
constrain something to be syntactically valid JSON and also to enforce specific
enum values or keys.第二种方法通常被称为“语法感知”或“受限解码”。采样器会屏蔽掉违反语法的标记。如果模型当前处于一个 JSON 对象内部，且模式规定只允许 oldText 和 newText，采样器可以阻止它输出 "in_file" 或 "type"。语法感知解码既可以用于约束语法上的有效 JSON，也可以用于强制执行特定的枚举值或键。
Without any form of constraints the model is merely following a learned convention.在没有任何形式约束的情况下，模型仅仅是在遵循一种习得的惯例。
Pi’s edit tool supports multiple exact string replacements in one call. That is
why the arguments contain an edits array. In the failing cases the model
produces entries like this:Pi 的编辑工具支持在一次调用中进行多次精确的字符串替换。这就是为什么参数包含一个 edits 数组的原因。在失败的情况下，模型产生的条目如下：
{
"oldText": "...",
"newText": "...",
"requireUnique": true
}
or this:或者这样：
{
"oldText": "...",
"newText": "...",
"oldText2": "",
"newText2": ""
}
Across repeated trials I saw a whole zoo of invented trailing keys: type,
id, kind, unique, requireUnique, matchCase, in_file,
forceMatchCount, children, notes, cost, oldText2, newText2,
oldText_2, newText_2, and even an event.0.additionalProperties key inside
the edit object itself.在多次试验中，我看到了各种凭空捏造的尾随键：type、id、kind、unique、requireUnique、matchCase、in_file、forceMatchCount、children、notes、cost、oldText2、newText2、oldText_2、newText_2，甚至在编辑对象内部还有一个 event.0.additionalProperties 键。
The most annoying part is that the actual oldText and newText payloads were
byte-correct in the invalid calls I inspected. The model had in fact produced
the right invocation but then added nonsense at the end of the object.最令人恼火的是，在我检查的无效调用中，实际的 oldText 和 newText 载荷在字节层面是正确的。模型实际上已经生成了正确的调用，但随后在对象末尾添加了无意义的内容。
The failure is also heavily context-dependent. A fresh single-turn prompt like “edit this file” did not reproduce it at all for me. An agentic history where the model had read files, diagnosed a problem and then composed a multi-line edit could reproduce it. And more annoyingly, not all transcripts will show that behavior. In fact, I needed Petr Baudis‘s transcripts to reproduce this for me at all! In that user’s session continuing the session caused Opus 4.8 to fail around 20% of the time. Stripping thinking blocks from history reduced the failure rate by half. Turning on strict tool invocation eliminated it in my runs.这种故障也高度依赖于上下文。一个诸如“编辑此文件”的全新单轮提示词完全无法复现该问题。只有在模型读取过文件、诊断出问题并编写了多行编辑的代理历史记录中，才能复现它。更令人沮丧的是，并非所有转录文本都会表现出这种行为。事实上，我需要 Petr Baudis 的转录文本才能复现它！在该用户的会话中，继续该会话会导致 Opus 4.8 有约 20% 的概率失败。从历史记录中剔除思维块（thinking blocks）可将故障率降低一半。在我的运行中，开启严格工具调用（strict tool invocation）则完全消除了该问题。
My strongest hypothesis is that this is not random deterioration but a training artifact.我最强烈的假设是，这并非随机的退化，而是一种训练伪影（training artifact）。
When older Anthropic models were trained, they were trained on some tools (some of which were documented). But that training did not yet have a user-shipped harness like Claude Code as the obvious target. Modern Anthropic models are most likely different because their post-training includes Claude Code or a harness that looks very similar. The model learns what a successful tool call looks like in that environment. It also learns what mistakes are tolerated by that environment.当旧版 Anthropic 模型接受训练时，它们学习了一些工具（其中一些是有文档记录的）。但当时的训练还没有像 Claude Code 这样用户可用的工具接口作为明确目标。现代 Anthropic 模型很可能不同，因为它们的后训练（post-training）包含了 Claude Code 或看起来非常相似的接口。模型学习了在该环境中成功的工具调用是什么样的，也学习了该环境能容忍哪些错误。
Claude Code’s own tools are comparatively flat. The ordinary edit tool is not
Pi’s nested edits[] shape; it is closer to file_path, old_string,
new_string, and an optional flag (replace_all). Looking at Claude Code’s
client is very instructive: it contains retry paths for malformed tool use,
parameter aliases, type coercions, Unicode repairs and filtering of unknown
keys. In other words, Anthropic’s own client appears to expect and accept a
fair amount of slop and repairs it, mostly silently.Claude Code 自身的工具相对扁平。普通的编辑工具不是 Pi 那种嵌套的 edits[] 形状；它更接近 file_path、old_string、new_string 和一个可选标志（replace_all）。查看 Claude Code 的客户端非常有启发性：它包含了针对格式错误工具使用的重试路径、参数别名、类型强制转换、Unicode 修复以及未知键的过滤。换句话说，Anthropic 自己的客户端似乎预期并接受相当多的“垃圾”输入，并会默默地修复它们。
If reinforcement learning happens in a harness like that, or a simulation of one, then slightly malformed tool calls can still complete the task and receive reward. The harness fully absorbs the error and there is little gradient against inventing an alias, adding a stray field or using a nearby parameter name.如果强化学习是在这样的接口（或其模拟环境）中进行的，那么稍微格式错误的工具调用仍然可以完成任务并获得奖励。接口完全吸收了错误，因此对于凭空捏造别名、添加多余字段或使用相近参数名称的行为，几乎没有梯度惩罚。
Worse, the model may become very strongly adapted to the canonical Claude Code edit tool shape. A different harness can present a tool with the same semantic intent but a different schema. Such a tool can increasingly be off-distribution. The better-trained model might actually fight you harder because its prior is stronger.更糟糕的是，模型可能会非常强烈地适应标准的 Claude Code 编辑工具形状。另一个接口可能会提供一个具有相同语义意图但模式不同的工具。这样的工具可能会越来越偏离其分布（off-distribution）。训练得更好的模型反而可能因为其先验知识更强而更难配合你。
This is not too surprising, but it is a change from how this was a few months ago. When Opus 4.5 launched, it adapted to other edit tools exceptionally well. In fact, I was pretty convinced that we’re on a good path where the models are more likely to adapt to any sort of tool shape that comes around for as long as the instructions are good.这并不令人惊讶，但这与几个月前的情况相比确实是一种变化。当 Opus 4.5 发布时，它对其他编辑工具的适应能力非常出色。事实上，我曾坚信我们正走在一条正确的道路上，即只要指令清晰，模型就更有可能适应任何出现的工具形状。
Now I’m somewhat worried about the track we’re on here. Alternative tool schemas might not just be unfamiliar. They might be implicitly punished by post-training that optimizes for one particular, forgiving tool ecology. And that ecology is not documented. While there is a text editor tool that is documented, you will see that this format is in fact not followed by Claude Code. What Claude Code does internally (which is a closed-source harness) is hidden from you.现在，我对我们所处的轨道感到有些担忧。替代性的工具模式可能不仅仅是不熟悉，它们还可能因为后训练过程优化了某一个特定的、宽容的工具生态系统而受到隐性惩罚。而这个生态系统并未被记录在案。虽然有一个有文档记录的文本编辑器工具，但你会发现 Claude Code 实际上并不遵循该格式。Claude Code 内部（这是一个闭源接口）的操作对你是隐藏的。
Claude Code is obviously closed-source but we can look at the minified code and get some idea of what it does. And honestly, it’s very forgiving of incoming data.Claude Code 显然是闭源的，但我们可以查看压缩后的代码，从而了解它的工作原理。老实说，它对传入的数据非常宽容。
For a start, Claude Code checks the model’s visible text for leaked <invoke
markup. It also emits some telemetry when that happens and then it has its
own state machine to retry such bad calls by pushing back to the model.首先，Claude Code 会检查模型可见文本中是否有泄露的 <invoke 标记。当这种情况发生时，它会发出一些遥测数据，然后通过自己的状态机将这些错误的调用推回给模型以进行重试。
It has explicit Unicode escape repair which fixes broken \uXXXX sequences and
lone surrogates in string values. It also has per-tool aliases for parameters.
For instance, Edit accepts old_str (presumably from the times when the models
were trained on the officially documented text editor tool), the newer old_string
from the schema, new_str/new_string, path as an alias for file_path, and some more.它具有显式的 Unicode 转义修复功能，可以修复字符串值中损坏的 \uXXXX 序列和孤立的代理对。它还为参数提供了针对特定工具的别名。例如，Edit 工具接受 old_str（大概是源于模型在官方文档记录的文本编辑器工具上进行训练的时期）、模式中较新的 old_string、new_str/new_string、作为 file_path 别名的 path 等等。
It also silently filters out unexpected keys and it does not use strict mode
either. The issue with strict mode is that Anthropic applies complexity
limits to the tool definitions that cause API requests to fail, so presumably
that’s why Claude Code does not attempt to use it.它还会默默地过滤掉意外的键，并且也没有使用严格模式（strict mode）。使用严格模式的问题在于，Anthropic 对工具定义施加了复杂性限制，这会导致 API 请求失败，所以想必这就是 Claude Code 不尝试使用它的原因。
Will this problem be with us in other harnesses too? One huge issue with Anthropic is that the models are completely closed, and so is the harness. Codex models are also closed, but at least the harness is not. We also have gpt-oss which is at least a bit interesting. The models are explicitly trained to use OpenAI’s harmony response format and there is a lot of documentation that at least tells us how OpenAI people think about this.这个问题是否也会出现在其他接口中？Anthropic 的一个巨大问题是模型是完全封闭的，接口也是如此。Codex 模型也是封闭的，但至少接口不是。我们还有 gpt-oss，它至少有点意思。这些模型被明确训练为使用 OpenAI 的 Harmony 响应格式，并且有大量文档至少告诉了我们 OpenAI 的人是如何思考这个问题的。
Harmony makes channels and tool-call content types part of the prompt format. A function call can look like this:Harmony 将通道和工具调用内容类型作为提示词格式的一部分。函数调用可能如下所示：
<|start|>assistant<|channel|>commentary to=functions.get_weather
<|constrain|>json<|message|>{"location":"San Francisco"}<|call|>
The important bit is <|constrain|>json. The model can express in-band that
this message body is JSON, and an inference stack can use that boundary to
switch into JSON-constrained sampling for the body of the tool call. Presumably
a bit of this also happens in Anthropic’s models, at least in strict mode
I would imagine.关键在于 <|constrain|>json。模型可以在带内表达此消息体是 JSON，推理栈可以使用该边界切换到 JSON 约束采样，用于工具调用的主体。想必 Anthropic 的模型中也发生了一些类似的情况，至少在严格模式下我想是这样。
The marker in harmony helps the sampler to detect when it needs to sample with a specific grammar, and because it is part of the transcript, it makes that rather easy to do. For hosted GPT models, there is also an option to provide a LARK grammar for custom tools that need to adhere to something like this.Harmony 中的标记有助于采样器检测何时需要使用特定语法进行采样，并且由于它是转录文本的一部分，这使得操作变得相当容易。对于托管的 GPT 模型，也有一个选项可以为需要遵守特定规则的自定义工具提供 LARK 语法。
Anthropic appears different from that, though maybe not entirely. If an array
of objects is represented as JSON, as it appears to be, then the model has to
write JSON inside the tool parameter. There is probably basic
grammar-constrained sampling going on, and that may partly explain the extra
keys. For a nested array parameter, that JSON includes escaped multi-line file
content inside string literals, inside one tag. The unexpected,
made-up keys appear exactly at the highest-entropy point of that task: after
closing a several-hundred-token escaped newText string, where the model must
decide } vs , "...".Anthropic 似乎与此不同，尽管可能并非完全不同。如果对象数组正如看起来的那样以 JSON 表示，那么模型必须在工具参数内编写 JSON。这可能存在基本的语法约束采样，这或许可以部分解释那些额外的键。对于嵌套数组参数，该 JSON 在一个标签内包含了字符串字面量中转义的多行文件内容。那些意想不到的、编造出来的键恰好出现在该任务熵最高的点：在关闭了一个包含数百个标记的转义 newText 字符串之后，模型必须决定是使用 } 还是 , "..."。
Opus 4.8 and Sonnet 5 seem to have much stronger priors about what an edit tool
call should look like and that prior appears to be Claude Code’s edit schema: a
flat old/new string pair, plus the optional replace_all flag. My guess is
that Opus has learned that an edit operation may have one extra optional field,
but under Pi’s nested oldText/newText shape it has no trained name for that
field. So it samples a plausible name fresh each time, which is why the
failures produce dozens of random keys rather than one stable alias.Opus 4.8 和 Sonnet 5 似乎对编辑工具调用应该是什么样子有着非常强烈的先验知识，而这种先验知识似乎就是 Claude Code 的编辑模式：一对扁平的 old/new 字符串，加上可选的 replace_all 标志。我的猜测是，Opus 已经学会了编辑操作可能有一个额外的可选字段，但在 Pi 的嵌套 oldText/newText 形状下，它并没有为该字段训练过名称。因此，它每次都会随机采样一个看似合理的名称，这就是为什么故障会产生几十个随机键，而不是一个稳定的别名。
As strict mode in Anthropic appears to fix this, I presume that on the server
side they are refusing to sample a key that is not permitted by the JSON schema
structure. That would also explain why they have limits to the complexity of
the tool definitions when strict mode is enabled.由于 Anthropic 的严格模式似乎解决了这个问题，我推测在服务器端，他们拒绝采样 JSON 模式结构不允许的键。这也解释了为什么在启用严格模式时，他们对工具定义的复杂性有限制。
So far, the Codex models I tested did not show this type of regression. I tested all available ones except 5.6, which I do not have access to yet.到目前为止，我测试的 Codex 模型没有表现出这种类型的回归。我测试了所有可用的模型，除了我还没有权限访问的 5.6。
The uncomfortable lesson is that tool schemas are not neutral, at least not on Anthropic models. We like to pretend that a schema is an abstract contract and the model is a general reasoner that will follow it, but that might no longer be the case for some of the tools.一个令人不安的教训是，工具模式并非中立的，至少在 Anthropic 模型上是这样。我们喜欢假装模式是一个抽象契约，模型是一个会遵循它的通用推理者，但这对于某些工具来说可能不再适用了。
Tool schemas are somewhere in the distribution and some shapes are close to what the model saw during post-training and some are far away. Some are easy for the provider’s hidden encoding (e.g. top-level attributes in ANTML), whereas some require the model to write large escaped JSON objects inside nested arrays after long multiline strings. The model may be smart enough to understand the schema and still be bad at sampling the exact shape under pressure.工具模式处于分布的某个位置，有些形状接近模型在后训练期间看到的，有些则相去甚远。有些对于提供商隐藏的编码（例如 ANTML 中的顶级属性）来说很容易，而有些则要求模型在长多行字符串后的嵌套数组内编写大型转义 JSON 对象。模型可能足够聪明以理解模式，但在压力下仍不擅长采样出精确的形状。
If this type of model behavior continues, I wonder what the implications for
harnesses are. Obviously one could turn on strict sampling in
Anthropic and the problem should go away. On the other hand, that the model
has this behavior shows the impact that reinforcement learning has on them.
Fighting that prior is probably futile if you want to get the best model performance.如果这种模型行为持续下去，我想知道这对接口意味着什么。显然，人们可以在 Anthropic 中开启严格采样，问题应该就会消失。另一方面，模型表现出这种行为显示了强化学习对它们的影响。如果你想获得最佳的模型性能，对抗那种先验知识可能是徒劳的。
Right now the reality is that Claude Code is not open source and we cannot really know what they are doing in their RL environments either. We cannot assume Claude-Code-trained behavior will transfer cleanly to your tools unless they are a close match. The more post-training happens inside one dominant harness, the more every other harness will have to inherit its quirks.目前的事实是 Claude Code 不是开源的，我们也不知道他们在强化学习环境中到底在做什么。除非你的工具与 Claude Code 的工具高度匹配，否则我们不能假设在 Claude Code 上训练出的行为能完美迁移。后训练在某一个主导接口中发生的次数越多，其他所有接口就越不得不继承它的怪癖。
I used to be more skeptical of strict grammar-constrained tool invocation because constrained decoding can have quality tradeoffs. I still think that can be true in general, but this bug moved my priors significantly. If the newest models get better at solving the task while getting worse at faithfully emitting an alternative tool schema, then the harness needs stronger guarantees somewhere.我过去对严格的语法约束工具调用持怀疑态度，因为受限解码可能会有质量权衡。我仍然认为这在总体上可能是正确的，但这个 Bug 显著改变了我的先验认知。如果最新的模型在解决任务方面变得更好，但在忠实输出替代工具模式方面变得更差，那么接口在某些地方就需要更强的保证。
If you want to find out more, or you want to discuss this, consider reading the issue on the Pi tracker.如果你想了解更多信息或进行讨论，可以考虑阅读 Pi 追踪器上的相关议题。