written on July 04, 2026写于 2026 年 7 月 4 日
A very strange Pi issue
sent me down a rabbit hole over the last two days. The short version is that
newer Claude models sometimes call Pi’s edit tool with extra, invented fields in
the nested edits[] array. And not Haiku or some small model: Opus 4.8. The
edit itself is usually correct but the arguments do not match the schema as
the model invents made-up keys and Pi thus rejects the tool call and asks to
try again.过去两天里，一个非常奇怪的 Pi 问题让我陷入了困境。简而言之，较新的 Claude 模型有时会在嵌套的 edits[] 数组中，向 Pi 的编辑工具调用中添加多余的、凭空捏造的字段。这并非 Haiku 或某些小模型的问题，而是 Opus 4.8。编辑内容本身通常是正确的，但参数与架构不匹配，因为模型捏造了虚假的键，导致 Pi 拒绝了该工具调用并要求重试。
That alone is not too surprising as models emit malformed tool calls sometimes. Particularly small ones. What surprised me is that this is getting worse with newer Anthropic models as both Opus 4.8 and Sonnet 5 show it but none of the older models. In other words, the SOTA models of the family are worse at this specific tool schema than their older siblings.单看这一点并不令人惊讶，因为模型有时确实会发出格式错误的工具调用，尤其是较小的模型。令我惊讶的是，随着 Anthropic 模型更新，这种情况反而愈演愈烈：Opus 4.8 和 Sonnet 5 都有这个问题，而旧模型却没有。换句话说，该系列中最先进（SOTA）的模型在处理这种特定工具架构时，表现反而不如它们的“前辈”。
In case you are curious about Fable: I intentionally did not test it because I was not sure if the classifiers they are running might downgrade me to Opus silently.如果你对 Fable 感兴趣：我特意没有测试它，因为我不确定他们运行的分类器是否会悄悄地将我降级为 Opus。
If you have not spent too much time looking at LLM tool calling internals, the important thing to understand is that tool calls are not magic and use some rather crude in-band signalling. The model receives a transcript, a system prompt and a list of available tools. The server munches that into a large prompt with special marker tokens. Because the model was trained and reinforced on examples of that format, at some point during generation it emits something that the API or client interprets as “call this tool with these arguments”.如果你没有花太多时间研究大语言模型（LLM）工具调用的内部机制，需要明白的重要一点是：工具调用并非魔法，而是使用了一些相当原始的带内信令（in-band signalling）。模型会接收到一个转录文本、一个系统提示词以及可用工具列表。服务器将其处理成一个带有特殊标记符号的长提示词。由于模型是在这种格式的示例上进行训练和强化学习的，因此在生成过程中的某个时刻，它会输出一些被 API 或客户端解释为“用这些参数调用此工具”的内容。
For a file edit tool, the intended invocation payload might say something like this:对于文件编辑工具，预期的调用负载可能如下所示：
{
"path": "some/file.py",
"edits": [
{
"oldText": "text to replace",
"newText": "replacement text"
}
]
}
A harness then validates the arguments, performs the edit, and feeds the result back into the model. If validation fails, the model sees an error and usually tries again.随后，一个工具框架（harness）会验证参数、执行编辑，并将结果反馈给模型。如果验证失败，模型会看到错误，通常会再次尝试。
How exactly that formatting happens is not known for the Anthropic models, but some people have gotten out “ANTML” markers and they at times do leak also into public communications. To the best of my knowledge, the call above would come out serialized like this from the model:Anthropic 模型具体是如何进行这种格式化的尚不为人知，但有些人已经提取出了“ANTML”标记，有时这些标记也会泄露到公开通信中。据我所知，上述调用从模型输出时会序列化为：
<antml:function_calls>
<antml:invoke name="edit">
<antml:parameter name="path">some/file.py</antml:parameter>
<antml:parameter name="edits">
[
{
"oldText": "text to replace",
"newText": "replacement text"
}
]
</antml:parameter>
</antml:invoke>
</antml:function_calls>
An important thing to note here is that this thing, while looking like XML, is not really XML. It’s just a thing they found convenient to tokenize and train on. The other thing to note is that a basic top-level string parameter appears in-line whereas an array of objects is implemented via JSON serialization. While I’m not entirely sure that this is how it works, there are some indications that this is not too far off. This will become relevant later.这里需要注意的一点是，虽然它看起来像 XML，但实际上并不是 XML。它只是开发者觉得方便标记和训练的一种格式。另一点需要注意的是，基本的顶级字符串参数是内联出现的，而对象数组则是通过 JSON 序列化实现的。虽然我不完全确定其运作机制，但有迹象表明事实相差不远。这一点稍后会变得很重要。
There are two very different ways to make the model produce a structure like this:让模型生成这种结构有两种截然不同的方法：
The second approach is what people usually refer to as grammar-aware or
constrained decoding. The sampler masks out tokens that would violate the
grammar. If the model is currently inside a JSON object and the schema says
only oldText and newText are allowed, the sampler can prevent it from
emitting "in_file" or "type". Grammar-aware decoding can be used both to
constrain something to be syntactically valid JSON and also to enforce specific
enum values or keys.第二种方法通常被称为语法感知（grammar-aware）或约束解码。采样器会屏蔽掉那些违反语法的标记。如果模型当前处于一个 JSON 对象中，而架构规定只允许 oldText 和 newText，采样器可以阻止它输出 "in_file" 或 "type"。语法感知解码既可用于限制输出为语法正确的 JSON，也可用于强制执行特定的枚举值或键。
Without any form of constraints the model is merely following a learned convention.在没有任何形式约束的情况下，模型仅仅是在遵循一种习得的惯例。
Pi’s edit tool supports multiple exact string replacements in one call. That is
why the arguments contain an edits array. In the failing cases the model
produces entries like this:Pi 的编辑工具支持在一次调用中进行多次精确的字符串替换。这就是为什么参数包含一个 edits 数组的原因。在失败的情况下，模型会产生如下条目：
{
"oldText": "...",
"newText": "...",
"requireUnique": true
}
or this:或者这样：
{
"oldText": "...",
"newText": "...",
"oldText2": "",
"newText2": ""
}
Across repeated trials I saw a whole zoo of invented trailing keys: type,
id, kind, unique, requireUnique, matchCase, in_file,
forceMatchCount, children, notes, cost, oldText2, newText2,
oldText_2, newText_2, and even an event.0.additionalProperties key inside
the edit object itself.在反复试验中，我看到了各种各样凭空捏造的尾随键：type、id、kind、unique、requireUnique、matchCase、in_file、forceMatchCount、children、notes、cost、oldText2、newText2、oldText_2、newText_2，甚至在编辑对象本身内部还有一个 event.0.additionalProperties 键。
The most annoying part is that the actual oldText and newText payloads were
byte-correct in the invalid calls I inspected. The model had in fact produced
the right invocation but then added nonsense at the end of the object.最令人恼火的是，在我检查的无效调用中，实际的 oldText 和 newText 负载在字节层面是完全正确的。模型实际上已经生成了正确的调用，但随后在对象末尾添加了无意义的内容。
The failure is also heavily context-dependent. A fresh single-turn prompt like “edit this file” did not reproduce it at all for me. An agentic history where the model had read files, diagnosed a problem and then composed a multi-line edit could reproduce it. And more annoyingly, not all transcripts will show that behavior. In fact, I needed Petr Baudis‘s transcripts to reproduce this for me at all! In that user’s session continuing the session caused Opus 4.8 to fail around 20% of the time. Stripping thinking blocks from history reduced the failure rate by half. Turning on strict tool invocation eliminated it in my runs.这种故障也高度依赖于上下文。对于像“编辑此文件”这样全新的单轮提示词，我完全无法复现它。但在一个代理式（agentic）的历史记录中，如果模型读取了文件、诊断了问题并随后编写了多行编辑，就能复现该问题。更令人恼火的是，并非所有的转录记录都会表现出这种行为。事实上，我需要 Petr Baudis 的转录记录才能复现它！在该用户的会话中，继续进行会话会导致 Opus 4.8 有约 20% 的失败率。从历史记录中剔除思考块（thinking blocks）将失败率降低了一半。而在我的运行中，开启严格工具调用（strict tool invocation）则完全消除了该问题。
My strongest hypothesis is that this is not random deterioration but a training artifact.我最强烈的假设是，这并非随机的退化，而是训练产物。
When older Anthropic models were trained, they were trained on some tools (some of which were documented). But that training did not yet have a user-shipped harness like Claude Code as the obvious target. Modern Anthropic models are most likely different because their post-training includes Claude Code or a harness that looks very similar. The model learns what a successful tool call looks like in that environment. It also learns what mistakes are tolerated by that environment.当旧的 Anthropic 模型进行训练时，它们是在某些工具（其中一些是有文档记录的）上进行训练的。但当时的训练还没有像 Claude Code 这样用户可用的工具框架作为明确的目标。现代 Anthropic 模型很可能有所不同，因为它们的训练后阶段（post-training）包含了 Claude Code 或看起来非常相似的框架。模型学会了在该环境中成功的工具调用是什么样子的，也学会了该环境能容忍哪些错误。
Claude Code’s own tools are comparatively flat. The ordinary edit tool is not
Pi’s nested edits[] shape; it is closer to file_path, old_string,
new_string, and an optional flag (replace_all). Looking at Claude Code’s
client is very instructive: it contains retry paths for malformed tool use,
parameter aliases, type coercions, Unicode repairs and filtering of unknown
keys. In other words, Anthropic’s own client appears to expect and accept a
fair amount of slop and repairs it, mostly silently.Claude Code 自身的工具相对扁平。普通的编辑工具不是 Pi 的嵌套 edits[] 形状；它更接近 file_path、old_string、new_string 和一个可选标志 (replace_all)。查看 Claude Code 的客户端非常有启发性：它包含了针对错误工具使用的重试路径、参数别名、类型强制转换、Unicode 修复以及对未知键的过滤。换句话说，Anthropic 自己的客户端似乎预期并接受了相当多的“粗糙”输入，并能基本静默地修复它们。
If reinforcement learning happens in a harness like that, or a simulation of one, then slightly malformed tool calls can still complete the task and receive reward. The harness fully absorbs the error and there is little gradient against inventing an alias, adding a stray field or using a nearby parameter name.如果强化学习是在这样的框架（或其模拟环境）中进行的，那么稍微格式错误的工具调用仍然可以完成任务并获得奖励。框架完全吸收了错误，因此对于捏造别名、添加多余字段或使用相近参数名，模型几乎不会受到任何惩罚。
Worse, the model may become very strongly adapted to the canonical Claude Code edit tool shape. A different harness can present a tool with the same semantic intent but a different schema. Such a tool can increasingly be off-distribution. The better-trained model might actually fight you harder because its prior is stronger.更糟糕的是，模型可能会非常强烈地适应标准的 Claude Code 编辑工具形状。不同的框架可能会提供语义意图相同但架构不同的工具。这样的工具可能会越来越处于分布之外（off-distribution）。训练得更好的模型反而可能更强烈地对抗你，因为它的先验知识更强。
This is not too surprising, but it is a change from how this was a few months ago. When Opus 4.5 launched, it adapted to other edit tools exceptionally well. In fact, I was pretty convinced that we’re on a good path where the models are more likely to adapt to any sort of tool shape that comes around for as long as the instructions are good.这并不令人惊讶，但这与几个月前的情况相比确实发生了变化。当 Opus 4.5 发布时，它对其他编辑工具的适应能力非常出色。事实上，我曾坚信我们正走在一条正确的道路上，即只要指令明确，模型就更有可能适应任何出现的工具形状。
Now I’m somewhat worried about the track we’re on here. Alternative tool schemas might not just be unfamiliar. They might be implicitly punished by post-training that optimizes for one particular, forgiving tool ecology. And that ecology is not documented. While there is a text editor tool that is documented, you will see that this format is in fact not followed by Claude Code. What Claude Code does internally (which is a closed-source harness) is hidden from you.现在，我开始担心我们所处的轨道。替代性的工具架构可能不仅是“不熟悉”的问题，它们还可能因为训练后阶段针对一种特定的、宽容的工具生态进行了优化，而受到隐性的惩罚。而且这种生态并没有文档记录。虽然有一个文本编辑器工具是有文档记录的，但你会发现 Claude Code 实际上并不遵循这种格式。Claude Code 内部（一个闭源框架）的操作对你来说是隐藏的。
Claude Code is obviously closed-source but we can look at the minified code and get some idea of what it does. And honestly, it’s very forgiving of incoming data.Claude Code 显然是闭源的，但我们可以查看其压缩代码，从而了解它的工作原理。老实说，它对传入的数据非常宽容。
For a start, Claude Code checks the model’s visible text for leaked <invoke
markup. It also emits some telemetry when that happens and then it has its
own state machine to retry such bad calls by pushing back to the model.首先，Claude Code 会检查模型可见文本中是否存在泄露的 <invoke 标记。当这种情况发生时，它还会发出一些遥测数据，并拥有自己的状态机，通过将错误反馈给模型来重试这些糟糕的调用。
It has explicit Unicode escape repair which fixes broken \uXXXX sequences and
lone surrogates in string values. It also has per-tool aliases for parameters.
For instance, Edit accepts old_str (presumably from the times when the models
were trained on the officially documented text editor tool), the newer old_string
from the schema, new_str/new_string, path as an alias for file_path, and some more.它具有显式的 Unicode 转义修复功能，可以修复字符串值中损坏的 \uXXXX 序列和孤立的代理对。它还为参数设置了针对特定工具的别名。例如，Edit 接受 old_str（大概源于模型在官方记录的文本编辑器工具上训练的时期）、架构中较新的 old_string、new_str/new_string、作为 file_path 别名的 path 等等。
It also silently filters out unexpected keys and it does not use strict mode
either. The issue with strict mode is that Anthropic applies complexity
limits to the tool definitions that cause API requests to fail, so presumably
that’s why Claude Code does not attempt to use it.它还会静默过滤掉意外的键，并且也没有使用严格模式（strict mode）。使用严格模式的问题在于，Anthropic 对工具定义施加了复杂性限制，这会导致 API 请求失败，所以这大概就是 Claude Code 不尝试使用它的原因。
Will this problem be with us in other harnesses too? One huge issue with Anthropic is that the models are completely closed, and so is the harness. Codex models are also closed, but at least the harness is not. We also have gpt-oss which is at least a bit interesting. The models are explicitly trained to use OpenAI’s harmony response format and there is a lot of documentation that at least tells us how OpenAI people think about this.这个问题会出现在其他框架中吗？Anthropic 的一个巨大问题是模型完全闭源，框架也是如此。Codex 模型也是闭源的，但至少框架不是。我们还有 gpt-oss，它至少有点意思。这些模型被明确训练为使用 OpenAI 的 harmony 响应格式，并且有大量的文档至少告诉了我们 OpenAI 的人是如何思考这个问题的。
Harmony makes channels and tool-call content types part of the prompt format. A function call can look like this:Harmony 将通道和工具调用内容类型作为提示词格式的一部分。函数调用可能如下所示：
<|start|>assistant<|channel|>commentary to=functions.get_weather
<|constrain|>json<|message|>{"location":"San Francisco"}<|call|>
The important bit is <|constrain|>json. The model can express in-band that
this message body is JSON, and an inference stack can use that boundary to
switch into JSON-constrained sampling for the body of the tool call. Presumably
a bit of this also happens in Anthropic’s models, at least in strict mode
I would imagine.关键部分是 <|constrain|>json。模型可以在带内明确表示此消息正文是 JSON，推理堆栈可以使用该边界切换到针对工具调用正文的 JSON 约束采样。大概 Anthropic 的模型中也发生了一些类似的情况，至少在严格模式下我想是这样的。
The marker in harmony helps the sampler to detect when it needs to sample with a specific grammar, and because it is part of the transcript, it makes that rather easy to do. For hosted GPT models, there is also an option to provide a LARK grammar for custom tools that need to adhere to something like this.Harmony 中的标记帮助采样器检测何时需要使用特定语法进行采样，而且由于它是转录文本的一部分，这使得操作变得相当容易。对于托管的 GPT 模型，还有一个选项可以为需要遵守类似规则的自定义工具提供 LARK 语法。
Anthropic appears different from that, though maybe not entirely. If an array
of objects is represented as JSON, as it appears to be, then the model has to
write JSON inside the tool parameter. There is probably basic
grammar-constrained sampling going on, and that may partly explain the extra
keys. For a nested array parameter, that JSON includes escaped multi-line file
content inside string literals, inside one tag. The unexpected,
made-up keys appear exactly at the highest-entropy point of that task: after
closing a several-hundred-token escaped newText string, where the model must
decide } vs , "...".Anthropic 的情况似乎有所不同，尽管可能不完全如此。如果对象数组表现为 JSON（看起来确实如此），那么模型必须在工具参数内编写 JSON。可能存在基础的语法约束采样，这或许可以部分解释那些多余的键。对于嵌套数组参数，该 JSON 在单个标签内包含了字符串字面量中转义的多行文件内容。那些意外的、捏造的键恰好出现在任务中熵最高的点：在关闭了一个包含数百个标记的转义 newText 字符串之后，模型必须在 } 和 , "..." 之间做出选择。
Opus 4.8 and Sonnet 5 seem to have much stronger priors about what an edit tool
call should look like and that prior appears to be Claude Code’s edit schema: a
flat old/new string pair, plus the optional replace_all flag. My guess is
that Opus has learned that an edit operation may have one extra optional field,
but under Pi’s nested oldText/newText shape it has no trained name for that
field. So it samples a plausible name fresh each time, which is why the
failures produce dozens of random keys rather than one stable alias.Opus 4.8 和 Sonnet 5 似乎对编辑工具调用的外观有着强得多的先验假设，而这个先验似乎就是 Claude Code 的编辑架构：扁平的 old/new 字符串对，加上可选的 replace_all 标志。我的猜测是，Opus 已经学会了编辑操作可能有一个额外的可选字段，但在 Pi 的嵌套 oldText/newText 形状下，它没有该字段的训练名称。因此，它每次都会即兴采样一个看起来合理的名称，这就是为什么失败产生的是数十个随机键，而不是一个稳定的别名。
As strict mode in Anthropic appears to fix this, I presume that on the server
side they are refusing to sample a key that is not permitted by the JSON schema
structure. That would also explain why they have limits to the complexity of
the tool definitions when strict mode is enabled.由于 Anthropic 中的严格模式似乎可以解决这个问题，我推测在服务器端，他们拒绝采样 JSON 架构结构不允许的键。这也解释了为什么在启用严格模式时，他们对工具定义的复杂性有限制。
So far, the Codex models I tested did not show this type of regression. I tested all available ones except 5.6, which I do not have access to yet.到目前为止，我测试的 Codex 模型没有表现出这种类型的回归。我测试了所有可用的版本，除了 5.6，我目前还没有访问权限。
The uncomfortable lesson is that tool schemas are not neutral, at least not on Anthropic models. We like to pretend that a schema is an abstract contract and the model is a general reasoner that will follow it, but that might no longer be the case for some of the tools.一个令人不安的教训是，工具架构并非中立的，至少在 Anthropic 模型上是这样。我们喜欢假装架构是一个抽象的契约，模型是一个遵循它的通用推理机，但这对于某些工具来说可能不再适用了。
Tool schemas are somewhere in the distribution and some shapes are close to what the model saw during post-training and some are far away. Some are easy for the provider’s hidden encoding (e.g. top-level attributes in ANTML), whereas some require the model to write large escaped JSON objects inside nested arrays after long multiline strings. The model may be smart enough to understand the schema and still be bad at sampling the exact shape under pressure.工具架构处于分布中的不同位置，有些形状接近模型在训练后阶段所见的内容，有些则相去甚远。有些对于提供商隐藏的编码（例如 ANTML 中的顶级属性）来说很容易，而有些则要求模型在长多行字符串后的嵌套数组中编写大型转义 JSON 对象。模型可能足够聪明去理解架构，却在压力下难以准确采样出预期的形状。
If this type of model behavior continues, I wonder what the implications for
harnesses are. Obviously one could turn on strict sampling in
Anthropic and the problem should go away. On the other hand, that the model
has this behavior shows the impact that reinforcement learning has on them.
Fighting that prior is probably futile if you want to get the best model performance.如果这种模型行为持续下去，我想知道这对框架意味着什么。显然，人们可以在 Anthropic 中开启严格采样，问题应该就会消失。另一方面，模型表现出这种行为，说明了强化学习对它们的影响。如果你想获得最佳的模型性能，对抗这种先验知识可能是徒劳的。
Right now the reality is that Claude Code is not open source and we cannot really know what they are doing in their RL environments either. We cannot assume Claude-Code-trained behavior will transfer cleanly to your tools unless they are a close match. The more post-training happens inside one dominant harness, the more every other harness will have to inherit its quirks.目前的事实是，Claude Code 不是开源的，我们也无法真正知道他们在 RL 环境中在做什么。我们不能假设在 Claude-Code 上训练的行为会完美迁移到你的工具上，除非它们非常匹配。在单一主导框架内进行的训练后阶段越多，其他所有框架就越不得不继承其怪癖。
I used to be more skeptical of strict grammar-constrained tool invocation because constrained decoding can have quality tradeoffs. I still think that can be true in general, but this bug moved my priors significantly. If the newest models get better at solving the task while getting worse at faithfully emitting an alternative tool schema, then the harness needs stronger guarantees somewhere.我曾经对严格的语法约束工具调用持怀疑态度，因为约束解码可能会有质量上的权衡。我仍然认为这在一般情况下可能是正确的，但这个 Bug 显著改变了我的先验认知。如果最新的模型在解决任务时表现得更好，而在忠实输出替代工具架构时表现得更差，那么框架就需要从其他地方获得更强的保证。
If you want to find out more, or you want to discuss this, consider reading the issue on the Pi tracker.如果你想了解更多信息，或者想讨论这个问题，可以考虑阅读 Pi 追踪器上的议题。