The Toolkit Pattern工具包模式

This is the third article in a series on agentic engineering and AI-driven development. Read part one here, part two here, and look for the next article on April 15 on O’Reilly Radar.这是关于代理工程和 AI 驱动开发系列的第三篇文章。阅读第一部分请点此，第二部分请点此，下一篇将在 4 月 15 日于 O’Reilly Radar 上发布。

The toolkit pattern is a way of documenting your project’s configuration so that any AI can generate working inputs from a plain-English description. You and the AI create a single file that describes your tool’s configuration format, its constraints, and enough worked examples that any AI can generate working inputs from a plain-English description. You build it iteratively, working with the AI (or, better, multiple AIs) to draft it. You test it by starting a fresh AI session and trying to use it, and every time that fails you grow the toolkit from those failures. When you build the toolkit well, your users will never need to learn how your tool’s configuration files work, because they describe what they want in conversation and the AI handles the translation. That means you don’t have to compromise on the way your project is configured, because the config files can be more complex and more complete than they would be if a human had to edit and understand them.工具包模式是一种记录项目配置的方法，使任何 AI 都能从纯英文描述生成可用的输入。您和 AI 共同创建一个文件，描述工具的配置格式、约束以及足够的示例，以便任何 AI 能从纯英文描述生成可用的输入。您通过与 AI（或更好的是多个 AI）迭代编写它。通过启动全新的 AI 会话并尝试使用它进行测试，每次失败都让您在这些失败中完善工具包。当工具包构建得好时，用户永远不需要学习配置文件的工作方式，因为他们只需在对话中描述需求，AI 完成翻译。这意味着您无需在项目配置方式上妥协，因为配置文件可以比人工编辑和理解时更复杂、更完整。

To understand why all of this matters, let me take you back to the mid-1980s.为了理解这一切为何重要，让我把时间倒回到 1980 年代中期。

I was 12 years old, and our family got an AT&T PC 6300, an IBM-compatible that came with a user’s guide roughly 159 pages long. Chapter 4 of that manual was called “What Every User Should Know.” It covered things like how to use the keyboard, how to care for your diskettes, and, memorably, how to label them, complete with hand-drawn illustrations and really useful advice, like how you should only use felt-tipped pens, never ballpoint, because the pressure might damage the magnetic surface.我当时 12 岁，家里买了台 AT&T PC 6300，这是一台兼容 IBM 的电脑，随附约 159 页的用户指南。手册第 4 章名为“每位用户都应了解的内容”。章节内容包括如何使用键盘、如何保养软盘，以及令人印象深刻的标记方法，配有手绘插图和非常实用的建议，例如只能使用毛笔尖的笔，绝不能使用圆珠笔，因为压力可能会损坏磁性表面。

I remember being fascinated by this manual. It wasn’t our first computer. I’d been writing BASIC programs and dialing into BBSs and CompuServe for a couple of years, so I knew there were all sorts of amazing things you could do with a PC, especially one with a blazing fast 8MHz processor. But the manual barely mentioned any of that. That seemed really weird to me, even as a kid, that you would give someone a manual that had a whole page on using the backspace key to correct typing mistakes (really!) but didn’t actually tell them how to use the thing to do anything useful.我记得对这本手册着迷。那并不是我们的第一台电脑。我已经写了几年 BASIC 程序，拨号进入 BBS 和 CompuServe，所以我知道 PC 能做很多惊人的事，尤其是配备 8MHz 超快处理器的机器。但手册几乎没有提到这些。这在我小时候就觉得很奇怪：手册里有整整一页教你如何使用退格键纠正打字错误（真的！），却没有告诉你如何用这台机器做任何有用的事。

That’s how most developer documentation works. We write the stuff that’s easy to write—installation, setup, the getting-started guide—because it’s a lot easier than writing the stuff that’s actually hard: the deep explanation of how all the pieces fit together, the constraints you only discover by hitting them, the patterns that separate a configuration that works from one that almost works. This is yet another “looking for your keys under the streetlight” problem: We write the documentation we write because it’s easiest to write, even if it’s not really the documentation our users need.这就是大多数开发者文档的工作方式。我们先写容易写的内容——安装、设置、入门指南——因为这比写真正困难的内容要容易得多：深入解释各部分如何协同工作、只有在实际碰到时才会发现的约束、区分可用配置与几乎可用配置的模式。这是又一个“在路灯下找钥匙”的问题：我们写文档是因为它最容易写，即使这并不是用户真正需要的文档。

Developers who came up through the Unix era know this well. Man pages were thorough, accurate, and often completely impenetrable if you didn’t already know what you were doing. The tar man page is the canonical example: It documents every flag and option in exhaustive detail, but if you just want to know how to extract a .tar.gz file, it’s almost useless. (The right flag is -xzvf in case you’re curious.) Stack Overflow exists in large part because man pages like tar’s left a gap between what the documentation said and what developers actually needed to know.经历过 Unix 时代的开发者对此深有体会。man 手册详尽、准确，但如果你不熟悉就几乎无法阅读。tar 的 man 手册就是典型例子：它详尽记录了每个标志和选项，但如果你只想知道如何解压 .tar.gz 文件，几乎毫无用处。（正确的标志是 -xzvf，供好奇的你参考。）Stack Overflow 的出现很大程度上是因为像 tar 这样的 man 手册在文档描述与开发者实际需求之间留下了空白。

And now we have AI assistants. You can ask Claude or ChatGPT about, say, Kubernetes, Terraform, or React, and you’ll actually get useful answers, because those are all established projects that have been written about extensively and the training data is everywhere.现在我们有了 AI 助手。你可以向 Claude 或 ChatGPT 询问 Kubernetes、Terraform 或 React 等，它们会给出有用的答案，因为这些项目都有大量文档，训练数据随处可见。

But AI hits a hard wall at the boundary of its training data. If you’ve built something new—a framework, an internal platform, a tool your team created—no model has ever seen it. Your users can’t ask their AI assistant for help, because the AI doesn’t know your thing even exists.但 AI 在其训练数据的边界上会遇到硬墙。如果你构建了全新的东西——框架、内部平台、团队自研工具——没有任何模型见过它。用户无法向 AI 助手求助，因为 AI 根本不知道你的东西的存在。

There’s been a lot of great work moving AI documentation in the right direction. AGENTS.md tells AI coding agents how to work on your codebase, treating the AI as a developer. llms.txt gives models a structured summary of your external documentation, treating the AI as a search engine. What’s been missing is a practice for treating the AI as a support engineer. Every project needs configuration: input files, option schemas, workflow definitions, usually in the form of a whole bunch of JSON or YAML files with cryptic formats that users have to learn before they can do anything useful.已经有很多优秀的工作让 AI 文档朝正确方向前进。AGENTS.md 告诉 AI 编码代理如何在你的代码库上工作，把 AI 当作开发者。llms.txt 为模型提供外部文档的结构化摘要，把 AI 当作搜索引擎。缺失的则是把 AI 当作支持工程师的实践。每个项目都需要配置：输入文件、选项模式、工作流定义，通常是一堆 JSON 或 YAML 文件，格式晦涩，用户必须先学会才能做任何有用的事。

The toolkit pattern solves that problem of getting AIs to write configuration files for a project that isn’t in its training data. It consists of a documentation file that teaches any AI enough about your project’s configuration that it can generate working inputs from a plain-English description, without your users ever having to learn the format themselves. Developers have been arriving at this same pattern (or something very similar) independently from different directions, but as far as I can tell, nobody has named it or described a methodology for doing it well. This article distills what I learned from building the toolkit for Octobatch pipelines into a set of practices you can apply to your own projects.工具包模式解决了让 AI 为不在其训练数据中的项目编写配置文件的问题。它由一个文档文件组成，教任何 AI 足够了解你的项目配置，以便它能从纯英文描述生成可用的输入，而用户无需自行学习格式。开发者们已经从不同方向独立得出了相同的模式（或非常相似的），但据我所知，还没有人给它命名或描述一套完整的方法论。本文提炼了我在为 Octobatch 流水线构建工具包时的经验，形成一套可用于自己项目的实践。

Build the AI its own manual为 AI 构建自己的手册

Traditionally, developers face a trade-off with configuration: keep it simple and easy to understand, or let it grow to handle real complexity and accept that it now requires a manual. The toolkit pattern emerged for me while I was building Octobatch, the batch-processing orchestrator I’ve been writing about in this series. As I described in the previous articles in this series, “The Accidental Orchestrator” and “Keep Deterministic Work Deterministic,” Octobatch runs complex multistep LLM pipelines that generate files or run Monte Carlo simulations. Each pipeline is defined using a complex configuration that consists of YAML, Jinja2 templates, JSON schemas, expression steps, and a set of rules tying it all together. The toolkit pattern let me sidestep that traditional trade-off.传统上，开发者在配置上面临两难：保持简单易懂，或让其成长以处理真实复杂性并接受需要手册的事实。工具包模式在我构建 Octobatch（本系列中一直在写的批处理编排器）时出现。正如我在本系列前两篇文章《意外的编排器》和《保持确定性工作确定性》中描述的，Octobatch 运行复杂的多步骤 LLM 流水线，生成文件或进行蒙特卡罗模拟。每条流水线使用由 YAML、Jinja2 模板、JSON 模式、表达式步骤以及一套规则组成的复杂配置。工具包模式让我规避了传统的两难。

As Octobatch grew more complex, I found myself relying on the AIs (Claude and Gemini) to build configuration files for me, which turned out to be genuinely valuable. When I developed a new feature, I would work with the AIs to come up with the configuration structure to support it. At first I defined the configuration, but by the end of the project I relied on the AIs to come up with the first cut, and I’d push back when something seemed off or not forward-looking enough. Once we all agreed, I would have an AI produce the actual updated config for whatever pipeline we were working on. This move to having the AIs do the heavy lifting of writing the configuration was really valuable, because it let me create a very robust format very quickly without having to spend hours updating existing configurations every time I changed the syntax or semantics.随着 Octobatch 越来越复杂，我开始依赖 AI（Claude 和 Gemini）为我构建配置文件，这的确非常有价值。当我开发新功能时，我会与 AI 合作确定支持该功能的配置结构。起初我自己定义配置，但项目后期我让 AI 给出第一版，我再根据感觉或前瞻性进行反馈。达成一致后，我让 AI 为我们正在开发的流水线生成实际的更新配置。让 AI 承担编写配置的重活非常有价值，因为它让我能够在不花费数小时更新现有配置的情况下，快速创建一个非常稳健的格式。

At some point I realized that every time a new user wanted to build a pipeline, they faced the same learning curve and implementation challenges that I’d already worked through with the AIs. The project already had a README.md file, and every time I modified the configuration I had an AI update it to keep the documentation up to date. But by this time, the README.md file was doing way too much work: It was really comprehensive but a real headache to read. It had eight separate subdocuments showing the user how to do pretty much everything Octobatch supported, and the bulk of it was focused on configuration, and it was becoming exactly the kind of documentation nobody ever wants to read. That particularly bothered me as a writer; I’d produced documentation that was genuinely painful to read.后来我意识到，每当新用户想要构建流水线时，他们都会面临我已经和 AI 走过的相同学习曲线和实现挑战。项目已经有了 README.md 文件，每次我修改配置时，我都会让 AI 更新它以保持文档同步。但此时 README.md 已经承担了太多工作：它内容非常全面，却让人阅读时头疼。它包含八个子文档，几乎覆盖了 Octobatch 支持的所有功能，重点是配置，正好成为了没人愿意阅读的那种文档。这让我作为写作者特别不舒服；我产出的文档真的让人痛苦。

Looking back at my chats, I can trace how the toolkit pattern developed. My first instinct was to build an AI-assisted editor. About four weeks into the project, I described the idea to Gemini:回顾我的聊天记录，我可以追溯到工具包模式的形成过程。我的第一直觉是构建一个 AI 辅助编辑器。项目进行约四周后，我向 Gemini 描述了这个想法：

I’m thinking about how to provide any kind of AI-assisted tool to help people create their own pipeline. I was thinking about a feature we would call “Octobatch Studio” where we make it easy to prompt for modifying pipeline stages, possibly assisting in creating the prompts. But maybe instead we include a lot of documentation in Markdown files, and expect them to use Claude Code, and give lots of guidance for creating it.我在思考如何提供任何形式的 AI 辅助工具，帮助人们创建自己的流水线。我在想一个我们会称之为 “Octobatch Studio” 的功能，让用户可以轻松提示修改流水线阶段，甚至帮助创建提示。但也许我们可以在 Markdown 文件中加入大量文档，让他们使用 Claude Code，并提供大量创建指导。

I can actually see the pivot to the toolkit pattern happening in real time in this later message I sent to Claude. It had sunk in that my users could use Claude Code, Cursor, or another AI as interactive documentation to build their configs exactly the same way I’ve been doing:我实际上可以在稍后给 Claude 的信息中看到向工具包模式的转变。那时我已经意识到我的用户可以使用 Claude Code、Cursor 或其他 AI 作为交互式文档，完全按照我一直在做的方式构建配置：

My plan is to use Claude Code as the IDE for creating new pipelines, so people who want to create them can just spin up Claude Code and start generating them. That means we need to give Claude Code specific context files to tell it everything it needs to know to create the pipeline YAML config with asteval expressions and Jinja2 template files.我的计划是使用 Claude Code 作为创建新流水线的 IDE，这样想创建流水线的人只需启动 Claude Code 并开始生成即可。这意味着我们需要为 Claude Code 提供特定的上下文文件，告诉它创建带有 asteval 表达式和 Jinja2 模板文件的流水线 YAML 配置所需的全部信息。

The traditional trade-off between simplicity and flexibility comes from cognitive overhead: the cost of holding all of a system’s rules, constraints, and interactions in your head while you work with it. It’s why many developers opt for simpler config files, so they don’t overload their users (or themselves). Once the AI was writing the configuration, that trade-off disappeared. The configs could get as complicated as they needed to be, because I wasn’t the one who had to remember how all the pieces fit together. At some point I realized the toolkit pattern was worth standardizing.传统的简易性与灵活性之间的权衡源于认知负荷：在使用系统时，需要在脑中保持所有规则、约束和交互的成本。这也是许多开发者倾向于使用更简单配置文件的原因，以免让用户（或自己）超负荷。一旦 AI 开始编写配置，这种权衡就消失了。配置可以变得任意复杂，因为我不再需要记住所有部件如何配合。于是我意识到工具包模式值得标准化。

That toolkit-based workflow—users describe what they want, the AI reads TOOLKIT.md and generates the config—is the core of the Octobatch user experience now. A user clones the repo and opens Claude Code, Cursor, or Copilot, the same way they would with any open source project. Every configuration prompt starts the same way: “Read pipelines/TOOLKIT.md and use it as your guide.” The AI reads the file, understands the project structure, and guides them step by step.基于工具包的工作流——用户描述需求，AI 阅读 TOOLKIT.md 并生成配置——现在是 Octobatch 用户体验的核心。用户克隆仓库并打开 Claude Code、Cursor 或 Copilot，就像使用任何开源项目一样。每个配置提示都以相同方式开始：“读取 pipelines/TOOLKIT.md 并以此为指南”。AI 读取文件，理解项目结构，并一步步引导用户。

To see what this looks like in practice, take the Drunken Sailor pipeline I described in “The Accidental Orchestrator.” It’s a Monte Carlo random walk simulation: A sailor leaves a bar and stumbles randomly toward the ship or the water. The pipeline configuration for that involves multiple YAML files, JSON schemas, Jinja2 templates, and expression steps with real mathematical logic, all wired together with specific rules.要看看实际效果，请参考我在《意外的编排器》中描述的 Drunken Sailor 流水线。它是一个蒙特卡罗随机漫步模拟：水手离开酒吧，随机朝船或水里走去。该流水线的配置涉及多个 YAML 文件、JSON 模式、Jinja2 模板以及带有真实数学逻辑的表达式步骤，全部通过特定规则连接。

Here’s the prompt that generated all of that. The user describes what they want in plain English, and the AI produces the entire configuration by reading TOOLKIT.md. This is the exact prompt I gave Claude Code to generate the Drunken Sailor pipeline—notice the first line of the prompt, telling it to read the toolkit file.以下是生成全部内容的提示。用户用纯英文描述需求，AI 通过读取 TOOLKIT.md 生成完整配置。这正是我给 Claude Code 的提示，用于生成 Drunken Sailor 流水线——注意提示的第一行，指示它读取工具包文件。

But configuration generation is only half of what the toolkit file does. Users can also upload TOOLKIT.md and PROJECT_CONTEXT.md (which has information about the project) to any AI assistant—ChatGPT, Gemini, Claude, Copilot, whatever they prefer—and use it as interactive documentation. A pipeline run finished with validation failures? Upload the two files and ask what went wrong. Stuck on how retries work? Ask. You can even paste in a screenshot of the TUI and say, “What do I do?” and the AI will read the screen and give specific advice. The toolkit file turns any AI into an on-demand support engineer for your project.但配置生成只是工具包文件功能的一半。用户还可以将 TOOLKIT.md 和 PROJECT_CONTEXT.md（包含项目信息）上传到任何 AI 助手——ChatGPT、Gemini、Claude、Copilot 等——并将其用作交互式文档。流水线运行出现验证失败？上传这两个文件并询问出错原因。重试机制不清楚？提问。甚至可以粘贴 TUI 截图并说：“我该怎么办？”AI 会读取屏幕并给出具体建议。工具包文件把任何 AI 变成项目的随需支持工程师。

What the Octobatch project taught me about the toolkit patternOctobatch 项目给我的工具包模式启示

Building the generative toolkit for Octobatch produced more than just documentation that an AI could use to create configuration files that worked; it also yielded a set of practices, and those practices turn out to be pretty consistent regardless of what kind of project you’re building. Here are the five that mattered most:为 Octobatch 构建生成式工具包不仅产生了 AI 可用来创建有效配置文件的文档，还形成了一套实践，这些实践在不同项目中都相当一致。以下是最重要的五点：

• Start with the toolkit file and grow it from failures. Don’t wait until the project is finished to write the documentation. Create the toolkit file first, then let each real failure add one principle at a time.从工具包文件开始，并从失败中迭代。不要等项目完成后才编写文档。先创建工具包文件，然后让每一次真实的失败添加一个原则。
• Let the AI write the config files. Your job is product vision—what the project should do and how it should feel. The AI’s job is translating that into valid configuration.让 AI 编写配置文件。你的工作是产品愿景——项目应做什么、应有什么感觉。AI 的工作是把它翻译成有效的配置。
• Keep guidance lean. State the principle, give one concrete example, move on. Every guardrail costs tokens, and bloated guidance makes AI performance worse.保持指导简洁。陈述原则，给出一个具体示例，然后继续。每个防护栏都会消耗 token，冗长的指导会降低 AI 性能。
• Treat every use as a test. There’s no separate testing phase for documentation. Every time someone uses the toolkit file to build something, that’s a test of whether the documentation works.把每一次使用都视为测试。没有单独的文档测试阶段。每次有人使用工具包文件构建东西，都是对文档是否有效的测试。
• Use more than one model. Different models catch different things. In a three-model audit of Octobatch, three-quarters of the defects were caught by only one model.使用多个模型。不同模型捕捉不同问题。在对 Octobatch 进行的三模型审计中，四分之三的缺陷仅被一个模型发现。

I’m not proposing a standard format for a toolkit file, and I think trying to create one would be counterproductive. Configuration formats vary wildly from tool to tool—that’s the whole problem we’re trying to solve—and a toolkit file that describes your project’s building blocks is going to look completely different from one that describes someone else’s. What I found is that the AI is perfectly capable of reading whatever you give it, and is probably better at writing the file than you are anyway, because it’s writing for another AI. These five practices should help build an effective toolkit regardless of what your project looks like.我并未提出工具包文件的标准格式，也认为尝试制定标准会适得其反。配置格式因工具而异——这正是我们要解决的问题——描述你项目构建块的工具包文件必然与描述他人项目的完全不同。我发现 AI 完全能够读取你提供的任何内容，而且在为另一个 AI 编写文件时往往比你更擅长。这五条实践应能帮助你构建有效的工具包，无论你的项目是什么样子。

Start with the toolkit file and grow it from failures从工具包文件开始，并从失败中迭代

You can start building a toolkit at any point in your project. The way it happened for me was organic: After weeks of working with Claude and Gemini on Octobatch configuration, the knowledge about what worked and what didn’t was scattered across dozens of chat sessions and context files. I wrote a prompt asking Gemini to consolidate everything it knew about the config format—the structure, the rules, the constraints, the examples, everything we’d talked about—into a single TOOLKIT.md file. That first version wasn’t great, but it was a starting point, and every failure after that made it better.你可以在项目的任何阶段开始构建工具包。对我而言，这一过程是自发的：在与 Claude 和 Gemini 合作数周后，关于配置的有效与无效的知识散落在数十个聊天会话和上下文文件中。我写了一个提示，让 Gemini 将它所知道的关于配置格式——结构、规则、约束、示例以及我们讨论的所有内容——整合成一个 TOOLKIT.md 文件。第一版并不完美，但它是起点，之后的每一次失败都让它变得更好。

I didn’t plan the toolkit from the beginning of the Octobatch project. It started because I wanted my users to be able to build pipelines the same way I had—by working with an AI—but everything they’d need to do that was spread across months of chat logs and the CONTEXT.md files I’d been maintaining to bootstrap new development sessions. Once I had Gemini consolidate everything into a single TOOLKIT.md file and had Claude review it, I treated it the way I treat any other code: Every time something broke, I found the root cause, worked with the AIs to update the toolkit to account for it, and verified that a fresh AI session could still use it to generate valid configuration.我并没有在 Octobatch 项目伊始就规划好工具包。它的出现是因为我希望用户能够像我一样通过 AI 构建流水线，而他们需要的所有信息分散在数月的聊天记录和我维护的 CONTEXT.md 文件中。等 Gemini 将所有内容整合成单一的 TOOLKIT.md 并让 Claude 审阅后，我像对待其他代码一样对待它：每当出现问题，我就找出根本原因，与 AI 合作更新工具包以解决，然后验证全新的 AI 会话仍能使用它生成有效配置。

That incremental approach worked well for me, and it let me test my toolkit the way I test any other code: try it out, find bugs, fix them, rinse, repeat.这种增量方法对我非常有效，让我能够像测试其他代码一样测试我的工具包：尝试、发现 bug、修复、重复。

You can do the same thing. If you’re starting a new project, you can plan to create the toolkit at the end. But it’s more effective to start with a simple version early and let it emerge over the course of development. That way you’re dogfooding it the whole time instead of guessing what users will need.你也可以这样做。如果你正在启动新项目，可以计划在结束时创建工具包。但更有效的做法是尽早用一个简单版本开始，让它在开发过程中自然演进。这样你就能在整个过程中自行使用它，而不是猜测用户需要什么。

Let the AI write the config files (but stay in control!)让 AI 编写配置文件（但保持控制！）

Early Octobatch pipelines had simple enough configuration that a human could read and understand them, but not because I was writing them by hand. One of the ground rules I set for the Octobatch experiment in AI-driven development was that the AIs would write all of the code, and that included writing all of the configuration files. The problem was that even though they were doing the writing, I was unconsciously constraining the AIs: pushing back on anything that felt too complex, steering toward structures I could still hold in my head.Octobatch 早期的流水线配置足够简单，以至于人类可以阅读和理解，但这并不是因为我手工编写。我的一个基本规则是：在 AI 驱动开发实验中，所有代码（包括所有配置文件）都由 AI 编写。问题在于，即使它们在写，我仍然在无意识地限制 AI：对任何感觉过于复杂的东西进行回退，倾向于我还能在脑中保持的结构。

At some point I realized my pushback was placing an artificial limit on the project. The whole point of having AIs write the config was that I didn’t need to keep every single line in my head—it was okay to let the AIs handle that level of complexity. Once I stopped constraining them, the cognitive overhead limit I described earlier went away. I could have full pipelines defined in config, including expression steps with real mathematical logic, without needing to hold all the rules and relationships in my head.后来我意识到我的回退人为地限制了项目。让 AI 编写配置的初衷是我不必把每一行都记在脑中——让 AI 处理这种复杂性。一旦我停止限制它们，我之前提到的认知负荷限制就消失了。我可以在配置中定义完整的流水线，包括带有真实数学逻辑的表达式步骤，而无需在脑中记住所有规则和关系。

Once the project really got rolling, I never wrote YAML by hand again. The cycle was always: need a feature, discuss it with Claude and Gemini, push back when something seemed off, and one of them produces the updated config. My job was product vision. Their job was translating that into valid configuration. And every config file they wrote was another test of whether the toolkit actually worked.项目真正起步后，我再也没有手写 YAML。循环始终是：需要功能 → 与 Claude 和 Gemini 讨论 → 对感觉不对的地方回退 → 其中一个生成更新的配置。我的工作是产品愿景，他们的工作是把它翻译成有效配置。每个他们写的配置文件都是对工具包是否真正有效的又一次测试。

This job delineation, however, meant inevitable disagreements between me and the AI, and it’s not always easy to find yourself disagreeing with a machine because they’re surprisingly stubborn (and often shockingly stupid). It required persistence and vigilance to stay in control of the project, especially when I turned over large responsibilities to the AIs.这种工作划分不可避免地导致我与 AI 之间出现分歧，而且当你发现自己与机器意见不合时并不容易，因为它们常常顽固（甚至令人震惊地愚蠢）。这需要坚持和警惕，以保持对项目的控制，尤其是在我把大量责任交给 AI 时。

The AIs consistently optimized for technical correctness—separation of concerns, code organization, effort estimation—which was great, because that’s the job I asked them to do. I optimized for product value. I found that keeping that value as my north star and always focusing on building useful features consistently helped with these disagreements.AI 一贯追求技术正确性——关注点分离、代码组织、工作量估算——这很好，因为这正是我让它们做的事。我则专注于产品价值。把价值作为北极星并始终专注于构建有用功能，帮助我在这些分歧中保持方向。

Keep guidance lean保持指导简洁

Once you start growing the toolkit from failures, the natural progression is to overdocument everything. Generative AIs are biased toward generating, and it’s easy to let them get carried away with it. Every bug feels like it deserves a warning, every edge case feels like it needs a caveat, and before long your toolkit file is bloated with guardrails that cost tokens without adding much value. And since the AI is the one writing your toolkit updates, you need to push back on it the same way you push back on architecture decisions. AIs love adding WARNING blocks and exhaustive caveats. The discipline you need to bring is telling them when not to add something.一旦你开始从失败中扩展工具包，自然的倾向是把所有东西都写进文档。生成式 AI 本身倾向于生成内容，容易让它们写得过于冗长。每个 bug 都像是需要警告，每个边缘情况都像是需要说明，结果你的工具包文件充斥着消耗 token 却价值不大的防护栏。而且因为 AI 在写工具包更新，你需要像对待架构决策一样对它们进行回退。AI 喜欢添加 WARNING 块和详尽的说明。你需要的纪律是告诉它们何时不该添加。

The right level is to state the principle, give one concrete example, and trust the AI to apply it to new situations. When Claude Code made a choice about JSON schema constraints that I might have second-guessed, I had to decide whether to add more guardrails to TOOLKIT.md. The answer was no—the guidance was already there, and the choice it made was actually correct. If you keep tightening guardrails every time an AI makes a judgment call, the signal gets lost in the noise and performance gets worse, not better. When something goes wrong, the impulse—for both you and the AI—is to add a WARNING block. Resist it. One principle, one example, move on.正确的做法是陈述原则，给出一个具体示例，然后信任 AI 将其应用到新情境。当 Claude Code 对 JSON 模式约束做出我可能会质疑的选择时，我必须决定是否在 TOOLKIT.md 中添加更多防护栏。答案是不要——指导已经在了，它的选择实际上是正确的。如果每次 AI 做出判断都收紧防护栏，信号会被噪声淹没，性能反而下降。当出现问题时，你和 AI 的冲动都是添加 WARNING 块。要抵制它。一个原则，一个示例，继续前进。

Treat every use as a test把每一次使用都视为测试

There was no separate “testing phase” for Octobatch’s TOOLKIT.md. Every pipeline that I created with it was a new test. After the very first version, I opened a fresh Claude Code session that had never seen any of my development conversations, pointed it at the newly minted TOOLKIT.md, and asked it to build a pipeline. The first time I tried it, I was surprised at how well it worked! So I kept using it, and as the project rolled along, I updated it with every new feature and tested those updates. When something failed, I traced it back to a missing or unclear rule in the toolkit and fixed it there.Octobatch 的 TOOLKIT.md 没有单独的“测试阶段”。我用它创建的每个流水线都是一次新测试。第一次版本后，我打开一个从未见过我任何开发对话的全新 Claude Code 会话，指向新生成的 TOOLKIT.md，让它构建流水线。第一次尝试时，我惊讶于它的表现！于是我继续使用它，随着项目推进，我在每个新功能后更新它并进行测试。出现错误时，我追溯到工具包中缺失或不清晰的规则并进行修正。

That’s the practical test for any toolkit: open a fresh AI session with no context beyond the file, describe what you want in plain English, and see if the output works. If it doesn’t, the toolkit has a bug.这就是任何工具包的实际测试：打开一个没有上下文的全新 AI 会话，使用纯英文描述需求，看看输出是否可用。如果不可用，工具包就有 bug。

Use more than one model使用多个模型

When you’re building and testing your toolkit, don’t just use one AI. Run the same task through a second model. A good pattern that worked for me was consistently having Claude generate the toolkit and Gemini check its work.在构建和测试工具包时，不要只用一个 AI。让第二个模型完成相同任务。我常用的模式是让 Claude 生成工具包，Gemini 检查其工作。

Different models catch different things, and this matters for both developing and testing the toolkit. I used Claude and Gemini together throughout Octobatch development, and I overruled both when they were wrong about product intent. You can do the same thing: If you work with multiple AIs throughout your project, you’ll start to get a feel for the different kinds of questions they’re good at answering.不同模型捕捉不同问题，这对开发和测试工具包都很重要。我在 Octobatch 开发期间一直同时使用 Claude 和 Gemini，并在它们对产品意图错误时予以纠正。你也可以这样做：如果在项目中使用多个 AI，你会逐渐了解它们各自擅长回答的不同问题类型。

When you have multiple models generate config from the same toolkit independently, you find out fast where your documentation is ambiguous. If two models interpret the same rule differently, the rule needs rewriting. That’s a signal you can’t get from using just one model.当多个模型独立地从同一工具包生成配置时，你会快速发现文档的歧义之处。如果两个模型对同一规则的解释不同，则该规则需要重写。这是单一模型无法提供的信号。

The manual, revisited手册，再次审视

That AT&T PC 6300 manual devoted a full page to labeling diskettes, which may have been overkill, but it got one thing right: it described the building blocks and trusted the reader to figure out the rest. It just had the wrong reader in mind.AT&T PC 6300 手册为标记软盘专门腾出整整一页，或许有点过度，但它做对了一件事：描述了构建块并让读者自行推断其余。只是它的目标读者选错了。

The toolkit pattern is the same idea, pointed at a different audience. You write a file that describes your project’s configuration format, its constraints, and enough worked examples that any AI can generate working inputs from a plain-English description. Your users never have to learn YAML or memorize your schema, because they have a conversation with the AI and it handles the translation.工具包模式是相同的理念，只是面向不同的受众。你编写一个文件，描述项目的配置格式、约束以及足够的示例，使任何 AI 能从纯英文描述生成可用输入。用户无需学习 YAML 或记忆你的模式，因为他们只需与 AI 对话，AI 完成翻译。

If you’re building a project and you want AI to be able to help your users, start here: write the toolkit file before you write the README, grow it from real failures instead of trying to plan it all upfront, keep it lean, test it by using it, and use more than one model because no single AI catches everything.如果你正在构建项目并希望 AI 能帮助用户，请从这里开始：在编写 README 之前先写工具包文件，从真实失败中迭代，而不是一次性规划，保持简洁，通过使用进行测试，并使用多个模型，因为没有单一 AI 能捕捉所有问题。

The AT&T manual’s Chapter 4 was called “What Every User Should Know.” Your toolkit file is “What Every AI Should Know.” The difference is that this time, the reader will actually use it.AT&T 手册第 4 章名为 “每位用户都应了解的内容”。你的工具包文件是 “每个 AI 都应了解的内容”。不同之处在于，这一次读者真的会使用它。

In the next article, I’ll start with a statistic about developer trust in AI-generated code that turned out to be fabricated by the AI itself—and use that to explain why I built a quality playbook that revives the traditional quality practices most teams cut decades ago. It explores an unfamiliar codebase, generates a complete quality infrastructure—tests, review protocols, validation rules—and finds real bugs in the process. It works across Java, C#, Python, and Scala, and it’s available as an open source Claude Code skill.在下一篇文章中，我将从一项关于开发者对 AI 生成代码信任度的统计数据说起——该数据实际上是 AI 自己捏造的——并用它解释为何我构建了一个质量手册，复活了数十年前大多数团队已放弃的传统质量实践。它会探索陌生代码库，生成完整的质量基础设施——测试、审查协议、验证规则——并在此过程中发现真实的 bug。它适用于 Java、C#、Python 和 Scala，并作为开源 Claude Code 技能提供。