This is the first article in a series on agentic engineering and AI-driven development. Look for the next article on March 19 on O’Reilly Radar.这是关于代理工程和 AI 驱动开发系列的第一篇文章。请关注 3 月 19 日在 O’Reilly Radar 上发布的下一篇文章。
There’s been a lot of hype about AI and software development, and it comes in two flavors. One says, “We’re all doomed, that tools like Claude Code will make software engineering obsolete within a year.” The other says, “Don’t worry, everything’s fine, AI is just another tool in the toolbox.” Neither is honest.关于 AI 与软件开发的炒作很多，且分为两种口味。一种说：“我们都完蛋了，像 Claude Code 这样的工具将在一年内让软件工程变得多余。”另一种说：“别担心，一切都好，AI 只是工具箱里的另一个工具。”这两种说法都不诚实。
I’ve spent over 20 years writing about software development for practitioners, covering everything from coding and architecture to project management and team dynamics. For the last two years I’ve been focused on AI, training developers to use these tools effectively, writing about what works and what doesn’t in books, articles, and reports. And I kept running into the same problem: I had yet to find anyone with a coherent answer for how experienced developers should actually work with these tools. There are plenty of tips and plenty of hype but very little structure, and very little you could practice, teach, critique, or improve.我在过去 20 多年里为从业者撰写软件开发相关内容，涵盖编码、架构、项目管理和团队动态等方方面面。过去两年，我专注于 AI，培训开发者有效使用这些工具，在书籍、文章和报告中写下哪些方法有效、哪些无效。我一直遇到同一个问题：我还没有找到任何人能给出经验丰富的开发者应该如何实际使用这些工具的连贯答案。虽然有大量技巧和炒作，但缺乏结构，几乎没有可以实践、教学、批评或改进的内容。
I’d been observing developers at work using AI with various levels of success, and I realized we need to start thinking about this as its own discipline. Andrej Karpathy, the former head of AI at Tesla and a founding member of OpenAI, recently proposed the term “agentic engineering” for disciplined development with AI agents, and others like Addy Osmani are getting on board. Osmani’s framing is that AI agents handle implementation but the human owns the architecture, reviews every diff, and tests relentlessly. I think that’s right.我观察到开发者在使用 AI 时成功程度各不相同，于是意识到我们需要把这视为一个独立的学科。特斯拉前 AI 负责人、OpenAI 创始成员之一 Andrej Karpathy 最近提出了“代理工程”（agentic engineering）这一术语，用于描述与 AI 代理进行有纪律的开发，Addy Osmani 等人也加入了进来。Osmani 的框架是：AI 代理负责实现，而人类负责架构，审查每一次差异，并进行不懈的测试。我认为这很对。
But I’ve spent a lot of the last two years teaching developers how to use tools like Claude Code, agent mode in Copilot, Cursor, and others, and what I keep hearing is that they already know they should be reviewing the AI’s output, maintaining the architecture, writing tests, keeping documentation current, and staying in control of the codebase. They know how to do it in theory. But they get stuck trying to apply it in practice: How do you actually review thousands of lines of AI-generated code? How do you keep the architecture coherent when you’re working across multiple AI tools over weeks? How do you know when the AI is confidently wrong? And it’s not just junior developers who are having trouble with agentic engineering. I’ve talked to senior engineers who struggle with the shift to agentic tools, and intermediate developers who take to it naturally. The difference isn’t necessarily the years of experience; it’s whether they’ve figured out an effective and structured way to work with AI coding tools. That gap between knowing what developers should be doing with agentic engineering and knowing how to integrate it into their day-to-day work is a real source of anxiety for a lot of engineers right now. That’s the gap this series is trying to fill.但在过去两年里，我花了大量时间教开发者使用 Claude Code、Copilot 的代理模式、Cursor 等工具，而我不断听到的反馈是，他们已经知道应该审查 AI 的输出、维护架构、编写测试、保持文档更新，并控制代码库。他们在理论上知道该怎么做，但在实践中卡住了：如何实际审查成千上万行 AI 生成的代码？在数周内跨多个 AI 工具工作时，如何保持架构的一致性？如何判断 AI 自信却错误？而且遇到困难的不仅是初级开发者。我也与在使用代理工具时感到困难的资深工程师以及自然适应的中级开发者交谈过。区别不一定在于经验年限，而在于他们是否找到了与 AI 编码工具高效、结构化的工作方式。开发者应该如何进行代理工程的认知与实际日常工作之间的差距，正是许多工程师当前焦虑的真实来源。这正是本系列想要填补的空白。
Despite what much of the hype about agentic engineering is telling you, this kind of development doesn’t eliminate the need for developer expertise; just the opposite. Working effectively with AI agents actually raises the bar for what developers need to know. I wrote about that experience gap in an earlier O’Reilly Radar piece called “The Cognitive Shortcut Paradox.” The developers who get the most from working with AI coding tools are the ones who already know what good software looks like, and can often tell if the AI wrote it.尽管关于代理工程的很多炒作告诉你，这种开发方式会消除对开发者专业知识的需求，事实恰恰相反。有效使用 AI 代理实际上提升了开发者需要掌握的门槛。我在早期的 O’Reilly Radar 文章《认知捷径悖论》中写到了这种经验差距。最能从 AI 编码工具中获益的开发者，往往已经知道优秀软件的样子，并且常常能判断出 AI 是否写出了好代码。
The idea that AI tools work best when experienced developers are driving them matched everything I’d observed. It rang true, and I wanted to prove it in a way that other developers would understand: by building software. So I started building a specific, practical approach to agentic engineering built for developers to follow, and then I put it to the test. I used it to build a production system from scratch, with the rule that AI would write all the code. I needed a project that was complex enough to stress-test the approach, and interesting enough to keep me engaged through the hard parts. I wanted to apply everything I’d learned and discover what I still didn’t know. That’s when I came back to Monte Carlo simulations.经验丰富的开发者驱动 AI 工具效果最佳的想法与我观察到的一致。这一点很真实，我想用其他开发者能理解的方式来证明：通过构建软件。因此，我开始构建一种针对开发者的具体、实用的代理工程方法，并将其付诸实践。我用它从零构建了一个生产系统，规则是 AI 编写所有代码。我需要一个足够复杂以检验该方法、又足够有趣以保持我在困难阶段的投入的项目。我想应用所学并发现自己仍未掌握的东西。于是，我回到了蒙特卡罗模拟。
The experiment实验
I’ve been obsessed with Monte Carlo simulations ever since I was a kid. My dad’s an epidemiologist—his whole career has been about finding patterns in messy population data, which means statistics was always part of our lives (and it also means that I learned SPSS at a very early age). When I was maybe 11 he told me about the drunken sailor problem: A sailor leaves a bar on a pier, taking a random step toward the water or toward his ship each time. Does he fall in or make it home? You can’t know from any single run. But run the simulation a thousand times, and the pattern emerges from the noise. The individual outcome is random; the aggregate is predictable.我从小就对蒙特卡罗模拟着迷。我的父亲是流行病学家——他的一生都在从混乱的人口数据中寻找模式，这意味着统计学一直是我们生活的一部分（也因此我很早就学会了使用 SPSS）。大约 11 岁时，他给我讲了酗酒水手问题：水手在码头的酒吧离开时，每一步随机向水里或向船走。会掉进水里还是回到家？单次运行无法得知。但如果模拟一千次，噪声中会出现模式。单个结果是随机的，整体是可预测的。
I remember writing that simulation in BASIC on my TRS-80 Color Computer 2: a little blocky sailor stumbling across the screen, two steps forward, one step back. The drunken sailor is the “Hello, world” of Monte Carlo simulations. Monte Carlo is a technique for problems you can’t solve analytically: You simulate them hundreds or thousands of times and measure the aggregate results. Each individual run is random, but the statistics converge on the true answer as the sample size grows. It’s one way we model everything from nuclear physics to financial risk to the spread of disease across populations.我记得在 TRS-80 Color Computer 2 上用 BASIC 编写了那个模拟：屏幕上一个方块水手蹒跚前行，两步前进，一步后退。酗酒水手是蒙特卡罗模拟的“Hello, world”。蒙特卡罗是一种对无法解析求解的问题的技术：你对其进行数百或数千次模拟并测量整体结果。每一次运行都是随机的，但随着样本量增大，统计数据会收敛到真实答案。这是我们从核物理到金融风险再到人口疾病传播建模的一种方式。
What if you could run that kind of simulation today by describing it in plain English? Not a toy demo but thousands of iterations with seeded randomness for reproducibility, where the outputs get validated and the results get aggregated into actual statistics you can use. Or a pipeline where an LLM generates content, a second LLM scores it, and anything that doesn’t pass gets sent back for another try.如果今天你可以用普通英文描述来运行这种模拟会怎样？不是玩具演示，而是成千上万次迭代、带有种子随机性的可复现运行，输出经过验证，结果汇总成可使用的实际统计数据。或者一个流水线：LLM 生成内容，第二个 LLM 打分，未通过的再返回重试。
The goal of my experiment was to build that system, which I called Octobatch. Right now, the industry is constantly looking for new real-world end-to-end case studies in agentic engineering, and I wanted Octobatch to be exactly that case study.我的实验目标是构建这样一个系统，我称之为 Octobatch。如今，业界不断寻找新的真实端到端案例研究来展示代理工程，我希望 Octobatch 正是这样一个案例。
I took everything I’d learned from teaching and observing developers working with AI, put it to the test by building a real system from scratch, and turned the lessons into a structured approach to agentic engineering I’m calling AI-driven development, or AIDD. This is the first article in a series about what agentic engineering looks like in practice, what it demands from the developer, and how you can apply it to your own work.我把从教学和观察开发者使用 AI 中学到的一切付诸实践，构建了一个从零开始的真实系统，并将经验转化为一种结构化的代理工程方法，我称之为 AI 驱动开发（AI-driven development，简称 AIDD）。这是系列文章的第一篇，介绍代理工程在实践中的样子、对开发者的要求以及如何将其应用到自己的工作中。
The result is a fully functioning, well-tested application that consists of about 21,000 lines of Python across several dozen files, backed by complete specifications, nearly a thousand automated tests, and quality integration and regression test suites. I used Claude Cowork to review all the AI chats from the entire project, and it turns out that I built the entire application in roughly 75 hours of active development time over seven weeks. For comparison, I built Octobatch in just over half the time I spent last year playing Blue Prince.结果是一个完整运行、经过充分测试的应用程序，约有 21,000 行 Python 代码，分布在数十个文件中，配有完整的规格说明、近千个自动化测试以及质量集成和回归测试套件。我使用 Claude Cowork 审查了整个项目的所有 AI 对话，结果显示我在七周内大约用了 75 小时的活跃开发时间完成了整个应用程序。相比之下，我去年玩《Blue Prince》时花的时间才是 Octobatch 开发时间的一半多一点。
But this series isn’t just about Octobatch. I integrated AI tools at every level: Claude and Gemini collaborating on architecture, Claude Code writing the implementation, LLMs generating the pipelines that run on the system they helped build. This series is about what I learned from that process: the patterns that worked, the failures that taught me the most, and the orchestration mindset that ties it all together. Each article pulls a different lesson from the experiment, from validation architecture to multi-LLM coordination to the values that kept the project on track.但本系列不仅仅关于 Octobatch。我在每个层面都集成了 AI 工具：Claude 与 Gemini 合作进行架构设计，Claude Code 编写实现，LLM 生成在其帮助构建的系统上运行的流水线。本系列分享我从这一过程中的收获：有效的模式、最有价值的失败、以及将一切联系在一起的编排思维。每篇文章都会从实验中提炼出不同的教训，从验证架构到多 LLM 协调，再到保持项目进度的价值观。
Agentic engineering and AI-driven development代理工程与 AI 驱动开发
When most people talk about using AI to write code, they mean one of two things: AI coding assistants like GitHub Copilot, Cursor, or Windsurf, which have evolved well beyond autocomplete into agentic tools that can run multifile editing sessions and define custom agents; or “vibe coding,” where you describe what you want in natural language and accept whatever comes back. These coding assistants are genuinely impressive, and vibe coding can be really productive.当大多数人谈论使用 AI 编写代码时，他们指的是两件事之一：GitHub Copilot、Cursor 或 Windsurf 等 AI 编码助手，这些工具已经超越了自动补全，成为可以运行多文件编辑会话并定义自定义代理的代理工具；或者是“氛围编码”，即用自然语言描述需求并接受返回的任何结果。这些编码助手确实令人印象深刻，氛围编码也可以非常高效。
Using these tools effectively on a real project, however, maintaining architectural coherence across thousands of lines of AI-generated code, is a different problem entirely. AIDD aims to help solve that problem. It’s a structured approach to agentic engineering where AI tools drive substantial portions of the implementation, architecture, and even project management, while you, the human in the loop, decide what gets built and whether it’s any good. By “structure,” I mean a set of practices developers can learn and follow, a way to know whether the AI’s output is actually good, and a way to stay on track across the life of a project. If agentic engineering is the discipline, AIDD is one way to practice it.然而，在真实项目中有效使用这些工具、在成千上万行 AI 生成代码中保持架构一致性，则是完全不同的问题。AIDD 旨在帮助解决这个问题。它是一种结构化的代理工程方法，AI 工具驱动实现、架构甚至项目管理的很大一部分，而你——人类环节——决定构建什么以及是否合格。所谓“结构”，指的是开发者可以学习并遵循的一套实践，判断 AI 输出是否真的好，以及在项目生命周期中保持进度的方式。如果说代理工程是一门学科，那么 AIDD 就是实践它的一种方式。
In AI-driven development, developers don’t just accept suggestions or hope the output is correct. They assign specific roles to specific tools: one LLM for architecture planning, another for code execution, a coding agent for implementation, and the human for vision, verification, and the decisions that require understanding the whole system.在 AI 驱动开发中，开发者不仅仅接受建议或盲目信任输出。他们为特定工具分配特定角色：一个 LLM 负责架构规划，另一个负责代码执行，编码代理负责实现，而人类负责愿景、验证以及需要整体系统理解的决策。
And the “driven” part is literal. The AI is writing almost all of the code. One of my ground rules for the Octobatch experiment was that I would let AI write all of it. I have high code quality standards, and part of the experiment was seeing whether AIDD could produce a system that meets them. The human decides what gets built, evaluates whether it’s right, and maintains the constraints that keep the system coherent.而“驱动”二字是字面意义。AI 几乎编写了所有代码。Octobatch 实验的一个基本规则是让我让 AI 编写全部代码。我对代码质量有很高的标准，实验的一部分是检验 AIDD 能否产出符合这些标准的系统。人类决定构建什么，评估是否正确，并维护保持系统一致性的约束。
Not everyone agrees on how much the developer needs to stay in the loop, and the fully autonomous end of the spectrum is already producing cautionary tales. Nicholas Carlini at Anthropic recently tasked 16 Claude instances to build a C compiler in parallel with no human in the loop. After 2,000 sessions and $20,000 in API costs, the agents produced a 100,000-line compiler that can build a Linux kernel but isn’t a drop-in replacement for anything, and when all 16 agents got stuck on the same bug, Carlini had to step back in and partition the work himself. Even strong advocates of a completely hands-off, vibe-driven approach to agentic engineering might call that a step too far. The question is how much human judgment you need to make that code trustworthy, and what specific practices help you apply that judgment effectively.并非所有人都认同开发者需要在环路中保持多少参与度，完全自主的极端已经出现了警示案例。Anthropic 的 Nicholas Carlini 最近让 16 个 Claude 实例并行构建 C 编译器，且没有人类参与。经过 2,000 次会话和 20,000 美元的 API 成本后，这些代理产出了一个 100,000 行的编译器，能够构建 Linux 内核，但并不是任何东西的直接替代品。当所有 16 个代理在同一个 bug 上卡住时，Carlini 必须亲自介入并重新划分工作。即使是完全放手、氛围驱动的代理工程的强力倡导者也可能认为这一步走得太远。关键在于你需要多少人类判断才能让代码可信，以及哪些具体实践能帮助你有效运用这种判断。
The orchestration mindset编排思维
If you want to get developers thinking about agentic engineering in the right way, you have to start with how they think about working with AI, not just what tools they use. That’s where I started when I began building a structured approach, and it’s why I started with habits. I developed a framework for these called the Sens-AI Framework, published as both an O’Reilly report (Critical Thinking Habits for Coding with AI) and a Radar series. It’s built around five practices: providing context, doing research before prompting, framing problems precisely, iterating deliberately on outputs, and applying critical thinking to everything the AI produces. I started there because habits are how you lock in the way you think about how you’re working. Without them, AI-driven development produces plausible-looking code that falls apart under scrutiny. With them, it produces systems that a single developer couldn’t build alone in the same time frame.如果你想让开发者以正确的方式思考代理工程，必须先从他们如何与 AI 合作的思维方式入手，而不仅仅是使用哪些工具。这正是我在构建结构化方法时的起点，也是我从习惯开始的原因。我为这些习惯开发了 Sens-AI 框架，并以 O’Reilly 报告（《使用 AI 编码的批判性思维习惯》）和 Radar 系列的形式发布。它围绕五项实践：提供上下文、在提示前进行研究、精确构建问题、对输出进行有意迭代、对 AI 产生的一切进行批判性思考。我从这里开始，因为习惯是锁定你对工作方式思考的方式。没有它们，AI 驱动开发只能产生看似合理但经不起审查的代码。有了它们，才能产出单个开发者在相同时间内无法独立完成的系统。
Habits are the foundation, but they’re not the whole picture. AIDD also has practices (concrete techniques like multi-LLM coordination, context file management, and using one model to validate another’s output) and values (the principles behind those practices). If you’ve worked with Agile methodologies like Scrum or XP, that structure should be pretty familiar: Practices tell you how to work day-to-day, and habits are the reflexes you develop so that the practices become automatic.习惯是基础，但并非全部。AIDD 还有实践（如多 LLM 协调、上下文文件管理、使用一个模型验证另一个模型输出的具体技术）和价值观（这些实践背后的原则）。如果你熟悉 Scrum 或 XP 等敏捷方法，这种结构应该很熟悉：实践告诉你日常如何工作，习惯是你培养的反射，使实践变得自动化。
Values often seem weirdly theoretical, but they’re an important piece of the puzzle because they guide your decisions when the practices don’t give you a clear answer. There’s an emerging culture around agentic engineering right now, and the values you bring to your project either match or clash with that culture. Understanding where the values come from is what makes the practices stick. All of that leads to a whole new mindset, what I’m calling the orchestration mindset. This series builds all four layers, using Octobatch as the proving ground.价值观常常显得理论化，但它们是拼图的重要部分，因为当实践无法给出明确答案时，它们指引你的决策。当前围绕代理工程正形成一种新文化，你带入项目的价值观要么与之匹配，要么冲突。了解价值观的来源是让实践落地的关键。所有这些汇聚成一种全新的思维方式，我称之为编排思维。本系列构建了这四层结构，以 Octobatch 为试验场。
Octobatch was a deliberate experiment in AIDD. I designed the project as a test case for the entire approach, to see what a disciplined AI-driven workflow could produce and where it would break down, and I used it to apply and improve the practices and values to make them effective and easy to adopt. And whether by instinct or coincidence, I picked the perfect project for this experiment. Octobatch is a batch orchestrator. It coordinates asynchronous jobs, manages state across failures, tracks dependencies between pipeline steps, and makes sure validated results come out the other end. That kind of system is fun to design but a lot of the details, like state machines, retry logic, crash recovery, and cost accounting, can be tedious to implement. It’s exactly the kind of work where AIDD should shine, because the patterns are well understood but the implementation is repetitive and error-prone.Octobatch 是一次有意的 AIDD 实验。我将项目设计为整个方法的测试案例，以观察有纪律的 AI 驱动工作流能产出什么以及会在哪些环节出现问题，并用它来应用和改进实践与价值观，使其有效且易于采纳。无论是本能还是巧合，我为这次实验挑选了完美的项目。Octobatch 是一个批处理编排器。它协调异步作业，管理失败时的状态，跟踪流水线步骤之间的依赖，并确保验证后的结果顺利输出。这类系统设计起来很有趣，但诸如状态机、重试逻辑、崩溃恢复和成本核算等细节实现起来往往繁琐。正是这种工作最适合 AIDD 发光发热，因为模式已被充分理解，而实现却重复且易出错。
Orchestration—the work of coordinating multiple independent processes toward a coherent outcome—evolved into a core idea behind AIDD. I found myself orchestrating LLMs the same way Octobatch orchestrates batch jobs: assigning roles, managing handoffs, validating outputs, recovering from failures. The system I was building and the process I was using to build it followed the same pattern. I didn’t anticipate it when I started, but building a system that orchestrates AI turns out to be a pretty good way to learn how to orchestrate AI. That’s the accidental part of the accidental orchestrator. That parallel runs through every article in this series.编排——将多个独立进程协调至一致结果的工作——演变为 AIDD 的核心理念。我发现自己在编排 LLM 时的方式与 Octobatch 编排批处理作业如出一辙：分配角色、管理交接、验证输出、从失败中恢复。我构建的系统和构建过程遵循相同的模式。起初我没有预料到，但构建一个编排 AI 的系统恰好成为学习如何编排 AI 的好方法。这就是意外指挥者的意外之处。这种平行贯穿本系列的每篇文章。
Want Radar delivered straight to your inbox? Join us on Substack. Sign up here.想要 Radar 直接送达你的收件箱吗？加入我们的 Substack。 点击此处注册。
The path to batch通往批处理的道路
I didn’t begin the Octobatch project by starting with a full end-to-end Monte Carlo simulation. I started where most people start: typing prompts into a chat interface. I was experimenting with different simulation and generation ideas to give the project some structure, and a few of them stuck. A blackjack strategy comparison turned out to be a great test case for a multistep Monte Carlo simulation. NPC dialogue generation for a role-playing game gave me a creative workload with subjective quality to measure. Both had the same shape: a set of structured inputs, each processed the same way. So I had Claude write a simple script to automate what I’d been doing by hand, and I used Gemini to double-check the work, make sure Claude really understood my ask, and fix hallucinations. It worked fine at small scale, but once I started running more than a hundred or so units, I kept hitting rate limits, the caps that providers put on how many API requests you can make per minute.我并不是一开始就用完整的端到端蒙特卡罗模拟来启动 Octobatch 项目。我从大多数人一样的起点开始：在聊天界面输入提示。我在尝试不同的模拟和生成思路，以为项目提供结构，最终有几个思路被保留下来。一个黑杰克策略比较成为多步蒙特卡罗模拟的绝佳测试案例。为角色扮演游戏生成 NPC 对话则提供了一个主观质量可衡量的创意工作负载。两者都有相同的形态：一组结构化输入，每个都以相同方式处理。于是我让 Claude 编写一个简单脚本，自动化我手动完成的工作，并使用 Gemini 进行双重检查，确保 Claude 真正理解我的需求并修正幻觉。小规模时运行良好，但当我开始运行超过一百个单元时，就频繁遇到速率限制——提供商对每分钟 API 请求数量的上限。
That’s what pushed me to LLM batch APIs. Instead of sending individual prompts one at a time and waiting for each response, the major LLM providers all offer batch APIs that let you submit a file containing all of your requests at once. The provider processes them on their own schedule; you wait for results instead of getting them immediately, but you don’t have to worry about rate caps. I was happy to discover they also cost 50% less, and that’s when I started tracking token usage and costs in earnest. But the real surprise was that batch APIs performed better than real-time APIs at scale. Once pipelines got past the 100- or 200-unit mark, batch started running significantly faster than real time. The provider processes the whole batch in parallel on their infrastructure, so you’re not bottlenecked by round-trip latency or rate caps anymore.这促使我转向 LLM 批处理 API。与一次发送单个提示并等待响应不同，主要的 LLM 提供商都提供批处理 API，允许一次提交包含所有请求的文件。提供商自行调度处理；你等待结果而不是立即获取，但无需担心速率上限。我很高兴发现它们的费用也降低了 50%，于是开始认真追踪 token 使用量和成本。但真正的惊喜是，批处理 API 在规模化时的表现优于实时 API。当流水线超过 100 或 200 单元时，批处理的运行速度显著快于实时。提供商在其基础设施上并行处理整个批次，你不再受往返延迟或速率上限的瓶颈。
The switch to batch APIs changed how I thought about the whole problem of coordinating LLM API calls at scale, and led to the idea of configurable pipelines. I could chain stages together: The output of one step could become the input to the next, and I could kick off the whole pipeline and come back to finished results. It turns out I wasn’t the only one making the shift to batch APIs. Between April 2024 and July 2025, OpenAI, Anthropic, and Google all launched batch APIs, converging on the same pricing model: 50% of the real-time rate in exchange for asynchronous processing.转向批处理 API 改变了我对大规模协调 LLM API 调用整个问题的思考方式，并引出了可配置流水线的想法。我可以将阶段串联起来：一个步骤的输出成为下一个步骤的输入，并且可以一次启动整个流水线，稍后返回查看完成结果。事实证明，我并不是唯一一个转向批处理 API 的人。2024 年 4 月至 2025 年 7 月期间，OpenAI、Anthropic 和 Google 都推出了批处理 API，采用相同的定价模型：实时费率的 50%，换取异步处理。
You probably didn’t notice that all three major AI providers released batch APIs. The industry conversation was dominated by agents, tool use, MCP, and real-time reasoning. Batch APIs shipped with relatively little fanfare, but they represent a genuine shift in how we can use LLMs. Instead of treating them as conversational partners or one-shot SaaS APIs, we can treat them as processing infrastructure, closer to a MapReduce job than a chatbot. You give them structured data and a prompt template, and they process all of it and hand back the results. What matters is that you can now run tens of thousands of these transformations reliably, at scale, without managing rate limits or connection failures.你可能没有注意到，三大 AI 提供商都发布了批处理 API。业界讨论的焦点一直是代理、工具使用、MCP 和实时推理。批处理 API 的发布相对低调，但它们代表了我们使用 LLM 的真正转变。我们不再把它们当作对话伙伴或一次性 SaaS API，而是把它们当作处理基础设施，更像是 MapReduce 作业而非聊天机器人。你提供结构化数据和提示模板，它们处理全部并返回结果。关键是，你现在可以可靠、规模化地运行数万次此类转换，而无需管理速率限制或连接失败。
Why orchestration?为什么要编排？
If batch APIs are so useful, why can’t you just write a for-loop that submits requests and collects results? You can, and for simple cases a quick script with a for-loop works fine. But once you start running larger workloads, the problems start to pile up. Solving those problems turned out to be one of the most important lessons for developing a structured approach to agentic engineering.如果批处理 API 如此有用，为什么不能直接写一个 for 循环提交请求并收集结果？可以，而且对于简单情况，一个带有 for 循环的快速脚本完全可行。但一旦你开始运行更大的工作负载，问题就会堆积。解决这些问题成为构建结构化代理工程方法的最重要经验之一。
First, batch jobs are asynchronous. You submit a job, and results come back hours later, so your script needs to track what was submitted and poll for completion. If your script crashes in the middle, you lose that state. Second, batch jobs can partially fail. Maybe 97% of your requests succeeded and 3% didn’t. Your code needs to figure out which 3% failed, extract them, and resubmit just those items. Third, if you’re building a multistage pipeline where the output of one step feeds into the next, you need to track dependencies between stages. And fourth, you need cost accounting. When you’re running tens of thousands of requests, you want to know how much you spent, and ideally, how much you’re going to spend when you first start the batch. Every one of these has a direct parallel to what you’re doing in agentic engineering: keeping track of the work multiple AI agents are doing at once, dealing with code failures and bugs, making sure the entire project stays coherent when AI coding tools are only looking at the one part currently in context, and stepping back to look at the wider project management picture.首先，批处理作业是异步的。你提交作业后，结果可能在数小时后返回，因此脚本需要跟踪已提交的内容并轮询完成情况。如果脚本在中途崩溃，你会失去状态。其次，批处理作业可能部分失败。也许 97% 的请求成功，3% 失败。你的代码需要找出这 3% 的失败项，提取并重新提交。第三，如果你在构建多阶段流水线，前一步的输出作为下一步的输入，你需要跟踪阶段之间的依赖关系。第四，你需要成本核算。当你运行数万次请求时，你想知道花了多少钱，理想情况下还能在启动批处理时预估费用。这些问题在代理工程中都有直接对应：跟踪多个 AI 代理的并行工作、处理代码错误和 bug、确保 AI 编码工具只关注当前上下文时项目整体的一致性，以及回顾更广阔的项目管理视角。
All of these problems are solvable, but they’re not problems you want to solve over and over (in both situations—when you’re orchestrating LLM batch jobs or orchestrating AI coding tools). Solving these problems in the code gave some interesting lessons about the overall approach to agentic engineering. Batch processing moves the complexity from connection management to state management. Real-time APIs are hard because of rate limits and retries. Batch APIs are hard because you have to track what’s in flight, what succeeded, what failed, and what’s next.所有这些问题都是可以解决的，但它们不是你想一次又一次解决的（无论是编排 LLM 批处理作业还是编排 AI 编码工具）。在代码中解决这些问题带来了关于整体代理工程方法的有趣经验。批处理将复杂性从连接管理转移到状态管理。实时 API 因速率限制和重试而困难。批处理 API 困难在于必须跟踪正在进行的、成功的、失败的以及接下来的任务。
Before I started development, I went looking for existing tools that handled this combination of problems, because I didn’t want to waste my time reinventing the wheel. I didn’t find anything that did the job I needed. Workflow orchestrators like Apache Airflow and Dagster manage DAGs and task dependencies, but they assume tasks are deterministic and don’t provide LLM-specific features like prompt template rendering, schema-based output validation, or retry logic triggered by semantic quality checks. LLM frameworks like LangChain and LlamaIndex are designed around real-time inference chains and agent loops—they don’t manage asynchronous batch job lifecycles, persist state across process crashes, or handle partial failure recovery at the chunk level. And the batch API client libraries from the providers themselves handle submission and retrieval for a single batch, but not multistage pipelines, cross-step validation, or provider-agnostic execution.在开始开发之前，我寻找现有工具来处理这些问题的组合，因为我不想浪费时间重新发明轮子。但我没有找到能满足我需求的工具。Apache Airflow 和 Dagster 等工作流编排器管理 DAG 和任务依赖，但它们假设任务是确定性的，且不提供 LLM 特有的功能，如提示模板渲染、基于模式的输出验证或由语义质量检查触发的重试逻辑。LangChain 和 LlamaIndex 等 LLM 框架围绕实时推理链和代理循环设计——它们不管理异步批处理作业的生命周期，也不在进程崩溃后持久化状态，或在块级别处理部分失败恢复。而各提供商的批处理 API 客户端库仅处理单个批次的提交和检索，未覆盖多阶段流水线、跨步骤验证或跨提供商的执行。
Nothing I found covered the full lifecycle of multiphase LLM batch workflows, from submission and polling through validation, retry, cost tracking, and crash recovery, across all three major AI providers. That’s what I built.我没有找到覆盖多阶段 LLM 批处理工作流完整生命周期的工具——从提交、轮询到验证、重试、成本追踪和崩溃恢复，跨所有三大 AI 提供商。这正是我构建的内容。
Lessons from the experiment实验的教训
The goal of this article, as the first one in my series on agentic engineering and AI-driven development, is to lay out the hypothesis and structure of the Octobatch experiment. The rest of the series goes deep on the lessons I learned from it: the validation architecture, multi-LLM coordination, the practices and values that emerged from the work, and the orchestration mindset that ties it all together. A few early lessons stand out, because they illustrate what AIDD looks like in practice and why developer experience matters more than ever.本文作为我关于代理工程和 AI 驱动开发系列的第一篇，旨在阐述 Octobatch 实验的假设和结构。系列的其余部分将深入探讨我从中学到的经验：验证架构、多 LLM 协调、从工作中产生的实践与价值观，以及将一切联系在一起的编排思维。以下几条早期教训尤为突出，因为它们展示了 AIDD 在实践中的样子以及为何开发者体验比以往任何时候都更重要。
- You have to run things and check the data. Remember the drunken sailor, the “Hello, world” of Monte Carlo simulations? At one point I noticed that when I ran the simulation through Octobatch, 77.5% of the sailors fell in the water. The results for a random walk should be 50/50, so clearly something was badly wrong. It turned out the random number generator was being re-seeded at every iteration with sequential seed values, which created correlation bias between runs. I didn’t identify the problem immediately; I ran a bunch of tests using Claude Code as a test runner to generate each test, run it, and log the results; Gemini looked at the results and found the root cause. Claude had trouble coming up with a fix that worked well, and proposed a workaround with a large list of preseeded random number values in the pipeline. Gemini proposed a hash-based fix reviewing my conversations with Claude, but it seemed overly complex. Once I understood the problem and rejected their proposed solutions, I decided the best fix was simpler than either of the AI’s suggestions: a persistent RNG per simulation unit that advanced naturally through its sequence. I needed to understand both the statistics and the code to evaluate those three options. Plausible-looking output and correct output aren’t the same thing, and you need enough expertise to tell the difference. (We’ll talk more about this situation in the next article in the series.)你必须运行并检查数据。还记得酗酒水手——蒙特卡罗模拟的“Hello, world”吗？有一次我注意到，当我通过 Octobatch 运行模拟时，77.5% 的水手掉进水里。随机游走的结果应该是 50/50，显然出了大问题。原来随机数生成器在每次迭代时都用顺序种子重新设定，导致运行之间出现相关偏差。我没有立刻发现问题；我使用 Claude Code 作为测试运行器生成每个测试、运行并记录结果，Gemini 查看结果并找出根本原因。Claude 难以提出有效的修复方案，建议使用大量预设随机数列表的变通办法；Gemini 提出了基于哈希的修复方案，但显得过于复杂。了解问题并拒绝它们的建议后，我决定采用最简单的方案：为每个模拟单元保持一个持久的 RNG，顺序自然递进。我需要同时理解统计学和代码才能评估这三种方案。看似合理的输出与正确的输出并不相同，你需要足够的专业知识来辨别差异。（我们将在系列的下一篇文章中进一步讨论此情形。）
- LLMs often overestimate complexity. At one point I wanted to add support for custom mathematical expressions in the analysis pipeline. Both Claude and Gemini pushed back, telling me, “This is scope creep for v1.0” and “Save it for v1.1.” Claude estimated three hours to implement. Because I knew the codebase, I knew we were already using asteval, a Python library that provides a safe, minimalistic evaluator for mathematical expressions and simple Python statements, elsewhere to evaluate expressions, so this seemed like a straightforward use of a library we’re already using elsewhere. Both LLMs thought the solution would be far more complex and time-consuming than it actually was; it took just two prompts to Claude Code (generated by Claude), and about five minutes total to implement. The feature shipped and made the tool significantly more powerful. The AIs were being conservative because they didn’t have my context about the system’s architecture. Experience told me the integration would be trivial. Without that experience, I would have listened to them and deferred a feature that took five minutes.LLM 常常高估复杂度。曾有一次，我想在分析流水线中加入对自定义数学表达式的支持。Claude 和 Gemini 都表示反对，告诉我“这属于 v1.0 的范围蔓延”，并建议“留到 v1.1”。Claude 估计实现需要三小时。因为我熟悉代码库，知道我们已经在其他地方使用了 asteval——一个提供安全、简约的数学表达式和简单 Python 语句求值的 Python 库——所以这看起来是对已有库的直接调用。两个 LLM 都认为解决方案会比实际更复杂、更耗时；实际上只用了两次 Claude Code 的提示（由 Claude 生成），总共约五分钟就实现了。该功能上线后显著提升了工具的能力。AI 之所以保守，是因为它们缺乏我对系统架构的上下文。我的经验告诉我，这种集成是微不足道的。若没有这种经验，我可能会听从它们的建议，推迟实现仅需五分钟的功能。
- AI is often biased toward adding code, not deleting it. Generative AI is, unsurprisingly, biased toward generation. So when I asked the LLMs to fix problems, their first response was often to add more code, adding another layer or another special case. I can’t think of a single time in the whole project when one of the AIs stepped back and said, “Tear this out and rethink the approach.” The most productive sessions were the ones where I overrode that instinct and pushed for simplicity. This is something experienced developers learn over a career: The most successful changes often delete more than they add—the PRs we brag about are the ones that delete thousands of lines of code.AI 倾向于添加代码，而不是删除代码。生成式 AI 自然偏向生成内容。因此，当我让 LLM 修复问题时，它们的第一反应往往是添加更多代码，增加另一层或另一个特例。我在整个项目中从未见过 AI 主动说“把这段代码删掉，重新思考方案”。最有效的会话是我克服这种本能，推动简化的时刻。这是经验丰富的开发者在职业生涯中学到的：最成功的改动往往是删除多于添加——我们引以为豪的 PR 往往是删除了数千行代码。
- The architecture emerged from failure. The AI tools and I didn’t design Octobatch’s core architecture up front. Our first attempt was a Python script with in-memory state and a lot of hope. It worked for small batches but fell apart at scale: A network hiccup meant restarting from scratch, a malformed response required manual triage. A lot of things fell into place after I added the constraint that the system must survive being killed at any moment. That single requirement led to the tick model (wake up, check state, do work, persist, exit), the manifest file as source of truth, and the entire crash-recovery architecture. We discovered the design by repeatedly failing to do something simpler.架构源于失败。AI 工具和我并没有在一开始就设计 Octobatch 的核心架构。我们的第一次尝试是一个带有内存状态的 Python 脚本，充满了希望。它在小批量时还能工作，但在规模化时崩溃：网络抖动导致必须从头重启，响应格式错误需要手动排查。后来我加入了系统必须随时能够被杀死的约束，这单一要求促成了 tick 模型（唤醒、检查状态、工作、持久化、退出）、将 manifest 文件作为真相来源以及完整的崩溃恢复架构。我们通过一次次的简单失败不断发现并完善设计。
- Your development history is a dataset. I just told you several stories from the Octobatch project, and this series will be full of them. Every one of those stories came from going back through the chat logs between me, Claude, and Gemini. With AIDD, you have a complete transcript of every architectural decision, every wrong turn, every moment where you overruled the AI and every moment where it corrected you. Very few development teams have ever had that level of fidelity in their project history. Mining those logs for lessons learned turns out to be one of the most valuable practices I’ve found.你的开发历史是一套数据集。我刚才给你讲了几个 Octobatch 项目的故事，这个系列将充满此类故事。所有这些故事都来源于我、Claude 和 Gemini 之间的聊天记录。使用 AIDD，你拥有每一次架构决策、每一次错误转向、每一次你否决 AI、每一次 AI 修正你的完整记录。很少有开发团队能拥有如此高保真的项目历史。挖掘这些日志以获取经验教训，成为我发现的最有价值的实践之一。
Near the end of the project, I switched to Cursor to make sure none of this was specific to Claude Code. I created fresh conversations using the same context files I’d been maintaining throughout development, and was able to bootstrap productive sessions immediately; the context files worked exactly as designed. The practices I’d developed transferred cleanly to a different tool. The value of this approach comes from the habits, the context management, and the engineering judgment you bring to the conversation, not from any particular vendor.项目后期，我切换到 Cursor，以确保这些经验并非特定于 Claude Code。我使用一直维护的上下文文件创建了全新的对话，并能够立即启动高效的会话；上下文文件如预期般工作。我的实践顺利迁移到不同的工具上。此方法的价值来源于习惯、上下文管理以及你在对话中带来的工程判断，而非任何特定供应商。
These tools are moving the world in a direction that favors developers who understand the ways engineering can go wrong and know solid design and architecture patterns…and who are okay letting go of control of every line of code.这些工具正将世界推向一个更有利于懂得工程可能出错、掌握扎实设计与架构模式的开发者的方向……并且愿意放手让 AI 编写每一行代码的方向。
What’s next接下来是什么
Agentic engineering needs structure, and structure needs a concrete example to make it real. The next article in this series goes into Octobatch itself, because the way it orchestrates AI is a remarkably close parallel to what AIDD asks developers to do. Octobatch assigns roles to different processing steps, manages handoffs between them, validates their outputs, and recovers when they fail. That’s the same pattern I followed when building it: assigning roles to Claude and Gemini, managing handoffs between them, validating their outputs, and recovering when they went down the wrong path. Understanding how the system works turns out to be a good way to understand how to orchestrate AI-driven development. I’ll walk through the architecture, show what a real pipeline looks like from prompt to results, present the data from a 300-hand blackjack Monte Carlo simulation that puts all of these ideas to the test, and use all of that to demonstrate ideas we can apply directly to agentic engineering and AI-driven development.代理工程需要结构，而结构需要具体案例来实现。系列的下一篇文章将深入探讨 Octobatch 本身，因为它编排 AI 的方式与 AIDD 要求开发者做的事情惊人地相似。Octobatch 为不同的处理步骤分配角色，管理它们之间的交接，验证输出，并在出现错误时进行恢复。这正是我在构建它时遵循的模式：为 Claude 和 Gemini 分配角色，管理它们之间的交接，验证输出，并在走错路时进行恢复。了解系统的工作原理正是理解如何编排 AI 驱动开发的好方法。我将展示架构，展示从提示到结果的真实流水线，呈现 300 手 Blackjack 蒙特卡罗模拟的数据，并用这些来演示可直接应用于代理工程和 AI 驱动开发的想法。
Later articles go deeper into the practices and ideas I learned from this experiment that make AI-driven development work: how I coordinated multiple AI models without losing control of the architecture, what happened when I tested the code against what I actually intended to build, and what I learned about the gap between code that runs and code that does what you meant. Along the way, the experiment produced some findings about how different AI models see code that I didn’t expect—and that turned out to matter more than I thought they would.后续文章将更深入探讨我从实验中学到的实践和理念，使 AI 驱动开发得以落地：我如何在不失控的前提下协调多个 AI 模型、我在测试代码与实际意图时的发现、以及我对运行代码与实现意图之间差距的领悟。在此过程中，实验还产生了一些关于不同 AI 模型如何看待代码的意外发现——这些发现比我预想的更为重要。
