TDD inside the agent loop - theater or actual value?在智能体循环中应用 TDD——是形式主义还是真有价值？
Birgitta is a Distinguished Engineer and AI-assisted delivery expert. She has over 20 years of experience as a software developer, architect and technical leader.
This article is part of “Exploring Gen AI”. A series capturing Thoughtworks technologists' explorations of using gen ai technology for software development.
The TDD (test-driven development) workflow can be used with AI-augmented coding in multiple ways:TDD（测试驱动开发）工作流可以通过多种方式与 AI 辅助编码相结合：
- Human writes the tests: A human defines the test scenarios in some form, be it in natural language, in BDD style, or directly in code. Then AI writes the implementation to make those tests pass (with maybe a first step that transforms the human's scenarios into code).人类编写测试：人类以某种形式（无论是自然语言、BDD 风格还是直接代码）定义测试场景，然后由 AI 编写实现以通过这些测试（可能包含一个将人类场景转换为代码的初始步骤）。
- Review checkpoint for the human: AI writes a failing test, human looks at it to review that the test is testing the wanted behavior, then AI writes the implementation人类审查检查点：AI 编写一个失败的测试，人类查看并确认该测试是否覆盖了预期的行为，随后 AI 再编写实现。
- Fully inside the agentic loop: Prompt an agent to write failing tests first, one by one, and then write the implementation and check that the previously failing test is green.完全在智能体循环内：提示智能体先逐个编写失败的测试，然后编写实现，并检查之前失败的测试是否变绿。
At this stage, that last usage is by far the most common one. But does it really make a difference, asking an agent to follow a TDD workflow fully inside its own loop? Does it really provide value, or is it one of the rare examples where what's good for the human might be irrelevant or bad for a coding agent?现阶段，最后一种用法最为常见。但让智能体在其循环内完全遵循 TDD 工作流真的有区别吗？它是否真的提供了价值，还是说这属于那种“对人类有益，但对编码智能体而言可能无关紧要甚至有害”的少数案例之一？
I created an exploratory evaluation setup to scratch the surface of this question and see what I would find. It is far from a comprehensive and structured eval result, but it did create some hypotheses to think about if you are working hard to get your agent to use TDD.我创建了一个探索性的评估设置来初步探讨这个问题，看看能发现什么。虽然这远非一个全面且结构化的评估结果，但它确实产生了一些假设，如果你正在努力让你的智能体使用 TDD，这些假设值得思考。
TLDR; Based on Opus's judgment of the quality of the outcomes, there was no clearly discernable difference based on TDD workflow versus no TDD workflow. On the contrary, more than once Opus ranked the non-TDD workflow solutions slightly higher in design and test quality. There was also no meaningful difference in mutation scores across the solutions.TL;DR（总结）：根据 Opus 对结果质量的判断，基于 TDD 工作流与非 TDD 工作流之间没有明显差异。相反，Opus 多次将非 TDD 工作流方案在设计和测试质量上的排名略微靠前。各方案之间的变异测试得分（mutation scores）也没有显著差异。
The setup设置
- Tasks: I created a small, medium and a larger task with the help of Claude, all green field implementations of a bit of business logic. I had it make a bunch of suggestions, asking for idiosyncratic and specific logic to increase the probability that there will be variance between solutions, and not just a repetition of something that is already dominant in the training data.任务：在 Claude 的帮助下，我创建了小型、中型和大型任务，全部都是从零开始实现一些业务逻辑。我让它提出了许多建议，要求其逻辑具有独特且特定的属性，以增加方案间产生差异的可能性，而不是仅仅重复训练数据中已有的主流内容。
- Instructions: In all runs, I included instructions to achieve at least 80% code coverage.指令：在所有运行中，我都包含了至少 80% 代码覆盖率的指令。
- Model: I used Sonnet 4.6 to generate the solutions.模型：我使用 Sonnet 4.6 来生成解决方案。
- Judgment of TDD adherence: Evaluation of adherence to TDD was also done by Sonnet 4.6.TDD 依从性判断：对 TDD 依从性的评估同样由 Sonnet 4.6 完成。
- Judgment of solutions: Opus 4.8 compared the quality of both solutions and their tests, without knowledge of how the solutions were created. I didn't give very specific inputs on what I consider to be good quality, as this was a very open exploration. And in my experience, the more specific I would have gotten, the more the model could have over-indexed unnecessarily on the quality criteria I list. Opus has shown to be quite a capable model in terms of judgment of code quality. For its ranking of the solutions, it created a rubric on the fly to pass to all subagents that were evaluating the individual solutions.方案判断：Opus 4.8 在不知晓方案如何创建的情况下，对比了两个方案及其测试的质量。由于这是一次非常开放的探索，我没有提供关于“何为高质量”的具体输入。根据我的经验，输入越具体，模型就越可能在我不经意间列出的质量标准上过度拟合。事实证明，Opus 在判断代码质量方面是一个相当有能力的模型。为了给方案排名，它即时创建了一套评分标准，并将其传递给评估各个方案的子智能体。
When you draw your own conclusions from my results, the main caveats to consider are:当你根据我的结果得出自己的结论时，需要考虑的主要注意事项有：
- This is obviously a very small sample size, so take it with a grain of salt样本量非常小，请谨慎参考。
- Judgment of what “quality” means was almost fully left to Opus (with only a few pointers about test quality)对“质量”含义的判断几乎完全留给了 Opus（仅有少量关于测试质量的提示）。
- None of the runs ever followed TDD perfectly, but pretty well没有一次运行能完美遵循 TDD，但总体表现还不错。
- The coding tasks given to the agents were all greenfield and relatively small, purely about business logic给智能体的编码任务都是从零开始的，且相对较小，纯粹涉及业务逻辑。
How good are agents even at TDD?智能体在 TDD 方面的表现究竟如何？
Before I even started, I needed to make sure the TDD instructions were actually followed. Historically that hasn't gone well for me: agents often write the implementation first and generate tests after, skip confirming the red step, or over-implement ahead of the current test so the next one passes without ever going red.在我开始之前，我需要确保 TDD 指令确实得到了遵循。从过往经验来看，这对我来说并不顺利：智能体通常会先编写实现后再生成测试，或者跳过确认“红灯”步骤，又或者在当前测试之前过度实现，导致下一个测试无需经过“红灯”阶段直接通过。
The prompt I ended up using worked well enough with Sonnet to use for the comparison, though all sessions showed some of these failures to an extent. For each TDD run, I had an independent agent judge how well the workflow was followed, based on the session transcript, so that I wouldn't accidentally take into account a run that didn't meaningfully do it.我最终使用的提示词在 Sonnet 上效果足够好，可以用于比较，尽管所有会话在某种程度上都表现出了一些失败。对于每次 TDD 运行，我都会让一个独立的智能体根据会话记录来判断工作流的遵循程度，这样我就不会意外地将那些未能有效执行的运行纳入考量。
Results结果
I created 5 batches of solutions, with two non-TDD and two TDD solutions each. In one batch, I also added two runs that were instructed to write the tests first, without full TDD discipline (no incremental red/green).我创建了 5 批解决方案，每批包含两个非 TDD 和两个 TDD 解决方案。在其中一批中，我还增加了两个被要求先写测试、但不遵循完整 TDD 规范（没有增量式的红/绿循环）的运行。
Across the small (1 batch) and medium (3 batches) tasks there was a bit of a pattern: Opus ranked the two non-TDD solutions #1 and #2, and the two TDD solutions #3 and #4. Only once - after I strengthened the TDD prompt with a more explicit refactor-and-design-review step - did a TDD solution rank #1. In that same batch, the other TDD solution, run with the identical prompt, ranked last though... For the larger task, TDD landed in the middle, while the two non-TDD runs took both the best and the worst spot.在小型（1 批）和中型（3 批）任务中，出现了一种模式：Opus 将两个非 TDD 方案排在第 1 和第 2 位，而将两个 TDD 方案排在第 3 和第 4 位。只有一次——在我加强了 TDD 提示词，增加了更明确的“重构与设计审查”步骤后——一个 TDD 方案才排在第 1 位。但在同一批次中，另一个使用相同提示词运行的 TDD 方案却排在最后……对于大型任务，TDD 处于中间位置，而两个非 TDD 运行分别占据了最好和最差的位置。
(Details in the appendix)（详情见附录）
Hypotheses假设
So in summary, both TDD and non-TDD scored both as a best and a worst solution across the batches, with TDD overall performing slightly worse.综上所述，TDD 和非 TDD 在各批次中都既有表现最好的方案，也有表现最差的方案，总体而言 TDD 表现略逊一筹。
Asked to look at the session traces to hypothesize about the results with knowledge of which workflow was used for which, Opus found that the non-TDD and test-first runs always created the full design (architecture, data types, edge cases, contracts) before writing any code or tests, rather than working through it one requirement/test at a time. That seemed to be the thing that moved the needle slightly towards comparatively better data models, more cross-cutting edge cases, and better completeness of the functionality.在被要求查看会话轨迹并结合已知工作流来对结果进行假设时，Opus 发现非 TDD 和“测试先行”运行总是会在编写任何代码或测试之前创建完整的设计（架构、数据类型、边缘情况、契约），而不是一次只解决一个需求/测试。这似乎是导致其数据模型相对更好、边缘情况考虑更全面、功能完整性更高的关键因素。
The TDD instructions actively work against such an up front design step. The design in those runs emerged from the sum of many locally-minimal decisions and was rarely revisited, so it tended to land on whatever shape the first test happened to lock in. Behaviour the agent didn't think to write a test for didn't get implemented at all.TDD 指令实际上阻碍了这种预先设计步骤。在那些运行中，设计是从许多局部最小决策的总和中产生的，且很少被重新审视，因此往往最终定格在第一个测试碰巧锁定的形状上。智能体没想过要写测试的行为根本就不会被实现。
When I chatted to Ivett Ördög about this, she had this theory: “The way AI agents were trained is that they have seen completed functions and descriptions of those functions. The number of actual step-by-step TDD examples they have seen is a tiny part of the training data. That means that the LLM has an internal representation of code that is a direct translation of requirements to code, and not a process of how to get to that representation.”当我与 Ivett Ördög 谈论此事时，她提出了一个理论：“AI 智能体的训练方式是基于已完成的函数及其描述。它们见过的实际的、循序渐进的 TDD 示例在训练数据中占比极小。这意味着大语言模型对代码的内部表示是‘需求到代码的直接转换’，而不是‘如何达到该表示的过程’。”
Goals of TDD - still achieved in the agent loop?TDD 的目标——在智能体循环中还能实现吗？
The following are my general reflections about using TDD in the agent loop, not only based on this experiment. I'm going through the ultimate goals I personally have when I use TDD, skipping some of the ones that are about having tests in the first place, and unit tests in particular (like refactoring safety net, living documentation, test coverage), focussing on the ones that are specific to the TDD workflow.以下是我对在智能体循环中使用 TDD 的总体反思，不仅基于本次实验。我将梳理我在使用 TDD 时追求的终极目标，跳过那些关于“首先要有测试”以及单元测试本身（如重构安全网、活文档、测试覆盖率）的目标，重点关注那些 TDD 工作流特有的目标。
Test first >> Avoiding tautology测试先行 >> 避免同义反复
Test-first makes it easier to assert the output I want, rather than restating the implementation. Such a test can never fail when the implementation is wrong as it was derived from the same logic it's supposedly checking. When the assertions are decoupled from the specific implementation path, the test can actually catch when the behaviour is not what I intended.测试先行使得断言我想要的输出变得更容易，而不是重述实现逻辑。如果实现错误，这种测试永远无法失败，因为它源自它所检查的同一逻辑。当断言与特定的实现路径解耦时，测试才能真正捕捉到行为是否符合我的意图。
Still achieved in the agent loop?
In my experiment, some TDD sessions had this problem anyway, in spite of writing the test first. In one particularly obvious example, tests checked the implementation's output against itself, re-running the same code to produce the “expected” answer (see 4. on this list of observations). Writing the test first doesn't reliably prevent this - it might make it less probable, which is all we can ever hope for anyway with LLMs, but from this small data set I can't draw any conclusions about that probability.在智能体循环中还能实现吗？
在我的实验中，尽管采用了测试先行，但一些 TDD 会话仍然存在这个问题。在一个特别明显的例子中，测试将实现的输出与自身进行对比，通过重新运行相同的代码来产生“预期”答案（见观察列表中的第 4 点）。测试先行并不能可靠地防止这种情况——它或许降低了发生的概率，但这已经是我们在 LLM 上所能期望的最好结果了，但从这个小数据集来看，我无法得出关于该概率的任何结论。
Test first >> Testability测试先行 >> 可测试性
Test-first ensures the code is designed to be testable from the start, rather than retrofitting tests that are more complex and brittle than necessary.测试先行确保代码从一开始就是为了可测试而设计的，而不是事后补救那些比必要情况更复杂且脆弱的测试。
Still achieved in the agent loop?
The results didn't give me any clear cut signals either way. For what it's worth, the size and nature of the tasks I chose didn't require a lot of design complexity that could have surfaced this. To an extent though, testability is a corollary to driving design (see below).在智能体循环中还能实现吗？
结果没有给出任何明确的信号。值得一提的是，我选择的任务规模和性质并未涉及太多可能显现出这一点的设计复杂性。不过在某种程度上，可测试性是驱动设计（见下文）的必然结果。
Red-green >> Test effectiveness红-绿循环 >> 测试有效性
Observing a test fail first, then succeed (red-green), proves it will actually catch a regression.观察测试先失败后成功（红-绿），证明了它确实能捕捉到回归问题。
Still achieved in the agent loop?
How much sense does this really make when the human is removed? Watching a test go red is only proof of anything if someone is checking why it went red. When the agent both writes the test and confirms it failed, a red test tells you the agent ran it and saw failure, not that the failure was for the right reason. The evaluations of TDD adherence in my experiment also show this: agents still sometimes skipped or faked the red step, or implemented ahead of the test so that it passed immediately. Regression effectiveness can be monitored and improved with mutation testing (as I wrote about here). Mutation scores across the solutions didn't show any signals that TDD runs produced meaningfully better mutation scores than non-TDD runs. I don't really care how regression quality was achieved, as long as I have a mechanism to see how good it is.在智能体循环中还能实现吗？
当人类被移除时，这还有多大意义？观察测试变红只有在有人检查它为何变红时才有意义。当智能体既编写测试又确认其失败时，红灯测试只能告诉你智能体运行了它并看到了失败，而不是失败的原因是正确的。我实验中对 TDD 依从性的评估也显示了这一点：智能体有时仍然会跳过或伪造红灯步骤，或者在测试之前就进行实现，使其直接通过。回归有效性可以通过变异测试来监控和改进（正如我在此处所写的）。各方案的变异测试得分并没有显示出 TDD 运行比非 TDD 运行产生明显更好的回归质量。只要我有机制来评估其质量，我并不关心回归质量是如何实现的。
Test first, red-green-refactor >> Driving better design测试先行，红-绿-重构 >> 驱动更好的设计
Writing the test first forces us to specify usage before implementation, pushing toward better interfaces and more modular code. The refactoring step in the TDD loop further pushes us to improve the design step by step.先写测试迫使我们在实现之前指定用法，推动更好的接口和更模块化的代码。TDD 循环中的重构步骤进一步推动我们一步步改进设计。
Still achieved in the agent loop?
The experiment at least hasn't demonstrated superior design in the TDD runs at all. I now even wonder if TDD makes it worse, based on Opus's scoring, as the non-TDD solutions more often than not were ranked higher, and the design flaws it listed made sense to me. But the data set is of course too small to definitively conclude anything. (If anybody has time and tokens to run a larger experiment, that would be very interesting!)在智能体循环中还能实现吗？
实验至少完全没有证明 TDD 运行具有卓越的设计。根据 Opus 的评分，我甚至怀疑 TDD 是否让情况变得更糟，因为非 TDD 方案往往排名更高，且它列出的设计缺陷对我来说很有道理。当然，数据集太小，无法得出确切结论。（如果有人有时间和 Token 来运行更大规模的实验，那将会非常有趣！）
When humans write a test first, it forces us to think about usage before implementation, we have to sit with the friction of specifying behaviour and expectations before knowing how to build it. An agent doesn't experience that and can write a test the same instant it plans an implementation. Without a human checkpoint between the two, is there really any purpose left to writing the test first?当人类先写测试时，它迫使我们在实现之前思考用法，我们必须忍受在不知道如何构建的情况下指定行为和期望的摩擦。智能体没有这种体验，它可以在计划实现的同时编写测试。如果没有人类在两者之间进行检查，先写测试还有意义吗？
Small steps >> YAGNI小步快跑 >> YAGNI（你不会需要它）
Writing only enough code to pass the next test is about restraint. It's supposed to stop us from building abstractions or handling cases nobody has asked for yet.只写足够的代码来通过下一个测试是一种克制。它旨在阻止我们构建没人要求的抽象或处理没人关心的边缘情况。
Still achieved in the agent loop?
This is a very human-centered benefit that gets lost when an agent does TDD by itself. We don't get to sit in that friction anymore where we really have to think about all the intricacies of what we're building. That is theoretically shifting to when we are writing the specs to give to an agent, but we don't have a TDD-like mechanism there that lets us think the spec through in small steps.
Couldn't an agent work in those small steps though and ask us questions whenever it finds something that might be unnecessary? In my general experience, they're not very good at that. And in the experiment as well, minimal-implementation instructions didn't reliably stop them from building more. They frequently overshot and implemented more than the current test demanded, because they had the full requirement available. We usually don't spoon-feed the spec one by one, that would be very inefficient.在智能体循环中还能实现吗？
这是一个非常以人为本的好处，当智能体自行进行 TDD 时，这一点就丢失了。我们不再处于那种必须思考所构建内容所有细节的摩擦之中。理论上，这转移到了我们编写规范给智能体时，但我们那里没有类似 TDD 的机制让我们分小步思考规范。
不过，智能体难道不能分小步工作，并在发现可能不必要的内容时问我们吗？根据我的普遍经验，它们在这方面做得并不好。在实验中也是如此，最小实现指令并不能可靠地阻止它们构建更多内容。它们经常越界并实现超过当前测试需求的功能，因为它们拥有完整的需求。我们通常不会逐一喂食规范，那效率太低了。
Small steps >> Fast, localized feedback小步快跑 >> 快速、局部的反馈
Taking one small step at a time means that when a test fails, I know almost exactly what caused it, as the only thing that changed since the last green state is the one thing you just wrote.采取小步快跑意味着当测试失败时，我几乎确切地知道是什么导致的，因为自上次绿灯状态以来唯一改变的就是你刚刚写的那一点点代码。
Still achieved in the agent loop?
The setup didn't show if agents got stuck debugging more frequently with versus without TDD. But in my general experience, agents are usually reasonably good at figuring out why a test is red, even without having taken small, deliberate steps to get there. I'm still doubtful if the times when they do get stuck could be meaningfully mitigated with small TDD steps, and if the overall cost/benefit comparison would hold up.在智能体循环中还能实现吗？
设置并未显示智能体在有无 TDD 的情况下调试卡住的频率有何不同。但根据我的普遍经验，即使没有采取细致、审慎的步骤，智能体通常也相当擅长找出测试变红的原因。我仍然怀疑它们卡住时是否可以通过微小的 TDD 步骤有效缓解，以及整体的成本/收益比较是否成立。
Small steps >> Confidence and learning小步快跑 >> 自信与学习
In Kent Beck's preface to “Test-driven Development by example”, his biggest rationale for TDD is “managing fear”. He says that the legitimate fear of hard problems makes developers tentative, less communicative, and avoidant of feedback. With TDD, each passing test shows us progress, so we can relax knowing that progress is locked in. The tests are a psychological mechanism that helps us keep going.在 Kent Beck 的《测试驱动开发》序言中，他进行 TDD 的最大理由是“管理恐惧”。他说，对难题的合理恐惧使开发人员犹豫不决、沟通减少且回避反馈。通过 TDD，每个通过的测试都向我们展示了进展，因此我们可以放松，因为知道进展已锁定。测试是一种帮助我们坚持下去的心理机制。
Still achieved in the agent loop?
This is very much about managing a human's fear and giving a human permission to relax. That doesn't transfer when the agent is doing TDD inside of the loop, as it doesn't give me the same control and trust as when I do it myself, step by step.在智能体循环中还能实现吗？
这主要关乎管理人类的恐惧并给予人类放松的许可。当智能体在循环内进行 TDD 时，这并不能转化，因为它没有给我像自己一步步操作时那样的控制感和信任感。
Costs成本
At least 3x the tokens至少 3 倍的 Token
See detailed numbers in the appendix.详细数据见附录。
Naturally, as a TDD workflow requires many more turns and tool calls, more tokens will be used. However, many of those will be cache hits, so note that the 3x or more factor of tokens aren't a direct representation of how much more costly it is. (Unfortunately, I didn't track cache hits during the experiment.)自然地，由于 TDD 工作流需要更多的轮次和工具调用，会使用更多的 Token。然而，其中许多会是缓存命中，所以请注意，3 倍或更多的 Token 因子并不能直接代表成本增加了多少。（遗憾的是，我在实验中没有跟踪缓存命中。）
Prompt maintenance and testing提示词维护与测试
TDD is a process that doesn't seem to “come natural” to models. It's like an uphill battle against the training data, and takes a lot of iterations on a prompt to get it to follow the process most of the time. For example, when I realised after my first batches that the agent didn't do much refactoring in the red-green-refactor loop, I changed the prompt to put more emphasis on that step, as it's of course crucial to TDD. I later asked Opus to look at those sessions and see if it found an improvement in refactoring efforts. It did report an increase in refactoring steps - however, it also listed some cases in which the agent set out to refactor, but decided the design was good enough even in cases where Opus thought it clearly wasn't (e.g. when everything was implemented in one big module, but could have clearly been split up into multiple responsibilities).TDD 是一个对模型来说似乎“并不自然”的过程。这就像是在对抗训练数据，需要大量的提示词迭代才能让它在大多数时候遵循该过程。例如，在第一批次后我意识到智能体在红-绿-重构循环中没有进行太多重构，于是我修改了提示词，更加强调这一步骤，因为它对 TDD 至关重要。后来我让 Opus 查看这些会话，看是否发现重构努力有所改进。它确实报告了重构步骤的增加——然而，它也列出了一些情况：智能体开始重构，但认为设计已经足够好了，即使在 Opus 认为明显不足的情况下也是如此（例如，当所有内容都实现在一个大模块中，而本可以清楚地拆分为多个职责时）。
TDD is a comparatively complex set of instructions with lots of variables, and consequently lots of variations in how agents interpret it. So I imagine this type of prompt to be even more volatile across models than simpler instructions are, meaning it takes effort to keep the prompt working across models and model releases.TDD 是一组相对复杂的指令，变量很多，因此智能体对其解释的变化也很多。所以我认为这种类型的提示词在不同模型间的波动性甚至比简单指令更大，这意味着需要投入精力来保持提示词在不同模型和模型版本间的有效性。
My conclusions我的结论
I think at this point there is generally more and more evidence that being overly specific about how we want a model to do something is not a sustainable approach. Instead, we should find as many ways as we can to monitor the outcomes and give feedback. That feedback should be automated wherever possible, and we need to carefully think about where we insert ourselves as arbiters of what is good and correct.我认为目前有越来越多的证据表明，对我们希望模型如何做某事过于具体并不是一种可持续的方法。相反，我们应该尽可能多地寻找监控结果并提供反馈的方法。这种反馈应尽可能自动化，我们需要仔细考虑我们在哪里介入作为好坏与正确的仲裁者。
Even though I am aware that my little eval is far from representing a broad perspective on the effectiveness of TDD, it definitely hasn't given me any new indications that all this effort is worth it. Especially not if we can find other ways to achieve the majority of TDD benefits.尽管我知道我的小评估远未代表对 TDD 有效性的广泛视角，但它绝对没有给我任何新的迹象表明所有这些努力是值得的。特别是如果我们能找到其他方法来实现 TDD 的大部分好处的话。
I personally have stopped telling my coding agents to write tests first, let alone do TDD (which I never did, to be honest), until I see evals or other strong arguments that convince me otherwise. I'm trying to focus instead on the benefits of TDD when I use it outside of the agent loop, and exploring alternative ways to achieve them.我个人已经停止要求我的编码智能体先写测试，更不用说做 TDD 了（说实话，我从未这样做过），直到我看到评估或其他强有力的论据来说服我。我正试图专注于我在智能体循环之外使用 TDD 时的好处，并探索实现这些好处的替代方法。
How to get good regression tests?如何获得好的回归测试？
...so that the agent and me get signals when existing functionality breaks……这样当现有功能损坏时，智能体和我都能收到信号。
I still care about solid regression tests, because even though an agent can of course fix red tests the wrong way around, at least the red test gives it a feedback signal to double check pre-existing requirements that might have broken. I monitor and improve regression quality with the help of mutation testing, instead of giving elaborate TDD instructions and hoping for the best.我仍然关心可靠的回归测试，因为即使智能体当然可以用错误的方式修复红灯测试，至少红灯测试给了它一个反馈信号，去重新检查可能已经损坏的既有需求。我通过变异测试来监控和改进回归质量，而不是给出复杂的 TDD 指令并祈祷最好的结果。
How to build regular refactoring into the process?如何将定期重构构建到流程中？
...so that the codebase remains easy to change……这样代码库保持易于更改。
Refactoring remains crucial, but the small steps of traditional TDD don't seem to be an efficient or effective way to do it in the agent loop. A few examples of triggers for refactorings: Give the agent access to static code analysis; run regular reviews of structure and modularity; develop team rituals to maintain a good understanding of the codebase and catch drift early; keep an eye on the trend of number of files touched per change, and number of tokens are for a change.重构仍然至关重要，但传统 TDD 的小步快跑似乎不是在智能体循环中执行它的有效方式。重构触发器的一些例子：让智能体访问静态代码分析；定期审查结构和模块化；制定团队仪式以保持对代码库的良好理解并尽早发现漂移；关注每次更改所触及的文件数量趋势以及每次更改的 Token 数量。
How to get confidence?如何获得信心？
...so that I am not afraid to push to production……这样我就不怕推送到生产环境。
The hardest question remains, how do we get that confidence that TDD was giving us, how do we manage fear, how do we lock in progress? I don't have a clear answer to that, but I'll just mention one of the things that seems like a good building block for that: I have recently tried out the Approved Scenarios approach that Ivett Ördög is advocating for. In my words (don't hold her to it), it's a form of semi-manual testing that is supported by a bespoke test runner for each application. That runner shows me functional test scenarios in an easy to think about way, and allows me to “freeze” expectations (scenarios / fixtures) in that runner after I have thoroughly confirmed them. Whenever those frozen expectations are violated in the future, I have to approve them again. My colleague Matteo Vaccari gave a great overview of his experiences with that approach here.最难的问题仍然是：我们如何获得 TDD 给我们的那种信心，我们如何管理恐惧，我们如何锁定进展？我没有明确的答案，但我只想提一件似乎是很好的构建模块的事情：我最近尝试了 Ivett Ördög 提倡的“批准场景”（Approved Scenarios）方法。用我的话来说（别让她负责），这是一种半手动测试，由每个应用程序的定制测试运行器支持。该运行器以一种易于思考的方式向我展示功能测试场景，并允许我在彻底确认后“冻结”该运行器中的期望（场景/固定装置）。每当未来这些冻结的期望被违反时，我都必须再次批准它们。我的同事 Matteo Vaccari 在这里提供了他使用该方法的经验概述。
Whatever ends up giving us trust and confidence in our software in the future - I think the role of TDD as we've known it is significantly smaller than pre-GenAI.无论未来是什么赋予了我们对软件的信任和信心——我认为 TDD 作为我们所知的角色比 GenAI 之前要小得多。
Appendix: Results, according to Opus evaluation附录：结果，根据 Opus 评估
You can find the full results in this repository.你可以在此仓库中找到完整结果。
- NT = No TDD instructionsNT = 无 TDD 指令
- T = TDD instructionsT = TDD 指令
- TF = Test-first instructionsTF = 测试先行指令
Token usage across all batches所有批次的 Token 使用情况
Broken down per task size:按任务规模细分：
| Task | NT avg tokens | T avg tokens | T / NT factor |
|---|---|---|---|
| Small | 119,815 (n=2) | 1,018,245 (n=2) | 8.50x |
| Medium | 736,486 (n=2) | 2,181,105 (n=6) | 2.96x |
| Large | 253,621 (n=2) | 1,239,408 (n=2) | 4.89x |
Only the medium size were done with added test-first instructions只有中等规模任务使用了附加的测试先行指令
Caveat: these numbers are only a rough proxy for session cost, not a measure of how much code or thinking went into a solution. The setup that recorded the token usage used the pi-coding-agent SDK's getSessionStats(), which sums input + output + cacheRead + cacheWrite usage across every assistant turn in the session. That is a running total across the whole conversation, since every turn re-reads the accumulated context and each of those re-reads (usually mostly served from cache) is counted again in that turn's cacheRead. So “Total Tokens” tracks more how many turns a session took, weighted by how large the context had grown by then. It therefore weights cheap cache-read tokens the same as expensive fresh tokens, so it likely overstates TDD's true dollar cost. With these small sample sizes, treat the multipliers as directional: TDD reliably cost several times more, how many times exactly is variable.注意事项：这些数字只是会话成本的一个粗略代理，而不是衡量一个解决方案投入了多少代码或思考的指标。记录 Token 使用情况的设置使用了 pi-coding-agent SDK 的 getSessionStats()，它汇总了会话中每次助手轮次的输入 + 输出 + 缓存读取 + 缓存写入使用情况。这是整个对话的运行总计，因为每一轮都会重新读取累积的上下文，并且每次重新读取（通常大部分由缓存提供）都会在该轮的缓存读取中再次计算。因此，“总 Token”更多地追踪会话花费了多少轮次，并按上下文增长的程度加权。因此，它将廉价的缓存读取 Token 与昂贵的新鲜 Token 同等加权，所以它可能夸大了 TDD 的真实美元成本。对于这些小样本量，请将乘数视为方向性的：TDD 确实可靠地花费了数倍的成本，具体是多少是可变的。
Medium task, round 1中等任务，第 1 轮
Task: Build a 4-stage Python pipeline (parse → aggregate → format → validate) that transforms raw ROW_ID:CATEGORY:VALUE:PERIOD strings into a plain-text report.任务：构建一个 4 阶段 Python 流水线（解析 → 聚合 → 格式化 → 验证），将原始 ROW_ID:CATEGORY:VALUE:PERIOD 字符串转换为纯文本报告。
The numbers数据
| ID | TDD | Test Count | Coverage | Mutation Score | Total Tokens | Turns | Tool Calls |
|---|---|---|---|---|---|---|---|
| NT1 | No | 75 | 100% | 84.2% | 769,814 | 31 | 37 |
| NT2 | No | 107 | 100% | 89.6% | 703,159 | 21 | 24 |
| T1 | Yes | 30 | 100% | 81.0% | 1,519,762 | 71 | 28 |
| T2 | Yes | 34 | 99% | 77.3% | 2,580,897 | 103 | 60 |
The overall verdict总体结论
| ID | TDD | Rank | Verdict |
|---|---|---|---|
| NT1 | No | 1 | Module-per-stage, dataclasses, `Decimal`; no correctness bugs, strongest error handling, only solution checking duplicate ROW_IDs; validation self-referential but harmless |
| NT2 | No | 2 | Module-per-stage, dataclasses, float/round; best-engineered and largest suite, but validator rejects its own valid fractional output (false-rejection bug); TOTAL row never validated |
| T1 | Yes | 3 | Single module, dicts, float; correct core stages but validation is circular (re-runs formatter); accepts `nan`/`inf`, ignores duplicate ROW_IDs |
| T2 | Yes | 4 | Single module, dicts, float; active TOTAL-row bug (headcount summed into dollars) enshrined by a test; missing validation check #3 entirely |
(This round's ranking was based on Verdict, test count, coverage and mutation score only — Opus's Design/Code/Test sub-scores were introduced starting with the next round.)（本轮排名仅基于结论、测试计数、覆盖率和变异得分 —— Opus 的设计/代码/测试子得分从下一轮开始引入。）
Medium task, round 2中等任务，第 2 轮
Task: Same 4-stage report pipeline as 01-medium, rerun with stricter TDD adherence and a new test-first variant added (NT1/NT2 reuse the same codebases from 01-medium).任务：与 01-medium 相同的 4 阶段报告流水线，以更严格的 TDD 依从性重新运行，并添加了新的测试先行变体（NT1/NT2 重用了 01-medium 的代码库）。
| ID | Approach | Test Count | Coverage | Total Tokens | Turns | Tool Calls |
|---|---|---|---|---|---|---|
| NT1 | No TDD | 107 | 100% | 703,159 | 21 | 20 |
| TF2 | Test-first | 90 | 92% | 619,531 | 27 | 26 |
| NT2 | No TDD | 75 | 100% | 769,814 | 31 | 30 |
| T2 | TDD | 29 | 98% | 2,099,280 | 96 | 95 |
| TF1 | Test-first | 62 | 99% | 268,323 | 17 | 16 |
| T1 | TDD | 25 | 100% | 2,017,739 | 90 | 89 |
| ID | Approach | Design | Code | Tests | Avg | Impl/Test LOC |
|---|---|---|---|---|---|---|
| NT1 | No TDD | 8 | 8 | 8 | 8.0 | 497 / 881 |
| TF2 | Test-first | 8 | 8 | 7 | 8.0 | 484 / 850 |
| NT2 | No TDD | 8 | 8 | 7 | 7.5 | 330 / 430 |
| T2 | TDD | 7 | 7 | 6 | 6.5 | 207 / 304 |
| TF1 | Test-first | 6 | 6 | 6 | 6.0 | 348 / 360 |
| T1 | TDD | 6 | 6 | 6 | 6.0 | 142 / 228 |
| ID | Approach | Rank | Verdict |
|---|---|---|---|
| NT1 | No TDD | 1 | Deepest suite, cleanest validation reusing formatter's layout; HEADCOUNT mixed into dollar totals unguarded by tests |
| TF2 | Test-first | 2 | `Decimal` throughout, strong parse/validate; check 3 is unreachable dead code, validate module cluttered |
| NT2 | No TDD | 3 | Clean design, `Decimal`, correct parse; thinner tests, one no-op test, validation self-referential |
| T2 | TDD | 4 | Clean happy path, correct formatting; crashes on malformed input instead of returning structured parse errors |
| TF1 | Test-first | 5 | Strongest parser of the single-file solutions; broken TOTAL row (headcount as dollars), dead scaffolding shipped |
| T1 | TDD | 6 | Most compact (142 LOC); crashes on malformed input, most tautological validation, thinnest test suite |
Medium task, round 3 (improved TDD instructions)中等任务，第 3 轮（改进的 TDD 指令）
Task: Same 4-stage report pipeline as 01-medium, rerun with an improved TDD prompt (emphasising upfront design and refactoring); NT1/NT2 again reuse the 01-medium codebases.任务：与 01-medium 相同的 4 阶段报告流水线，使用改进的 TDD 提示词（强调预先设计和重构）重新运行；NT1/NT2 再次重用了 01-medium 的代码库。
| ID | TDD | Test Count | Coverage | Mutation Score | Total Tokens | Turns | Tool Calls |
|---|---|---|---|---|---|---|---|
| T1 | Yes | 51 | 100% | 90.2% | 3,447,283 | 117 | 116 |
| NT1 | No | 107 | 100% | 89.6% | 703,159 | 21 | 20 |
| NT2 | No | 75 | 100% | 84.2% | 769,814 | 31 | 30 |
| T2 | Yes | 43 | 100% | 81.1% | 1,421,671 | 61 | 60 |
| ID | TDD | Design | Code | Tests | Avg |
|---|---|---|---|---|---|
| --- | --- | ------ | ---- | ----- | ---- |
| T1 | Yes | 8 | 8 | 7 | 7.67 |
| NT1 | No | 8 | 8 | 6 | 7.33 |
| NT2 | No | 8 | 7 | 6 | 7.0 |
| T2 | Yes | 7 | 7 | 6 | 6.67 |
| Rank | ID | TDD | Headline weakness |
|---|---|---|---|
| 1 | T1 | Yes | HEADCOUNT-only TOTAL row printed as `$`; validation is a substring check, not arithmetic |
| 2 | NT1 | No | Fractional HEADCOUNT → spurious ValidationError on valid input; tests lean on monkeypatching |
| 3 | NT2 | No | `NaN`/`Infinity` crash the pipeline instead of a ParseError; validation re-runs the formatter |
| 4 | T2 | Yes | Validation is tautological (checks aggregate against itself); width check is only a comment |
Small task小型任务
Task: Build a Python module that validates medical appointment slot codes in DAY-TIME-ROOM-CHECKSUM format, returning a structured result identifying which rule failed and why.任务：构建一个 Python 模块，验证 DAY-TIME-ROOM-CHECKSUM 格式的医疗预约槽位代码，返回结构化结果，识别哪个规则失败以及原因。
| ID | TDD | Test Count | Coverage | Mutation Score | Total Tokens | Turns | Tool Calls |
|---|---|---|---|---|---|---|---|
| NT1 | No | 61 | 100% | 89.6% | 122,108 | 10 | 15 |
| NT2 | No | 58 | 100% | 92.3% | 117,522 | 10 | 20 |
| T1 | Yes | 21 | 100% | 93.6% | 894,451 | 55 | 37 |
| T2 | Yes | 20 | 100% | 93.2% | 1,142,039 | 68 | 26 |
| ID | TDD | Design | Code | Tests | Avg |
|---|---|---|---|---|---|
| NT1 | No | 8 | 9 | 9 | 8.67 |
| NT2 | No | 8 | 8 | 8 | 8.0 |
| T1 | Yes | 7 | 8 | 7 | 7.33 |
| T2 | Yes | 6 | 7 | 7 | 6.67 |
| ID | TDD | Rank | Verdict |
|---|---|---|---|
| NT1 | No | 1 | Best overall — dataclass result, no bugs, 61 reason-asserting tests |
| NT2 | No | 2 | Very close — dataclass result, but a Unicode-digit spec deviation |
| T1 | Yes | 3 | Correct & clean, but dict result + fewer tests + dead code |
| T2 | Yes | 4 | Weakest design (free-text error) + a genuine crash bug |
Larger task大型任务
Task: Build an in-memory Python loyalty points engine with tiered earn rates (Bronze/Silver/Gold), trailing-365-day spend tracking for tier recalculation, and point redemption.任务：构建一个内存中的 Python 忠诚度积分引擎，具有分层赚取率（青铜/白银/黄金）、用于层级重新计算的过去 365 天消费跟踪，以及积分兑换功能。
| ID | TDD | Test Count | Coverage | Mutation Score | Total Tokens | Turns | Tool Calls |
|---|---|---|---|---|---|---|---|
| NT2 | No | 69 | 100% | 86.9% | 322,148 | 14 | 13 |
| T2 | Yes | 22 | 99% | 85.6% | 1,225,517 | 63 | 62 |
| T1 | Yes | 21 | 99% | 85.2% | 1,253,300 | 67 | 66 |
| NT1 | No | 74 | 99% | 89.4% | 185,094 | 11 | 9 |
| ID | TDD | Design | Code | Tests | Correctness | Avg |
|---|---|---|---|---|---|---|
| NT2 | No | 8 | 9 | 8 | 8 | 8.25 |
| T2 | Yes | 7 | 7 | 8 | 8 | 7.5 |
| T1 | Yes | 7 | 7 | 7 | 9 | 7.5 |
| NT1 | No | 8 | 7 | 6 | 6 | 6.75 |
| ID | TDD | Rank | Verdict |
|---|---|---|---|
| NT2 | No | 1 | Only solution with real input validation; precise boundary tests; minor out-of-order purchase edge cases only |
| T2 | Yes | 2 | Clean typed data model, all core rules correct; no error handling, duplicate purchase ID bug, dead state fields |
| T1 | Yes | 3 | Most functionally correct (no bugs found on probing); untyped nested dicts, vestigial structure, fewest tests |
| NT1 | No | 4 | Highest design score, 74 tests — but two High bugs: wrong batch draw-down order; future-dated points counted as spendable |
Acknowledgements致谢
Thanks to Ivett Ördög, Matteo Vaccari, Dan Mutton, Lukasz Plotnicki, and Emily Bache, for taking the time to review, and for the feedback and valuable discussions that helped improve this post.感谢 Ivett Ördög、Matteo Vaccari、Dan Mutton、Lukasz Plotnicki 和 Emily Bache 花时间进行审查，以及感谢有助于改进本文的反馈和宝贵讨论。
GenAI was used for research, pulling together ideas into structure, and polishing the language.GenAI 用于研究、将想法整理成结构以及润色语言。
latest article (Aug 10):最新文章（8 月 10 日）：
previous article:上一篇文章：