已置顶已置顶
Continual Learning for Agents智能体的持续学习
Everyone talks about Continual Learning as if it means one thing only: updating model weights. But there's an inconvenient truth about the agent ecosystem — the vast majority of agents in production today leverage closed frontier models. When you don't own the weights, you certainly can't fine-tune them. For most agent builders, weight-level continual learning is off the table, especially when working at the very frontier of capabilities (think Fable 5 or GPT 5.6).每个人都把持续学习理解为同一件事：更新模型权重。但关于智能体生态系统，有一个不容忽视的事实——当今生产环境中绝大多数智能体都依赖封闭前沿模型。当你无法拥有权重时，自然也就无法微调它们。对大多数智能体构建者而言，权重级别的持续学习是不可能的，尤其是在处理最前沿的能力时（比如Fable 5或GPT 5.6）。
That doesn't mean agents can't learn. Agentic systems can improve at three layers — model, harness, and context [0] — and the last two are fully within your control. This is where the a massive (but often overlooked) opportunity lies: harness-level learning lets you mine production traces to systematically improve the code, tools, and instructions that power every instance of your agent, while context-level learning lets you personalize at the agent, user, and org level, so your product gets better with every interaction. Do all the above, and you will be compounding improvements that you can ship daily. 但这并不意味着智能体无法学习。智能体系统可以在三个层面得到改进——模型、框架和上下文[0]——而后两者完全在你的掌控之中。这里蕴藏着一个巨大（但常被忽视）的机遇：框架层面的学习让你能够挖掘生产轨迹，系统性地改进驱动每个智能体实例的代码、工具和指令；而上下文层面的学习则让你可以在智能体、用户和组织层面进行个性化，从而使你的产品在每次交互中不断优化。做到以上所有，你将获得复合改进，并能每日交付。
In the rest of this article, I'll walk through how we've been applying continual learning to Replit Agent for the past year, and share all the lessons we learned along the way.在本文剩余部分，我将介绍我们过去一年如何将持续学习应用于Replit Agent，并分享一路走来学到的所有经验。
Evaluating and improving Replit Agent at scale大规模评估和改进Replit Agent
Most Replit Agent users start with an idea. They describe the goal in natural language — without a repo, test suite, or chosen framework — and expect the agent to turn it into a functioning app. The result might be a website, slide deck, mobile app, several connected artifacts, or something else entirely.大多数Replit Agent用户从一个想法开始。他们用自然语言描述目标——没有仓库、测试套件或选定框架——并期望智能体将其变成可运行的应用程序。结果可能是一个网站、幻灯片、移动应用、几个相互关联的产物，或者别的什么。
Vibe coders are not usually checking diffs or test output. Success for Replit Agent is deceptively simple: the app should work when users click around.Vibe编码者通常不会检查差异或测试输出。Replit Agent的成功定义简单得令人难以置信：用户点击时，应用应该能正常工作。
That changes the job of evaluation. A single score can help with a specific shipping decision, but it cannot tell us, week over week, whether Replit Agent is getting better for users. To answer that question, evaluation must become part of the improvement loop.这改变了评估的任务。一个分数可以帮助做出特定的发布决策，但无法告诉我们周复一周Replit Agent是否对用户变得更好。要回答这个问题，评估必须成为改进循环的一部分。
NASA Lewis Research Center, Central Control Room in the Engine Research Building, 1968 — Measurement matters when it changes what ships.NASA Lewis研究中心，发动机研究大楼中央控制室，1968年——当测量影响发布内容时，测量才有意义。
Evaluation has to do more now评估现在必须做得更多
Agent evaluation used to look like a one-way process: run the eval, produce a score, and make a shipping call. This works when releases are slow and the thing being measured rarely changes. It breaks down when models, prompts, tools, and product surfaces are all changing quickly.智能体评估过去看起来像是一个单向过程：运行评估，产生分数，做出发布决定。当发布速度慢且被评估对象很少变化时，这很有效。但当模型、提示、工具和产品表面都在快速变化时，这种方法就失效了。
The old loop made evaluation feel bounded. But Replit Agent changes too quickly for a single score to carry the whole decision. A score can compare two candidates on one slice of tasks. It cannot explain what users care about, where production is breaking, or what to improve next.旧循环让评估显得有边界。但Replit Agent变化太快，单一分数无法承担整个决策。一个分数可以比较两个候选者在某一任务切片上的表现，却无法解释用户关心什么、生产环境哪里出问题、下一步该改进什么。
Evaluation had to move from launch check to improvement loop.评估必须从发布检查转变为改进循环。
The old evaluation job ends at a human shipping decision; the new one feeds a continuous system that learns from production and ships improved agents.旧评估工作以人类的发布决策结束；新评估工作则为一个从生产中学习并发布改进后智能体的连续系统提供输入。
The system has two measurement pillars and one optimization loop. Offline benchmarks tell us whether candidate changes can complete simulated app-building tasks before we ship them. Online A/B tests and production traces show how real users are affected after the changes ship. Those signals then flow back into evals and shipping decisions.该系统有两个测量支柱和一个优化循环。离线基准测试告诉我们候选变更在发布前能否完成模拟应用构建任务。在线A/B测试和生产轨迹显示变更发布后真实用户受到的影响。这些信号随后反馈到评估和发布决策中。
No layer is enough on its own. Benchmarks catch regressions before release. A/B tests show whether production behavior moved. Trace clusters explain the failures under aggregate metrics. Human judgment keeps the improvement loop pointed at the right product and engineering outcomes. The shape is analogous to the Swiss cheese model in safety engineering: each layer has holes, but together they catch more than any one layer can.没有一个层面是单独足够的。基准测试在发布前捕获回归。A/B测试显示生产行为是否发生变化。轨迹聚类解释聚合指标下的失败原因。人工判断确保改进循环指向正确的产品和工程成果。其形态类似于安全工程中的瑞士奶酪模型：每个层都有漏洞，但共同作用能捕获比任何单层更多的漏洞。
Existing benchmarks stop short of the user现有基准测试止步于用户
Agentic coding benchmarks such as SWE-bench [1] and Terminal-Bench [2] grade code in constrained, repeatable environments. These benchmarks are valuable and widely adopted, but they miss the signal a vibe coder cares about.诸如SWE-bench [1]和Terminal-Bench [2]等智能体编码基准测试在受限、可重复的环境中评估代码。这些基准测试很有价值且被广泛采用，但它们错过了vibe编码者关心的信号。
Replit Agent often creates the codebase from scratch. Users do not bring fixed routes, function signatures, selectors, or tests; they bring a product request. The agent chooses the stack, schema, routes, components, and interaction flows.Replit Agent经常从头创建代码库。用户不会带来固定的路由、函数签名、选择器或测试；他们带来的是产品需求。智能体选择技术栈、模式、路由、组件和交互流程。
That creates a functional correctness gap. An agent can satisfy the local constraints of a coding benchmark and still fail at what the user sees: whether the finished app does what was asked. For vibe coding, the evaluation target is the artifact itself: does it load, does the core workflow work, and does the result match the request?这就产生了一个功能正确性差距。智能体可以满足编码基准测试的局部约束，但仍然在用户所见方面失败：完成的应用是否按要求运行。对于vibe编码，评估目标是产物本身：它能否加载，核心工作流是否正常，结果是否符合请求。
Introducing ViBenchViBench介绍
The need for this style of end-to-end evaluation is precisely why we built 正是对这种端到端评估风格的需求，促使我们构建了 [3], our public benchmark for vibe coding, measuring a simple but important signal: does the application built by the agent meet the spec?[3]，我们为vibe编码设计的公开基准测试，衡量一个简单但重要的信号：智能体构建的应用是否满足规格？
ViBench starts with a plain-English product requirements document (PRD) drawn from anonymized Replit production traces. From there, the agent receives the PRD and builds a running app from scratch, without being constrained to the scaffolding, routes, or references that traditional coding benchmarks require.ViBench从一份从Replit生产轨迹中匿名提取的纯英文产品需求文档（PRD）开始。然后，智能体接收PRD并从头构建一个可运行的应用，而不受传统编码基准测试所需的脚手架、路由或参考的限制。
However, the same flexibility that makes ViBench realistic demands an equally flexible eval agent, one that stays grounded in the PRD. In SWE-bench-style benchmarks, the project already exists, so the evaluation surface is fixed. In vibe coding, the agent picks the stack, routes, components, and flow. Evaluation has to explore whatever it invented.然而，使ViBench逼真的灵活性同样需要一个灵活的评估智能体，该智能体必须始终基于PRD。在SWE-bench风格的基准测试中，项目已经存在，因此评估面是固定的。在vibe编码中，智能体选择技术栈、路由、组件和流程。评估必须探索它所发明的一切。
To that end, each ViBench task pairs the PRD with a set of 为此，每个ViBench任务将PRD与一组natural-language自然语言 test plans that describe the feature-level interactions and assertions the finished app must satisfy. The eval agent uses Playwright as a flexible backbone, which lets it exercise complex features such as offline simulation, file manipulation, and multi-tenancy. Because it doesn't know the app's locators or structure 测试计划配对，这些测试计划描述了完成的应用必须满足的功能级交互和断言。评估智能体使用Playwright作为灵活的骨干，这使其能够执行复杂功能，如离线模拟、文件操作和多租户。由于它不知道应用的选择器或结构a priori先验地, it works in a notebook environment, progressively discovering how the app is built and interacting with it step by step, an approach drawn from Replit's earlier research on automated self-testing [4].，它在类似笔记本的环境中工作，逐步发现应用是如何构建的并与之交互，这种方法源自Replit早期关于自动化自测试的研究[4]。
Running ViBench, and our evals in general, at Replit scale also demands strong infrastructure support [5]. Internally, we lean on the same production infrastructure that lets us spin up isolated, well-resourced sandboxes for building apps and running our agents. Because we can quickly fork those sandboxes [6], we run much of the evaluation in parallel, without risking cross-evaluation contamination.在Replit规模上运行ViBench及我们的通用评估也需要强大的基础设施支持[5]。在内部，我们依赖相同的生产基础设施，这些基础设施使我们能够快速为构建应用和运行智能体创建隔离、资源充足的沙箱。由于我们可以快速分叉这些沙箱[6]，我们并行运行大部分评估，避免了交叉评估污染。
Beyond building apps from scratch, the same ViBench foundation, a natural-language PRD graded by natural-language test plans, adapts to a range of vibe-coding scenarios. To evaluate how an agent works 除了从头构建应用，相同的ViBench基础——由自然语言测试计划评分的自然语言PRD——也适用于多种vibe编码场景。为了评估智能体如何within在 an existing app, closer to Replit's mid-trajectory workloads, we start it on an existing codebase and measure how well it ships feature extensions from a feature PRD. That codebase can come from our own reference implementations or from apps the agent vibe-coded itself, which we call Vibe-to-ref and Vibe-on-Vibe in our publication. When we ship new product surfaces, the same backbone lets us quickly derive new problems to evaluate novel interaction patterns, as we did for Agent 4's parallel-and-merge and subagent decompositions.现有应用中工作（更接近Replit的中途轨迹工作负载），我们将其启动在现有代码库上，并衡量其从功能PRD交付功能扩展的能力。该代码库可以来自我们自己的参考实现，也可以来自智能体自身vibe编码的应用，我们在论文中称之为Vibe-to-ref和Vibe-on-Vibe。当我们发布新的产品表面时，同样的骨干让我们能够快速衍生新问题来评估新颖的交互模式，正如我们对Agent 4的并行合并和子智能体分解所做的那样。
ViBench keeps the behavioral evaluator fixed while varying the input and construction strategy.ViBench保持行为评估器固定，同时改变输入和构建策略。
Early ViBench results gave us two useful lessons. First, frontier coding-benchmark scores do not always transfer to full app building, especially for open-weight models. Second, most models get worse when extending their own code, as errors often compound. Together, those lessons give us a better hill to climb: not just writing code that passes tests, but building apps that can survive the next user request.早期的ViBench结果给了我们两个有用的教训。首先，前沿编码基准测试的分数并不总能转化为完整的应用构建，尤其是对于开源权重模型。其次，大多数模型在扩展自己的代码时表现更差，因为错误往往会累加。这些教训共同为我们指明了更好的改进方向：不仅是编写能通过测试的代码，更是构建能经受住下一个用户请求的应用。
A/B is how we keep ourselves honestA/B测试是我们保持诚实的方式
We trust offline evals deeply, but they are not the only judge. We have seen enough agent updates look good in controlled settings, only to regress real user behavior, to know that production needs its own measurement layer.我们非常信任离线评估，但它们不是唯一的评判标准。我们见过太多智能体更新在受控环境中表现良好，却在真实用户行为上出现回归，因此知道生产环境需要自己的测量层面。
Users are unscripted, always on, and operating at a scale no offline benchmark can fully reproduce. They abandon projects, change their minds, combine features in surprising ways, and discover failure modes we did not know to test.用户是无剧本的、始终在线的，并且以离线基准测试无法完全复现的规模运行。他们放弃项目、改变主意、以令人惊讶的方式组合功能，并发现我们未曾想到要测试的失败模式。
So we A/B most agent-affecting updates: prompts, tools, harness revisions, model swaps, and larger behavior changes. Multiple experiments often run concurrently — with attribution kept clear to avoid hiding interaction effects. A/B tests surface user behavior, sentiment, and success: did users keep going, did cost behave unexpectedly, did sentiment move, and did users ship something?因此，我们对大多数影响智能体的更新进行A/B测试：提示、工具、框架修订、模型更换以及更大的行为变化。多个实验通常同时运行——属性保持清晰以避免隐藏交互效应。A/B测试揭示用户行为、情感和成功：用户是否继续使用，成本是否异常，情感是否变化，用户是否发布了某些内容。
A/B tests give us a controlled read on production behavior, but the aggregate metrics do not explain themselves. Is this a win? What’s the root behavior change?A/B测试让我们对生产行为有了受控的解读，但聚合指标本身无法解释。这是一个胜利吗？根本行为变化是什么？
A challenge with A/B testing is that results are hard to interpret. If run duration goes up, did the agent do more useful work, or did it get stuck? If cost goes down, did we improve efficiency, or did the agent silently stop doing something valuable? If sentiment drops, which use cases regressed, which failure modes are new, and which users gave up?A/B测试的一个挑战是结果难以解释。如果运行时长增加，是智能体做了更多有用工作，还是它卡住了？如果成本下降，是我们提高了效率，还是智能体默默地不再做某些有价值的事情？如果情感下降，哪些用例出现了回归，哪些失败模式是新的，以及哪些用户放弃了？
Telescope: what is breakingTelescope：什么出了问题
A/B testing tells us when production behavior moved. Telescope — our system for trace analysis and clustering — helps explain why.A/B测试告诉我们生产行为何时发生了变化。Telescope——我们的轨迹分析和聚类系统——则有助于解释原因。
At production scale, no engineer can read every trace. Telescope organizes repeated patterns into issue clusters that engineers and agents can act on. It summarizes failure trajectories, embeds them, clusters similar cases, and classifies new sessions as the distribution changes. The goal is not just to count failures, but to discover the ones hidden in plain sight.在生产规模下，没有工程师能阅读每条轨迹。Telescope将重复出现的模式组织成问题集群，工程师和智能体可以据此采取行动。它总结失败轨迹，嵌入它们，对相似案例进行聚类，并在分布变化时对新会话进行分类。目标不仅仅是统计失败，而是发现那些隐藏在眼皮底下的问题。
GIF
Clustering what we didn’t know to look for.聚类我们不知道要寻找的问题。
Telescope uses short, evidence-grounded facets inspired by the same bottom-up approach as Clio [7]. For traces, it reconstructs the session from user messages, visible agent replies, tool calls, errors, metadata, and other context. From there, Telescope summarizes what went wrong, embeds those summaries, and uses density-based clustering [8] to form emergent issue groups.Telescope使用简短、基于证据的面（facets），受Clio [7]中相同的自下而上方法启发。对于轨迹，它从用户消息、可见的智能体回复、工具调用、错误、元数据和其他上下文中重建会话。然后，Telescope总结出错之处，嵌入这些总结，并使用基于密度的聚类[8]形成新兴问题组。
Facets make investigation faster, especially when clustering alone is not enough. When support reports point to a broad issue, such as port failures, engineers and agents can search the compact layer first, explore the relevant facets, and then drill into representative sessions with the logs and observability context needed to explain it.Facets使调查更快，尤其是在聚类本身不够时。当支持报告指向一个广泛问题（如端口故障）时，工程师和智能体可以先搜索紧凑层面，探索相关facets，然后深入代表性会话，并利用日志和可观测性上下文来解释问题。
In aggregate, the same structure turns scattered failures into product questions: which workflows dominate, which get abandoned, what breaks repeatedly, and whether a mitigation is shrinking the intended cluster.在聚合层面，相同的结构将分散的失败转化为产品问题：哪些工作流占主导地位，哪些被放弃，哪些反复出现故障，以及缓解措施是否正在缩小目标集群。
For more on this underlying architecture, see the in-depth post on 有关此底层架构的更多信息，请参阅我们来自Braintrust的合作者关于 from our collaborators at Braintrust [9].的深入文章 [9]。
The loop: from evidence to agent improvements循环：从证据到智能体改进
Once measurement exists, the bottleneck moves. ViBench, A/B tests, and Telescope can tell us what failed, where it failed, and how often it is happening. We still have to turn that evidence into plausible fixes.一旦测量到位，瓶颈就转移了。ViBench、A/B测试和Telescope可以告诉我们什么失败了、在哪里失败以及发生频率。我们仍然需要将这些证据转化为可行的修复方案。
We turn to a self-improvement loop to address this. The operating principle is simple: if agents are useful for building software, they should also be useful for improving the agent. Each pass starts by reading production logs, trace clusters, and recent failures to find a hypothesis worth chasing. Then it builds a candidate, opens a draft PR with the reasoning attached, measures the result against ViBench, A/B results, trajectory data, and recent baselines, and recommends whether to ship, iterate, or drop it.我们通过一个自我改进循环来解决这个问题。操作原理很简单：如果智能体对构建软件有用，那么它们也应该对改进智能体本身有用。每一轮从读取生产日志、轨迹集群和近期失败开始，寻找值得追寻的假设。然后构建候选方案，附带推理过程打开一个草稿PR，根据ViBench、A/B结果、轨迹数据和近期基线衡量结果，并建议是发布、迭代还是放弃。
The optimization loop discovers problems, proposes agent changes, evaluates them, and decides whether to ship, iterate, or drop.优化循环发现问题，提出智能体变更，评估它们，并决定是发布、迭代还是放弃。
Shipping does not become automatic. The loop can prepare the evidence and first-pass implementation; engineers still review the result and own the launch decision.发布不会自动进行。该循环可以准备证据和初步实现；工程师仍然审查结果并拥有发布决策权。
Each run records what it tried and what happened, including failures. That record improves the loop over time: future runs can reuse what worked, avoid known dead ends, and propose changes that generalize.每次运行记录它尝试了什么以及发生了什么，包括失败。该记录会随着时间的推移改进循环：未来的运行可以重用有效的方法，避免已知的死胡同，并提出具有泛化性的变更。
Agent iteration gets faster without giving up engineering control. Given a new model, product surface, or reliability goal, the loop can proactively find prompt edits, skill proposals, tool fixes, and harness changes while engineers keep the system pointed toward the larger product optimum.智能体迭代变得更快，同时不放弃工程控制。给定一个新的模型、产品表面或可靠性目标，该循环可以主动发现提示编辑、技能提议、工具修复和框架变更，而工程师则让系统保持指向更大的产品最优解。
A concrete example一个具体例子
One recent run started with a small but growing Telescope cluster. Environment setup was silently degrading across a long tail of cold-start scenarios. These sessions were not obvious from aggregate metrics, but the cluster showed a pattern worth investigating.最近一次运行从一个虽小但不断增长的Telescope集群开始。环境设置在一系列冷启动场景中悄无声息地降级。这些会话在聚合指标中并不明显，但该集群显示出一个值得调查的模式。
After surfacing the pattern, the loop read the affected trajectories, proposed a patch, added a regression test, and ran the candidate against ViBench to confirm that the happy path did not regress. Engineers reviewed the evidence, approved the change, and pushed it to production the same day.在发现该模式后，循环读取受影响的轨迹，提出一个补丁，添加一个回归测试，并在ViBench上运行候选方案以确认正常路径没有回归。工程师审查了证据，批准了变更，并在同一天将其推送到生产环境。
After the patch shipped, sentiment recovered and affected users were unblocked. This is the shape we want — a loop that finds a real failure pattern, connects it to affected users, proposes the right level of fix, and brings back enough evidence for a human to decide whether to ship.补丁发布后，情感恢复，受影响的用户被解封。这正是我们想要的形态——一个循环，能够发现真实的失败模式，将其与受影响的用户联系起来，提出适当程度的修复，并带回足够的证据供人类决定是否发布。
Where human taste still matters most人类判断仍然最重要的地方
Much of this can run autonomously: clustering failures, proposing hypotheses, building candidates, running evals, and assembling evidence. Humans still set the direction and gate most exits, including:这个过程的大部分可以自主运行：聚类失败、提出假设、构建候选方案、运行评估和汇编证据。人类仍然设定方向并把关大多数出口，包括：
- Hypothesis selection. A system can surface a thousand failures, but humans decide which questions deserve the loop's overnight budget. Not every cluster is equally important, and not every regression points to the right product problem.假设选择。一个系统可以浮现上千个失败，但人类决定哪些问题值得循环花费整夜的计算预算。并非每个集群都同等重要，也并非每次回归都指向正确的产品问题。
- Implementation architecture. Traces might show that users are abandoning a workflow, but deciding whether to smooth that path, change the agent's behavior, or redesign the surface is an engineering and product judgment.实现架构。轨迹可能显示用户正在放弃一个工作流，但决定是平滑该路径、改变智能体行为还是重新设计表面，则是一个工程和产品判断。
- Eval curation. This is not administrative work; it shapes the hill the agent climbs. If the eval rewards the wrong behavior, the optimization loop will faithfully optimize toward the wrong thing.评估策展。这不是行政工作；它塑造了智能体攀登的山丘。如果评估奖励了错误的行为，优化循环会忠实地朝着错误的方向优化。
- Launch approval. Shipping an agent change is not just reading a number. Launch approval means reading the evidence, understanding the blast radius, deciding whether the risk is acceptable, and owning the rollout.发布批准。发布智能体变更不仅仅是读取一个数字。发布批准意味着阅读证据，理解影响范围，决定风险是否可接受，并拥有发布权。
That balance matters: the loop can do more of the search, measurement, and synthesis. Engineers still choose the direction, make the product calls, and decide what ships.这种平衡至关重要：循环可以执行更多的搜索、测量和综合工作。工程师仍然选择方向，做出产品决策，并决定发布什么。
Closing the loop闭环
Evaluation is no longer just a gate before launch. It helps decide what to fix, what to test, and what to release.评估不再只是发布前的一道关卡。它有助于决定修复什么、测试什么以及发布什么。
The work is not to produce a better number. It is to turn user failures into better releases, so more ideas become apps people are proud to publish.这项工作不是为了产生一个更好的数字。而是为了将用户失败转化为更好的发布，从而让更多的想法变成用户自豪地发布的应用。
We're excited to keep pushing the frontier of autonomous agents, with a focus on reliability for the most complex coding tasks. If you are interested in working on autonomous coding agents, I'm always hiring on the Replit AI team — reach out to 我们很高兴能继续推动自主智能体的前沿发展，重点关注最复杂编码任务的可靠性。如果你对从事自主编码智能体工作感兴趣，我一直在Replit AI团队招聘——请通过
Authors作者: , , ,
References参考文献
想发布自己的文章？想发布自己的文章？
升级为 Premium升级为 Premium