Using LLM-as-a-judge scoring to measure your software factory利用“大模型即评判者”（LLM-as-a-judge）评分来衡量你的软件工厂

If you want to get the most out of coding agents in your organization, you need to stop guessing how well your agents are performing, and start measuring.如果你想在组织内充分利用编程智能体，就必须停止猜测它们的表现，转而进行量化衡量。

There are a couple of ways to do this. The typical approach is DORA metrics: PR merge rate, cycle time, defect rate, time to fix errors, etc. If those metrics are all going in the right direction as you increase agent usage, it's a good signal you are getting value from coding agents.实现这一目标有几种方法。典型的方法是使用 DORA 指标：PR 合并率、周期时间、缺陷率、错误修复时间等。如果你在增加智能体使用量的同时，这些指标都在向好的方向发展，那么这说明你正从编程智能体中获得价值。

There’s a second approach though, that you should also consider: directly scoring coding agents using LLM-as-a-judge. Since agents provide a complete digital record of their work, you can examine and grade past sessions, see where they are deficient, and adjust going forward.不过，还有第二种方法值得考虑：直接使用“大模型即评判者”来为编程智能体评分。由于智能体的工作提供了完整的数字记录，你可以检查并评估过去的会话，找出不足之处，并据此进行调整。

Scoring forms the basis of agentic self-improvement, where observer agents automatically suggest changes to improve agent ROI based on past scores of how the factory is performing.评分构成了智能体自我改进的基础，观察者智能体可以根据工厂表现的历史评分，自动建议改进措施，从而提升智能体的投资回报率（ROI）。

There are a few prerequisites for setting up an effective scoring system. I’ll illustrate the primitives using the built-in scoring infrastructure in Warp Factories, but you can also create something similar on your own.建立有效的评分系统有几个前提条件。我将使用 Warp Factories 内置的评分基础设施来演示这些基本要素，但你也可以自行创建类似的系统。

Here is the tl;dr:以下是摘要：

1. Build a record of prior agent traces that your scorers can grade.建立一份可供评分员评估的过往智能体轨迹记录。
2. Define "scoring agents" using the criteria your team wants to track and improve (efficiency, code quality, verbosity, etc.)使用团队想要跟踪和改进的标准（效率、代码质量、详细程度等）来定义“评分智能体”。
3. Decide on a sampling strategy.确定抽样策略。
4. Automate scoring by scheduling scoring agents to grade past agent sessions.通过安排评分智能体对过去的智能体会话进行评估，实现评分自动化。
5. Add an “observer” loop of self-improvement agents that examine scores and suggest changes to improve them.增加一个由自我改进智能体组成的“观察者”循环，它们会检查评分并提出改进建议。
6. Use scorers as the basis of benchmarking to compare different model configurations in your factory.将评分员作为基准测试的基础，以比较工厂中不同的模型配置。

Let’s take a closer look:让我们深入了解一下：

First, you need a record of prior agent traces that your scorers can grade. These traces should include not just the agent conversation, but the agent’s entire “input and output;” and they should be stored in the cloud and be accessible via API so agents can analyze them.首先，你需要一份可供评分员评估的过往智能体轨迹记录。这些轨迹不仅应包含智能体的对话，还应包含智能体完整的“输入和输出”；它们应当存储在云端并通过 API 访问，以便智能体进行分析。

“Inputs” are prompts, tool calls and MCP results, input images, etc. “Outputs” should include all artifacts created by the agent like PRs, specs, screenshots, etc; anything that would be helpful in judging whether the agent did its job. In Warp Factories we automatically store all this info and make it API accessible (potentially in a company’s own storage). Depending on your factory approach, you may have to do some infrastructure work to set this up.“输入”是指提示词、工具调用和 MCP 结果、输入图像等。“输出”应包括智能体创建的所有工件，如 PR、规范、截图等；任何有助于判断智能体是否完成工作的内容。在 Warp Factories 中，我们会自动存储所有这些信息并使其可通过 API 访问（可能存储在公司自己的存储空间中）。根据你的工厂方案，你可能需要进行一些基础设施建设来完成此设置。

Second, you need a way of defining and triggering “scoring agents.” A scoring agent takes a prior agent trace as an input and returns a grade. Each scorer typically focuses on a single dimension like cost or quality, and is defined by a prompt, classification instructions, and a judge model to use. You’ll also need a place to store and view the aggregate scores. Again, this is built into our factory infra; if you are building your own you’ll want to use some sort of cron-based cloud agent to score prior runs.其次，你需要一种定义和触发“评分智能体”的方法。评分智能体接收过往的智能体轨迹作为输入，并返回一个评分。每个评分员通常专注于一个维度（如成本或质量），并由提示词、分类说明和所使用的评判模型来定义。你还需要一个地方来存储和查看汇总后的评分。同样，这已内置于我们的工厂基础设施中；如果你是自行构建，则需要使用某种基于定时任务的云端智能体来对过往运行进行评分。

You can define scoring agents along different dimensions:你可以从不同维度定义评分智能体：

• Task compliance: did the agent complete the task per the user’s request?任务合规性：智能体是否按照用户的要求完成了任务？
• Efficiency: did the agent complete the task efficiently, or did it do a bunch of unnecessary work?效率：智能体是否高效完成了任务，还是做了大量不必要的工作？
• Verbosity: did the agent emit the right number of tokens in completing the task?详细程度：智能体在完成任务时输出的 Token 数量是否合适？
• Quality: for a coding task, was the quality of the code good? Did it match expected conventions?质量：对于编程任务，代码质量是否良好？是否符合预期的规范？
• Custom dimensions for your org, like whether the agents used the right internal MCPs and Skills针对你所在组织自定义的维度，例如智能体是否使用了正确的内部 MCP 和技能。

For example, here’s the definition for a custom scorer that checks for redundant test creation, a common failure mode we were seeing in our internal factory.例如，这里有一个自定义评分员的定义，用于检查冗余测试的创建，这是我们在内部工厂中发现的一种常见故障模式。

It contains a judging prompt (our full redundant tests scoring rubric for reference):它包含一个评判提示词（供参考的完整冗余测试评分准则）：

Along with a set of output classifications – what counts as a “pass” –以及一组输出分类——即什么算作“通过”——

and a sampling rate, indicating what percent of runs to score.还有一个抽样率，表明需要对百分之多少的运行进行评分。

When a scoring agent runs, it loads an agent trace, brings all its inputs and outputs into context, and then prompts an LLM to judge the run. The output is a classification like in the above example.当评分智能体运行时，它会加载智能体轨迹，将其所有输入和输出纳入上下文，然后提示大模型对该次运行进行评判。输出结果即为上述示例中的分类。

You won’t necessarily want to score every run, since scoring itself costs money. Instead, you’ll want to (third) decide on a sampling strategy. It could be percent-based, it could be classifier based (e.g. “score all my front-end tasks”), etc. For our internal factory, scoring currently accounts for about 3% of total token costs – that’s a reasonable amount to get visibility into agent performance.你未必想对每一次运行都进行评分，因为评分本身会产生费用。因此，你需要（第三步）确定抽样策略。可以是基于百分比的，也可以是基于分类器的（例如“对所有前端任务进行评分”），等等。对于我们的内部工厂，评分目前约占总 Token 成本的 3%——这对于获取智能体性能的可见性来说是一个合理的比例。

Over time, (fourth) you’ll build up a corpus of your scored runs. At the simplest level, you can use these just like DORA as another measurement of the efficacy of your factory. You can graph how the metrics are changing over time, catch regressions when they get worse, etc. Depending on how your factory is set up, you can try to correlate changes to models, skills and context with improvements (and regressions).随着时间的推移，（第四步）你将建立起一个已评分运行的语料库。在最简单的层面上，你可以像使用 DORA 指标一样，将其作为衡量工厂效能的另一种指标。你可以绘制指标随时间变化的图表，在指标恶化时捕捉回归问题等。根据你的工厂设置方式，你可以尝试将模型、技能和上下文的变化与改进（及回归）关联起来。

In the above graph you can see that our scorer thinks we are mostly avoiding redundant tests, but there are a few failing runs every day. To investigate, you can click into the failures and examine what the coding agent did and also examine the scorer run itself, since it’s just another agent, to understand why it thinks these coding agent runs produced redundant tests. You may notice patterns, and then adjust the skills which drive your agents, so that they write tests more sparingly.在上面的图表中，你可以看到我们的评分员认为我们基本避免了冗余测试，但每天仍有几次失败的运行。为了进行调查，你可以点击进入失败项，检查编程智能体的具体操作，并检查评分员本身的运行情况（因为它也只是另一个智能体），以了解它为什么认为这些编程智能体的运行产生了冗余测试。你可能会发现其中的规律，然后调整驱动智能体的技能，使它们在编写测试时更加谨慎。

Once you get a feel for checking your scorers by hand, you’ll probably want to (fifth) automate how they are used, and create an actual learning loop. In Warp Factories we call this “self-improvement,” and you can learn more about it here. The tl;dr is that scorers can be input into another agent loop that synthesizes their output in batch and creates updates to the factory definition automatically.一旦你习惯了手动检查评分员，你可能就会想要（第五步）自动化它们的使用方式，并创建一个实际的学习循环。在 Warp Factories 中，我们称之为“自我改进”，你可以在此处了解更多信息。简而言之，评分结果可以输入到另一个智能体循环中，该循环会批量合成这些输出，并自动创建对工厂定义的更新。

Scorers also (sixth) form the basis of more advanced optimizations like benchmarking, where you test different model configurations against your factory to optimize its cost and performance. If you want to learn about benchmarking, check out this post.评分员还（第六步）构成了更高级优化（如基准测试）的基础，你可以在基准测试中针对工厂测试不同的模型配置，以优化其成本和性能。如果你想了解基准测试，请查看这篇文章。

In sum, if you aren’t currently scoring your coding agents, you are missing a crucial layer of visibility into how they are performing and how you might improve them. It’s a bit of work to set up, but in an age where more and more of your company’s software production depends on how efficiently your agents work, it’s well worth the effort to gain that visibility.总之，如果你目前没有对编程智能体进行评分，那么你就缺失了关于它们表现如何以及如何改进它们的关键可见性层。虽然设置起来需要一些工作，但在公司软件生产越来越依赖智能体工作效率的时代，为了获得这种可见性，这些努力是非常值得的。

If you are interested in learning more about Warp Factories and how they are helping companies scale development on open, observable infrastructure, you can request early access here. We are offering up to $10k in usage to qualified companies.如果你有兴趣了解更多关于 Warp Factories 的信息，以及它们如何帮助公司在开放、可观测的基础设施上扩展开发，可以在此处申请抢先体验。我们为符合条件的公司提供最高 1 万美元的使用额度。