Designing Efficient Verifiers for Legal Agents为法律智能体设计高效的验证器

Can We Run More Efficient Verification?我们能否运行更高效的验证？

The cost of running frontier verifiers creates a practical question for teams running legal-agent evaluation or training legal agents with RL:运行前沿验证器的成本给进行法律智能体评估或使用 RL 训练法律智能体的团队带来了一个现实问题：

How can you best reduce the cost of verifiers while remaining close to frontier performance?如何在保持接近前沿模型性能的同时，最大限度地降低验证器成本？

We study two different methods of doing more efficient verification:我们研究了两种更高效的验证方法：

1. Use fewer tokens使用更少的 token
2. Use cheaper tokens使用更便宜的 token

First method we explore: using fewer tokens. In order to use fewer tokens, we propose running verifiers in batch. That is - rather than using an LLM call for each criterion independently, we can ask it to judge the full rubric in a single batch call.我们探索的第一种方法：使用更少的 token。为了减少 token 使用量，我们建议批量运行验证器。也就是说——不必为每条标准单独调用 LLM，而是可以在一次批量调用中让它评判完整的评分标准。

• Per-criterion scoring: run one judge call for each rubric requirement.逐条标准评分：为每条评分标准要求运行一次裁判调用。
• Batch scoring: run one judge call for the task and ask the judge to label every rubric requirement at once.批量评分：为任务运行一次裁判调用，并要求裁判同时标注每条评分标准要求。

Second method we explore: using cheaper tokens. To use cheaper tokens we can test cheaper models during verification. We used Opus 4.7 per-criterion as the reference and compared GPT-5.5, Sonnet 4.6, DeepSeek v4 Flash, and Claude Haiku 4.5 across per-criterion and batch scoring.我们探索的第二种方法：使用更便宜的 token。为了使用更便宜的 token，我们可以在验证期间测试更便宜的模型。我们以 Opus 4.7 逐条标准评分作为参考，并比较了 GPT-5.5、Sonnet 4.6、DeepSeek v4 Flash 和 Claude Haiku 4.5 在逐条标准和批量评分中的表现。

Experiments to Measure Efficiency across Verifier Designs衡量不同验证器设计效率的实验

To run our verifier experiments, we first needed to produce a set of outputs for the verifier to evaluate.  To create these outputs, we ran an agent (powered by Kimi K2.6) over 40 public LAB tasks across the following practice areas: Corporate M&A, Tax, Emerging Companies/VC, and Trusts and Estates.为了运行验证器实验，我们首先需要生成一组供验证器评估的输出。为了创建这些输出，我们运行了一个由 Kimi K2.6 驱动的智能体，完成了 40 个公开的 LAB 任务，涵盖以下业务领域：公司并购、税务、新兴公司/风险投资以及信托与遗产。

Across these 40 tasks were 2,348 individual rubric criteria - each is scored as pass/fail by a verifier.  We first run Opus-4.7 across all criteria as a baseline.  This gives us a baseline to compare against when testing GPT-5.5, Sonnet 4.6, Haiku 4.5, and DeepSeek-V4-Flash as other verifier options.  Every verifier run will produce the same 2,348 criteria scores (pass/fail) and we can use these to study how they compare.这 40 个任务中共有 2,348 条单独的评分标准——每条标准由验证器评定为通过/未通过。我们首先使用 Opus-4.7 对所有标准进行基线运行。这为我们提供了基准，以便在测试 GPT-5.5、Sonnet 4.6、Haiku 4.5 和 DeepSeek-V4-Flash 作为其他验证器选项时进行比较。每次验证器运行都会产生相同的 2,348 条标准分数（通过/未通过），我们可以用这些分数来研究它们的比较情况。

For each verifier run, we measured:对于每次验证器运行，我们测量了：

• Agreement: how often it matched Opus per-criterion labels.一致性：与 Opus 逐条标准标签的匹配频率。
• False pass: how often it passed a criterion that Opus failed.误判通过：Opus 未通过但验证器通过的标准频率。
• False fail: how often it failed a criterion that Opus passed.误判未通过：Opus 通过但验证器未通过的标准频率。
• Cost: observed token cost for the 40-task verifier run.成本：运行 40 个任务验证器的观测 token 成本。

We paid particular attention to false passes. In real world settings, a failed criterion can be escalated for further review. That is usually preferable to letting a criterion pass when it should fail in a domain like legal.我们特别关注误判通过。在现实场景中，未通过的标准可以升级进行进一步审核。在法律等领域，这通常比让本应未通过的标准通过更可取。

Verification, like most agent-system design, is a tradeoff between performance, cost, and time. Per-criterion verification gives the judge a narrower decision window, but it requires many more calls. Batch verification is cheaper and faster, but the judge has to track the full rubric at once.验证与大多数智能体系统设计一样，是性能、成本和时间之间的权衡。逐条标准验证为裁判提供了更窄的决策窗口，但需要更多的调用次数。批量验证更便宜、更快，但裁判必须同时追踪完整的评分标准。

The chart below shows cost versus label drift. The x-axis is verifier cost per 1,000 rubric criteria. The y-axis is disagreement with Opus per-criterion labels, or 100% - agreement. Lower and further left is better.下图展示了成本与标签漂移的关系。x 轴是每 1,000 条评分标准的验证器成本。y 轴是与 Opus 逐条标准标签的不一致率，即 100% 减去一致性。越低越靠左越好。

Some takeaways:一些关键发现：

1. Across the board, running in batch mode has lower match rates than running in per-criterion mode.  But running batch is an order of magnitude cheaper to run for the same model as it saves on repeated input token costs.总体而言，批量运行的匹配率低于逐条标准运行。但对于同一模型，批量运行的成本要低一个数量级，因为它节省了重复的输入 token 成本。
2. Even frontier models like GPT-5.5 and Opus disagree on labels - they only have a 95.7% match rate.  This means that some of the datapoints may not be sufficiently specified for models to apply them as consistently as experts. This also means that targeting 100% match rate may not be realistic, and a match rate of 95.7% may be a reasonable upper bound.即使是 GPT-5.5 和 Opus 这样的前沿模型在标签上也会存在分歧——它们的匹配率仅为 95.7%。这意味着某些数据点可能没有被充分明确，导致模型无法像专家一样一致地应用它们。这也意味着追求 100% 的匹配率可能不现实，而 95.7% 的匹配率可能是一个合理的上限。
3. DeepSeek is a strong approximation of Opus as a verifier, both running one criterion at a time and running in batch mode.  It can also be run 3 orders of magnitude more cheaply which makes it a good candidate for large data and training domains where you need to run verification at scale.作为验证器，DeepSeek 是 Opus 的强大近似，无论是一次运行一条标准还是批量运行。它的运行成本也可以低 3 个数量级，这使其成为需要大规模验证的大数据领域和训练领域的良好候选方案。
4. Haiku was cheaper than Opus and Sonnet, but much more permissive. Its false-pass rates were 48.4% per-criterion and 34.7% batch, which is the wrong failure mode for legal verification.Haiku 比 Opus 和 Sonnet 更便宜，但也更宽松。它的误判通过率分别为逐条标准 48.4% 和批量 34.7%，这对法律验证来说是错误的失效模式。

Cost savings on post-training后训练的成本节省

Verifiers are not just used for evals. They are also used for post-training, and verification costs are amplified here, due to the multiple rollouts per task. LLM-as-judge systems turn task rubrics into reward signals, and cheaper reward signals make it practical to run more experiments, audit more rollouts, and iterate faster.验证器不仅用于评估。它们也用于后训练，而验证成本在这里会被放大，因为每个任务有多次 rollout。LLM-as-judge 系统将任务评分标准转化为奖励信号，而更便宜的奖励信号使运行更多实验、审核更多 rollout 和更快迭代成为可能。

A quick pass at extrapolating costs show that DeepSeek can be run 60-1000x cheaper than frontier verifiers at scale.  This becomes especially important in domains that are not easily programmatically verifiable and require some amount of LLM as a Judge to produce a reward signal.快速推算成本表明，DeepSeek 在大规模运行时可以比前沿验证器便宜 60-1000 倍。这在那些不易通过程序验证、需要一定程度的 LLM as a Judge 来产生奖励信号的领域中尤为重要。

Tuning Verifier Behavior from Traces基于追踪调优验证器行为

The results keep the prompt fixed per model and per verifier architecture (per-criterion vs batch). One additional lever we tested was targeted prompt tuning.上述结果中，每个模型和每种验证器架构（逐条标准 vs 批量）的提示词是固定的。我们测试的另一个额外手段是有针对性的提示词调优。

In order to test the effects of prompt tuning, we ran an auto-research loop on the previous results of DeepSeek compared to Opus. We looked at why & how DeepSeek diverged and tweaked the prompt over several runs. We told it optimize for false-pass rate.为了测试提示词调优的效果，我们针对 DeepSeek 与 Opus 的先前结果运行了一个自动研究循环。我们分析了 DeepSeek 为何以及如何产生分歧，并在多次运行中调整了提示词。我们指示它优化误判通过率。

One key reason for some of the DeepSeek errors with the default prompt was that DeepSeek was too willing to pass criteria when the answer was related to the requirement but did not satisfy every material part. The final prompt made the verifier decompose each piece of each criterion more explicitly as a checklist and instructed it to be cautious if the information present wasn’t totally clear. This reduced DeepSeek false-pass rates in both scoring modes: from 10.7% to 9.5% per-criterion and from 15.6% to 14.2% in batch.使用默认提示词时，DeepSeek 出现部分错误的一个关键原因是，当答案与要求相关但并未满足每一个实质部分时，DeepSeek 太容易让标准通过。最终提示词让验证器将每条标准的每个部分更明确地分解为一个检查清单，并指示它在现有信息不完全明确时保持谨慎。这降低了 DeepSeek 在两种评分模式下的误判通过率：逐条标准从 10.7% 降至 9.5%，批量评分从 15.6% 降至 14.2%。

Mining traces for data and doing targeted distillation of behavior via prompting continues to be an effective strategy for improving verifiers and agents in general.从追踪中挖掘数据，并通过提示词进行有针对性的行为提炼，仍然是改进验证器和智能体的有效策略。

Building Better Agents & More Efficient Verification Systems for the Legal Domain为法律领域构建更优秀的智能体和更高效的验证系统

Verifiers are one piece of the puzzle for building world class legal agents. Open model verifiers give us a cost-performance tradeoff that allows teams to run evals and do RL post-training orders of magnitude more cheaply, and often makes it feasible to attempt in the first place.  We also find that simple methods like batching verification work reasonably well and provide another order of magnitude reduction in cost.验证器是构建世界一流法律智能体的重要一环。开源模型验证器为我们提供了成本与性能的权衡，使团队能够以低几个数量级的成本运行评估和进行 RL 后训练，而且往往使其在一开始就成为可能。我们还发现，批量验证等简单方法效果相当好，并能进一步将成本降低一个数量级。

Open models also give firms the opportunity to fine-tune bespoke verifiers for their most crucial domains. A lot of work assumes that frontier closed models are the gold standard to distill towards, but even Opus, GPT-5.5, and Sonnet disagree on roughly 4-5% of labels in this study. We feel there’s more work to be done to challenge this belief further.开源模型还使企业有机会针对其最关键的领域微调定制验证器。许多工作假设前沿闭源模型是提炼的黄金标准，但即使在本研究中，Opus、GPT-5.5 和 Sonnet 在约 4-5% 的标签上也存在分歧。我们认为还需要做更多工作来进一步挑战这一观念。

We’re excited to partner with Harvey to push forward research on better verification systems at scale. In future work, we’re excited to study the impact of fine-tuning verifiers and their impact on post-training and running evals at scale.我们很荣幸能与 Harvey 合作，推动大规模更优验证系统的研究。在未来的工作中，我们很期待研究微调验证器的影响，以及它们对后训练和大规模运行评估的影响。

‍