LLM-as-a-Verifier

Test-Time Scaling测试时扩展

LLM-as-a-Verifier achieves state-of-the-art performance across coding, robotics, and medical domains: Terminal-Bench V2 (86.5%), SWE-Bench Verified (78.2%), RoboRewardBench (87.4%), and MedAgentBench (73.3%). LLM作为验证器在编程、机器人技术和医疗领域均达到了最先进性能：Terminal-Bench V2（86.5%）、SWE-Bench Verified（78.2%）、RoboRewardBench（87.4%）和MedAgentBench（73.3%）。

Reinforcement Learning强化学习

LLM-as-a-Verifier as a dense reward signal improves the sample efficiency of both off-policy and on-policy RL algorithms. On LIBERO, LLM-as-a-Verifier achieves ≈ 1.8× higher sample efficiency than sparse reward baselines when fine-tuning a π₀ policy with SAC. On the MATH reasoning benchmark, it achieves ≈ 1.1× higher sample efficiency when fine-tuning Qwen3-8B with GRPO. LLM作为验证器作为密集奖励信号，提高了离策略和在线策略强化学习算法的样本效率。在LIBERO上，使用SAC微调π0策略时，LLM作为验证器实现了比稀疏奖励基线高约1.8倍的样本效率。在MATH推理基准上，使用GRPO微调Qwen3-8B时，实现了约1.1倍的样本效率提升。

Progress Tracking (Coding)进度跟踪（编程）

SUCCESS FAILED Task: pytorch-model-cli — run MNIST inference · Terminus-2 · Gemini 2.5 Pro

step 10/10

SUCCESS

FAILED

We observe a strong correlation between the chronological progression of code generation steps and the scores from LLM-as-a-Verifier. The successful trajectory of the pytorch-model-cli task follows a coherent sequence of events—Read model.py → Install g++ compiler → Install CPU-only torch → Update hidden_dim → DONE and exhibits consistently increasing verifier scores. In contrast, the failed trajectory is characterized by erroneous behaviors—it unnecessarily installs the large torchvision package, which exhausts the available disk space and hits a compilation error—resulting in significantly lower scores. 我们观察到代码生成步骤的时间顺序与LLM作为验证器的得分之间存在强相关性。pytorch-model-cli任务的成功轨迹遵循连贯的事件序列——读取model.py → 安装g++编译器 → 安装仅CPU的torch → 更新hidden_dim → 完成，并显示出持续上升的验证器得分。相反，失败轨迹的特点是错误行为——它不必要地安装了庞大的torchvision包，耗尽了可用磁盘空间并遇到编译错误——导致得分显著降低。

Progress Tracking (Robotics)进度跟踪（机器人技术）

Task: Corn onto Plate任务：玉米上盘

LLM-as-a-VerifierRoboRewardBench · Qwen 3.6

frame 1

LLM-as-a-Verifier generates a smooth, temporally aligned progress signal over the course of robot rollout while remaining robust to failure modes such as incorrect intent or imprecise grasping. The example above illustrates the “Corn on Plate” task from the RoboRewardBench-OOD dataset. LLM作为验证器在机器人部署过程中生成平滑且时间对齐的进度信号，同时对错误意图或抓取不精确等故障模式保持鲁棒性。上面的示例展示了来自RoboRewardBench-OOD数据集的“玉米上盘”任务。

Three Simple APIs $pip install llm-verifier三个简单的API $pip install llm-verifier

llm_verifier.selectllm_verifier.select

🏆 Best-of-N selection🏆 N选1最佳选择

from llm_verifier import select best = select(task, cands)from llm_verifier import select best = select(task, cands)

Score N candidates and return the best one — test-time scaling in a single call.对N个候选进行评分并返回最佳的一个——一次调用即可实现测试时扩展。

llm_verifier.comparellm_verifier.compare

⚖️ Pairwise comparison⚖️ 成对比较

from llm_verifier import compare best = compare(task, a, b)from llm_verifier import compare best = compare(task, a, b)

Directly score two candidates head-to-head with fine-grained preferences.直接对两个候选进行头对头评分，提供细粒度偏好。

llm_verifier.trackllm_verifier.track

📈 Progress tracking📈 进度跟踪

from llm_verifier import track scores = track(task, traj)from llm_verifier import track scores = track(task, traj)

Get a temporally aligned progress score over a rollout for monitoring or RL.获取部署过程中时间对齐的进度得分，用于监控或强化学习。

How does LLM-as-a-Verifier work?LLM作为验证器如何工作？

The key idea behind it is simple其核心思想很简单

Use fine-grained scoring granularity (e.g., 1–20 instead of the standard 1–5 scale)使用细粒度评分（例如1-20分，而非标准的1-5分制）
Take the expectation over the full logprob distribution of score tokens对评分令牌的完整对数概率分布取期望
Scale repeated evaluation and criteria decomposition扩展重复评估和标准分解

Scaling pre-training, post-training, and test-time compute have become the central paradigms for improving the capabilities of large language models (LLMs). In this work, we identify verification—the ability to determine the correctness of a solution—as a new scaling axis. To unlock this and demonstrate its effectiveness, we introduce LLM-as-a-Verifier, a general-purpose verification framework that provides fine-grained feedback for agentic tasks without requiring additional training. Unlike standard LM judges that prompt LLMs to produce discrete scores for candidate solutions, LLM-as-a-Verifier computes the expectation over the distribution of scoring-token logits to generate continuous scores. This probabilistic formulation substantially reduces tie rates when comparing complex solutions and enables verification to scale along multiple dimensions: (1) score granularity, (2) repeated evaluation, and (3) criteria decomposition. To make verification scaling practical, we further introduce a cost-efficient ranking algorithm for selecting the best solution among candidates using the preference probabilities derived from the verifier's continuous scores. LLM-as-a-Verifier achieves state-of-the-art performance on Terminal-Bench V2 (86.5%), SWE-Bench Verified (78.2%), RoboRewardBench (87.4%), and MedAgentBench (73.3%). Beyond verification, the fine-grained signals can serve as a proxy for estimating task progress, enabling safe deployment of agents; we provide extensions for Claude Code and Codex. Finally, we show that LLM-as-a-Verifier can be used as a dense reward signal for RL, improving the sample efficiency of SAC and GRPO on robotics and mathematical reasoning benchmarks.

Motivation动机

Generation is Not the Bottleneck — Verification Is生成不是瓶颈——验证才是

The key finding is that most agents already “know” how to solve the tasks. On Terminal-Bench, if you repeatedly sample 100 trajectories per task, they can outperform Claude Mythos and nearly solve the entire benchmark. But the problem is that they don't know which one is correct, particularly when dealing with long-horizon tasks. While standard LLM-as-a-Judge can be used here, they fail to provide sufficiently fine-grained feedback. When comparing complex solutions, judges often assign the same score, resulting in a tie and failing to discriminate between them. Coarse scoring leads to 27% ties on Terminal-Bench V2. 关键发现是，大多数智能体已经“知道”如何解决任务。在Terminal-Bench上，如果对每个任务重复采样100条轨迹，它们可以超越Claude Mythos并几乎解决整个基准。但问题在于它们不知道哪一个是正确的，尤其是在处理长周期任务时。虽然标准的LLM作为评判者可以用于此，但它们无法提供足够细粒度的反馈。在比较复杂解决方案时，评判者通常给出相同的分数，导致平局而无法区分。粗粒度评分在Terminal-Bench V2上导致27%的平局。

Methodology方法论

Fine-Grained Reward Estimation细粒度奖励估计

Given a task prompt $x$, a language model $p_\theta$, a criterion $c$, and two candidate trajectories $\tau_i$ and $\tau_j$, we construct scoring prompts and obtain their conditional distributions $p_{\theta}(v \mid x,c,\tau_i)$ and $p_{\theta}(v \mid x, c, \tau_j)$ by extracting the logprobs from <score_A> and <score_B> tags using the following prompt: 给定任务提示$x$、语言模型$p_\theta$、标准$c$以及两个候选轨迹$\tau_i$和$\tau_j$，我们构建评分提示，并通过使用以下提示从<score_A>和<score_B>标签中提取对数概率，得到它们的条件分布$p_{\theta}(v \mid x,c,\tau_i)$和$p_{\theta}(v \mid x, c, \tau_j)$：

You are an expert [domain] reviewer. You will see a task description and two trajectories. Evaluation Criteria: [domain specific criteria] Task: {task prompt} Trajectory A: {A} Trajectory B: {B} Carefully analyze each trajectory, then provide your final scores: <score_A> INTEGER_1_TO_20 </score_A> <score_B> INTEGER_1_TO_20 </score_B> Rating Rules: Rate correctness on a 1–20 scale based on evaluation criteria (1 = incorrect, 10 = borderline, 20 = correct) Note: We use a letter-based scale instead of digits to enable logprob extraction for granularity scaling.您是一位专家[领域]评审员。您将看到一个任务描述和两个轨迹。评估标准：[领域特定标准] 任务：{任务提示} 轨迹A：{A} 轨迹B：{B} 仔细分析每个轨迹，然后给出您的最终评分： <score_A> 整数_1_到_20 </score_A> <score_B> 整数_1_到_20 </score_B> 评分规则：根据评估标准对正确性进行1-20分评分（1=不正确，10=临界，20=正确）注意：我们使用基于字母的尺度而非数字，以便提取对数概率实现粒度扩展。

Rather than collapsing each distribution to a single discrete score, we approximate the reward of a trajectory as: 我们不是将每个分布压缩为单个离散分数，而是将轨迹的奖励近似为：

\[ R(x, \tau) = \frac{1}{CK} \sum_{c=1}^{C} \sum_{k=1}^{K} \sum_{g=1}^{G} p_{\theta}(v_g \mid x, c, \tau)\,\phi(v_g) \] \[ R(x, \tau) = \frac{1}{CK} \sum_{c=1}^{C} \sum_{k=1}^{K} \sum_{g=1}^{G} p_{\theta}(v_g \mid x, c, \tau)\,\phi(v_g) \]

where $C$ is the number of evaluation criteria, $K$ is the number of repeated verifications, $G$ is the number of score tokens (granularity level), $p_{\theta}(v_g \mid x, c, \tau)$ is the probability assigned by model $\theta$ to score token $v_g$, and $\phi(v_g)$ maps each score token to a scalar value. 其中$C$是评估标准的数量，$K$是重复验证的次数，$G$是评分令牌的数量（粒度级别），$p_{\theta}(v_g \mid x, c, \tau)$是模型$\theta$分配给评分令牌$v_g$的概率，而$\phi(v_g)$将每个评分令牌映射为一个标量值。

Probabilistic Pivot Tournament (PPT)概率枢轴锦标赛（PPT）

To select the best of $N$ candidates under a constrained verification budget, PPT compares each candidate against a small set of $k \ll N$ pivots, cutting cost from $\mathcal{O}(N^2)$ to $\mathcal{O}(Nk)$. 为了在有限的验证预算下从$N$个候选中选择最佳，PPT将每个候选与一个小型$k \ll N$枢轴集进行比较，将成本从$\mathcal{O}(N^2)$降至$\mathcal{O}(Nk)$。

(1) Candidates: the candidate pool $\{\tau_1,\dots,\tau_N\}$ to be ranked.（1）候选：待排序的候选池$\{\tau_1,\dots,\tau_N\}$。

(2) Ring pass: a random Hamiltonian cycle scores the $N$ adjacent pairs so every candidate appears once in the “A” slot and once in “B”, canceling the model's positional bias.（2）环传递：一个随机哈密顿环对$N$个相邻对进行评分，使得每个候选在“A”槽和“B”槽中各出现一次，从而消除模型的位置偏差。

(3) Pivot selection: candidates are ranked by their ring-pass scores $w_{(i)}$, and the top-$k$ candidates form the pivot set $\mathcal{P}$.（3）枢轴选择：根据环传递得分$w_{(i)}$对候选进行排序，前$k$个候选构成枢轴集$\mathcal{P}$。

(4) Pivot tournament: every non-pivot–vs–pivot and pivot–vs–pivot pair is scored with the fine-grained reward estimation above, concentrating the budget on uncertain top candidates and cutting cost from $\mathcal{O}(N^2)$ to $\mathcal{O}(Nk)$.（4）枢轴锦标赛：每个非枢轴-枢轴对和枢轴-枢轴对都使用上述细粒度奖励估计进行评分，将预算集中在不确定的顶级候选上，并将成本从$\mathcal{O}(N^2)$降至$\mathcal{O}(Nk)$。

(5) Selection: comparisons are aggregated into weight $w_i$ and count $c_i$; the candidate with the highest normalized $w_i/c_i$ is returned.（5）选择：比较结果聚合成权重$w_i$和计数$c_i$；返回归一化$w_i/c_i$最高的候选。

Analysis分析

Verification as a New Scaling Axis验证作为新的扩展轴

We find that verification accuracy consistently improves as we scale across multiple dimensions: (1) the granularity of score tokens, (2) the number of repeated evaluations, and (3) the decomposition of evaluation criteria. 我们发现，随着在多个维度上的扩展，验证准确性持续提高：（1）评分令牌的粒度，（2）重复评估的次数，以及（3）评估标准的分解。

What drives granularity scaling?粒度扩展的驱动力是什么？

Better separation between positive and negative solutions. To isolate why finer granularity improves verification, we decompose the pairwise score gap between correct ($s_c$) and incorrect ($s_i$) trajectories into a signal and a noise component. We define the signal-to-noise ratio as below, where $\mathbb{E}(s_c-s_i)$ captures how strongly the verifier prefers the correct trajectory over the incorrect one (signal strength), and the denominator, $\mathrm{Var}(s_c - s_i)$, captures how inconsistent that preference is across pairs (noise). 更好地区分正解和负解。为了隔离为什么更细的粒度能提高验证效果，我们将正确轨迹（$s_c$）和错误轨迹（$s_i$）之间的成对分数差距分解为信号和噪声成分。我们定义信噪比如下，其中$\mathbb{E}(s_c-s_i)$捕捉验证器对正确轨迹相对于错误轨迹的偏好强度（信号强度），分母$\mathrm{Var}(s_c - s_i)$捕捉这种偏好跨成对的不一致性（噪声）。

\[ \mathrm{SNR}(G) = \frac{\mathbb{E}[s_c - s_i]}{\sqrt{\mathrm{Var}(s_c - s_i)}} \] \[ \mathrm{SNR}(G) = \frac{\mathbb{E}[s_c - s_i]}{\sqrt{\mathrm{Var}(s_c - s_i)}} \]

Signal-to-noise ratio grows with the number of scoring tokens $G$ (Terminal-Bench V2, $k{=}16$).
Granularity $G$	1	4	16	20
SNR (k=16)	0.775	0.786	0.797	0.799

What drives repeated evaluation?重复评估的驱动力是什么？

Variance reduction. LLM-as-a-Verifier consistently outperforms LLM-as-a-Judge , achieving 77.4% verification accuracy while eliminating ties entirely across all repeated verification budgets. Even at $k = 16$, where repeated verification reduces judge ties, the verifier still maintains 7.2% higher accuracy. 方差减少。LLM作为验证器始终优于LLM作为评判者，在所有重复验证预算下实现77.4%的验证准确率，同时完全消除平局。即使在$k = 16$时，重复验证减少了评判者的平局，验证器仍然保持7.2%的更高准确率。

Verifier vs Judge: accuracy and tie rate across repeated evaluations

What drives criteria decomposition?标准分解的驱动力是什么？

Complexity reduction. Granularity and repeated evaluation both assume the rubric itself is adequate. In long-horizon agentic tasks, a judgment like “is this trajectory correct?” conflates several logically distinct factors, and a verifier asked a compound question often latches onto whichever factor is most salient in the prompt. We instead replace the single monolithic rubric with an ensemble over $C$ simpler sub-criteria. For code-agent trajectories we decompose correctness into three factors that are each easier to verify — Specification (all task requirements satisfied), Output (final output format matches the expected result), and Errors (no failure signals in logs and tool outputs) — and average the expected scores across criteria. Any single criterion alone reaches 75.2–76.4% accuracy; their ensemble reaches 78.3%. 复杂性降低。粒度和重复评估都假设评分细则本身是充分的。在长周期智能体任务中，像“这个轨迹正确吗？”这样的判断混淆了几个逻辑上不同的因素，而一个被问到复合问题的验证器往往会抓住提示中最突出的因素。我们转而用由$C$个更简单的子标准组成的集成取代单一的完整评分细则。对于代码智能体轨迹，我们将正确性分解为三个更易验证的因素——规范（所有任务要求均满足）、输出（最终输出格式与预期结果匹配）和错误（日志和工具输出中没有失败信号）——并对各标准的期望分数取平均。任何单个标准单独达到75.2-76.4%的准确率；它们的集成达到78.3%。

Case Study: Terminal-Bench案例研究：Terminal-Bench

To concretely illustrate how scaling granularity to $G{=}20$ and our probabilistic formulation sharpen the verifier's signal, we analyze a representative trajectory pair from the query-optimize task on Terminal-Bench V2. The agent is given a slow SQL query and asked to produce an equivalent optimized version; both candidates run faster, but only one validates equivalence against the canonical database. Over 100 repeated evaluations, a discrete 1–5 judge collapses these nuanced assessments into ties (88/100). Taking the expectation over the same 5-point distribution eliminates ties entirely and ranks the correct trajectory higher in 69/100 runs; scaling granularity to $G{=}20$ sharpens the signal further, ranking it strictly higher in 77/100 runs. 为了具体说明将粒度扩展到$G=20$以及我们的概率公式如何增强验证器的信号，我们分析了来自Terminal-Bench V2上query-optimize任务的一对代表性轨迹。智能体获得一个慢速SQL查询，并要求生成一个等效的优化版本；两个候选都运行得更快，但只有一个针对规范数据库验证了等价性。在100次重复评估中，一个离散的1-5分评判者将这些细微的评估压缩为平局（88/100）。对同一个5点分布取期望完全消除了平局，并在69/100次运行中将正确轨迹排在更高位置；将粒度扩展到$G=20$进一步增强了信号，在77/100次运行中将其严格排得更高。

Method	correct > incorrect ✅	correct = incorrect ⚖️	correct < incorrect ❌
Judge (discrete, G=5)	12/100	88/100	0/100
Verifier (continuous, G=5)	69/100	0/100	31/100
Verifier (continuous, G=20)	77/100	0/100	23/100

Verifier (Continuous) vs. Judge (Discrete)

Left: The verifier achieves 74.7% at k=1 and improves to 77.5% at k=16, consistently outperforming the discrete judge across all evaluation budgets. A single-pass verifier already beats a heavily ensembled judge. Right: The judge produces ties in 26.7% of comparisons at k=1 due to coarse discrete scoring, while the continuous verifier yields zero ties.

Experiments实验

Test-Time Scaling Results测试时扩展结果

Per-Benchmark Performance各基准性能

Across challenging benchmarks such as Terminal-Bench V2, SWE-Bench Verified, and MedAgentBench, LLM-as-a-Verifier outperforms frontier models including Claude Opus 4.8, GPT 5.5, and Gemini models. Results for Terminal-Bench and SWE-Bench are reported from the official leaderboards. 在Terminal-Bench V2、SWE-Bench Verified和MedAgentBench等具有挑战性的基准上，LLM作为验证器优于包括Claude Opus 4.8、GPT 5.5和Gemini模型在内的前沿模型。Terminal-Bench和SWE-Bench的结果来自官方排行榜。

Benchmark	Baseline models (accuracy)			LLM-as-a-Verifier
Benchmark	#1	#2	#3	Pass@1	Oracle	Ours
Terminal-Bench V2	GPT-5.5 (84.7%)	Opus 4.7 (80.2%)	Gemini 3.1 Pro (80.2%)	83.1%	92.1%	86.5%
SWE-Bench Verified	Opus 4.5 (76.8%)	Gemini 3 Flash (75.8%)	MiniMax M2.5 (75.8%)	76.1%	84.4%	78.2%
MedAgentBench	Opus 4.8 (70.2%)	Gemini 3.5 Flash (66.3%)	GPT-5.5 (65.1%)	70.2%	75.0%	73.3%

Preference Accuracy on RoboRewardBenchRoboRewardBench上的偏好准确率

For RoboRewardBench, we curate pairs of rollout videos that follow the same natural-language instruction but make different amounts of progress; the reward model must output a preference indicating which rollout makes more progress. LLM-as-a-Verifier outperforms fine-tuned robotics reward models. 对于RoboRewardBench，我们策划了遵循相同自然语言指令但取得不同进展的成对部署视频；奖励模型必须输出一个偏好，指示哪个部署取得了更多进展。LLM作为验证器优于微调的机器人奖励模型。

Method	Accuracy (%)
TOPReward	74.7
Robometer-4B	78.8
RoboReward-8B	81.4
LLM-as-a-Judge (Discrete)	70.8
LLM-as-a-Verifier (Ours)	87.4

Progress Tracking Results进度跟踪结果

We quantify progress tracking with the Value-Order Correlation (VOC) — the Spearman rank correlation between a step's chronological index and the verifier's predicted value for the prefix ending at that step. A verifier that tracks progress assigns monotonically higher scores to later prefixes of a successful rollout ($\mathrm{VOC}\to 1$): 我们使用价值顺序相关性（VOC）来量化进度跟踪——即步骤的时间索引与验证器对截至该步骤的前缀的预测值之间的Spearman秩相关。一个跟踪进度的验证器会对成功部署的后期前缀分配单调递增的分数（$\mathrm{VOC}\to 1$）：

\[ \mathrm{VOC} = \mathrm{rank\text{-}correlation}\!\left( \mathrm{argsort}(s_{t_1}, s_{t_2}, \cdots, s_{t_K}),\ (t_1, t_2, \cdots, t_K) \right) \] \[ \mathrm{VOC} = \mathrm{rank\text{-}correlation}\!\left( \mathrm{argsort}(s_{t_1}, s_{t_2}, \cdots, s_{t_K}),\ (t_1, t_2, \cdots, t_K) \right) \]

VOC on Terminal-Bench V2在Terminal-Bench V2上的VOC

Successful rollouts show near-monotonic progress, while failed rollouts correlate more weakly. 成功部署显示出近乎单调的进展，而失败部署的相关性较弱。

Trajectory outcome	Spearman VOC
Successful	0.848
Failed	0.769
Success − Failed (gap)	+0.079

VOC on RoboRewardBench在RoboRewardBench上的VOC

LLM-as-a-Verifier attains the highest correlation between step index and predicted progress, outperforming fine-tuned robotics reward models. LLM作为验证器在步骤索引与预测进度之间达到最高相关性，优于微调的机器人奖励模型。

Method	Spearman VOC
LLM-as-a-Verifier (Qwen 3.6 35B)	0.966
RoboReward-8B	0.877
Robometer-4B	0.780
TOPReward (Qwen 3.6)	0.565

Reinforcement Learning Results强化学习结果

The fine-grained verifier score is a drop-in dense reward for both off-policy and on-policy RL, improving sample efficiency by ≈1.8× on LIBERO and ≈1.1× on MATH. 细粒度验证器分数是可即插即用的密集奖励，适用于离策略和在线策略强化学习，在LIBERO上提高了约1.8倍的样本效率，在MATH上提高了约1.1倍。

Off-policy RL: dense progress rewards for DSRL-SAC离策略强化学习：用于DSRL-SAC的密集进度奖励

When fine-tuning a $\pi_0$ policy on LIBERO with SAC, we relabel each rollout with the verifier's per-step progress score $\rho_t$ as a dense shaped reward, then store the transitions in the replay buffer $\mathcal{D}$ and train the SAC critic on returns sampled from $\mathcal{D}$: 当使用SAC在LIBERO上微调$\pi_0$策略时，我们使用验证器的每步进度得分$\rho_t$作为密集塑形奖励重新标记每个部署，然后将转换存储在回放缓冲区$\mathcal{D}$中，并在从$\mathcal{D}$采样的回报上训练SAC评论家：

\[ r_t = r^{\text{env}}_t + \lambda\,\rho_t \qquad\quad \mathcal{D} \leftarrow \mathcal{D} \cup \{(s_t, a_t, r_t, s_{t+1})\} \] \[ r_t = r^{\text{env}}_t + \lambda\,\rho_t \qquad\quad \mathcal{D} \leftarrow \mathcal{D} \cup \{(s_t, a_t, r_t, s_{t+1})\} \]

Sample Efficiency on LIBERO ($\pi_0$ + DSRL-SAC)在LIBERO上的样本效率（$\pi_0$ + DSRL-SAC）

Environment timesteps required to reach each target success rate (averaged over $n{=}5$ seeds). 达到每个目标成功率所需的环境时间步（在$n=5$个种子上平均）。

Target SR (%)	Sparse	LLM-as-a-Verifier	Steps saved	Speedup
20	132,800	74,300	58,500	1.79×
40	309,400	168,500	140,900	1.84×
60	993,800	600,000	393,800	1.66×

On-policy RL: dense reasoning rewards for GRPO在线策略强化学习：用于GRPO的密集推理奖励

When fine-tuning Qwen3-8B on MATH with GRPO, sparse correctness rewards give no gradient when every sampled answer is wrong. We score each completion's reasoning trace with the Probabilistic Pivot Tournament (PPT) and add this reasoning-quality score to the correctness and format rewards to provide additional signal: 当使用GRPO在MATH上微调Qwen3-8B时，如果每个采样答案都错误，稀疏的正确性奖励不会提供梯度。我们使用概率枢轴锦标赛（PPT）对每个完成的推理轨迹进行评分，并将这个推理质量分数添加到正确性和格式奖励中，以提供额外信号：

\[ r_i = r_{\mathrm{correct},i} + r_{\mathrm{format},i} + \beta\, r_{\mathrm{reasoning},i} \] \[ r_i = r_{\mathrm{correct},i} + r_{\mathrm{format},i} + \beta\, r_{\mathrm{reasoning},i} \]

Sample Efficiency on MATH (Qwen3-8B + GRPO)在MATH上的样本效率（Qwen3-8B + GRPO）

Sampled completions required to reach each target success rate (averaged over $n{=}3$ seeds; each step samples 1024). 达到每个目标成功率所需的采样完成数（在$n=3$个种子上平均；每一步采样1024个）。

Target SR (%)	Sparse	LLM-as-a-Verifier	Completions saved	Speedup
20	40,890	36,010	4,880	1.14×
40	47,990	43,450	4,540	1.10×
60	55,860	50,780	5,080	1.10×

Zero-Shot Robotics Reward Modeling

On RoboRewardBench, the verifier must integrate visual context across multi-frame videos to reason about physical progress. Applied zero-shot with a Qwen 3.6 35B VLM, LLM-as-a-Verifier reaches 87.4% trajectory preference accuracy—outperforming the discrete judge (70.8%) and reward models trained specifically on robotics data, including RoboReward-8B (81.4%), Robometer-4B (78.8%), and TOPReward (74.7%)—despite using no fine-tuning. It also reduces MAE against human annotations from 1.11 to 0.72.

Appendix附录

Recovering Continuous Rewards for Logit-Restricted Frontier Models为对数限制的前沿模型恢复连续奖励

Some frontier models (e.g., GPT-5.5, Claude Opus) expose only sampled completions and withhold token-level logprobs, which our continuous reward requires. A simple two-stage workaround recovers most of the calibrated signal: the closed model supplies domain-specific reasoning, and an open verifier supplies the calibrated probability distribution it withholds. 一些前沿模型（例如GPT-5.5、Claude Opus）只暴露采样完成结果，并隐藏令牌级别的对数概率，而我们的连续奖励需要这些。一个简单的两阶段变通方法恢复了大部分校准信号：闭源模型提供领域特定推理，开源的验证器提供其隐藏的校准概率分布。

🔒

Closed Frontier Model闭源前沿模型

GPT-5.5 · Claude Opus
logprobs withheldGPT-5.5 · Claude Opus对数概率隐藏

→reasoning + draft score

🔓

Open Verifier开源验证器

Gemini 2.5 Flash (G=20)
reads <score_A>/<score_B> logprobsGemini 2.5 Flash (G=20)读取<score_A>/<score_B>对数概率

→expectation over logits

📈

Continuous Reward连续奖励

$R(x,\tau)$$R(x,\tau)$

On Terminal-Bench V2, routing GPT-5.5's reasoning through Gemini 2.5 Flash recovers a +5.2-point accuracy gain over directly using the closed model's integer scores (80.1% vs. 74.9%) and eliminates its 10.9% tie rate entirely — without any access to the frontier model's logits. 在Terminal-Bench V2上，将GPT-5.5的推理通过Gemini 2.5 Flash路由，相比直接使用闭源模型的整数分数，恢复了+5.2个百分点的准确率提升（80.1% vs. 74.9%），并完全消除了其10.9%的平局率——且无需访问前沿模型的logits。

$K$	GPT-5.5 (Discrete)		GPT-5.5 → Gemini 2.5 Flash (Continuous)
$K$	Accuracy (%)	Tie rate (%)	Accuracy (%)	Tie rate (%)
1	74.9	10.9	80.1	0.0
2	76.3	9.1	80.5	0.0
4	77.6	7.0	81.0	0.0
8	78.4	5.8	80.9	0.0
16	79.1	5.0	81.2	0.0

BibTeXBibTeX

@misc{kwok2026llmasaverifiergeneralpurposeverificationframework,
      title={LLM-as-a-Verifier: A General-Purpose Verification Framework},
      author={Jacky Kwok and Shulu Li and Pranav Atreya and Yuejiang Liu and Yixing Jiang and Chelsea Finn and Marco Pavone and Ion Stoica and Azalia Mirhoseini},
      year={2026},
      eprint={2607.05391},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2607.05391},
}

① Score Granularity

② Repeated Evaluation

③ Criteria Decomposition

BibTeXBibTeX

\(K\)	GPT-5.5 (Discrete)		GPT-5.5 → Gemini 2.5 Flash (Continuous)
\(K\)	Accuracy (%)	Tie rate (%)	Accuracy (%)	Tie rate (%)
1	74.9	10.9	80.1	0.0
2	76.3	9.1	80.5	0.0
4	77.6	7.0	81.0	0.0
8	78.4	5.8	80.9	0.0
16	79.1	5.0	81.2	0.0