The Agent Said It Was Done. The Database Disagreed. 代理说任务已完成。数据库却不这么认为。
Figure 1: ThinkingBox runs an agent against isolated MCP tool sessions, then grades the terminal backend state and side effects it leaves behind. From our ThinkingBox paper.图 1：ThinkingBox 在独立的 MCP 工具会话中运行代理，然后对其终端后端状态及留下的副作用进行评分。摘自我们的 ThinkingBox 论文。
This is a joint blog by Microsoft and Hugging Face, special thanks to Tommy Guy (founder at Enderis AI, previously Microsoft), Sergio Paniego from Hugging Face and our former interns Zhuochun Li (University of Pittsburgh), Ali Keramati (UC Irvine), Youngmin Ko (Northwestern) for co-authoring/reviewing efforts. 这是一篇由微软与 Hugging Face 联合发布的博客。特别感谢 Tommy Guy（Enderis AI 创始人，前微软员工）、来自 Hugging Face 的 Sergio Paniego，以及我们前实习生 Zhuochun Li（匹兹堡大学）、Ali Keramati（加州大学欧文分校）、Youngmin Ko（西北大学）在撰写和审阅方面所做的贡献。
A customer writes in. Her $745 kitchen appliance has been stuck in a courier "exception" at a Nashville distribution center, fifteen days past its estimated delivery date.一位客户发来咨询。她价值 745 美元的厨房电器在纳什维尔配送中心陷入了快递“异常”状态，距离预计送达日期已经过去了十五天。
The AI agent does careful work. Nine tool calls: it pulls the order, checks tracking, looks up her customer profile, searches the refund policy twice, confirms no ticket exists, opens one, documents the timeline, and reads the policy correctly; her account segment genuinely does not qualify for late-delivery compensation.AI 代理工作得十分仔细。它进行了九次工具调用：调取订单、检查物流、查看客户档案、两次搜索退款政策、确认没有现存工单、新建一个工单、记录时间线，并准确解读了政策；她的账户细分领域确实不符合延迟交付赔偿的条件。
Then it closes the ticket as resolved and replies “ Since your query is resolved, is there anything I may assist you with? ”随后，它将工单关闭并标记为已解决，回复道：“既然您的问题已解决，还有什么我可以帮您的吗？”
Two things are wrong. The carrier exception is still open, so the required end state was on hold, pending resolution. And the customer never got a real answer to what she actually asked.这里有两个问题。快递异常状态仍未解除，因此所需的最终状态应为“挂起”，等待解决。而且，客户对于她实际提出的问题始终没有得到真正的答复。
An AI grader checking tool calls would see nine well-formed ones. The grader checking whether the agent wrote to the database would see that too. The database is what disagrees.检查工具调用的 AI 评分器会看到九次格式正确的调用。检查代理是否向数据库写入数据的评分器也会看到同样的结果。但数据库给出的反馈却截然不同。
That gap is what ThinkingBox measures. Across 507 stateful business workflows, each run 20 times against various LLM models, it grades agents on terminal backend state and side effects. This post covers what we found, what consistency costs, and how to run the benchmark yourself through OpenEnv.这种差距正是 ThinkingBox 所衡量的指标。通过 507 个有状态的业务工作流，每项工作流针对各种 LLM 模型运行 20 次，它会对代理的终端后端状态和副作用进行评分。本文涵盖了我们的发现、一致性所带来的成本，以及如何通过 OpenEnv 自行运行该基准测试。
You can run this one yourself: the example above is adapted from a benchmark task sandbox_external_retail_group1.py:test_case_ST003_006, and the executable check that fails is a single field: the ticket's status is solved where the required end state is hold. The full trace is in Appendix D.4, Case 3 of our paper.您可以亲自运行此测试：上述示例改编自基准测试任务 sandbox_external_retail_group1.py:test_case_ST003_006，其中失败的可执行检查仅涉及一个字段：工单状态为“已解决”，而要求的最终状态应为“挂起”。完整的追踪记录位于我们论文的附录 D.4，案例 3 中。
Contents目录
- A tool call is not an outcome工具调用不等同于结果
- One success is not reliability一次成功不代表可靠
- Can you depend on the model behind your agent?您能依赖代理背后的模型吗？
- What consistency costs一致性的代价
- Failure signatures失败特征
- How it works工作原理
- Run it yourself自行运行
- Where this goes next后续方向
Want to try it before reading the results? Skip to section Run it yourself.想在阅读结果前先试一试？请直接跳转至“自行运行”部分。
A tool call is not an outcome
Final responses and valid tool calls are only proxies. An agent can sound correct while leaving the wrong value, changing the wrong record, or creating an extra side effect. Only the records it leaves behind settle the question.最终响应和有效的工具调用仅仅是代理表现的代理指标。代理听起来可能很正确，但却可能留下了错误的值、修改了错误的记录，或产生了额外的副作用。只有它留下的记录才能最终判定结果。
The gap is substantial. In a common-set ablation covering 121,680 valid trials across 12 LLM models, 79,853 attempts failed the executable checks. Of those failures, 67.24% still terminated cleanly, invoked a state-changing tool, and reported no final tool error. Executable checks nevertheless found wrong field values in 77.61% of them, unintended extra effects in 43.30%, and missing required effects in 25.36%. Those state-check findings overlap.这种差距非常显著。在一项涵盖 12 个 LLM 模型、共 121,680 次有效试验的通用集消融研究中，有 79,853 次尝试未通过可执行检查。在这些失败案例中，67.24% 的尝试依然顺利终止、调用了改变状态的工具，并且未报告任何最终工具错误。尽管如此，可执行检查发现其中 77.61% 存在错误的字段值，43.30% 产生了意料之外的额外影响，25.36% 缺失了必要的影响。这些状态检查结果存在重叠。
A trajectory is a claim. Database state is the evidence. Repetition is the trust test.轨迹是主张，数据库状态是证据，重复测试则是信任的试金石。
One success is not reliability
An agent that processes a refund correctly once and mishandles it the next four times is not a working refund agent. So every task runs 20 independent times, each from an identical clean backend, and we report three different things:一个能正确处理一次退款，但在接下来的四次中处理失误的代理，并不是一个合格的退款代理。因此，每个任务都要在相同的干净后端环境下独立运行 20 次，我们报告三个不同的指标：
Table 1: The three numbers we report, and the question each one answers.表 1：我们报告的三个数字，以及每个数字所回答的问题。
| Metric | What it measures | What it answers |
|---|---|---|
| pass@1 | Share of all attempts that succeeded | How does it usually do? |
| pass@20 | Share of tasks solved at least once in 20 tries | Can it ever do this? Breadth. |
| Observed 20/20 | Tasks that actually passed all 20 recorded attempts | Can it always be correct? |
We use observed 20/20 in this blog post as the literal count of how many of the 507 tasks passed 20 out of 20. No estimator, no smoothing. 我们在本文中使用观察到的 20/20 作为 507 个任务中有多少个任务在 20 次试验中全部成功的字面计数。没有估算，没有平滑处理。
Starting with the familiar view. The table below reports pass@1, the single-attempt score estimate, broken out by domain. This is the number most leaderboards publish, and on its own it reads like an ordinary capability ranking.从熟悉的视角开始。下表报告了按领域细分的 pass@1（单次尝试评分估计）。这是大多数排行榜发布的数据，仅凭此数据，它看起来就像一个普通的性能排名。
Table 2: ThinkingBox-Bench pass@1 (%) by domain. Each model is evaluated on every task for 20 repeated trials. Bold marks the group leader; underline marks the runner-up. The standard errors for the single attempt score estimates are provided in Table 4 in our ThinkingBox paper.表 2：按领域划分的 ThinkingBox-Bench pass@1 (%)。每个模型都在每个任务上进行了 20 次重复试验评估。粗体表示组内领先者；下划线表示亚军。单次尝试评分估计的标准误差详见我们 ThinkingBox 论文中的表 4。
| Model | Retail (98) | Auto insurance (100) | Travel (104) | Neobank (104) | Consulting (101) | Overall, task-weighted (507) |
|---|---|---|---|---|---|---|
| Proprietary models | ||||||
| Claude Opus 5.5 | 80.97 | 68.40 | 54.28 | 71.25 | 61.58 | 67.16 |
| Claude Opus 5 | 80.71 | 65.80 | 49.95 | 70.62 | 66.19 | 66.50 |
| GPT-5.4 | 76.33 | 62.65 | 68.12 | 65.34 | 54.60 | 65.36 |
| GPT-5.6 Sol | 67.65 | 65.30 | 60.34 | 59.09 | 57.52 | 61.91 |
| Claude Sonnet 4.6 | 72.35 | 54.40 | 58.94 | 56.39 | 54.31 | 59.19 |
| GPT-6 Astra | 71.73 | 46.55 | 55.87 | 60.87 | 56.83 | 58.31 |
| GPT-5.2 | 70.20 | 22.40 | 53.70 | 51.15 | 34.06 | 46.28 |
| Claude Opus 4.6 | 68.62 | 8.30 | 21.11 | 35.67 | 27.82 | 32.09 |
| o3-pro | 37.70 | 2.95 | 17.31 | 24.28 | 14.60 | 19.31 |
| Grok-4.3 | 43.93 | 2.60 | 15.14 | 1.78 | 9.55 | 14.38 |
| Open-weight models | ||||||
| Kimi-K3 | 82.24 | 50.80 | 61.83 | 41.35 | 51.63 | 57.37 |
| Qwen3.8-27B | 64.03 | 47.85 | 53.41 | 47.88 | 45.69 | 51.70 |
| DeepSeek-V4-Pro | 68.21 | 29.65 | 43.13 | 44.86 | 31.04 | 43.26 |
| Kimi-K2.6 | 53.72 | 24.50 | 39.52 | 33.65 | 37.33 | 37.66 |
| GLM-5.1 | 58.67 | 25.70 | 35.43 | 13.27 | 34.06 | 33.19 |
| Qwen3.6-27B | 43.11 | 29.00 | 46.39 | 27.84 | 18.37 | 32.94 |
| Qwen3.5-9B | 19.90 | 0.70 | 4.71 | 1.15 | 2.33 | 5.65 |
| Mistral-Large-3 | 11.28 | 1.30 | 8.99 | 1.15 | 0.74 | 4.66 |
Claude Opus 5.5 leads overall at 67.16%, two-thirds of a point above Claude Opus 5. Kimi-K3 is the strongest open-weights model, within a point of GPT-6-Astra. Domain matters just as much: Claude Opus 4.6 scores 68.62% on retail but 8.30% on auto insurance.Claude Opus 5.5 以 67.16% 的成绩整体领先，比 Claude Opus 5 高出三分之二个百分点。Kimi-K3 是最强的开源权重模型，与 GPT-6-Astra 的差距不到一个百分点。领域同样重要：Claude Opus 4.6 在零售领域得分为 68.62%，但在汽车保险领域仅为 8.30%。
One good run tells you a model can do the work. It does not tell you whether it will do it again. So run every task 20 times and ask how much of that score survives.一次良好的运行只能说明模型能完成这项工作，并不能说明它是否能再次完成。因此，请将每个任务运行 20 次，看看该分数中有多少能够保持不变。
Figure 2: How much of each model's single-attempt score survives 20 repeats.图 2：每个模型的单次尝试得分在 20 次重复后保留了多少。
Only three hold on to most of their pass@1 scores: GPT-6 Astra retains 78% of its single-attempt rate, and Claude Opus 5.5 and Claude Opus 5 each retain 71%. At the other end, GLM-5.1, Kimi-K2.6 and DeepSeek-V4-Pro each keep about 8%.只有三个模型保留了大部分 pass@1 得分：GPT-6 Astra 保留了 78% 的单次尝试成功率，Claude Opus 5.5 和 Claude Opus 5 各保留了 71%。而在另一端，GLM-5.1、Kimi-K2.6 和 DeepSeek-V4-Pro 各自仅保留了约 8%。
The gap between what a model can do once and what it does every time is the whole story.模型一次能做什么与它每次都能做什么之间的差距，才是整个故事的核心。
Can you depend on the model behind your agent?
Figure 3: Breadth and consistency pull apart. Twelve of the eighteen models are shown; six below 33% pass@1 are omitted for legibility.图 3：广度与一致性的分化。图中展示了 18 个模型中的 12 个；为清晰起见，省略了 pass@1 低于 33% 的 6 个模型。
Kimi-K3 has the broadest coverage of any model we tested. It solves 93.89% of the benchmark at least once: 476 of 507 tasks. Only 31 tasks defeat it entirely, the lowest count in the field. On retail workflows it leads outright at 82.24% pass@1, ahead of every proprietary model.Kimi-K3 拥有我们测试过的模型中最广泛的覆盖范围。它至少成功解决了一次 93.89% 的基准测试任务：507 个任务中有 476 个。只有 31 个任务能完全难倒它，这是该领域中的最低计数。在零售工作流中，它以 82.24% 的 pass@1 成绩领先，超过了所有专有模型。
Kimi-K3 is also among the least consistent. Just 68 of 507 tasks, 13.41%, succeed in all 20 attempts.但 Kimi-K3 也是最不一致的模型之一。在 507 个任务中，仅有 68 个（13.41%）在全部 20 次尝试中均获得成功。
Claude Opus 5 inverts this. It solves fewer tasks at least once (79.09%; 106 defeat it entirely) but completes 47.53% of the benchmark on every single attempt.Claude Opus 5 的情况则相反。它至少成功解决一次的任务较少（79.09%；有 106 个任务完全难倒了它），但在全部 20 次尝试中都能完成 47.53% 的基准测试任务。
A newer model does not fix this. Claude Opus 5.5 scores higher than Claude Opus 5 on every-attempt average, 67.16% against 66.50%, and solves more tasks at least once. It passes exactly the same number of tasks on all 20 attempts: 241. Half a point of headline accuracy bought no additional dependability at all.更新的模型并不能解决这个问题。Claude Opus 5.5 在全尝试平均得分上比 Claude Opus 5 更高（67.16% 对 66.50%），并且至少成功解决了一次的任务更多。但在全部 20 次尝试中通过的任务数量完全相同：241 个。半个百分点的头条准确率提升并没有带来任何额外的可靠性。
- Kimi-K3 solves 75 more tasks at least once than Opus 5.Kimi-K3 至少成功解决一次的任务比 Opus 5 多 75 个。
- Opus 5 solves 173 more tasks consistently than Kimi-K3.Opus 5 稳定完成的任务比 Kimi-K3 多 173 个。
If you are choosing a model for work that touches real records, pass@20 is the wrong column to look at.如果您正在为涉及真实记录的工作选择模型，pass@20 并不是您应该关注的指标。
What consistency costs
Capability comparisons usually stop at the score. For anyone deploying, the relevant question is what a successful unit of work costs. We measure that as cost per successful task attempt. We say task attempt because every benchmark task is run repeatedly and cost is incurred per attempt, so pass@1 is the matching quality denominator.能力比较通常止步于分数。对于任何部署者来说，相关的问题是：一个成功的任务单元需要付出多少成本。我们将其衡量为“单次成功任务尝试的成本”。我们之所以说“任务尝试”，是因为每个基准测试任务都会重复运行，且成本按尝试次数计算，因此 pass@1 是匹配的质量分母。
We took each model's recorded token usage from its full 507 × 20 campaign and priced it at undiscounted list rates available on OpenRouter+, reversing promotional discounts and excluding endpoints that declare quantization. Input, output and cache rates all come from one provider endpoint per model.我们采用了每个模型在 507 × 20 次完整活动中记录的代币使用量，并按照 OpenRouter+ 上提供的未折扣标价进行定价，剔除了促销折扣，并排除了声明量化的端点。输入、输出和缓存费率均来自每个模型的一个提供商端点。
Then we divided one run's cost by the number of attempts that succeeded:然后，我们将一次运行的成本除以成功的尝试次数：
Cost per successful task attempt = estimated cost for 507 attempts, one per task ÷ (507 × pass@1)单次成功任务尝试成本 = 507 次尝试（每个任务一次）的预估成本 ÷ (507 × pass@1)
This is a comparative efficiency index, not an invoice, and not the price of serving one production request. It also prices single successes, not consistency. We price consistency next.这是一个比较效率指数，而非发票，也不是服务一个生产请求的价格。它定价的是单次成功，而非一致性。我们接下来将对一致性进行定价。
Example: GPT-5.4 costs $43.49 for 507 attempts (one attempt per task) and has 65.36% pass@1, so $43.49 ÷ (507 × 0.6536) = $0.131 per successful task attempt.示例：GPT-5.4 进行 507 次尝试（每个任务一次）的成本为 43.49 美元，pass@1 为 65.36%，因此 43.49 美元 ÷ (507 × 0.6536) = 0.131 美元/成功任务尝试。
Pareto cost frontier 帕累托成本前沿
A model is on the frontier if no other model is both no more expensive and at least as accurate. Three models qualify; every other model is dominated on at least one axis.如果没有任何其他模型在价格不更高的情况下准确率至少相同，则该模型处于前沿。有三个模型符合条件；其他所有模型在至少一个维度上都处于劣势。
Figure 4: Cost per successful task attempt against pass@1. Ringed dots are pareto cost frontier models.图 4：单次成功任务尝试成本与 pass@1 的对比。带圆圈的点是帕累托成本前沿模型。
The frontier has three steps. GPT-5.6 Sol has the lowest cost per success at $0.127; GPT-5.4 raises pass@1 by 3.45 percentage points for $0.004 more per success; Claude Opus 5.5 adds another 1.80 points at $0.276 per success. Each of the three remains on the cost frontier line because no cheaper model matches its pass@1.前沿有三个步骤。GPT-5.6 Sol 的单次成功成本最低，为 0.127 美元；GPT-5.4 的 pass@1 提高了 3.45 个百分点，但每次成功多花费 0.004 美元；Claude Opus 5.5 又提高了 1.80 个百分点，每次成功花费 0.276 美元。这三个模型中的每一个都保持在成本前沿线上，因为没有更便宜的模型能达到与之相当的 pass@1。
Claude Opus 5 is the clearest case: at $0.475 per successful attempt and 66.50% pass@1, it is both more expensive and less accurate than Claude Opus 5.5 at $0.276 and 67.16%.Claude Opus 5 是最明显的案例：每次成功尝试成本为 0.475 美元，pass@1 为 66.50%，相比之下，Claude Opus 5.5 每次成功尝试成本为 0.276 美元，pass@1 为 67.16%，前者既昂贵又不准确。
Now price consistency 现在为一致性定价
Cost per success rewards a model that is cheap and often right. It does not reward a model that is right every time. So we also compute cost per dependable task: the cost of the full 20-run campaign divided by the number of tasks the model passed on all 20 attempts.单次成功成本奖励的是既便宜又经常正确的模型。它不奖励每次都正确的模型。因此，我们还计算了“可靠任务成本”：整个 20 次运行活动的成本除以模型在全部 20 次尝试中均通过的任务数量。
Cost per dependable task = estimated cost of 20 runs of 507 attempts ÷ tasks passing 20/20可靠任务成本 = 20 次 507 次尝试运行的预估成本 ÷ 20/20 通过的任务数
Example: GPT-6-Astra costs 20 × $86.03 = $1,720.60 for the campaign and passes 231 tasks on every attempt, so $1,720.60 ÷ 231 = $7.45 per dependable task.示例：GPT-6-Astra 该活动的成本为 20 × 86.03 美元 = 1,720.60 美元，并且在每次尝试中通过了 231 个任务，因此 1,720.60 美元 ÷ 231 = 7.45 美元/可靠任务。
Table 3: The nine lowest costs per dependable task among models with at least one observed 20/20 task, sorted low to high. Estimated $, not actual cloud bills.表 3：在至少有一个观察到 20/20 任务的模型中，可靠任务成本最低的九个模型，按从低到高排序。为预估美元金额，非实际云账单。
| Model | Tasks passing 20/20 | Est. cost, 20 runs | Cost per dependable task |
|---|---|---|---|
| GPT-5.4 | 128 (25.25%) | $869.80 | $6.80 |
| GPT-6 Astra | 231 (45.56%) | $1,720.60 | $7.45 |
| Claude Opus 5.5 | 241 (47.53%) | $1,880.77 | $7.80 |
| GPT-5.6 Sol | 82 (16.17%) | $800.00 | $9.76 |
| Claude Opus 5 | 241 (47.53%) | $3,206.00 | $13.30 |
| Claude Sonnet 4.6 | 102 (20.12%) | $1,587.60 | $15.56 |
| GPT-5.2 | 44 (8.68%) | $878.00 | $19.95 |
| Kimi-K3 | 68 (13.41%) | $1,406.40 | $20.68 |
| Qwen3.8-27B | 38 (7.50%) | $925.80 | $24.36 |
Now rank by consistency. GPT-5.4 is the cheapest at $6.80, though only 128 tasks meet the bar. GPT-6 Astra reaches 231 at $7.45, and Claude Opus 5.5 the joint-highest 241 at $7.80.现在按一致性排名。GPT-5.4 最便宜，为 6.80 美元，尽管只有 128 个任务达到标准。GPT-6 Astra 达到 231 个任务，成本为 7.45 美元，Claude Opus 5.5 达到并列最高的 241 个任务，成本为 7.80 美元。
None of the three dominates the others: each additional dependable task costs more. Claude Opus 5 also passes 241, but at $13.30, so Opus 5.5 dominates it outright. GPT-5.6 Sol, the cheapest per single success at $0.127, costs $9.76 per dependable task. The cheapest way to get a right answer is not the cheapest way to get a dependable one.这三个模型中没有一个能完全占据优势：每增加一个可靠任务都需要付出更多成本。Claude Opus 5 也通过了 241 个任务，但成本为 13.30 美元，因此 Opus 5.5 在此维度上完全胜出。单次成功成本最低的 GPT-5.6 Sol（0.127 美元）在可靠任务上的成本为 9.76 美元。获得正确答案的最便宜方式，并非获得可靠答案的最便宜方式。
Failure signatures
We assign each failed trace one deterministic diagnostic signature, and the headline is actionable: roughly four in five failures are tool handling, not reasoning. Across an ablation study in Table 5 of our paper:我们为每个失败的轨迹分配一个确定性的诊断特征，其核心结论是可操作的：大约五分之四的失败是工具处理问题，而非推理问题。根据我们论文表 5 中的消融研究：
| Failure signature | Share of failures |
|---|---|
| Tool usage | 79.9% |
| Wrong state updates | 10.3% |
| Incomplete user resolutions | 7.0% |
| No state-changing action | 2.9% |
These are unweighted averages of per-model shares and observable labels, not unique causal explanations.这些是各模型份额和可观察标签的未加权平均值，并非唯一的因果解释。
The practical pattern is simple: agents usually get far enough to attempt the workflow, then fail to recover from tool errors, failed preconditions, or empty lookups. That is a retry and error-recovery problem before it is a model problem.实际模式很简单：代理通常能进行到足以尝试工作流的程度，然后因无法从工具错误、前提条件失败或空查询中恢复而失败。这是一个重试和错误恢复问题，而非模型问题。
Difficulty also changes by domain: across the models listed in Table 2 above, retail averages 59.52% pass@1 while auto insurance averages 33.83%.难度也因领域而异：在上述表 2 所列模型中，零售领域的平均 pass@1 为 59.52%，而汽车保险领域的平均 pass@1 为 33.83%。
What to do about it. Treat the 20/20 rate as a design input, not a verdict. The same signal the benchmark grades on is available in production: check the terminal state before you commit, not the model's summary of it.应对之道：将 20/20 的成功率视为设计输入，而非最终判决。基准测试评分所依据的信号在生产环境中同样可用：在提交前检查终端状态，而不是检查模型对其的总结。
Classify tool and system errors so retries target the recoverable ones. Cut the tool surface to what the workflow needs. And require human approval on the changes you cannot cheaply reverse. We have not measured the lift from any of these on this benchmark, which is exactly the kind of thing the environment now makes testable.对工具和系统错误进行分类，以便重试能够针对可恢复的错误。将工具范围缩减至工作流所需的程度。对于无法低成本撤销的更改，要求人工审批。我们尚未测量这些措施在基准测试中的提升效果，但这恰恰是该环境现在可以测试的内容。
How it works
ThinkingBox is the agent sandbox, while ThinkingBox-Bench is a dataset benchmark to evaluate agents. The diagram at the top of this post shows the loop; here is what each part does.ThinkingBox 是代理沙盒，而 ThinkingBox-Bench 是用于评估代理的数据集基准。本文顶部的图表展示了循环过程；以下是各部分的功能。
Figure 5: The sandbox loop from panel A in Figure 1 above: isolated tool session, terminal database state, side effects, executable judges.图 5：图 1 A 面板中的沙盒循环：隔离工具会话、终端数据库状态、副作用、可执行判断器。
Each task defines a starting backend state, a user goal, the available MCP tools, the domain policy, and executable checks over the terminal state. A simulated user holds private context (a booking reference, a preference, or a date of birth) and releases it only when asked.每个任务都定义了起始后端状态、用户目标、可用的 MCP 工具、领域政策以及针对终端状态的可执行检查。模拟用户持有私有上下文（预订参考号、偏好或出生日期），且仅在被问及时才会释放。
Every attempt gets an isolated MCP session with freshly initialized state. Two attempts of the same task never share a database row or cached tool state, which is what makes 20-trial comparison meaningful.每一次尝试都会获得一个具有初始状态的独立 MCP 会话。同一任务的两次尝试绝不会共享数据库行或缓存的工具状态，这正是 20 次试验比较具有意义的原因。
At the end, a side-effect extractor derives what actually changed, and deterministic judges compare it against the required end state, accepting any trajectory that produces the right outcome while rejecting wrong, missing or extra effects. For requirements with no clean database value ("did the agent disclose this is not guaranteed?"), a narrow binary rubric question handles the semantics. 477 of the 507 tasks are graded on state alone; 30 add response rubrics.最后，副作用提取器会推导出实际发生了什么变化，确定性判断器将其与要求的最终状态进行比较，接受任何产生正确结果的轨迹，同时拒绝错误、缺失或额外的效果。对于没有明确数据库值的要求（“代理是否披露了这无法保证？”），则通过一个狭义的二元规则问题来处理语义。507 个任务中有 477 个仅根据状态评分；30 个增加了响应规则。
The trust boundary: the model sees tasks, dialogue and tool schemas. Golden state, assertions, grading internals and credentials stay on the evaluator side.信任边界：模型可以看到任务、对话和工具架构。黄金状态、断言、评分内部逻辑和凭据保留在评估者一侧。
Run it yourself
ThinkingBox is now on Hugging Face, both the harness and the dataset. ThinkingBox-Bench now sits behind the OpenEnv interface, and each finished episode returns a binary pass/fail reward. The released adapter is designed for evaluation; separate, non-benchmark scenarios can use the same interface in training workflows.ThinkingBox 现在已在 Hugging Face 上线，包括工具套件和数据集。ThinkingBox-Bench 现在位于 OpenEnv 接口之后，每个完成的情节都会返回一个二元通过/失败奖励。发布的适配器专为评估而设计；独立的非基准测试场景可以在训练工作流中使用相同的接口。
Before you start 开始之前
Tested on Linux and WSL, with Python 3.11+, uv and Docker. You also need a thinkingbox-data checkout at the pinned release and model endpoints for the agent, simulated user and judge. One endpoint can serve all three roles, which is the simplest way to start. The OpenEnv image starts only the OpenEnv API; everything else you run yourself.已在 Linux 和 WSL 上测试，需 Python 3.11+、uv 和 Docker。您还需要在固定版本处检出 thinkingbox-data，并为代理、模拟用户和判断器准备模型端点。一个端点可以同时担任这三个角色，这是最简单的入门方式。OpenEnv 镜像仅启动 OpenEnv API；其他所有内容均由您自行运行。
Install 安装
# 1. OpenEnv + the ThinkingBox environment
git clone https://github.com/huggingface/OpenEnv
cd OpenEnv
uv sync --project envs/thinkingbox_env --frozen
# 2. The executable benchmark, at the pinned release
git clone https://github.com/microsoft/thinkingbox-data
git -C thinkingbox-data checkout thinkingbox-bench-v1.0
# 3. The ThinkingBox CLI, which provides `tb`
uv tool install "thinkingbox @ git+https://github.com/microsoft/thinkingbox"
Start Typesense 启动 Typesense
In a second terminal, start Typesense 30.1 and wait for its health check:在第二个终端中，启动 Typesense 30.1 并等待其健康检查通过：
mkdir -p .typesense-data
docker run --rm -d --name thinkingbox-typesense \
-p 8108:8108 \
-v "$PWD/.typesense-data:/data" \
typesense/typesense:30.1 \
--data-dir /data --api-key=Fake --enable-cors
until curl -fsS http://127.0.0.1:8108/health; do sleep 1; done
Start the MCP servers 启动 MCP 服务器
In a third terminal, start the Session Proxy and MCP servers.在第三个终端中，启动会话代理和 MCP 服务器。
cd OpenEnv
tb mcp-start --host 127.0.0.1 --port 7111 \
--servers "$PWD/thinkingbox-data/servers/servers.yaml"
curl -fsS http://127.0.0.1:7111/health
Start the OpenEnv server 启动 OpenEnv 服务器
Back in the first terminal, start the OpenEnv server against a ThinkingBox YAML config naming your three models (config guide):回到第一个终端，针对命名了您三个模型的 ThinkingBox YAML 配置启动 OpenEnv 服务器（配置指南）：
OPENENV_TB_CONFIG="$PWD/thinkingbox.yaml" \
uv run --project envs/thinkingbox_env --frozen server
Check readiness 检查就绪状态
Gate on readiness before running anything. It returns 503 until its observable data, configuration and Session Proxy checks pass. It cannot observe Typesense or live-probe every model endpoint, so confirm those separately:在运行任何内容之前，先检查就绪状态。在可观察数据、配置和会话代理检查通过之前，它会返回 503。它无法观察 Typesense 或实时探测每个模型端点，因此请单独确认这些内容：
curl -sS http://127.0.0.1:8000/ready
Score an episode 为情节评分
Now score a real episode. example_usage.py only resets and lists tools; for agent actions, effects and assertions use the packaged evaluator:现在为真实情节评分。example_usage.py 仅重置并列出工具；对于代理操作、效果和断言，请使用打包的评估器：
echo "- sandbox_external_retail_group1.py:test_case_ST002_001" > one_task.yaml
uv run --project envs/thinkingbox_env thinkingbox-eval \
one_task.yaml \
--config "$PWD/thinkingbox.yaml" \
--output results.jsonl \
--errors-output errors.jsonl \
--repeat 1 --message-timeout 1800
The OpenEnv adapter writes operational failures to an errors sidecar so they can be rerun rather than silently mixed with model outcomes. A canonical result must resolve or explicitly account for those attempts; we counted system errors as unsuccessful trials.OpenEnv 适配器将操作失败写入错误侧边栏，以便它们可以被重新运行，而不是与模型结果静默混合。规范结果必须解决或明确说明这些尝试；我们将系统错误计为不成功的试验。
Runs are gated on a pinned framework commit, a pinned data release and a bundle hash, so a canonical result is verifiable rather than asserted.运行基于固定的框架提交、固定的数据发布和捆绑哈希，因此规范结果是可验证的，而非断言的。
Where this goes next
The useful part of this work is not our pass@1 leaderboard. It is the environment.这项工作的有用之处不在于我们的 pass@1 排行榜，而在于这个环境。
If you are evaluating an agent that touches real records:如果您正在评估一个触及真实记录的代理：
- Inspect a failure. Find a run that terminated cleanly and still failed, and look at what actually changed in the database. It reframes what your own evals measure.检查失败案例。找到一个顺利终止但仍然失败的运行，看看数据库中实际发生了什么变化。这重新定义了您自己的评估指标。
- Reproduce one task through OpenEnv with your own model.通过 OpenEnv 使用您自己的模型复现一个任务。
- Report a repeat metric, and define it. Whatever k your use case justifies; say whether you are reporting best-of-k or every-of-k, and how you computed it.报告重复指标，并对其进行定义。无论您的用例证明 k 是多少；说明您报告的是 best-of-k 还是 every-of-k，以及您是如何计算的。
Further details can be found in the following links:更多详情请见以下链接：
- Environment: envs/thinkingbox_env环境：envs/thinkingbox_env
- OpenEnv: https://huggingface.co/docs/openenv/environments/thinkingboxOpenEnv：https://huggingface.co/docs/openenv/environments/thinkingbox
- Framework: microsoft/thinkingbox · tutorial框架：microsoft/thinkingbox · 教程
- Benchmark: microsoft/thinkingbox-data · v1.0 release基准测试：microsoft/thinkingbox-data · v1.0 发布
- Dataset viewer: microsoft/ThinkingBox-Bench数据集查看器：microsoft/ThinkingBox-Bench
- Paper: arXiv:2608.19741 or HF论文：arXiv:2608.19741 或 HF
- RL training (coming soon): microsoft/thinkingbox-training强化学习训练（即将推出）：microsoft/thinkingbox-training
ThinkingBox code is MIT-licensed; the benchmark data is CDLA-Permissive-2.0; the OpenEnv environment ships under OpenEnv's BSD-3-Clause.ThinkingBox 代码采用 MIT 许可；基准测试数据采用 CDLA-Permissive-2.0；OpenEnv 环境基于 OpenEnv 的 BSD-3-Clause 发布。
Disclaimer: every task in the public benchmark is a synthetic reconstruction. The workflows and policies are modeled on real AI agentic enterprise patterns; the customers are not real.免责声明：公共基准测试中的每个任务都是合成重建。工作流和政策基于真实的 AI 代理企业模式建模；客户并非真实存在。
ThinkingBox and ThinkingBox-Bench is built by the Microsoft Copilot Studio team in partnership with Toloka with collaborators from the University of Pittsburgh, Northwestern University, Columbia University and UC Irvine who interned at Microsoft. Questions are welcome in the comments section below or on githubThinkingBox 和 ThinkingBox-Bench 由微软 Copilot Studio 团队与 Toloka 合作构建，合作者包括在微软实习的匹兹堡大学、西北大学、哥伦比亚大学和加州大学欧文分校的学生。欢迎在下方评论区或 GitHub 上提出问题。
+ OpenRouter cost snapshot taken on Sept 20th 2026; Opus 5.5 pricing as per the Anthropic site.+ OpenRouter 成本快照拍摄于 2026 年 9 月 20 日；Opus 5.5 定价以 Anthropic 网站为准。
