Enterprise AI benchmarks are broken
Every enterprise software vendor claims its AI agent is production-ready, but how to measure it remains up in the air. 每家企业软件供应商都声称其 AI 智能体已“可投入生产”，但如何衡量这一标准仍悬而未决。
The enterprise AI platform DevRev, led by Nutanix co-founder Dheeraj Pandey, argues that while benchmarks like TAU-Bench and Agent’s Last Exam exist, none of them really test for the work employees actually do, where the ability to work across large context windows is often more important than pure reasoning power.由 Nutanix 联合创始人 Dheeraj Pandey 领导的企业 AI 平台 DevRev 指出，虽然目前已有 TAU-Bench 和 Agent’s Last Exam 等基准测试，但没有一个能真正测试员工的实际工作场景。在这些场景中，处理大规模上下文窗口的能力往往比单纯的推理能力更为重要。
This week, DevRev released the first version of its Enterprise AI Agent Benchmark, an open test that makes the dataset, evaluation harness, judging criteria, results, and raw traces available, so any organization can run it against its own systems.本周，DevRev 发布了其“企业 AI 智能体基准测试”（Enterprise AI Agent Benchmark）的首个版本。这是一项开放式测试，公开了数据集、评估工具、评分标准、测试结果及原始追踪记录，以便任何组织都能将其应用于自身的系统进行测试。
The release covers only the two lowest tiers, L1 and L2, of a four-level framework, and DevRev is counting on the community to help build out the more advanced tiers.此次发布的版本仅涵盖了四级框架中最低的两个级别（L1 和 L2），DevRev 希望借助社区的力量来共同构建更高级别的测试。
DevRev built the benchmark on Terminal Bench, the evaluation harness from the Laude Institute, whose researchers are affiliated with UC Berkeley and Stanford. It had the methodology reviewed by Alexandros Dimakis, a UC Berkeley professor and co-founder of Bespoke Labs, which builds tasks for several of the benchmarks the frontier labs invest in.该基准测试基于 Laude Institute 的评估工具 Terminal Bench 构建，该机构的研究人员与加州大学伯克利分校和斯坦福大学有合作关系。此外，其方法论还经过了加州大学伯克利分校教授、Bespoke Labs 联合创始人 Alexandros Dimakis 的评审；Bespoke Labs 为多家前沿实验室投资的基准测试提供任务构建支持。
The idea for a new test, Pandey and DevRev CTO Ahmed Bashir tell The New Stack, started as irritation. Sitting through conference after conference where every booth advertised the same thing, Bashir kept asking who was checking.Pandey 和 DevRev 首席技术官 Ahmed Bashir 告诉 The New Stack，这一新测试的构想源于他们的不满。在参加了一场又一场会议，看到每个展位都在宣传同样的内容后，Bashir 不断追问：到底谁在进行核查？
“People could say that they have industry-leading AI, and I said, ‘Based on which metric?’ … There is no method. It’s ad hoc.”“人们可以说他们拥有行业领先的 AI，但我会问：‘基于什么指标？’……目前没有任何方法，全都是临时凑出来的。”
“People could say that they have industry-leading AI, and I said, ‘Based on which metric?'” says Bashir. “There is no method. It’s ad hoc. It’s built on this idea that it’s a marketing concept. There is no benchmark. In a world of hundreds, if not thousands, of benchmarks that relate to AI, was there one that actually came to the essence of what enterprise was looking for?”“人们可以说他们拥有行业领先的 AI，但我会问：‘基于什么指标？’”Bashir 说道，“目前没有任何方法，全都是临时凑出来的。这完全是基于一种营销概念。根本没有基准测试。在当今成百上千甚至数以万计的 AI 相关基准测试中，难道就没有一个能真正触及企业核心需求的吗？”
The data is the hard part数据才是难点
The benchmarks that matter to the frontier labs reward exotic reasoning, and enterprise work rarely looks like that, the DevRev team argues.DevRev 团队认为，前沿实验室所看重的基准测试往往奖励奇特的推理能力，而企业工作很少是那样的。
“We are not going to get closer to benchmarking the needs of enterprise by creating more complex tasks,” Bashir says. “What we needed to do was combine a moderate level of task complexity with an inordinate amount of complexity in terms of data organization, and data organization is about the source of the information, the shape of the information, and the permissions that control access to the information.”“通过创造更复杂的任务，我们无法更接近企业需求的基准测试，”Bashir 说，“我们需要做的是，将适度的任务复杂度与极高的数据组织复杂度结合起来。数据组织涉及信息的来源、信息的形态，以及控制信息访问的权限。”
“We are not going to get closer to benchmarking the needs of enterprise by creating more complex tasks.”“通过创造更复杂的任务，我们无法更接近企业需求的基准测试。”
Most enterprise tasks aren’t as hard as a chemistry problem. They’re moderate tasks that eat time because the data they run on is scattered across systems, shaped inconsistently, and locked behind permissions — data silos that never quite went away. Task complexity is a solved-enough problem, but data complexity, Bashir says, is “quite novel” as a thing to measure: “the data itself can be organized, shaped, and access controlled in ways that make it complex to access.”大多数企业任务并不像化学难题那样复杂。它们只是些平庸的任务，之所以耗时，是因为它们所依赖的数据分散在各个系统中、格式不统一，且被权限锁死——那些从未真正消失的数据孤岛。任务复杂度是一个已经得到充分解决的问题，但 Bashir 表示，数据复杂度作为一个衡量指标是“相当新颖”的：“数据本身在组织、形态和访问控制方式上，都可能变得难以获取。”
To turn this into a benchmark, DevRev built what it calls scale-invariant ground truth. The dataset models a single mid-size software company at four sizes: 1x, 4x, 16x, and 64x. The correct answer to every task remains the same across all four. The additional data is noise that a correctly operating agent should never surface. At 1x, roughly 40 percent of the tickets are relevant to a given question. At 16x, only 2.5 percent are.为了将其转化为基准测试，DevRev 构建了所谓的“尺度不变事实基准”（scale-invariant ground truth）。该数据集模拟了一家单一中型软件公司在四种规模下的情况：1x、4x、16x 和 64x。对于每个任务，无论规模如何，正确答案始终保持不变。额外增加的数据只是干扰项，运行正常的智能体不应提取这些数据。在 1x 规模下，约 40% 的工单与给定问题相关；而在 16x 规模下，这一比例仅为 2.5%。
The team argues that this design targets the shortcut most systems rely on. Agents are good at finding a needle in a haystack when the context window is small, but that falls apart at production volume.该团队认为，这种设计针对的是大多数系统所依赖的“捷径”。智能体在上下文窗口较小时擅长“大海捞针”，但一旦达到生产规模，这种能力就会失效。
The benchmark scores agents on three axes: precision (does it get the right answer with a verifiable source and an auditable path?), efficiency (does the cost track the question or the data size?), and safety (are permission boundaries respected and every action traceable?).该基准测试从三个维度对智能体进行评分：精确度（是否通过可验证的来源和可审计的路径得出正确答案？）、效率（成本是随问题复杂度还是数据规模而变？）以及安全性（是否遵守权限边界且每个操作皆可追溯？）。
An independent LLM judge verifies results against published criteria, and every run has to submit its traces alongside its score.一个独立的 LLM（大语言模型）裁判会根据已发布的标准核实结果，且每次运行都必须提交其追踪记录及评分。
Four levels, borrowed from self-driving借鉴自动驾驶的四个级别
The L1–L4 framework describes graduated autonomy the way the levels of self-driving do.L1–L4 框架借鉴了自动驾驶的分级方式，描述了渐进式的自主能力。
Bashir describes an L1 task as retrieval and synthesis across “a maximum of two or three sources,” well-bounded and single- or few-turn. An L2 task “usually has to do with multi-step thinking, but it’s still single domain,” he says, where the planning is evident, but the agent still isn’t taking action. L3 moves up to cross-domain problem-solving over a mix of structured and unstructured data, and L4 is full autonomy, the level where, as Pandey puts it, “you have to change code for L4 to work.”Bashir 将 L1 任务描述为跨“最多两到三个来源”的检索与综合，边界明确且仅涉及单轮或少量轮次对话。L2 任务“通常涉及多步思考，但仍属于单一领域”，他说，此时规划过程清晰可见，但智能体尚未采取实际行动。L3 提升至跨领域的问题解决，涵盖结构化与非结构化数据的混合处理；而 L4 则是完全自主，正如 Pandey 所言：“要让 L4 工作，你必须修改代码。”
Pandey frames the progression as a “trifecta” of search, answers, and actions. Enterprise search, he says, “has been commoditized to a large extent,” which is why L1 is where most vendors already play. L2 is “where is my order, where is my cancellation, where is my refund.”Pandey 将这一演进过程概括为搜索、回答和行动的“三位一体”。他认为，企业搜索“在很大程度上已经商品化”，这就是为什么大多数供应商目前处于 L1 阶段。L2 则是“我的订单在哪里、如何取消、如何退款”这类问题。
Actions, the part buyers actually want, start at L3, but DevRev argues most of the market is still stuck at search.行动（用户真正想要的部分）始于 L3，但 DevRev 认为市场大部分仍停留在搜索阶段。
L3 and L4, larger datasets, and voice scenarios are on the roadmap, but Bashir says the company held them back to draw third-party submissions first. “We’d love to see a progression in the submission, so that it’s not just our submission but also third-party submissions,” says Bashir.L3 和 L4、更大的数据集以及语音场景都在路线图中，但 Bashir 表示公司暂缓了这些功能的发布，以便先吸引第三方提交测试。“我们希望能看到提交内容的演进，不仅仅是我们自己的，还有第三方的提交，”Bashir 说。
Same model, different outcomes相同的模型，不同的结果
To show the benchmark in practice, DevRev ran its own agent, Computer (an AI agent that works across enterprise systems on top of a shared, permission-aware memory layer), head-to-head against Claude Code. Both ran on the same Opus model family, were judged the same way, and performed identical L1–L2 tasks via protocol-faithful replicas of vendor APIs and Model Context Protocol (MCP) servers. Computer ranked first overall.为了展示该基准测试的实际应用，DevRev 将其自有的智能体 Computer（一个在共享、具备权限感知能力的内存层之上，跨企业系统工作的 AI 智能体）与 Claude Code 进行了对比测试。两者均运行在相同的 Opus 模型系列上，采用相同的评分方式，并通过供应商 API 和模型上下文协议 (MCP) 服务器的忠实副本执行相同的 L1–L2 任务。最终 Computer 排名第一。
In the L1–L2 results, DevRev’s Computer was more accurate by 22 to 35 points at every data scale tested, and it reached those answers on far fewer tokens: 268,000 per correct answer against Claude Code’s 902,000, or 3.4 times fewer. Scaling the dataset from 1x to 64x increased Computer’s token consumption by 11 percent, while Claude Code’s increased by 55 percent.在 L1–L2 的结果中，DevRev 的 Computer 在每个测试数据规模下的准确率都高出 22 到 35 个百分点，且在得出答案时消耗的 Token 远少于 Claude Code：每个正确答案消耗 26.8 万个 Token，而 Claude Code 消耗 90.2 万个，即效率高出 3.4 倍。当数据集从 1x 扩展到 64x 时，Computer 的 Token 消耗量仅增加了 11%，而 Claude Code 则增加了 55%。
Because both agents used the same Opus-class model, DevRev attributes the gap to data-retrieval architecture rather than the model itself. The benchmark sorts enterprise AI systems into two kinds: those whose costs rise with data volume, and those whose costs track the complexity of the question. DevRev’s contention is that the first kind doesn’t survive contact with a real enterprise’s data.由于两个智能体均使用相同的 Opus 级模型，DevRev 将这一差距归因于数据检索架构，而非模型本身。该基准测试将企业 AI 系统分为两类：一类是成本随数据量增加而上升的，另一类是成本随问题复杂度而变化的。DevRev 认为，第一类系统无法在真实的企业数据环境中生存。
A vendor’s own yardstick供应商自己的标尺
A vendor grading its own benchmark deserves skepticism, but DevRev did bring in third-party experts to validate it.供应商评价自己的基准测试难免让人怀疑，但 DevRev 确实引入了第三方专家进行验证。
The benchmark’s central variable is noisy data at scale, which rewards selective retrieval and punishes loading everything into context. Not coincidentally, that’s the exact problem DevRev’s architecture is built to solve. The comparison target, Claude Code, is a coding agent pressed into service as a stand-in for a general enterprise agent, not a product Anthropic markets for this job, though Anthropic’s Cowork is built on the same foundation.该基准测试的核心变量是规模化的干扰数据，这奖励了选择性检索，并惩罚了将所有内容加载到上下文中的做法。并非巧合的是，这正是 DevRev 架构旨在解决的问题。对比目标 Claude Code 是一个被用作通用企业智能体替代品的编码智能体，并非 Anthropic 专门为此类工作销售的产品（尽管 Anthropic 的 Cowork 建立在相同基础上）。
Bashir says the company had its tasks vetted outside its own walls, choosing to “submit these tasks for review by not just the Laude Institute, but also Bespoke Labs, so it has been independently verified.”Bashir 表示，公司在内部之外对任务进行了审查，选择“将这些任务提交给 Laude Institute 以及 Bespoke Labs 进行评审，因此它已经过独立验证”。
Publishing the dataset, the harness, and its own traces lets anyone rerun the tests or contest the scores, and the company invites competitors to submit to the public leaderboard. That raises the floor on trust, but it doesn’t change the fact that DevRev chose which axis to measure.发布数据集、评估工具及其自身的追踪记录，让任何人都可以重新运行测试或对分数提出质疑，公司也邀请竞争对手在公共排行榜上提交结果。这提高了信任门槛，但无法改变 DevRev 选择衡量指标这一事实。
DevRev argues that this is the first open, vendor-neutral enterprise benchmark. But similar efforts exist. Salesforce’s CRMArena-Pro, out in 2025, simulates Salesforce orgs and grades agents on accuracy, cost, and safety. Sierra’s tau-bench tests tool-agent-user interaction and policy adherence in enterprise-style domains. Both come from vendors, too.DevRev 称这是第一个开放、供应商中立的企业基准测试。但类似的努力也存在。Salesforce 将于 2025 年推出的 CRMArena-Pro 可模拟 Salesforce 组织，并从准确性、成本和安全性方面对智能体进行评分。Sierra 的 tau-bench 则测试企业级领域中的工具-智能体-用户交互和策略合规性。这些也同样来自供应商。
What’s different in DevRev’s benchmark is the data-scale axis and the decision to open-source everything down to the traces, not the idea of an enterprise agent benchmark. Meanwhile, SWE-bench, the coding benchmark that set the template, has been saturating, with frontier models clustered near the top and contamination caveats piling up.DevRev 基准测试的不同之处在于数据规模维度以及开源所有内容（包括追踪记录）的决定，而不是“企业智能体基准测试”这一概念本身。同时，作为基准模板的编码测试 SWE-bench 已趋于饱和，前沿模型聚集在顶端，且污染问题日益严重。
It’s the memory, not the model关键在于内存，而非模型
On the benchmark, Anthropic’s Opus 4.6 and Opus 4.8 perform almost identically. “Both models can perform the task, provided that they have the data,” Bashir says. The models have gotten good enough at enterprise-grade tasks that a newer one barely moves the score, which means the differentiation now comes from somewhere else: the harness.在基准测试中，Anthropic 的 Opus 4.6 和 Opus 4.8 表现几乎相同。“只要有数据，两个模型都能完成任务，”Bashir 说。模型在企业级任务上的表现已经足够好，以至于新版本对分数的提升微乎其微，这意味着差异化现在来自其他方面：评估工具。
“As you have more data sources, the ability to, at any given time, have a deep understanding of all the sources goes down … You’ll start forgetting things that you remembered.”“随着数据来源的增加，在任何给定时间对所有来源进行深度理解的能力就会下降……你会开始遗忘之前记住的事情。”
Bashir points to a minor Claude Code release that shipped prompt improvements aimed specifically at MCP. “It’s actually getting better outside of the release of models,” he says. The frontier labs are optimizing how agents handle data, he argues, not just how they reason.Bashir 指出 Claude Code 的一个小版本更新专门针对 MCP 进行了提示词改进。“它实际上是在模型发布之外变得更好的，”他说。他认为，前沿实验室正在优化智能体处理数据的方式，而不仅仅是优化它们的推理方式。
That matters because, in Bashir’s description, MCP “is a hands-off relationship with the source data,” effectively stateless. “As you have more data sources, the ability to, at any given time, have a deep understanding of all the sources goes down,” he says. “If you have three or four sources, you’re going to start struggling. You’ll start forgetting things that you remembered.”这很重要，因为按照 Bashir 的描述，MCP 是“一种与源数据保持距离的关系”，本质上是无状态的。“随着数据来源的增加，在任何给定时间对所有来源进行深度理解的能力就会下降，”他说，“如果你有三四个来源，就会开始感到吃力。你会开始遗忘之前记住的事情。”
