Enterprise AI benchmarks are broken
Every enterprise software vendor claims its AI agent is production-ready, but how to measure it remains up in the air. 每个企业软件供应商都声称其AI代理已准备好投入生产，但如何衡量它仍然悬而未决。
The enterprise AI platform DevRev, led by Nutanix co-founder Dheeraj Pandey, argues that while benchmarks like TAU-Bench and Agent’s Last Exam exist, none of them really test for the work employees actually do, where the ability to work across large context windows is often more important than pure reasoning power.由Nutanix联合创始人Dheeraj Pandey领导的企业AI平台DevRev认为，虽然存在TAU-Bench和Agent's Last Exam等基准测试，但其中没有一个真正测试员工实际所做的工作，而在这些工作中，跨大上下文窗口工作的能力往往比纯粹的推理能力更重要。
This week, DevRev released the first version of its Enterprise AI Agent Benchmark, an open test that makes the dataset, evaluation harness, judging criteria, results, and raw traces available, so any organization can run it against its own systems.本周，DevRev发布了其企业AI代理基准测试的首个版本，这是一个开放的测试，公开了数据集、评估工具、评判标准、结果和原始追踪，因此任何组织都可以在自己的系统上运行它。
The release covers only the two lowest tiers, L1 and L2, of a four-level framework, and DevRev is counting on the community to help build out the more advanced tiers.该版本仅涵盖四级框架中最低的两个层级L1和L2，DevRev寄希望于社区帮助构建更高级的层级。
DevRev built the benchmark on Terminal Bench, the evaluation harness from the Laude Institute, whose researchers are affiliated with UC Berkeley and Stanford. It had the methodology reviewed by Alexandros Dimakis, a UC Berkeley professor and co-founder of Bespoke Labs, which builds tasks for several of the benchmarks the frontier labs invest in.DevRev在Terminal Bench（Laude Institute的评估工具）上构建了该基准测试，Laude Institute的研究人员来自加州大学伯克利分校和斯坦福大学。其方法论由加州大学伯克利分校教授、Bespoke Labs联合创始人Alexandros Dimakis审核，Bespoke Labs为前沿实验室投资的多个基准测试构建任务。
The idea for a new test, Pandey and DevRev CTO Ahmed Bashir tell The New Stack, started as irritation. Sitting through conference after conference where every booth advertised the same thing, Bashir kept asking who was checking.Pandey和DevRev首席技术官Ahmed Bashir告诉The New Stack，新测试的想法源于烦恼。在参加一个又一个会议时，每个展位都在宣传同样的事情，Bashir不断追问谁在验证。
“People could say that they have industry-leading AI, and I said, ‘Based on which metric?’ … There is no method. It’s ad hoc.”“人们可以说他们拥有行业领先的AI，而我说，‘基于什么指标？’……没有方法。都是临时性的。”
“People could say that they have industry-leading AI, and I said, ‘Based on which metric?'” says Bashir. “There is no method. It’s ad hoc. It’s built on this idea that it’s a marketing concept. There is no benchmark. In a world of hundreds, if not thousands, of benchmarks that relate to AI, was there one that actually came to the essence of what enterprise was looking for?”“人们可以说他们拥有行业领先的AI，而我说，‘基于什么指标？’”Bashir说道，“没有方法。都是临时性的。它建立在这是一个营销概念的想法上。没有基准测试。在一个有数百甚至数千个与AI相关的基准测试的世界里，是否有一个真正触及了企业所寻找的本质？”
The data is the hard part数据是难点
The benchmarks that matter to the frontier labs reward exotic reasoning, and enterprise work rarely looks like that, the DevRev team argues.DevRev团队认为，对前沿实验室重要的基准测试奖励的是奇特的推理，而企业工作很少是这样的。
“We are not going to get closer to benchmarking the needs of enterprise by creating more complex tasks,” Bashir says. “What we needed to do was combine a moderate level of task complexity with an inordinate amount of complexity in terms of data organization, and data organization is about the source of the information, the shape of the information, and the permissions that control access to the information.”“我们不会通过创建更复杂的任务来更接近企业需求的基准测试，”Bashir说，“我们需要做的是将适度的任务复杂性与在数据组织方面极度的复杂性结合起来，而数据组织涉及信息来源、信息形状以及控制信息访问的权限。”
“We are not going to get closer to benchmarking the needs of enterprise by creating more complex tasks.”“我们不会通过创建更复杂的任务来更接近企业需求的基准测试。”
Most enterprise tasks aren’t as hard as a chemistry problem. They’re moderate tasks that eat time because the data they run on is scattered across systems, shaped inconsistently, and locked behind permissions — data silos that never quite went away. Task complexity is a solved-enough problem, but data complexity, Bashir says, is “quite novel” as a thing to measure: “the data itself can be organized, shaped, and access controlled in ways that make it complex to access.”大多数企业任务并不像化学问题那样难。它们是适度的任务，之所以耗费时间，是因为它们所运行的数据分散在各系统中、形状不一致、且被权限所锁定——这些数据孤岛从未真正消失。任务复杂性是一个已基本解决的问题，但数据复杂性，Bashir说，作为一种衡量对象是“相当新颖的”：“数据本身可以被组织、塑形和访问控制，使其变得复杂难以访问。”
To turn this into a benchmark, DevRev built what it calls scale-invariant ground truth. The dataset models a single mid-size software company at four sizes: 1x, 4x, 16x, and 64x. The correct answer to every task remains the same across all four. The additional data is noise that a correctly operating agent should never surface. At 1x, roughly 40 percent of the tickets are relevant to a given question. At 16x, only 2.5 percent are.为了将其转化为基准测试，DevRev构建了所谓的不变尺度基准真相。该数据集模拟了一个中等规模的软件公司，有四种规模：1x、4x、16x和64x。每个任务在不同规模下的正确答案保持不变。额外的数据是噪音，一个正确运行的代理绝不应将其呈现出来。在1x规模下，大约40%的工单与给定问题相关。在16x规模下，只有2.5%是相关的。
The team argues that this design targets the shortcut most systems rely on. Agents are good at finding a needle in a haystack when the context window is small, but that falls apart at production volume.团队认为，这种设计针对的是大多数系统依赖的捷径。当上下文窗口很小时，代理擅长在干草堆里找针，但在生产规模下就不行了。
The benchmark scores agents on three axes: precision (does it get the right answer with a verifiable source and an auditable path?), efficiency (does the cost track the question or the data size?), and safety (are permission boundaries respected and every action traceable?).该基准测试从三个维度评分代理：精确性（是否获得正确答案，带有可验证的来源和可审计的路径？）、效率（成本是否与问题或数据大小挂钩？）和安全性（是否尊重权限边界，每个动作是否可追溯？）。
An independent LLM judge verifies results against published criteria, and every run has to submit its traces alongside its score.一个独立的LLM评判根据发布的标准验证结果，每次运行都必须提交其追踪记录以及得分。
Four levels, borrowed from self-driving四个层级，借鉴自自动驾驶
The L1–L4 framework describes graduated autonomy the way the levels of self-driving do.L1-L4框架描述了分级自主性，就像自动驾驶的层级一样。
Bashir describes an L1 task as retrieval and synthesis across “a maximum of two or three sources,” well-bounded and single- or few-turn. An L2 task “usually has to do with multi-step thinking, but it’s still single domain,” he says, where the planning is evident, but the agent still isn’t taking action. L3 moves up to cross-domain problem-solving over a mix of structured and unstructured data, and L4 is full autonomy, the level where, as Pandey puts it, “you have to change code for L4 to work.”Bashir将L1任务描述为跨“最多两三个来源”的检索和综合，边界清晰且单轮或少数轮次。L2任务“通常涉及多步思考，但仍然是单一领域”，他说，其中规划是明显的，但代理仍然没有采取行动。L3升级为跨领域的问题解决，涉及结构化和非结构化数据的混合，L4则是完全自主，正如Pandey所说，“你必须更改代码才能让L4工作。”
Pandey frames the progression as a “trifecta” of search, answers, and actions. Enterprise search, he says, “has been commoditized to a large extent,” which is why L1 is where most vendors already play. L2 is “where is my order, where is my cancellation, where is my refund.”Pandey将这一演进描述为搜索、答案和行动的“三重奏”。他说，企业搜索“在很大程度上已经被商品化”，这就是为什么L1是大多数供应商已经涉足的领域。L2是“我的订单在哪里，我的取消在哪里，我的退款在哪里。”
Actions, the part buyers actually want, start at L3, but DevRev argues most of the market is still stuck at search.行动，即买家真正想要的部分，始于L3，但DevRev认为大多数市场仍停留在搜索阶段。
L3 and L4, larger datasets, and voice scenarios are on the roadmap, but Bashir says the company held them back to draw third-party submissions first. “We’d love to see a progression in the submission, so that it’s not just our submission but also third-party submissions,” says Bashir.L3和L4、更大的数据集以及语音场景都在路线图上，但Bashir表示公司暂缓了这些，以首先吸引第三方提交。“我们希望看到提交方面的进展，这样不仅是我们提交，还有第三方提交，”Bashir说。
Same model, different outcomes相同模型，不同结果
To show the benchmark in practice, DevRev ran its own agent, Computer (an AI agent that works across enterprise systems on top of a shared, permission-aware memory layer), head-to-head against Claude Code. Both ran on the same Opus model family, were judged the same way, and performed identical L1–L2 tasks via protocol-faithful replicas of vendor APIs and Model Context Protocol (MCP) servers. Computer ranked first overall.为了展示基准测试的实际应用，DevRev将其自己的代理Computer（一个在共享且感知权限的内存层之上跨企业系统工作的AI代理）与Claude Code进行了正面比较。两者运行在相同的Opus模型系列上，以相同的方式评判，并通过供应商API和模型上下文协议（MCP）服务器的协议忠实复刻执行相同的L1-L2任务。Computer总体排名第一。
In the L1–L2 results, DevRev’s Computer was more accurate by 22 to 35 points at every data scale tested, and it reached those answers on far fewer tokens: 268,000 per correct answer against Claude Code’s 902,000, or 3.4 times fewer. Scaling the dataset from 1x to 64x increased Computer’s token consumption by 11 percent, while Claude Code’s increased by 55 percent.在L1-L2结果中，DevRev的Computer在每个测试数据规模下准确率高出22至35个百分点，并且使用更少的token达到这些答案：每个正确答案268,000个token，而Claude Code为902,000个，少了3.4倍。将数据集从1x扩展到64x，Computer的token消耗增加了11%，而Claude Code增加了55%。
Because both agents used the same Opus-class model, DevRev attributes the gap to data-retrieval architecture rather than the model itself. The benchmark sorts enterprise AI systems into two kinds: those whose costs rise with data volume, and those whose costs track the complexity of the question. DevRev’s contention is that the first kind doesn’t survive contact with a real enterprise’s data.由于两个代理使用相同的Opus类模型，DevRev将差距归因于数据检索架构而非模型本身。该基准测试将企业AI系统分为两类：成本随数据量上升的系统，以及成本随问题复杂性变化的系统。DevRev的主张是，第一种系统无法在真实企业数据的接触中存活。
A vendor’s own yardstick供应商自己的标尺
A vendor grading its own benchmark deserves skepticism, but DevRev did bring in third-party experts to validate it.供应商对自己的基准测试进行评分值得怀疑，但DevRev确实引入了第三方专家进行验证。
The benchmark’s central variable is noisy data at scale, which rewards selective retrieval and punishes loading everything into context. Not coincidentally, that’s the exact problem DevRev’s architecture is built to solve. The comparison target, Claude Code, is a coding agent pressed into service as a stand-in for a general enterprise agent, not a product Anthropic markets for this job, though Anthropic’s Cowork is built on the same foundation.基准测试的核心变量是大规模噪音数据，这奖励选择性检索，惩罚将所有内容加载到上下文中。这并非巧合，这正是DevRev架构旨在解决的问题。比较对象Claude Code是一个编码代理，被用作通用企业代理的替代品，并非Anthropic为此任务推广的产品，尽管Anthropic的Cowork建立在相同的基础上。
Bashir says the company had its tasks vetted outside its own walls, choosing to “submit these tasks for review by not just the Laude Institute, but also Bespoke Labs, so it has been independently verified.”Bashir表示，公司将其任务提交给外部审核，选择“将这些任务提交给不仅Laude Institute，还有Bespoke Labs进行审查，因此得到了独立验证。”
Publishing the dataset, the harness, and its own traces lets anyone rerun the tests or contest the scores, and the company invites competitors to submit to the public leaderboard. That raises the floor on trust, but it doesn’t change the fact that DevRev chose which axis to measure.公开数据集、评估工具及其自身追踪记录，任何人都可以重新运行测试或质疑分数，公司邀请竞争对手提交到公开排行榜。这提高了信任底线，但并未改变DevRev选择了测量哪些维度的事实。
DevRev argues that this is the first open, vendor-neutral enterprise benchmark. But similar efforts exist. Salesforce’s CRMArena-Pro, out in 2025, simulates Salesforce orgs and grades agents on accuracy, cost, and safety. Sierra’s tau-bench tests tool-agent-user interaction and policy adherence in enterprise-style domains. Both come from vendors, too.DevRev辩称这是第一个开放、供应商中立的企业基准测试。但类似的努力也存在。Salesforce的CRMArena-Pro于2025年发布，模拟Salesforce组织，并根据准确性、成本和安全性对代理进行评分。Sierra的tau-bench测试工具-代理-用户交互和企业风格领域中的策略遵守。两者也都来自供应商。
What’s different in DevRev’s benchmark is the data-scale axis and the decision to open-source everything down to the traces, not the idea of an enterprise agent benchmark. Meanwhile, SWE-bench, the coding benchmark that set the template, has been saturating, with frontier models clustered near the top and contamination caveats piling up.DevRev基准测试的不同之处在于数据规模轴以及将所有内容开源到追踪记录的决定，而非企业代理基准测试的概念。与此同时，SWE-bench，这个设定模板的编码基准测试，已经趋于饱和，前沿模型聚集在顶部，污染警告不断堆积。
It’s the memory, not the model是记忆，而非模型
On the benchmark, Anthropic’s Opus 4.6 and Opus 4.8 perform almost identically. “Both models can perform the task, provided that they have the data,” Bashir says. The models have gotten good enough at enterprise-grade tasks that a newer one barely moves the score, which means the differentiation now comes from somewhere else: the harness.在基准测试中，Anthropic的Opus 4.6和Opus 4.8表现几乎相同。“两个模型都能执行任务，只要它们有数据，”Bashir说道。模型在企业级任务上已经足够好，以至于较新的模型几乎不影响分数，这意味着差异化现在来自其他地方：评估工具。
“As you have more data sources, the ability to, at any given time, have a deep understanding of all the sources goes down … You’ll start forgetting things that you remembered.”“当你有更多数据源时，在任何给定时间深入理解所有源的能力就会下降……你会开始忘记你曾经记住的事情。”
Bashir points to a minor Claude Code release that shipped prompt improvements aimed specifically at MCP. “It’s actually getting better outside of the release of models,” he says. The frontier labs are optimizing how agents handle data, he argues, not just how they reason.Bashir指出，Claude Code的一个小版本发布了专门针对MCP的提示改进。“它实际上在模型发布之外也在变得更好，”他说。他认为，前沿实验室正在优化代理处理数据的方式，而不仅仅是它们如何推理。
That matters because, in Bashir’s description, MCP “is a hands-off relationship with the source data,” effectively stateless. “As you have more data sources, the ability to, at any given time, have a deep understanding of all the sources goes down,” he says. “If you have three or four sources, you’re going to start struggling. You’ll start forgetting things that you remembered.”这很重要，因为在Bashir的描述中，MCP“与源数据是一种不干涉的关系”，实际上是无状态的。“当你有更多数据源时，在任何给定时间深入理解所有源的能力就会下降，”他说。“如果你有三四个源，你就会开始挣扎。你会开始忘记你曾经记住的事情。”
