Evaluating AI Agents Live at the Grounded Reasoning Cup

This year, Databricks hosted the inaugural Grounded Reasoning Cup, a first-of-its-kind live AI competition to evaluate AI agents’ ability to reason over complex, enterprise-style document collections. By testing agents on a newly released corpus under live competition conditions, the Grounded Reasoning Cup was designed to help answer one of the hardest questions in AI evaluation: how well do performance improvements on a benchmark generalize to similar, real-world tasks?今年，Databricks 举办了首届 Grounded Reasoning Cup（基础推理杯），这是一场同类首创的现场 AI 竞赛，旨在评估 AI 智能体在复杂的企业级文档集合中进行推理的能力。通过在现场竞赛条件下对新发布的语料库进行测试，Grounded Reasoning Cup 旨在帮助回答 AI 评估中最棘手的问题之一：基准测试中的性能提升在多大程度上可以泛化到类似的现实任务中？

The competition brought together 11 top academic teams from across the U.S. and Canada, paired with resources and mentorship from frontier labs including OpenAI, Anthropic, and Google DeepMind. Over the course of two months, teams developed and optimized their agents on OfficeQA, our flagship grounded-reasoning benchmark designed to reflect economically valuable enterprise workflows. On competition day, they were challenged to apply those systems in real time to a newly released grounded-reasoning benchmark, OfficeQA Pro V2, designed to test whether their improvements generalized.本次竞赛汇集了来自美国和加拿大的 11 支顶尖学术团队，并获得了包括 OpenAI、Anthropic 和 Google DeepMind 在内的前沿实验室提供的资源与指导。在两个月的时间里，各团队在我们的旗舰基础推理基准 OfficeQA 上开发并优化了各自的智能体，该基准旨在反映具有经济价值的企业工作流程。在比赛当天，他们面临的挑战是将这些系统实时应用于新发布的 OfficeQA Pro V2 基础推理基准，以测试其改进是否具有泛化能力。

Stanford won with a system that achieved 63.3% accuracy, beating out the average team by approximately +22 points, and the average frontier agent offline baseline by approximately +35 points. The top teams demonstrated substantial gains through document preprocessing, targeted retrieval, parallel agents, structured tool use, and verification. At the same time, 18.8% of questions went unsolved by every team, underscoring how much headroom remains in enterprise grounded reasoning.斯坦福大学凭借一套准确率达到 63.3% 的系统夺得冠军，比平均团队成绩高出约 22 个百分点，比平均离线基准前沿智能体高出约 35 个百分点。顶尖团队通过文档预处理、目标检索、并行智能体、结构化工具使用和验证，展现了显著的性能提升。与此同时，所有团队均未能解决 18.8% 的问题，这凸显了企业基础推理领域仍有巨大的提升空间。

Expand

In this blog post, we recap the competition and discuss agent optimization strategies and insights from the Grounded Reasoning Cup’s winning teams: Stanford, University of Massachusetts Amherst, and Yale.在这篇博文中，我们将回顾本次竞赛，并探讨来自 Grounded Reasoning Cup 获胜团队（斯坦福大学、马萨诸塞大学阿默斯特分校和耶鲁大学）的智能体优化策略与见解。

In general, we find the following:总的来说，我们发现了以下几点：

• Generalization requires representative, held-out evaluations. Techniques developed on OfficeQA did not always transfer reliably to our new benchmark. This stresses the importance of utilizing held out test sets like OfficeQA Pro V2 to ensure solutions generalize to new examples.泛化需要具有代表性的留出评估。在 OfficeQA 上开发的技术并不总是能可靠地迁移到我们的新基准测试中。这强调了利用像 OfficeQA Pro V2 这样的留出测试集的重要性，以确保解决方案能够泛化到新的示例中。
• Agent performance depends on the full system, not only the model. The average gap between the top scoring and lowest scoring teams using the same model was 30.4 points. Parsing, retrieval, tool use, verification, parallelism, and operational infrastructure all made the difference in whether agents could successfully complete end-to-end grounded reasoning tasks.智能体的性能取决于整个系统，而不仅仅是模型。使用相同模型的最高分团队与最低分团队之间的平均差距为 30.4 分。解析、检索、工具使用、验证、并行处理以及运营基础设施，共同决定了智能体能否成功完成端到端的基础推理任务。
• Enterprise grounded reasoning remains far from solved. Even winning teams struggled with many of the benchmark’s retrieval, parsing, and analytical demands, leaving substantial room for continued research and improvement. We encourage practitioners to use the publicly available OfficeQA benchmark suite to continue advancing this work.企业基础推理远未得到解决。即使是获胜团队在面对基准测试的许多检索、解析和分析需求时也感到吃力，这为持续的研究和改进留下了巨大的空间。我们鼓励从业者使用公开的 OfficeQA 基准套件来继续推动这项工作。

The Competition Setup竞赛设置

The goal of the Grounded Reasoning Cup was to bring together top academic teams to develop generalizable approaches to grounded reasoning – a common task in enterprise settings that involves answering complex questions using evidence from large, often proprietary, document collections.Grounded Reasoning Cup 的目标是汇集顶尖学术团队，开发可泛化的基础推理方法——这是企业环境中的一项常见任务，涉及使用来自大型（通常是专有的）文档集合的证据来回答复杂问题。

Teams of 2-4 people representing their academic institution were paired with an industry partner from OpenAI, Anthropic, or Google DeepMind, who provided access to their models and mentorship throughout the development period. Teams had approximately two months to build an agent using any approaches they saw fit, with the one constraint that they must use their partner lab’s model family exclusively to power their agent. During this period, they used the OfficeQA benchmark to evaluate new techniques they believed would generalize to similar grounded reasoning tasks.由 2-4 人组成的学术机构团队与来自 OpenAI、Anthropic 或 Google DeepMind 的行业合作伙伴配对，后者在开发期间提供模型访问权限和指导。团队有大约两个月的时间使用他们认为合适的任何方法构建智能体，唯一的限制是他们必须仅使用其合作伙伴实验室的模型系列来驱动其智能体。在此期间，他们使用 OfficeQA 基准来评估他们认为可以泛化到类似基础推理任务中的新技术。

On the day of the competition, teams were then tasked with applying their agents in real time on a new, freshly released benchmark. The competition was governed by the following rules:比赛当天，团队的任务是将他们的智能体实时应用于一个新发布的基准测试。比赛遵循以下规则：

• Benchmark release: The new corpus (the U.S. Treasury’s Accounts of Receipts and Expenditures) was released just 36 hours before the competition. This gave teams time to process and index the data while limiting the opportunity for methods to overfit to the new benchmark.基准发布：新的语料库（美国财政部的收支账目）在比赛开始前 36 小时才发布。这给了团队处理和索引数据的时间，同时限制了方法对新基准进行过拟合的机会。
• Design Constraints: Teams could use any agent framework, corpus version(s), retrieval strategy, tool-use setup, or human-in-the-loop workflow, so long as they used models from their assigned industry lab partner.设计限制：团队可以使用任何智能体框架、语料库版本、检索策略、工具使用设置或人在回路工作流程，只要他们使用分配的行业实验室合作伙伴的模型即可。
• Format: The competition consisted of six 15-minute rounds with 15 questions per round. Rounds became progressively harder over the course of the event.格式：比赛包含六个 15 分钟的回合，每回合 15 个问题。随着比赛进行，回合难度逐渐增加。
• Scoring: Teams received 1 point for each correct answer. To incentivize low latency, they were also awarded a 0.25-point speed bonus if they were the first to answer a given question correctly. Points were worth 2x points in the final round, which consisted of the most challenging questions. Each team was also allowed 3 resubmissions throughout the competition, which they could choose to apply to correct a previous answer.评分：团队每答对一个问题得 1 分。为了激励低延迟，如果他们是第一个正确回答给定问题的团队，还会获得 0.25 分的速度奖励。最后一轮包含最具挑战性的问题，分值加倍。每个团队在整个比赛过程中还有 3 次重新提交的机会，可用于更正之前的答案。

Team Performance & Learnings团队表现与经验教训

The competition made one thing clear: grounded reasoning over enterprise-style document corpora has improved since we released the OfficeQA benchmark 7 months ago, but it is still far from solved. The average team score was ~41%, while the top three teams exceeded 50% accuracy with the Stanford team winning the competition with 63.3% accuracy. These results point towards impressive work from top teams, as well as plenty of remaining headroom to explore.比赛清楚地表明了一点：自 7 个月前我们发布 OfficeQA 基准以来，针对企业级文档语料库的基础推理能力有所提高，但仍远未得到解决。团队平均得分约为 41%，前三名团队的准确率超过了 50%，其中斯坦福大学团队以 63.3% 的准确率赢得了比赛。这些结果既体现了顶尖团队令人印象深刻的工作，也表明仍有大量的提升空间值得探索。

Winning Team Strategies获胜团队策略

While each took a unique approach, several patterns emerged across the agents built by the top three teams. Strong systems tended to combine careful document preprocessing, targeted retrieval, structured tool use, and answer verification steps. In many cases, performance depended less on a single model call and more on the surrounding system: how documents were parsed, how evidence was retrieved, how intermediate calculations were performed, and how answers were verified before submission. While these were qualities the most performant systems generally had in common, they also each employed distinct and creative strategies, as described below.虽然每个团队都采取了独特的方法，但在前三名团队构建的智能体中出现了一些共同模式。强大的系统往往结合了细致的文档预处理、目标检索、结构化工具使用和答案验证步骤。在许多情况下，性能与其说是取决于单一的模型调用，不如说是取决于周边的系统：如何解析文档、如何检索证据、如何执行中间计算，以及在提交前如何验证答案。虽然这些是高性能系统普遍具备的特质，但正如以下所述，每个团队也都采用了独特且富有创造性的策略。

Expand

1st Place: Stanford University第一名：斯坦福大学

Expand

The Stanford team’s winning approach came from turning common grounded reasoning failure modes into reusable operating procedures for their Claude Opus 4.8 Claude Code agent to learn from and apply on the competition questions. During their development on the public OfficeQA benchmark, including experiments ablating with Opus 4.8 and even Fable 5 (while available), the team repeatedly traced wrong answers back to the agent’s exact misstep, and then turned those patterns into skills for table localization, answer formatting, clarifications on common financial wordings, etc. The team also integrated skills that would decide when to search across parsed corpus text and markdown-style document representations and when to fall back to the source PDFs if the parsed text lacked full context. By competition day, Stanford had prepared their agent with a playbook of ~100+ skills. They led all teams in accuracy in the 57 correct answers of 88 they attempted.斯坦福大学团队的获胜方法是将常见的错误推理模式转化为可复用的操作流程，供其 Claude Opus 4.8 Claude Code 智能体学习并在比赛问题中应用。在公共 OfficeQA 基准测试的开发过程中，包括使用 Opus 4.8 甚至 Fable 5（在可用时）进行消融实验，团队反复将错误答案追溯到智能体的具体失误，然后将这些模式转化为表格定位、答案格式化、常见财务措辞澄清等技能。团队还集成了多种技能，以决定何时搜索解析后的语料库文本和 Markdown 风格的文档表示，以及何时在解析后的文本缺乏完整上下文时回退到原始 PDF。到比赛当天，斯坦福大学已经为他们的智能体准备了一套包含 100 多种技能的“剧本”。在他们尝试的 88 个问题中，有 57 个回答正确，准确率领先所有团队。

Despite this in-depth preparation, Stanford still had to adapt its strategy during the competition. In the first three rounds, the team used another Claude Code agent as a verifier to re-extract intermediate values, check common failure modes such as tracing revised values through data lineage and handling unit scaling, and patch calculations when it identified discrepancies. But after earning only two speed bonuses in those rounds, Stanford removed the extra verification pass for the final three. The change significantly reduced latency, helping the team earn speed bonuses on 14 questions and fuel its comeback. The verifier nevertheless proved decisive in the final round, when Stanford switched it back on to correct an answer through a final resubmission, ultimately securing their win.尽管准备充分，斯坦福大学在比赛期间仍不得不调整策略。在前三轮中，团队使用另一个 Claude Code 智能体作为验证器来重新提取中间值，检查常见错误模式（例如通过数据血缘追踪修订值和处理单位缩放），并在发现差异时修补计算结果。但在这些轮次中仅获得两次速度奖励后，斯坦福大学在后三轮中取消了额外的验证步骤。这一改变显著降低了延迟，帮助团队在 14 个问题上获得了速度奖励，并推动了他们的反超。不过，验证器在最后一轮中发挥了决定性作用，当时斯坦福大学重新开启了验证器，通过最后一次重新提交纠正了一个答案，最终确保了胜利。

2nd Place: UMass Amherst第二名：马萨诸塞大学阿默斯特分校

Expand

The UMass team made a bet on speed. They used Claude Opus 4.8 Fast as their primary model and preprocessed the corpus to create a metadata catalog that enabled rapid search and filtering over parsed documents. To improve answer quality while preserving low latency, they ran three agents in parallel on each question, followed by a final Opus verification call to select the best answer.马萨诸塞大学（UMass）团队在速度上进行了博弈。他们使用 Claude Opus 4.8 Fast 作为主模型，并预处理了语料库以创建一个元数据目录，从而能够对解析后的文档进行快速搜索和过滤。为了在保持低延迟的同时提高答案质量，他们对每个问题并行运行三个智能体，随后进行一次最终的 Opus 验证调用以选择最佳答案。

This strategy gave UMass the fastest average submission time for correct answers: four minutes, less than half the team average of eight minutes and 30 seconds. As a result, they earned 36 speed bonuses, worth 0.25 points each, for being the first team to answer correctly, more than twice Stanford’s second-place total of 16. These bonuses helped them build a 10.25-point lead over Stanford by halftime and preserve a 3.75-point advantage entering the final round. UMass held that lead until the final 56 seconds of the competition, when Stanford’s slower, more accurate agent proved decisive on the most challenging questions and pulled ahead to win by 1.75 points.这一策略使 UMass 获得了最快的正确答案平均提交时间：4 分钟，不到团队平均时间（8 分 30 秒）的一半。因此，他们因率先正确回答而获得了 36 次速度奖励（每项 0.25 分），是第二名斯坦福大学总数（16 次）的两倍多。这些奖励帮助他们在半场时领先斯坦福 10.25 分，并在进入最后一轮时保持了 3.75 分的优势。UMass 一直保持领先直到比赛最后 56 秒，当时斯坦福大学较慢但更准确的智能体在最具挑战性的问题上发挥了决定性作用，最终以 1.75 分的优势反超获胜。

UMass’s approach demonstrates how faster models, document preprocessing, and test-time scaling can work together to achieve very low latency without sacrificing too much accuracy.UMass 的方法展示了更快的模型、文档预处理和测试时缩放（test-time scaling）如何协同工作，在不牺牲过多准确率的情况下实现极低的延迟。

3rd Place: Yale University第三名：耶鲁大学

Expand

The Yale team built a multi-arm verification harness designed to remain robust when any individual agent failed. The system ran four independent arms in parallel, spanning two agentic strategies. Two arms used autonomous ReAct agents: one powered by Gemini 3.1 Pro and the other by Gemini 3.5 Flash. The remaining arms used a more structured planner-verifier pipeline, with Gemini 3.1 Pro powering all LLM calls. In this pipeline, the planner inspected and contextualized the source documents, assembling a scratchpad containing the evidence needed to answer the question. The verifier then checked the cited sources and performed the final calculations.耶鲁大学团队构建了一个多臂验证框架，旨在当任何单个智能体失败时仍能保持稳健。该系统并行运行四个独立的分支，涵盖两种智能体策略。其中两个分支使用自主 ReAct 智能体：一个由 Gemini 3.1 Pro 驱动，另一个由 Gemini 3.5 Flash 驱动。其余分支使用更结构化的规划器-验证器流水线，所有 LLM 调用均由 Gemini 3.1 Pro 驱动。在此流水线中，规划器检查并情境化源文档，组装一个包含回答问题所需证据的草稿板。随后，验证器检查引用的来源并执行最终计算。

A Gemini 3.1 Pro meta-verifier reviewed the answers and reasoning produced by all four arms and selected the final response. To reduce the risk of introducing a new unsupported answer, the meta-verifier could only select an answer proposed by one of the existing arms. When it could not do so, the system fell back to a majority vote. By combining distinct agent architectures with independent, partially decorrelated failure modes, Yale emphasized consistent, grounded verification. The approach earned the team third place, with 49 of 90 questions answered correctly and a top-three correctness ranking in four of the competition’s six rounds.一个 Gemini 3.1 Pro 元验证器负责审查所有四个分支产生的答案和推理，并选择最终响应。为了降低引入新的不受支持答案的风险，元验证器只能选择现有分支之一提出的答案。当无法做到这一点时，系统会回退到多数投票。通过将不同的智能体架构与独立的、部分去相关的故障模式相结合，耶鲁大学强调了持续、扎实的验证。这种方法为团队赢得了第三名，90 个问题中有 49 个回答正确，并在比赛的六个回合中有四轮的正确率排名前三。

Summary of Winning Team Strategies获胜团队策略总结

One of the core takeaways from these winning approaches is that applying an agent successfully to end-to-end grounded reasoning tasks requires holistic systems thinking, rather than model choice alone. Across the winning approaches, a few key design levers stood out as especially important:这些获胜方法的核心收获之一是：将智能体成功应用于端到端的基础推理任务需要整体的系统思维，而不仅仅是模型选择。在获胜方案中，几个关键的设计杠杆显得尤为重要：

• Parsing quality: Top teams used pre-parsed documents and often augmented these parsed representations to include additional metadata (e.g. descriptions of charts or page level metadata) and preserved paths back to source PDFs to fall back to when parsed text was incomplete. In our own studies on OfficeQA Pro V2, we find that pre-parsing documents with ai_parse contributes significantly to Genie’s performance improvement of 24.0 pt over baseline frontier agents.解析质量：顶尖团队使用预解析文档，并经常增强这些解析表示以包含额外元数据（例如图表描述或页面级元数据），并保留回溯到原始 PDF 的路径，以便在解析文本不完整时进行回退。在我们对 OfficeQA Pro V2 的研究中，我们发现使用 ai_parse 预解析文档对 Genie 的性能提升（较基准前沿智能体提升 24.0 分）有显著贡献。
• Retrieval quality: Strong systems didn't rely on generic top-k chunk search. They instead used some form of lexical retrieval (e.g. grep), often paired with dense retrieval for a hybrid approach.检索质量：强大的系统并不依赖通用的 top-k 分块搜索。他们通常采用某种形式的词汇检索（例如 grep），并经常与密集检索配对，形成混合方法。
• Tool use: Successful agents generally delegated search, document inspection, calculations, comparison, and submission to specialized tools.工具使用：成功的智能体通常将搜索、文档检查、计算、比较和提交任务委托给专用工具。
• Verification strategies: Several teams built in some form of explicit verification check via a verification agent or LLM call to boost answer quality.验证策略：几个团队通过验证智能体或 LLM 调用内置了某种形式的显式验证检查，以提高答案质量。
• Robust harness: Under live deadline pressure, operational details like preprocessing, retry logic, parallelism, and submission scaffolding were necessary to ensure correct answers actually made it through in time.稳健的框架：在现场截止日期的压力下，预处理、重试逻辑、并行处理和提交脚手架等运营细节对于确保正确答案能及时提交至关重要。

These factors determined whether the model could reliably find the right evidence, perform the right calculation, and submit the right answer under deadline pressure.这些因素决定了模型能否在截止日期压力下可靠地找到正确证据、执行正确计算并提交正确答案。

Impact and Next Steps影响与后续步骤

In summary, the results of the Grounded Reasoning Cup demonstrate a number of broader lessons:综上所述，Grounded Reasoning Cup 的结果展示了许多更广泛的经验教训：

• Generalization must be tested, not assumed. Improvements on OfficeQA did not always transfer to OfficeQA Pro V2, and frontier agents still answered fewer than half of its questions correctly on average. Progress on a familiar benchmark does not necessarily translate to a new corpus or task distribution.泛化必须经过测试，而不是假设。在 OfficeQA 上的改进并不总是能迁移到 OfficeQA Pro V2，且前沿智能体平均仍只有不到一半的问题回答正确。在熟悉基准上的进步并不一定能转化为新的语料库或任务分布上的进步。
• Representative, held-out evaluations are critical. They should evaluate the forms of generalization that matter, ideally keeping core qualities of the task fixed (i.e. grounded reasoning over documents), while varying elements of the task that can change, like the corpus. In our post describing OfficeQA Pro V2, we discuss how we leverage synthetic data, combined with our knowledge of customer workflows, to scale these representative evaluations quickly.具有代表性的留出评估至关重要。它们应该评估重要的泛化形式，理想情况下保持任务的核心质量不变（即基于文档的基础推理），同时改变任务中可能变化的元素，例如语料库。在我们介绍 OfficeQA Pro V2 的文章中，我们讨论了如何结合客户工作流程的知识，利用合成数据快速扩展这些代表性评估。
• Agent performance depends on the full system. Performance depends not only on the model, but also on parsing, retrieval, tool use, verification, and the quality of the underlying data and infrastructure. Winning teams optimized these components holistically. Similarly, Genie significantly outperforms baseline frontier agents when using tools like ai_parse to pre-parse documents, and additional harness optimization strategies.智能体性能取决于整个系统。性能不仅取决于模型，还取决于解析、检索、工具使用、验证以及底层数据和基础设施的质量。获胜团队全面优化了这些组件。同样，当使用 ai_parse 等工具预解析文档以及采用额外的框架优化策略时，Genie 的表现显著优于基准前沿智能体。

We are grateful to all of the teams and industry sponsors, including OpenAI, Google DeepMind, and Anthropic, who participated in Databricks’s inaugural Grounded Reasoning Cup and helped push the field forward. We also thank USAFacts for their continued partnership, from helping us identify a new corpus and create relevant questions for OfficeQA Pro V2, to partnering with us on the competition itself. Finally, we thank the U.S. Treasury for helping identify the U.S. Accounts of Receipts and Expenditures as the foundation for the competition benchmark, and for releasing the dataset as a cohesive collection for the first time.我们感谢所有参与 Databricks 首届 Grounded Reasoning Cup 并推动该领域发展的团队和行业赞助商，包括 OpenAI、Google DeepMind 和 Anthropic。我们也感谢 USAFacts 的持续合作伙伴关系，从帮助我们确定新语料库并为 OfficeQA Pro V2 创建相关问题，到在比赛本身与我们合作。最后，我们感谢美国财政部帮助确定将美国收支账目作为竞赛基准的基础，并首次将该数据集作为统一集合发布。

Authors: Krista Opsahl-Ong, Arnav Singhvi, Josh Joseph, Jasmine Collins, Ivan Zhou, Brooke Wenig, Denny Lee, Michael Bendersky, Erich Elsen, Xing Chen, Matei Zaharia作者：Krista Opsahl-Ong, Arnav Singhvi, Josh Joseph, Jasmine Collins, Ivan Zhou, Brooke Wenig, Denny Lee, Michael Bendersky, Erich Elsen, Xing Chen, Matei Zaharia