Introducing OfficeQA Pro V2: A New Benchmark for Enterprise Grounded-Reasoning

Today, we are releasing OfficeQA Pro V2, a new benchmark designed to evaluate whether AI agents can generalize to unfamiliar, enterprise-style grounded-reasoning tasks.今天，我们发布了 OfficeQA Pro V2，这是一个全新的基准测试，旨在评估 AI 智能体是否能够泛化到陌生的、企业级的扎根推理任务中。

Seven months ago, we introduced the OfficeQA benchmark to measure how well AI systems answer analytical questions using evidence from large document collections, an extremely common and important enterprise task that we found agents struggled with. Since its introduction, OfficeQA, and its frontier subset, OfficeQA Pro, has become an important measure for frontier model and agent capabilities, driving progress in document retrieval, parsing, and analytical reasoning. But this progress raises a fundamental question: do these improvements reflect broader advances in grounded reasoning, or progress specific to one corpus and task distribution? This distinction matters in enterprise settings, where agents rarely operate on a single, stable document collection.七个月前，我们推出了 OfficeQA 基准测试，用于衡量 AI 系统利用大型文档集合中的证据回答分析性问题的能力——这是我们发现智能体难以应对的一种极其常见且重要的企业任务。自发布以来，OfficeQA 及其前沿子集 OfficeQA Pro 已成为衡量前沿模型和智能体能力的重要指标，推动了文档检索、解析和分析推理方面的进展。但这种进展引发了一个根本性问题：这些改进反映的是扎根推理能力的广泛提升，还是仅针对特定语料库和任务分布的进步？在企业环境中，这种区别至关重要，因为智能体很少在单一、稳定的文档集合上运行。

Our new benchmark, OfficeQA Pro V2, is designed to test that generalization directly. We initially developed OfficeQA Pro V2 as the benchmark for the inaugural Databricks Grounded Reasoning Cup, a live competition in which 11 academic teams, supported by OpenAI, Anthropic, and Google DeepMind, built agents that were evaluated on a previously unseen corpus and set of grounded-reasoning tasks. The benchmark contains 90 questions grounded in approximately 120,000 pages from the U.S. Treasury’s Accounts of Receipts and Expenditures, which was released as a cohesive dataset by the U.S. Treasury for the first time in conjunction with the 250th anniversary of the United States. Now, we are excited to release the benchmark to AI practitioners more broadly to use for their own development.我们的新基准测试 OfficeQA Pro V2 旨在直接测试这种泛化能力。我们最初将 OfficeQA Pro V2 开发为首届 Databricks 扎根推理杯（Grounded Reasoning Cup）的基准测试。这是一场现场竞赛，11 支学术团队在 OpenAI、Anthropic 和 Google DeepMind 的支持下构建了智能体，并在一个此前未见的语料库和一系列扎根推理任务上进行了评估。该基准测试包含 90 个问题，这些问题基于美国财政部《收支账目》（Accounts of Receipts and Expenditures）中约 120,000 页的文档。这是美国财政部为纪念美国建国 250 周年，首次将这些文档作为统一数据集发布。现在，我们很高兴向更广泛的 AI 从业者发布该基准测试，供其在开发中使用。

Expand

OfficeQA Pro V2 remains challenging for today’s AI systems. Out-of-the-box frontier agents using Claude Code with Claude Opus 4.8 and Claude Fable 5, or Codex with GPT-5.5 and GPT-5.6 Sol, achieved an average accuracy of 37.5%. Agents developed specifically for the Grounded Reasoning Cup performed better, averaging 41.1%, with the winning team reaching 63.3%. We also evaluated Databricks Genie, Databricks’ AI agent designed to answer complex questions over enterprise data, using the same underlying models. Genie, which also uses Databricks’ ai_parse to pre-parse the document corpus, improved accuracy by an average of 24.0 percentage points over the default harnesses, with the strongest configuration reaching 60%. Together, these results show that grounded reasoning remains far from solved, but that the right agent harness can unlock significant gains from existing frontier models.OfficeQA Pro V2 对当前的 AI 系统来说仍然具有挑战性。使用 Claude Code 配合 Claude Opus 4.8 和 Claude Fable 5，或使用 Codex 配合 GPT-5.5 和 GPT-5.6 Sol 的现成前沿智能体，平均准确率仅为 37.5%。专门为扎根推理杯开发的智能体表现更好，平均准确率为 41.1%，获胜团队达到了 63.3%。我们还使用相同的底层模型评估了 Databricks Genie（Databricks 旨在回答企业数据复杂问题的 AI 智能体）。Genie 还利用 Databricks 的 ai_parse 对文档语料库进行了预解析，其准确率比默认工具平均提高了 24.0 个百分点，最强配置达到了 60%。这些结果共同表明，扎根推理远未得到解决，但合适的智能体工具可以从现有的前沿模型中获得显著的性能提升。

Below, we describe in more detail how agents perform on OfficeQA Pro V2, how it was built, how it differs from the original OfficeQA benchmark.下面，我们将更详细地介绍智能体在 OfficeQA Pro V2 上的表现、该基准测试的构建方式，以及它与最初的 OfficeQA 基准测试有何不同。

Agent Performance on OfficeQA Pro V2智能体在 OfficeQA Pro V2 上的表现

Expand

We first evaluated frontier models using each model provider’s associated harness, with Codex for GPT models and Claude Code for Claude models, using the same deterministic exact-match metric with 0.0% tolerance metric as used for OfficeQA. Across the five models evaluated under both harnesses, these baseline configurations achieved 26.0% accuracy on average. Performance varied substantially by model, and higher cost did not consistently translate into greater accuracy. For example, Sonnet 5 on Claude Code scored 15.6% at $5.01 per rollout, while GPT-5.6 Sol on Codex reached 33.3% at a comparable $4.70.我们首先使用每个模型提供商的相关工具评估了前沿模型（GPT 模型使用 Codex，Claude 模型使用 Claude Code），并使用了与 OfficeQA 相同的确定性精确匹配指标（0.0% 容差）。在两种工具下评估的五个模型中，这些基准配置的平均准确率仅为 26.0%。性能因模型而异，且更高的成本并不总是意味着更高的准确率。例如，使用 Claude Code 的 Sonnet 5 得分为 15.6%，单次运行成本为 5.01 美元；而使用 Codex 的 GPT-5.6 Sol 在类似的 4.70 美元成本下达到了 33.3%。

Using these models within Genie produced substantial gains. Across matched model comparisons, Genie improved accuracy by 24.0 percentage points on average (a 92% relative improvement). Genie configurations using GPT-5.6 Luna, GPT-5.6 Terra, and Claude Fable 5 also dominate the cost-quality Pareto frontier, demonstrating that these improvements do not require trading efficiency for accuracy. In the case of Claude Fable 5, switching to Genie improves performance by 14.4 percentage points (+32% relative) while reducing cost by ~9x. From inspecting traces, we find that Fable 5 is prone to looping attempts to parse many documents, which leads to long rollouts costing $37.36 on average. Genie parses documents using ai_parse document, which enables the model to efficiently identify the correct information from each page, thereby saving cost and improving performance.在 Genie 中使用这些模型产生了显著的收益。在匹配的模型比较中，Genie 的平均准确率提高了 24.0 个百分点（相对提升 92%）。使用 GPT-5.6 Luna、GPT-5.6 Terra 和 Claude Fable 5 的 Genie 配置也在成本-质量帕累托前沿上占据主导地位，证明这些改进无需以牺牲效率为代价来换取准确性。以 Claude Fable 5 为例，切换到 Genie 后，性能提升了 14.4 个百分点（相对提升 +32%），同时成本降低了约 9 倍。通过检查执行轨迹，我们发现 Fable 5 容易陷入解析大量文档的循环中，导致平均单次运行成本高达 37.36 美元。Genie 使用 ai_parse 文档解析功能，使模型能够高效地从每一页中识别正确信息，从而节省成本并提高性能。

While these harness improvements lead to substantial gains, significant headroom remains in OfficeQA Pro V2. Systems continue to exhibit similar failure modes to those we observed in the original OfficeQA – parsing fidelity, incorrect temporal reconciliation as accounting conventions change over time, and misinterpretation of entity scope or category granularity.虽然这些工具的改进带来了显著的收益，但 OfficeQA Pro V2 仍有很大的提升空间。系统继续表现出与我们在最初的 OfficeQA 中观察到的类似的失败模式——解析保真度问题、随着会计惯例随时间变化而导致的错误时间对账，以及对实体范围或类别粒度的误解。

Building a New Grounded Reasoning Benchmark构建新的扎根推理基准测试

To truly test generalization, we needed a new corpus that still reflected the same enterprise-relevant skills OfficeQA was designed to measure: parsing complex documents, retrieving the right evidence, and performing analytical reasoning grounded in that information.为了真正测试泛化能力，我们需要一个新的语料库，它必须依然体现 OfficeQA 设计初衷所要衡量的企业相关技能：解析复杂文档、检索正确的证据，以及基于这些信息执行分析推理。

We worked with the U.S. Treasury to build the benchmark around a new corpus: the U.S. Federal Accounts of Receipts and Expenditures, which the Treasury was preparing to release as a unified dataset for the first time in conjunction with the 250th anniversary of the United States. The corpus consists of roughly 1,400 PDFs and 120,000 pages containing detailed U.S. accounting records, spanning 1793 through 2024.我们与美国财政部合作，围绕一个新的语料库构建了该基准测试：美国联邦《收支账目》。这是财政部为纪念美国建国 250 周年，首次准备将其作为统一数据集发布。该语料库由约 1,400 份 PDF 和 120,000 页文档组成，包含了从 1793 年到 2024 年详细的美国会计记录。

Like the U.S. Treasury Bulletins used in the original OfficeQA benchmark, this new corpus reflects many of the challenges common across enterprise document collections: dense tables and charts, values revised over time, evolving reporting conventions, and deep institutional knowledge. These complexities also translate into a challenging evaluation: as with the original OfficeQA benchmark, we find that baseline frontier agents struggle to achieve consistently high accuracy on OfficeQA Pro V2.与最初 OfficeQA 基准测试中使用的《美国财政部公报》（U.S. Treasury Bulletins）一样，这个新语料库反映了企业文档集合中常见的许多挑战：密集的表格和图表、随时间修订的数值、不断演变的报告惯例以及深厚的机构知识。这些复杂性也转化为极具挑战性的评估：与最初的 OfficeQA 基准测试一样，我们发现基础前沿智能体难以在 OfficeQA Pro V2 上实现持续的高准确率。

Scaling OfficeQA Pro V2 with Synthetic Data利用合成数据扩展 OfficeQA Pro V2

Expand

When we created the original OfficeQA benchmark in late 2025, the process was highly manual. Human annotators constructed questions and answers by hand while combing through a corpus comprising 89,000 U.S. Treasury Bulletin pages, often requiring numerous rounds of human review in order to ensure question quality. Although this approach eventually produced a high-quality benchmark, it was time-consuming, costly, and difficult to scale. Since then, we have developed techniques to automate the creation of rigorous benchmarks much more efficiently and reliably.当我们于 2025 年底创建最初的 OfficeQA 基准测试时，过程是高度人工化的。人类标注员在梳理包含 89,000 页《美国财政部公报》的语料库时，手工构建问题和答案，通常需要多轮人工审查以确保问题质量。尽管这种方法最终产生了一个高质量的基准测试，但它耗时、昂贵且难以扩展。从那时起，我们开发了更高效、更可靠地自动化创建严谨基准测试的技术。

To build OfficeQA Pro V2, we leveraged asynth, our internal library for building synthetic data-generation pipelines. This enabled us to quickly and scalably create a diverse, verifiable benchmark that accurately reflected the challenges of grounded reasoning using a systematic process for generating and vetting questions:为了构建 OfficeQA Pro V2，我们利用了 asynth（我们内部用于构建合成数据生成流水线的库）。这使我们能够通过系统化的生成和审查流程，快速且可扩展地创建一个多样化、可验证的基准测试，准确反映扎根推理的挑战：

1. Inputs. To promote sample diversity, the synthesizer was seeded with a randomly selected time period, analytical method, target number of source documents, and salient topic drawn from a list developed with our partner, USAFacts, to reflect questions that analysts might naturally ask of the corpus.输入。为了提升样本多样性，合成器被植入了随机选择的时间段、分析方法、目标源文档数量以及从我们与合作伙伴 USAFacts 开发的列表中提取的显著主题，以反映分析师在面对该语料库时可能自然提出的问题。
2. Synthesis. For each candidate question, a synthesis agent searched the corpus for evidence matching these criteria and generated a sample supported by a traceable chain of grounded reasoning.合成。对于每个候选问题，合成智能体会在语料库中搜索符合这些标准的证据，并生成一个由可追溯的扎根推理链支持的样本。
3. Verification. Next, samples went through several quality gates to audit for source fidelity, analytical relevance, diversity, and answerability. Independent solver agents attempted the question from scratch, producing alternate solution trajectories, which were reviewed by a verification agent. Only candidates with verifiably correct answers advanced to the next stage.验证。接下来，样本会通过多个质量关卡，以审计源保真度、分析相关性、多样性和可回答性。独立的求解智能体从头开始尝试回答该问题，生成备选的求解轨迹，并由验证智能体进行审查。只有答案可验证正确无误的候选样本才能进入下一阶段。
4. Final Review. As a final quality-control step, samples were manually reviewed alongside solutions produced by additional frontier models. Reviewers examined the question, answer, reasoning, and cited evidence together, retaining only samples whose interpretation and ground truth were unambiguously correct.最终审查。作为最后的质量控制步骤，样本会与由其他前沿模型生成的解决方案一起进行人工审查。审查人员共同检查问题、答案、推理和引用的证据，仅保留那些解释和事实依据都绝对正确无误的样本。

The resulting questions require the same core grounded-reasoning capabilities measured by the original OfficeQA benchmark: retrieval across multiple source documents, analytical reasoning, and specialized capabilities such as supplemental web search and multimodal interpretation of figures. Example questions from the benchmark, as well as details about its composition, can be seen below.最终生成的问题需要最初 OfficeQA 基准测试所衡量的核心扎根推理能力：跨多个源文档的检索、分析推理，以及补充性网络搜索和图表多模态解读等专业能力。该基准测试中的示例问题及其构成详情如下所示。

Example Questions问题示例

1. Low Difficulty: Compute Chief Justice Earnings: “What is the total nominal dollar value for John Jay's earnings as Chief Justice of the Supreme Court, according to the Account of Receipts and Expenditures of the United States during the back half of CY 1793, and which warrant numbers corresponded to the payments? Return your answer as comma-separated values in enclosed brackets in the order of his earnings value, and the warrant numbers in calendar year order.”1. 低难度：计算首席大法官薪酬：“根据 1793 年下半年《美国收支账目》，约翰·杰伊（John Jay）作为最高法院首席大法官的薪酬名义总价值是多少？哪些授权书编号对应这些付款？请将答案以逗号分隔值的形式放在括号内，顺序为薪酬价值在前，授权书编号按日历年顺序排列。”

Expand

This question requires locating a single 1793 Account of Receipts and Expenditures page and reading the four warrant payments recorded to Chief Justice John Jay (highlighted in red) while correctly interpreting that "do." is shorthand for "ditto," indicating the same data category as the preceding row. The question then requires understanding that the “back half of CY 1793” scopes the answer to only the July and November payments. Finally, retrieval of corresponding warrant numbers and a simple sum is required to compute the final answer.这个问题需要定位 1793 年《收支账目》中的特定页面，阅读记录给首席大法官约翰·杰伊的四笔授权书付款（以红色高亮显示），同时正确解读“do.”是“ditto”的缩写，表示与前一行相同的数据类别。然后，问题需要理解“1793 年下半年”将答案范围限定在 7 月和 11 月的付款上。最后，需要检索相应的授权书编号并进行简单的加法计算，得出最终答案。

2. Medium Difficulty: Forecast Increase in Medicare Outlays: “Between FY1990 and FY1994 inclusive, Medicare outlays grew rapidly as the program expanded. Fitting an OLS linear regression to the reported annual Medicare outlay figures for those fiscal years, what is the estimated average annual dollar increase in Medicare outlays reported in millions of dollars, rounded to two decimal places?”2. 中等难度：预测医疗保险支出的增长：“在 1990 财年至 1994 财年（含）期间，随着计划的扩大，医疗保险支出迅速增长。对这些财年报告的年度医疗保险支出数字进行 OLS 线性回归拟合，以百万美元为单位报告的医疗保险支出估计平均年度美元增长额是多少（四舍五入到小数点后两位）？”

Expand

This question requires retrieving the Medicare outlay figure from the "Outlays by Function" table in five separate annual Combined Statements (FY1990–FY1994, highlighted in red). Importantly, the agent must read each year’s value from the latest revised prior-year column, rather than the first reported figure. Finally, the agent must fit the series with a linear regression model to estimate the average annual increase.这个问题需要从五个独立的年度《综合报表》（1990 财年至 1994 财年，以红色高亮显示）的“按功能划分的支出”表中检索医疗保险支出数字。重要的是，智能体必须从最新的修订后上一年列中读取每年的数值，而不是第一个报告的数字。最后，智能体必须用线性回归模型拟合该序列，以估计平均年度增长额。

3. High Difficulty: Forecast Increase of Outlays Across Agencies: “For the combined net outlays of the Departments of Commerce, Interior, and State, the Environmental Protection Agency, and the National Aeronautics and Space Administration rolled up from their constituent Treasury Account Symbol (TAS) families as reported by the U.S. Treasury, fit the standard nonseasonal Theta method on the reported outlays: estimate the θ=0 component as an OLS linear trend, construct the θ=2 transformed series, forecast that component with simple exponential smoothing initialized at its first value, and choose α over [0, 1], allowing α = 1.0. Using this forecasting and averaging the θ=0 and θ=2 one-step-ahead forecasts, what is the FY1989 forecast in millions of dollars rounded to the nearest hundredth?”3. 高难度：预测各机构支出的增长：“对于美国财政部报告的由其组成部分财政账户符号（TAS）系列汇总的商务部、内政部、国务院、环境保护署和国家航空航天局的合并净支出，对报告的支出应用标准的非季节性 Theta 方法：将 θ=0 分量估计为 OLS 线性趋势，构建 θ=2 转换序列，使用在其第一个值处初始化的简单指数平滑法预测该分量，并从 [0, 1] 中选择 α，允许 α = 1.0。使用此预测方法并对 θ=0 和 θ=2 的一步预测值进行平均，1989 财年的预测值（以百万美元为单位，四舍五入到小数点后两位）是多少？”

Expand

This question requires assembling an eight-year series (FY1981–FY1988) of net outlays for five different agencies (Commerce, Interior, State, EPA, and NASA) from eight separate Combined Statements. Again, the agent must use only the latest reported figures for each year. The resulting series then must be used to forecast net outlays for FY1989, using a specific forecasting method.这个问题需要从八份独立的《综合报表》中汇总五个不同机构（商务部、内政部、国务院、环保署和 NASA）八年（1981 财年至 1988 财年）的净支出序列。同样，智能体必须仅使用每年的最新报告数字。然后，必须使用特定的预测方法，利用所得序列来预测 1989 财年的净支出。

Benchmark Details基准测试详情

Expand

OfficeQA Pro V2 consists of 90 questions, all of which require evidence from the benchmark’s corpus. As part of our synthetic-data verification pipeline, we filtered out questions that could be answered using parametric knowledge or web search alone.OfficeQA Pro V2 由 90 个问题组成，所有问题都需要基准测试语料库中的证据。作为合成数据验证流水线的一部分，我们过滤掉了那些仅凭参数化知识或网络搜索即可回答的问题。

Although built on a new corpus, OfficeQA Pro V2 preserves the core enterprise grounded-reasoning capabilities measured by the original OfficeQA benchmark, both across the full question set, OfficeQA Full, and within its frontier-difficulty subset, OfficeQA Pro. 7% of questions require visual understanding of charts, graphs, or figures, compared with approximately 3% in OfficeQA Pro. Another 10% require supplemental information obtained through web search (such as inflation indices, GDP series, or population figures), compared with 21.8% in OfficeQA Pro and 15.9% in OfficeQA Full. Together, roughly one in six OfficeQA Pro V2 questions requires at least one of these specialized capabilities beyond text retrieval and analysis.尽管构建在新的语料库之上，但 OfficeQA Pro V2 保留了最初 OfficeQA 基准测试所衡量的核心企业扎根推理能力，无论是在完整问题集（OfficeQA Full）还是其前沿难度子集（OfficeQA Pro）中都是如此。7% 的问题需要对图表或图形进行视觉理解，而 OfficeQA Pro 中约为 3%。另有 10% 的问题需要通过网络搜索获得补充信息（如通胀指数、GDP 系列或人口数据），而 OfficeQA Pro 中为 21.8%，OfficeQA Full 中为 15.9%。总的来说，大约六分之一的 OfficeQA Pro V2 问题需要除文本检索和分析之外的至少一种专门能力。

Compared to OfficeQA Pro, OfficeQA Pro V2 requires evidence from substantially more source documents per question. OfficeQA Pro questions draw on approximately 2 Treasury Bulletin documents on average, whereas OfficeQA Pro V2 questions require evidence from 6.7 source documents on average, with a median of 5.5 and a maximum of 24. In total, 74.4% of OfficeQA Pro V2 questions require four or more sources, compared with 62.4% in OfficeQA Pro and 56.1% in OfficeQA Full.与 OfficeQA Pro 相比，OfficeQA Pro V2 每个问题需要来自更多源文档的证据。OfficeQA Pro 问题平均利用约 2 份《财政部公报》文档，而 OfficeQA Pro V2 问题平均需要 6.7 份源文档的证据，中位数为 5.5，最多达 24 份。总计 74.4% 的 OfficeQA Pro V2 问题需要四个或更多源文档，而在 OfficeQA Pro 中这一比例为 62.4%，在 OfficeQA Full 中为 56.1%。

Taken together, these characteristics make OfficeQA Pro V2 a more demanding test of end-to-end grounded reasoning than the original benchmark, while preserving the same realistic mix of analytical, multimodal, and external-knowledge requirements found in enterprise workflows. To understand why the benchmark remains challenging, it is useful to examine how dramatically the underlying documents and reporting conventions have evolved over time.总而言之，这些特性使 OfficeQA Pro V2 成为比原始基准测试更严苛的端到端扎根推理测试，同时保留了企业工作流中常见的分析、多模态和外部知识需求的现实组合。为了理解为什么该基准测试仍然具有挑战性，考察底层文档和报告惯例随时间的演变过程非常有用。

A Corpus Spanning over Two Centuries跨越两个多世纪的语料库

Spanning 232 years of U.S. federal financial reporting, the corpus also captures the evolution of document structures, accounting conventions, terminology, and institutions over more than two centuries. For instance, records in the late 1700s and early 1800s often record payments to individuals. These documents often include wide landscape foldouts, archaic typography like the use of long-s in words such as “Treaſury” or “Preſident”, and historical writing conventions like recording “do./ditto”, which pose novel challenges for modern parsing solutions.该语料库跨越了 232 年的美国联邦财务报告，捕捉了两个多世纪以来文档结构、会计惯例、术语和机构的演变。例如，18 世纪末和 19 世纪初的记录经常记录对个人的付款。这些文档通常包含宽大的横向折页、古老的排版（如在“Treaſury”或“Preſident”等词中使用长 s），以及历史性的书写惯例（如记录“do./ditto”），这对现代解析方案构成了新的挑战。

Expand

By the mid-nineteenth century, the data was presented in two-page T-account formats and dense, ruled tables, while the reporting period changed from calendar to fiscal years. The early 1900s then introduced new financial terms like opening and closing balances, multi-year appropriations, and surplus and deficit conventions.到了 19 世纪中叶，数据以两页 T 型账户格式和密集的规则表格呈现，而报告期也从日历年改为财年。20 世纪初引入了新的财务术语，如期初和期末余额、多年期拨款以及盈余和赤字惯例。

Expand

By the 1980s, the Combined Statement had evolved into an annual report with a detailed appendix. The reports became digital-native in the early 2000s and began incorporating visual charts alongside a standardized table of contents.到 20 世纪 80 年代，《综合报表》已演变为带有详细附录的年度报告。这些报告在 21 世纪初实现了数字化，并开始在标准目录旁边加入视觉图表。

Expand

OfficeQA Pro V2 builds on the complexities of the Accounts of Receipts and Expenditures, creating an especially demanding stress test for grounded reasoning. Across the corpus, the same financial concept may change in name, location, table schema, unit, time basis, and level of aggregation. In addition to parsing documents and retrieving values, this means agents must reconcile information across changing reporting conventions – a complexity that is typical in enterprise settings.OfficeQA Pro V2 基于《收支账目》的复杂性，为扎根推理创造了一个特别严苛的压力测试。在整个语料库中，同一个财务概念在名称、位置、表格模式、单位、时间基准和汇总级别上可能会发生变化。除了解析文档和检索数值外，这意味着智能体必须在不断变化的报告惯例之间进行对账——这在企业环境中是一种典型的复杂性。

Conclusion & Acknowledgements结论与致谢

The benchmark and parsed corpus are now publicly available on Hugging Face, with evaluation code available on GitHub. We encourage researchers and practitioners to evaluate their own agentic systems on OfficeQA Pro V2, and recommend its use as a test set for OfficeQA Pro. For enterprise developers, OfficeQA Pro V2 can serve as a testbed for building agents that must answer complex questions across large, heterogeneous document collections. Teams can use it to compare models and agent architectures, identify whether failures originate in parsing, retrieval, reasoning, or verification, and measure how changes to their systems affect accuracy, latency, and cost before applying those approaches to their own enterprise data. Finally, we leave readers with three takeaways:该基准测试和已解析的语料库现已在 Hugging Face 上公开，评估代码可在 GitHub 上获取。我们鼓励研究人员和从业者在 OfficeQA Pro V2 上评估自己的智能体系统，并推荐将其用作 OfficeQA Pro 的测试集。对于企业开发者而言，OfficeQA Pro V2 可以作为构建智能体的试验台，这些智能体必须能够回答跨大型、异构文档集合的复杂问题。团队可以使用它来比较模型和智能体架构，识别失败是源于解析、检索、推理还是验证，并衡量系统变更如何影响准确性、延迟和成本，然后再将这些方法应用于自己的企业数据。最后，我们给读者留下三个要点：

1. Representative evaluations are essential, and synthetic data can help scale them. Frontier agents still struggle with enterprise workflows out of the box, making it critical to evaluate systems on tasks that reflect real-world complexity and test whether improvements generalize to unfamiliar settings. At Databricks, we combine synthetic data techniques with our understanding of customer workflows to build representative evaluations like OfficeQA Pro V2 more efficiently and at greater scale.具有代表性的评估至关重要，合成数据有助于扩展这些评估。前沿智能体在现成状态下仍然难以应对企业工作流，因此在反映现实世界复杂性的任务上评估系统，并测试改进是否能泛化到陌生环境中，这一点至关重要。在 Databricks，我们将合成数据技术与我们对客户工作流的理解相结合，以更高效、更大规模地构建像 OfficeQA Pro V2 这样具有代表性的评估。
2. An optimized agent harness is critical for achieving the best performance. Moving from model providers’ baseline harnesses to Databricks Genie improved performance by 24.0 percentage points on average (+92% relative) across evaluated models. These results demonstrate that document processing, retrieval, tool use, and orchestration choices can substantially affect both the accuracy and efficiency of grounded-reasoning systems.优化的智能体工具对于实现最佳性能至关重要。从模型提供商的基准工具转向 Databricks Genie，在评估的模型中平均性能提高了 24.0 个百分点（相对提升 +92%）。这些结果表明，文档处理、检索、工具使用和编排选择可以显著影响扎根推理系统的准确性和效率。
3. Grounded reasoning still has significant room for improvement. Although harness optimizations produced substantial gains over the out-of-the-box baselines, even the strongest systems continued to struggle with challenges such as faithful document parsing, reconciling values across time, and retrieving the latest revised figures. Continued progress will require innovations across the full agent stack, from parsing and retrieval to analysis and verification.扎根推理仍有很大的改进空间。尽管工具优化比现成基准产生了显著收益，但即使是最强大的系统在忠实文档解析、跨时间对账数值以及检索最新修订数字等挑战上仍然感到吃力。持续的进步将需要整个智能体栈的创新，从解析和检索到分析和验证。

We thank USAFacts for their continued partnership on the OfficeQA benchmark suite, including helping us identify the new corpus, develop representative analytical topics and questions, and organize the Grounded Reasoning Cup. We also thank the U.S. Treasury for helping identify the U.S. Accounts of Receipts and Expenditures as the foundation for the competition benchmark, and for releasing the dataset as a cohesive collection for the first time.我们感谢 USAFacts 在 OfficeQA 基准测试套件上的持续合作，包括帮助我们识别新语料库、开发具有代表性的分析主题和问题，以及组织扎根推理杯。我们还要感谢美国财政部帮助确定将美国《收支账目》作为竞赛基准的基础，并首次将该数据集作为统一集合发布。

Authors: Krista Opsahl-Ong, Arnav Singhvi, Josh Joseph, Jasmine Collins, Ivan Zhou, Shubham Toshniwal, Michael Bendersky, Erich Elsen,  Xing Chen, Matei Zaharia作者：Krista Opsahl-Ong, Arnav Singhvi, Josh Joseph, Jasmine Collins, Ivan Zhou, Shubham Toshniwal, Michael Bendersky, Erich Elsen, Xing Chen, Matei Zaharia

If you want to learn more and get started with OfficeQA Pro V2, check out the benchmark on Hugging Face.如果您想了解更多信息并开始使用 OfficeQA Pro V2，请查看 Hugging Face 上的基准测试。