Introducing Data-eng-bench: Why You Need "Data-Native" Harnesses for Data Engineering

As AI agents move from writing individual functions to owning end-to-end workflows, data teams face a harsh reality: general-purpose coding agents, regardless of their proficiency in Python or SQL, can struggle with production-grade data engineering. They often fail to complete tasks or incur high costs. This is a demanding test of an agent's ability to navigate a large warehouse, reason about business logic and handle edge cases. This is precisely the kind of work that has historically been difficult to measure.随着 AI 智能体从编写单个函数转向掌控端到端工作流，数据团队面临着一个残酷的现实：通用的编程智能体，无论其 Python 或 SQL 水平如何，在处理生产级数据工程时都可能力不从心。它们往往无法完成任务，或者导致高昂的成本。这对智能体在大型仓库中导航、推理业务逻辑以及处理边缘情况的能力提出了严峻考验。而这恰恰是历史上难以量化的工作。

To measure this capability, we're open sourcing data-eng-bench, a benchmark for repository-level data engineering created in joint work with Bespoke Labs. Tasks in data-eng-bench hand an agent a live dbt project connected to an enterprise-scale data warehouse and ask it to build and fix real data pipelines. Resulting dbt models are then evaluated against the business rules and edge cases a working pipeline must satisfy. Relative to ADE-Bench, one of few other open benchmarks in this domain, data-eng-bench offers higher scale (103 tasks versus 65 tasks as of August 10, 2026), robust tests that check invariants of the output data pipeline, and more complex task specifications.为了衡量这种能力，我们开源了 data-eng-bench，这是一个由我们与 Bespoke Labs 共同创建的、针对仓库级数据工程的基准测试。data-eng-bench 中的任务会为智能体提供一个连接到企业级数据仓库的实时 dbt 项目，并要求其构建和修复真实的数据流水线。随后，生成的 dbt 模型将根据工作流水线必须满足的业务规则和边缘情况进行评估。与该领域为数不多的其他开源基准测试 ADE-Bench 相比，data-eng-bench 提供了更高的规模（截至 2026 年 8 月 10 日，包含 103 个任务，而 ADE-Bench 为 65 个）、用于检查输出数据流水线不变量的稳健测试，以及更复杂的任务规范。

Our testing on data-eng-bench reveals a clear divide between generic agent harnesses, including Claude Code and OpenAI Codex, and the data-native agent harness Snowflake CoCo. Specifically, CoCo leverages its understanding of the data platform to achieve matching or higher quality (that is, task completion rate) while incurring significantly lower cost. In more detail:我们在 data-eng-bench 上的测试揭示了通用智能体框架（包括 Claude Code 和 OpenAI Codex）与数据原生智能体框架 Snowflake CoCo 之间的明显鸿沟。具体而言，CoCo 利用其对数据平台的理解，在实现同等或更高质量（即任务完成率）的同时，大幅降低了成本。详细情况如下：

• The harness matters for quality; the effect varies by model: Holding the harness fixed at CoCo, Pass@1 varies drastically across models: from 73.8% with Opus 5 to 64.1% with GPT 5.6 Sol to 56.6% with Sonnet 5, a 17-point spread. The impact of the harness on quality depends on the model: Opus 5 performs the best with CoCo, dropping by ~4pp in Pass@1 with Claude Code; Sonnet 5 performs equally well with both CoCo and Claude Code; GPT 5.6 Sol performs the best with CoCo, dropping by 3.6pp with Codex.框架对质量至关重要，且效果因模型而异：在框架固定为 CoCo 的情况下，Pass@1 在不同模型间差异显著：从 Opus 5 的 73.8%，到 GPT 5.6 Sol 的 64.1%，再到 Sonnet 5 的 56.6%，跨度达到 17 个百分点。框架对质量的影响取决于模型：Opus 5 在 CoCo 下表现最佳，使用 Claude Code 时 Pass@1 下降了约 4 个百分点；Sonnet 5 在 CoCo 和 Claude Code 下表现相当；GPT 5.6 Sol 在 CoCo 下表现最佳，使用 Codex 时下降了 3.6 个百分点。
• The harness matters significantly for cost efficiency: CoCo achieves higher quality at lower cost than other harnesses: With Opus 5, CoCo reports a 4pp higher Pass@1 at 3.9x lower cost than Claude Code. With Sonnet 5, CoCo reports the same Pass@1 at 2.3x lower cost than Claude Code. With GPT 5.6 Sol, CoCo reports a 3.6pp higher Pass@1 than Codex with Codex incurring 1.5x the cost of CoCo. CoCo completes tasks with 1.5x fewer tool operations and 2.2x fewer agent steps than Claude Code. Task solving patterns show that CoCo adopts a more efficient exploration and validation strategy, requiring 1.7x fewer SQL queries and 1.2x fewer file reads during these phases. CoCo also stays on scope in 2.2x more instances, skipping DuckDB cross-validation that Claude Code performs by default, a step the benchmark tasks do not require since the backend is specified to Snowflake.框架对成本效率的影响显著：与其他框架相比，CoCo 以更低的成本实现了更高的质量：使用 Opus 5 时，CoCo 的 Pass@1 比 Claude Code 高出 4 个百分点，成本却降低了 3.9 倍。使用 Sonnet 5 时，CoCo 的 Pass@1 与 Claude Code 相同，成本降低了 2.3 倍。使用 GPT 5.6 Sol 时，CoCo 的 Pass@1 比 Codex 高出 3.6 个百分点，而 Codex 的成本是 CoCo 的 1.5 倍。与 Claude Code 相比，CoCo 完成任务所需的工具操作减少了 1.5 倍，智能体步骤减少了 2.2 倍。任务解决模式显示，CoCo 采用了更高效的探索和验证策略，在这些阶段所需的 SQL 查询减少了 1.7 倍，文件读取减少了 1.2 倍。CoCo 在 2.2 倍多的实例中保持在任务范围内，跳过了 Claude Code 默认执行的 DuckDB 交叉验证——由于后端指定为 Snowflake，基准测试任务并不要求执行此步骤。

While these results indicate that frontier agents have made big strides in tackling data engineering tasks, there is still headroom for further improvement. The strongest configuration we benchmarked, Snowflake CoCo with Opus 5, successfully solves 73.8% of tasks on the first attempt (average Pass@1 across 3 trials), but only 64.1% of tasks pass on all three runs (Pass^3). Sonnet 5 reports Pass^3 at 40.8% and GPT 5.6 Sol reports Pass^3 at <56% across harnesses.虽然这些结果表明前沿智能体在处理数据工程任务方面取得了巨大进步，但仍有进一步提升的空间。我们测试的最强配置——搭载 Opus 5 的 Snowflake CoCo——在首次尝试中成功解决了 73.8% 的任务（3 次试验的平均 Pass@1），但只有 64.1% 的任务在全部三次运行中均通过（Pass^3）。Sonnet 5 的 Pass^3 为 40.8%，GPT 5.6 Sol 在各框架下的 Pass^3 均低于 56%。

In the rest of this blog, we provide an overview of the data-eng-bench benchmark, report how frontier agents perform on it in terms of quality, token and cost efficiency, and what it means for teams adopting coding agents for data engineering tasks.在本博客的其余部分，我们将概述 data-eng-bench 基准测试，报告前沿智能体在质量、Token 消耗和成本效率方面的表现，并探讨这对采用编程智能体进行数据工程任务的团队意味着什么。

Get the benchmark.获取基准测试。

The data-eng-bench benchmarkdata-eng-bench 基准测试

We designed data-eng-bench to mirror how enterprise data engineering teams actually operate: many source systems, layered staging-to-mart development and a shared project that must be maintained by many users. Here is a breakdown:我们设计 data-eng-bench 的初衷是映射企业数据工程团队的实际运作方式：多个源系统、分层的从暂存区到数据集市的开发模式，以及需要由多用户维护的共享项目。具体分解如下：

One shared data warehouse: Every task runs against a single, persistent retail data warehouse — 579 source tables across 19 schemas, roughly 8,000 columns in total — spanning orders, finance, procurement, marketing, inventory and more. That's broader than prior data engineering benchmarks, and each task requires navigating the same schema with different requirements.一个共享数据仓库：每个任务都在同一个持久化的零售数据仓库上运行——涵盖 19 个架构下的 579 个源表，总计约 8,000 列——涉及订单、财务、采购、营销、库存等。这比以往的数据工程基准测试范围更广，且每个任务都需要在相同的架构下处理不同的需求。

103 tasks of two variants: Each task gives the agent an instruction in natural language, a starting dbt project, and the data warehouse, then asks it to produce or correct models that materialize the right tables. The two variants are as follows:103 个任务，分为两种类型：每个任务为智能体提供自然语言指令、一个起始 dbt 项目以及数据仓库，然后要求其生成或修正模型以物化正确的表。这两种类型如下：

• Build (84 tasks): Author new models and keep the existing pipeline running. Build tasks can be further divided into (a) Greenfield: Scaffold a brand new dbt project from an empty state; and (b) Brownfield: Add new models into an existing multi-layer project while reusing and preserving dozens of production models.构建（84 个任务）：编写新模型并保持现有流水线运行。构建任务可进一步细分为：(a) Greenfield（绿地）：从空状态搭建全新的 dbt 项目；(b) Brownfield（棕地）：在现有分层项目中添加新模型，同时重用并保留数十个生产模型。
• Fix (19 tasks): Diagnose and repair a subtly broken production model.修复（19 个任务）：诊断并修复一个存在细微故障的生产模型。

Real dbt mechanics, not free-form SQL: Agents work through dbt primitives: source declarations (45 tasks), reusable macros (13 tasks) and per-model materializations across staging, intermediate and mart layers. And 82% of gold solutions wire models together through explicit ref dependencies, averaging roughly nine ref calls each, which the agent must resolve into a coherent, compilable DAG. Those DAGs are meaningfully large: a median of four models per solution, and up to 42 for the biggest pipelines.真实的 dbt 机制，而非自由格式的 SQL：智能体通过 dbt 原语进行工作：源声明（45 个任务）、可重用宏（13 个任务）以及跨暂存层、中间层和数据集市层的逐模型物化。82% 的黄金解决方案通过显式的 ref 依赖关系连接模型，平均每个解决方案包含约 9 个 ref 调用，智能体必须将其解析为连贯、可编译的 DAG。这些 DAG 的规模相当大：每个解决方案中位数包含 4 个模型，最大的流水线多达 42 个。

Diverse, high-difficulty business rules: The tasks live in domains where data engineering is proven difficult — finance (ledger reconciliation, revenue recognition, multi-currency settlement), inventory (LIFO/FIFO costing, turnover, stockout risk), marketing (multi-touch attribution, campaign ROI) and customer analytics (RFM segmentation, churn, lifetime value). The business rules depend on the order and timing of events, accumulated or allocated values across many interdependent tables, and edge cases that should be handled exactly as defined.多样化且高难度的业务规则：任务所在的领域被公认为数据工程的难点——财务（总账对账、收入确认、多币种结算）、库存（LIFO/FIFO 成本核算、周转率、缺货风险）、营销（多触点归因、营销活动 ROI）和客户分析（RFM 分段、流失率、生命周期价值）。业务规则取决于事件的顺序和时间、跨多个相互依赖表的累积或分配值，以及必须严格按照定义处理的边缘情况。

Graded by what the pipeline does, not what it looks like: Each task ships with a hidden verifier suite of 10–50 assertions that materializes the agent's models and interrogates the resulting tables directly. Those assertions encode the invariants a correct solution must satisfy (for example, output grain and column contracts, formula-level correctness, edge-case handling, idempotency across re-runs) and for the hardest tasks, independently recompute the expected result in Python.根据流水线的功能而非外观进行评分：每个任务都附带一套包含 10–50 个断言的隐藏验证程序，用于物化智能体的模型并直接查询生成的表。这些断言编码了正确解决方案必须满足的不变量（例如，输出粒度和列契约、公式级正确性、边缘情况处理、跨重运行的幂等性），对于最困难的任务，还会使用 Python 独立重新计算预期结果。

Scoring is two-fold: At the test level, assertions provide partial credit (the fraction that pass). At the task level, a task counts as resolved only if every assertion passes — so a pipeline that's directionally right but wrong at the edges earns no task-level credit. This lets us separate "can generate a plausible model" from "got the whole pipeline correct" and pinpoint exactly where solutions fail.评分分为两个维度：在测试层面，断言提供部分分数（通过的比例）。在任务层面，只有当所有断言都通过时，任务才算解决——因此，一个方向正确但在边缘情况处理上错误的流水线将无法获得任务级分数。这使我们能够将“能否生成看似合理的模型”与“能否正确完成整个流水线”区分开来，并精确定位解决方案失败的具体位置。

Measuring agent quality and cost衡量智能体的质量与成本

We evaluate a combination of harnesses and models on data-eng-bench, isolating how much of an agent's performance comes from the scaffold versus the underlying model.我们在 data-eng-bench 上评估了多种框架与模型的组合，以隔离智能体的性能有多少源于框架，又有多少源于底层模型。

We test three models spanning proprietary families — Opus 5 (Anthropic), Sonnet 5 (Anthropic) and GPT 5.6 Sol (OpenAI), run with several harnesses that differ in context management, planning and tool interfaces: Snowflake CoCo, Claude Code and Codex.¹ We recommend Code mode (cortex --mode code via CLI) while using CoCo on dbt tasks for quality/cost balance. For every combination, we report two quality metrics: Pass@1 and Pass^3 rates. Pass@1 and Pass^3 capture complementary dimensions of quality, measuring the capability ceiling and consistency across runs respectively. We also report the average cost per trial (in $) and the average number of total tokens (across input, cache and output) per trial.我们测试了三个跨自有系列的模型——Opus 5 (Anthropic)、Sonnet 5 (Anthropic) 和 GPT 5.6 Sol (OpenAI)，并运行了在上下文管理、规划和工具接口方面有所不同的多个框架：Snowflake CoCo、Claude Code 和 Codex。在 dbt 任务中使用 CoCo 时，我们建议使用代码模式（通过 CLI 执行 cortex --mode code）以平衡质量与成本。对于每种组合，我们报告两个质量指标：Pass@1 和 Pass^3 率。Pass@1 和 Pass^3 分别从能力上限和跨运行一致性两个互补维度衡量质量。我们还报告了每次试验的平均成本（以美元计）以及每次试验的平均总 Token 数（包括输入、缓存和输出）。

Tab 1. Strict, task-level resolve rates (%). Pass@1 = mean single-attempt pass rate (averaged across three attempts); Pass^3 = resolved if all three attempts succeed. Cost multipliers are set to 1 for each combination of CoCo with the 3 models (Opus 5, Sonnet 5, GPT 5.6 Sol); the cost multiplier for other harnesses in conjunction with the corresponding model is compared against this baseline. For example, Claude Code with Opus 5 is 3.9x more costly than CoCo with Opus 5.表 1. 严格的任务级解析率 (%)。Pass@1 = 平均单次尝试通过率（取三次尝试的平均值）；Pass^3 = 三次尝试全部成功即视为解决。每种 CoCo 与 3 个模型（Opus 5、Sonnet 5、GPT 5.6 Sol）的组合成本乘数设为 1；其他框架与相应模型结合的成本乘数均以此为基准进行比较。例如，使用 Opus 5 的 Claude Code 的成本是使用 Opus 5 的 CoCo 的 3.9 倍。

A key takeaway is that the harness matters significantly for quality and cost efficiency. On data-eng-bench, CoCo consistently achieves higher quality at lower cost than other harnesses:一个关键结论是，框架对质量和成本效率的影响非常显著。在 data-eng-bench 上，CoCo 始终以比其他框架更低的成本实现更高的质量：

• With Opus 5, CoCo reports a 4pp higher Pass@1 at 3.9x lower cost than Claude Code.使用 Opus 5 时，CoCo 的 Pass@1 比 Claude Code 高出 4 个百分点，成本降低了 3.9 倍。
• With Sonnet 5, CoCo reports the same Pass@1 at 2.3x lower cost than Claude Code.使用 Sonnet 5 时，CoCo 的 Pass@1 与 Claude Code 相同，成本降低了 2.3 倍。
• With GPT 5.6 Sol, CoCo reports a 3.6pp higher Pass@1 at 1.5x lower cost than Codex使用 GPT 5.6 Sol 时，CoCo 的 Pass@1 比 Codex 高出 3.6 个百分点，成本降低了 1.5 倍。

We now delve into why CoCo is more cost efficient than Claude Code.我们现在深入探讨为什么 CoCo 比 Claude Code 更具成本效益。

With Opus 5, CoCo requires 1.5x fewer tool operations and 2.2x fewer agent steps than Claude Code to solve a task on average. The agent spends most of its time on the predevelopment exploration and postdevelopment validation phases, under both harnesses. Overall, CoCo issues 1.7x fewer SQL queries, concentrates file reads to the exploration phase and produces 1.5x fewer file writes. The tool call patterns in different phases on task execution reveal the strategies adopted by Opus 5 under the two harnesses:使用 Opus 5 时，CoCo 平均比 Claude Code 少用 1.5 倍的工具操作和 2.2 倍的智能体步骤即可解决任务。在两种框架下，智能体大部分时间都花在开发前的探索阶段和开发后的验证阶段。总体而言，CoCo 发出的 SQL 查询减少了 1.7 倍，将文件读取集中在探索阶段，产生的文件写入减少了 1.5 倍。任务执行不同阶段的工具调用模式揭示了 Opus 5 在两种框架下采取的策略：

1. Phase 1 — Predevelopment setup and exploration: CoCo requires 1.3x fewer tool operations before writing or editing the first SQL file. Both agents spend similar effort reading existing models and project config (6.6 vs 7.1 file reads). The divergence is in SQL: CoCo uses 2x fewer queries for schema exploration and source data profiling before writing.阶段 1——开发前设置与探索：在编写或编辑第一个 SQL 文件之前，CoCo 所需的工具操作减少了 1.3 倍。两个智能体在阅读现有模型和项目配置上花费的精力相当（6.6 次 vs 7.1 次文件读取）。分歧在于 SQL：CoCo 在编写前进行架构探索和源数据分析时使用的查询次数减少了 2 倍。
2. Phase 2 — dbt model development: Both agents primarily write SQL files (4.4 vs 5.6 file writes). Claude Code additionally reads more files during this phase — consulting existing staging models and macros mid-draft — while CoCo completes all reading in Phase 1 and writes without look-back.阶段 2——dbt 模型开发：两个智能体主要进行 SQL 文件写入（4.4 次 vs 5.6 次文件写入）。Claude Code 在此阶段额外读取了更多文件——在草拟过程中查阅现有的暂存模型和宏——而 CoCo 则在阶段 1 完成了所有读取，并在编写时无需回头。
3. Phase 3 — Build, validate and iterate on Snowflake: CoCo requires 1.7x fewer tool operations, with the gap driven primarily by Claude Code issuing 1.5x more SQL queries to verify the built models and 3x more shell setup commands around each build iteration.阶段 3——在 Snowflake 上构建、验证与迭代：CoCo 所需的工具操作减少了 1.7 倍，差距主要源于 Claude Code 为验证构建的模型发出的 SQL 查询次数多出 1.5 倍，且在每次构建迭代时发出的 Shell 设置命令多出 3 倍。
4. Phase 4 — Cross-validation with DuckDB: Interestingly, Claude Code performs DuckDB cross-dialect validation in 2.25x more trials than CoCo. This step is unnecessary given the backend is specified as Snowflake and evaluates the agent's ability to avoid irrelevant details in the task instructions.阶段 4——使用 DuckDB 进行交叉验证：有趣的是，Claude Code 在 2.25 倍多的试验中执行了 DuckDB 跨方言验证。鉴于后端已指定为 Snowflake，此步骤是不必要的，这评估了智能体在任务指令中忽略无关细节的能力。

These differences reflect two distinct strategies. CoCo follows a plan-then-execute approach — it front-loads exploration, writes directly, verifies once and stops. Claude Code follows an explore-and-refine approach — it interleaves reading with writing throughout development, wraps each phase with additional verification passes and treats cross-dialect validation as a default rather than an optional step.这些差异反映了两种截然不同的策略。CoCo 遵循“先规划后执行”的方法——预先进行探索，直接编写，验证一次即停止。Claude Code 遵循“探索与细化”的方法——在整个开发过程中交替进行读取和写入，在每个阶段前后包裹额外的验证步骤，并将跨方言验证视为默认步骤而非可选步骤。

Get started开始使用

Data-eng-bench is open source. Whether you build agents, harnesses or the models underneath them, it's a realistic, hard-to-saturate testbed for measuring autonomous data engineering.Data-eng-bench 是开源的。无论您是构建智能体、框架还是底层模型，它都是衡量自主数据工程的、现实且难以饱和的测试平台。

• Explore the benchmark: here探索基准测试：此处
• Contribute: add tasks, harnesses or model results and tell us what you find.贡献：添加任务、框架或模型结果，并告诉我们您的发现。

We're excited to see how far agents can go on the work that quietly powers every analytics stack. Try data-eng-bench, run your own model–harness combinations and share your results!我们很高兴看到智能体在默默支撑每个分析栈的工作上能走多远。尝试 data-eng-bench，运行您自己的模型-框架组合并分享您的结果！

¹ Tests were conducted between 08/03/2026 - 08/05/2026 using CoCo v0.26.0804 in code mode, Codex v0.146.0 (tested with GPT 5.6 Sol), and Claude Code v2.1.221 (tested with Opus 5 and Sonnet 5), all under default configurations. CoCo runs natively within Snowflake, with direct access to the warehouse schema and metadata; Claude Code and Codex interact with the warehouse via standard SQL interfaces. All harnesses used the same 103 tasks and were evaluated by the same automated verifier suite.1 测试于 2026 年 8 月 3 日至 2026 年 8 月 5 日期间进行，使用的环境为代码模式下的 CoCo v0.26.0804、Codex v0.146.0（使用 GPT 5.6 Sol 测试）以及 Claude Code v2.1.221（使用 Opus 5 和 Sonnet 5 测试），均采用默认配置。CoCo 在 Snowflake 内部原生运行，可直接访问仓库架构和元数据；Claude Code 和 Codex 通过标准 SQL 接口与仓库交互。所有框架均使用相同的 103 个任务，并由同一套自动化验证程序进行评估。