Benchmarking Coding Agents on Databricks’ Multi-Million Line Codebase

At Databricks, the way we build software is changing quickly as we aggressively adopt AI for engineering. The landscape of models and harnesses for code authoring has rapidly expanded in the last year, giving developers more choices than ever. With more options, it has become increasingly important to understand which coding agents offer the best performance on real-world coding tasks as well as understanding how task-performance varies with price.在 Databricks，随着我们积极采用人工智能进行工程开发，软件构建方式正在快速变化。过去一年中，用于代码编写的模型和工具框架迅速增多，给开发者带来了比以往更多的选择。选项增多后，了解哪些编码代理在实际编码任务中表现最佳，以及任务性能如何随价格变化，变得愈发重要。

This article shares the results and methodology of the internal coding benchmark we built at Databricks, which evaluates tools on actual coding tasks our engineers performed on the Databricks codebase. Tasks featured edits against a multi-million line codebase covering many popular languages (Python, Go, Typescript, Scala, etc.) and both tasks and solutions were carefully reviewed to ensure accuracy. This isn't meant to be comprehensive, but the exercise surfaced insights that have already made our engineering team meaningfully more efficient with coding agents. Below, you can see how models and harnesses scored on the overall benchmark:本文分享了我们在 Databricks 构建的内部编码基准测试的结果和方法，该测试基于工程师在 Databricks 代码库上执行的实际编码任务来评估工具。任务涉及对数百万行代码库进行编辑，涵盖多种流行语言（Python、Go、TypeScript、Scala 等），并且对任务和解决方案都进行了仔细审查以确保准确性。这并不是一个全面性的测试，但通过这一实践，我们获得了深刻见解，已经显著提升了工程团队使用编码代理的效率。下面你可以看到各模型和工具框架在整体基准测试中的得分：

Figure 1: Cost vs. Performance on our benchmark图 1：基准测试中的成本 vs 性能

The main conclusions from our analysis were:我们的分析得出的主要结论如下：

1. The Pareto frontier for coding tasks (i.e. best quality for a given cost) includes models from OpenAI, Anthropic, and open source. This means today, only a mix of tools can provide frontier performance.编码任务的帕累托前沿（即在给定成本下获得最佳质量）包括来自 OpenAI、Anthropic 和开源社区的模型。这意味着当前只有混合使用多种工具才能达到前沿性能。
2. Open models, and GLM 5.2 in particular, are now able to handle even the highest level of task difficulty.开源模型，尤其是 GLM 5.2，现已能够处理最高难度的任务。
3. The token price of a model is a poor indicator of actual costs incurred on end-to-end tasks. Larger models can be far more token efficient and have lower overall costs.模型的 token 价格并不能很好地反映端到端任务的实际成本。较大的模型在 token 效率上可能远胜一筹，且总体成本更低。
4. The harness a model is called from dramatically impacts cost and quality. In many cases, simple harnesses like Pi performed best on our workloads.调用模型的工具框架对成本和质量影响巨大。在许多情况下，像 Pi 这样的简单框架在我们的工作负载中表现最佳。

Let’s dive a bit deeper on each one.我们来逐一深入探讨。

Models cluster into rough “capability tiers”模型大致可分为“能力层级”

Specific results being a couple points off can often even out in real world tasks. We focused more on the thematic patterns that help us reason about which models to use for various tasks. In fact, the results showed clear clustering of the models and harnesses into 3 capability tiers.个别结果相差几个百分点在实际任务中往往可以抹平。我们更关注那些有助于推断不同任务应使用何种模型的总体模式。事实上，结果显示模型和工具框架清晰地聚类为三个能力层级。

Figure 2: Three distinct capability tiers emerged in our overall results, with nuance in which models were effective in each group图 2：整体结果中出现了三个不同的能力层级，各层级内有效模型略有差异

At the upper end of performance, we see that the most intelligent models are very effective at solving all kinds of problems, but they’re very expensive. Medium and lower intelligence models are still highly effective at the common tasks, and in many cases, they’re also significantly cheaper. 在性能高端，我们看到最智能的模型在解决各种问题方面非常有效，但成本极高。中等和较低智能的模型在常见任务中仍然高效，而且在许多情况下成本大幅降低。

Day to day, engineers do a lot of different things that vary significantly in complexity: common operational tasks like flipping a flag or updating configs don’t require extremely intelligent models, but deeper design explorations do. However, in the past, our default models were always the most expensive ones. Based on this analysis we determined we should push more work to the Haiku and GPT 5.4 Mini class of models.在日常工作中，工程师处理着复杂度各异的众多事务：像切换功能开关或更新配置这样的常见操作任务并不需要极其智能的模型，但深入的设计探索则需要。然而在过去，我们的默认模型总是最昂贵的。基于这一分析，我们决定将更多工作分配给 Haiku 和 GPT 5.4 Mini 这类模型。

Open models are here for coding开源模型已可用于编码

There’s been a lot of excitement about GLM 5.2, and our results showed evidence that GLM can be a daily driver model for a lot of our developers. It landed in the top capability tier, statistically tied with Opus 4.8 on quality, but costing $1.28/task against Opus’s $1.94.人们对 GLM 5.2 充满期待，我们的结果也表明 GLM 可以成为许多开发者的日常主力模型。它进入了最高能力层级，在质量上与 Opus 4.8 统计上持平，但每任务成本仅为 1.28 美元，而 Opus 为 1.94 美元。

The GLM quality scores are consistent with qualitative feedback we’ve gotten from internal developers who have been piloting GLM for daily development. Because of its great performance for everyday coding tasks, we’ve been focused on serving GLM with the best performance, and the evidence shows it’s time to start deploying these as daily drivers for coding. GLM 的质量评分与我们内部试用 GLM 进行日常开发的开发者给出的定性反馈一致。由于其在日常编码任务中的出色表现，我们一直致力于以最佳性能提供 GLM 服务，而证据表明，是时候将其部署为编码日常主力模型了。

Price-per-task vs price-per-token每任务成本 vs 每 token 成本

Developers often eyeball token costs to determine how expensive a model will be when completing coding tasks. We found, however, that token costs are often a poor indicator of overall task costs, due to variance in reasoning efficiency amongst models. This underscores the need for task-level benchmarking, since task shape and complexity may be different in different contexts.开发者通常通过 token 成本来估算模型完成编码任务的费用。然而我们发现，由于模型间推理效率的差异，token 成本往往无法很好地反映整体任务成本。这凸显了进行任务级基准测试的必要性，因为不同场景下任务的形态和复杂度可能不同。

As an example, Sonnet 5 is ~1.7x cheaper per token than Opus 4.8, but, on our tasks, we found that Sonnet cost $2.09/task vs Opus’s $1.94, while scoring six points lower on task completion (81% vs 87%). This was mostly because Sonnet 5 worked longer and read more to get there, consuming 1.9x more tokens. 例如，Sonnet 5 每 token 成本比 Opus 4.8 便宜约 1.7 倍，但在我们的任务中，我们发现 Sonnet 每任务成本为 2.09 美元，而 Opus 为 1.94 美元，同时 Sonnet 的任务完成得分低 6 分（81% vs 87%）。这主要是因为 Sonnet 5 工作时间更长、读取更多，消耗了 1.9 倍的 token。

Harnesses have a major impact on efficiency工具框架对效率有重大影响

When we ran the same model with the same thinking effort through two different harnesses (Claude Code/Codex vs Pi), we observed that the cost per task differed significantly (more than 2x in some cases), while quality remained the same. The main difference came down to how much context each harness fed the model on each turn.当我们使用相同的模型和相同的思考努力，通过两个不同的工具框架（Claude Code/Codex 和 Pi）运行时，观察到每任务成本差异显著（某些情况下超过 2 倍），而质量保持不变。主要区别在于每个工具框架每轮向模型提供的上下文数量。

Pi sent about 3x less context per turn. It managed context better, keeping a tighter working set and finishing the tasks in fewer runs.
Pi 每轮发送的上下文减少了约 3 倍。它更好地管理了上下文，保持了更紧凑的工作集，并以更少的运行次数完成任务。

The lesson here isn’t that one harness is always cheaper or that native harnesses are worse. Instead, model choice is only one piece of the puzzle. Establishing this flexibility is why we invested in Omnigent to make mode-and-harness swaps seamless. 这里的教训并非某个工具框架总是更便宜，或者原生工具框架更差。相反，模型选择只是拼图的一部分。正是为了建立这种灵活性，我们投资了 Omnigent，以实现模型和工具框架的无缝切换。

Why build your own benchmark?为什么要构建自己的基准测试？

Public benchmarks like SWE-Bench and TerminalBench are useful, but they can’t answer the questions we had. There are a few reasons for this:像 SWE-Bench 和 TerminalBench 这样的公共基准测试很有用，但它们无法回答我们面临的问题。原因有以下几点：

• The tasks are public, so the solutions leak into training data over time.任务是公开的，因此解决方案会逐渐泄露到训练数据中。
• We found the results weren’t representative for our codebase, which spans 10+ languages and many services written in Scala, Go, Rust, Java and Python, Bazel, Protobuf, and more.我们发现结果对我们的代码库没有代表性，该代码库涵盖 10 多种语言以及用 Scala、Go、Rust、Java 和 Python、Bazel、Protobuf 等编写的众多服务。

By building a benchmark on our own PRs, we can make these decisions with higher confidence that we won’t hamper our developers by rolling out optimizations.通过基于我们自己的 PR 构建基准测试，我们可以更有信心地做出这些决策，确保推出优化不会妨碍开发者。

How we built the benchmark我们如何构建基准测试

We used Unity AI Gateway to capture logs of all our coding interactions, which enabled us to analyze the complexity of the tasks engineers tackle using coding agents. There was a significant diversity in the task complexity, and about a quarter were tagged as low-complexity work and ~60% as medium complexity.我们使用 Unity AI Gateway 捕获所有编码交互的日志，从而能够分析工程师使用编码代理处理的任务的复杂度。任务复杂度差异很大，大约四分之一被标记为低复杂度工作，约 60% 为中等复杂度。

However, expensive models are the default models engineers use, so there was clearly a huge opportunity for improving efficiency.然而，昂贵的模型是工程师使用的默认模型，因此显然存在巨大的效率提升机会。

Task Construction任务构建

Our engineers merge thousands of code changes a day, so we already have a great dataset to build off. A good pull request is a rich artifact, with commits that show iteration from the developer, review by humans, and tests that help verify a code change is faithful to its intent. However, we needed several quality checks and filters to construct a high-quality benchmark out of them:我们的工程师每天合并数千次代码变更，因此我们已经拥有一个很好的数据集来构建基准。一个好的拉取请求是丰富的产物，包含显示开发者迭代的提交、人工审查以及帮助验证代码变更符合意图的测试。然而，我们需要多项质量检查和过滤来从中构建高质量的基准测试：

• Recency: We pull from recent history so the tasks reflect how we build today, including the frameworks, patterns, and conventions currently in use.时效性：我们从近期历史中选取，以便任务反映我们当前的构建方式，包括当前使用的框架、模式和约定。
• Human written: Bot commits, service accounts, fully AI generated changes, and auto-generated changes were filtered. 人工编写：过滤掉机器人提交、服务账户、完全由 AI 生成的变更以及自动生成的变更。
• Associated high quality test suite: We filtered for PRs that included high-quality tests for validating the code changes.关联的高质量测试套件：我们筛选出包含高质量测试以验证代码变更的 PR。
• Self-contained: The changes were confined to a few modules. 自包含：变更仅限于少数模块。
• Representative of typical tasks: We selected PRs from a distribution of tasks across the full stack: Scala backend services, Rust systems code, the React and TypeScript frontend, protobuf and gRPC contracts, and Bazel configs.代表典型任务：我们从整个技术栈的任务分布中挑选 PR：Scala 后端服务、Rust 系统代码、React 和 TypeScript 前端、protobuf 和 gRPC 合约以及 Bazel 配置。

Once we had candidate PRs, we focused on constructing well-specified tasks by:获得候选 PR 后，我们专注于构建描述清晰的任务，方法如下：

1. Gleaning the intent and summarizing it as a prompt. We read the PR to understand what it was actually for and then describe the outcome we want. Usually, that meant rewriting the PR description by stating the problem or goal, naming any constraints, and removing the description of the solution. It’s important to remove, for example, explanations of why a bug fix is the right one, since that makes the task too easy.提炼意图并总结为提示。我们阅读 PR 以了解其实际目的，然后描述我们想要的结果。通常这意味着重写 PR 描述，说明问题或目标，列出任何约束，并删除解决方案的描述。例如，删除关于为什么某个 bug 修复是正确做法的解释很重要，因为那会使任务过于简单。
2. Splitting out the relevant tests. The non-test files were the change the model has to reproduce on its own, so we set the test files aside and ensured we could compile that. Our build system can already determine which tests depend on the files that were touched in the original PR, so we ran all those test targets in full.分离相关测试。非测试文件是模型必须自行复现的变更，因此我们将测试文件放在一边，并确保可以编译。我们的构建系统已经能够确定哪些测试依赖于原始 PR 中修改的文件，因此我们完整运行了所有这些测试目标。

What came out of this exercise was a single task in the benchmark. Here’s a simplified example:这一过程最终形成了基准测试中的一个任务。以下是一个简化示例：

While we used scripting and AI to generate candidate tasks, we evaluated each sample by hand. In some cases, we found that tests in the original PR needed to be rewritten to allow for an alternative implementation or to be more rigorous, which we did manually (without AI). Similarly, we also found cases that required improving the task description to make them well-specified.虽然我们使用脚本和 AI 生成候选任务，但我们对每个样本都进行了手动评估。在某些情况下，我们发现原始 PR 中的测试需要重写，以允许替代实现或更加严格，我们手动进行了这些修改（不借助 AI）。同样，我们也发现了需要改进任务描述以使其更明确的情况。

Figure 3: A before-and-after from our test suite: the previous test anchored on verifying exact string match which resulted in some failures when the model tried to solve the task. This wasn’t a great way to test non-deterministic output so it was rewritten to grade behavior instead.图 3：测试套件的前后对比：之前的测试侧重于验证精确的字符串匹配，导致模型尝试解决任务时出现一些失败。这不是测试非确定性输出的好方法，因此被重写为评估行为而非精确匹配。

We instantiated the coding agent harnesses and models using their standard, out of the box setups, with all common tools that Databricks engineers would have available to them.我们使用标准的开箱即用设置实例化了编码代理工具框架和模型，并提供了 Databricks 工程师可用的所有常见工具。

When the agent explicitly said that it had completed the task, we checkpointed that code, patched the tests that were held out, and evaluated the tests to determine whether that task is a “pass” for that model + harness combination. We did not use an LLM judge to evaluate correctness, since we’ve found that this rewards sounding right over being right.当代理明确表示任务完成时，我们检查点该代码，应用之前保留的测试补丁，并评估测试以确定该模型+工具框架组合是否通过任务。我们没有使用 LLM 裁判来评估正确性，因为我们发现这会更倾向于听起来正确的答案而非真正正确的答案。

Additional Guardrails额外防护措施

In our early experiments, a few model scores looked too good to be true, so we manually inspected the traces to understand what happened in these agent trajectories. What we saw was that due to our original setup, the “correct” implementation was still recoverable in the Git history of the worktree! Every task had originated from a merged commit, so nothing stopped an agent with a shell from walking forward through git history to find it. To fix this issue, we sealed git history: for the length of each run, we cut the working copy off from the repository entirely.在早期实验中，几个模型得分高得离谱，因此我们手动检查了轨迹以了解这些代理路径中发生了什么。我们发现，由于最初的设置，“正确”的实现仍可在工作树的 Git 历史中恢复！每个任务都源自一个已合并的提交，因此拥有 shell 的代理完全可以沿着 Git 历史向前找到它。为解决这个问题，我们封闭了 Git 历史：在每次运行期间，我们将工作副本与仓库完全隔离。

What’s next?下一步计划？

We started with a simple question: can we use coding agents more efficiently? The answer is a definitive yes, and because we can be data driven, we can start building capabilities to select the right models automatically and track efficiency.我们从一个简单的问题开始：我们能否更高效地使用编码代理？答案是肯定的，而且由于我们可以数据驱动，我们可以开始构建自动选择合适模型并跟踪效率的能力。

Any company can do the same. Any team with a backlog of merged PRs is sitting on a benchmark already that no model has trained on, graded by the tests your team wrote. We’re actively adding more tasks (especially harder ones), and we plan to run every new agent/harness through it and feel more confident about our choices.任何公司都可以这样做。任何拥有大量已合并 PR 的团队都已经拥有一个基准测试，该测试尚未被任何模型训练过，并由你团队编写的测试进行评分。我们正在积极添加更多任务（尤其是更难的），并计划让每个新代理/工具框架通过它进行测试，从而对我们的选择更有信心。

At Databricks, we’ve always been wary of lock-in, not just to vendors, but to assumptions that make teams less flexible over time. That same instinct shaped our early bets on open formats and standards, and it shapes how we’re approaching AI now: measure what actually works on the code we ship, give engineers room to move across models and harnesses with consistent guardrails, and do optimizations to use AI effectively.在 Databricks，我们一直警惕锁定，不仅是对供应商的锁定，也包括那些会让团队随着时间推移失去灵活性的假设。同样的直觉塑造了我们早期对开放格式和标准的押注，也塑造了我们当前应对 AI 的方式：衡量哪些方法在我们的代码中真正有效，为工程师提供在一致防护措施下跨模型和工具框架切换的空间，并进行优化以有效使用 AI。

In a follow-up blog, we’ll talk more about how we’re using intelligent routing features in Unity AI Gateway and in Omnigent to help our developers use the most intelligent agents while being efficient.在后续博客中，我们将进一步讨论如何利用 Unity AI Gateway 和 Omnigent 中的智能路由功能，帮助开发者在保持效率的同时使用最智能的代理。