arXiv is now an independent nonprofit! Learn more
License: CC BY 4.0
arXiv:2607.06906v1 [cs.AI] 08 Jul 2026

The Harness Effect: How Orchestration Design Sets
the Token Economics of Enterprise Agentic AI
Harness 效应:编排设计如何决定企业智能体 AI 的 Token 经济学

Muayad Sayed Ali, Aliaksandra Novik, Anji Boddupally, Artem Yavorskyi, Chris Nickerson, Daniel Rica, Emily DuGranrut, Felix Leung, Garrett Prince, Grace Barnett, Heath Robinson, Hosain Al Ahmad, Jesse Resnick, Juan Carlos Farah, Jyothi Swaroop Meruga, Leonid Kuznetsov, Brock Perry, Luke Gorham, Marie Schmoll, Michael Paciullo, Saumya Das, Sharath Sheripally, Tommy Griscom, Mykyta Osadchyi, Neha Mantri, Nick Westrum, Olivia Benowitz, Parikshith Kulkarni, Radik Chernyshov, Rakshith Vasudev, Rohith Nadimpally, Vikas Gangadevi, and Waseem AlShikh Writer, Inc. {muayad, aliaksandra, anji, artem, chris, daniel, emily, felix, garrett, grace, heath, hosain, jesse, juan, jyothi, leonid, brock, luke, marie, michael, saumya, sharath, tommy, mykyta, neha, nick, olivia, parikshith, radik, rakshith, rohith, vikas, waseem}@writer.com
(July 2026)(2026 年 7 月)
Abstract摘要

The dominant pattern in agentic AI development is what we call token maxing: buying capability with tokens—longer reasoning traces, more agent turns, wider tool payloads, larger replayed contexts—so that tokens per task grow faster than task value. Falling per-token prices mask the pattern without fixing it; total spend rises anyway. We argue that the decisive lever against token maxing is the harness: the orchestration layer that assembles context, exposes tools, sequences turns, delegates work, and carries the observability and governance surface an enterprise deployment runs on. To isolate this layer we run a controlled swap: the same 22 locked evaluation tasks on the same six foundation models (Claude Sonnet 4.6, Gemini 3.1, Gemini Flash 3.5, Qwen 3.6, GLM 5.1, and Palmyra X6), changing only the orchestration layer: a conventional production agent loop (the frozen baseline) versus the Writer Agent Harness. Holding models constant, placing the harness at the core of execution cuts blended cost per task by 41% ($0.21 \rightarrow $0.12), median wall-clock by 44% (48 s \rightarrow 27 s), and tokens per task by 38% (14.2k \rightarrow 8.8k), while headline task-completion quality holds at parity (0.78 \rightarrow 0.81, directional at this sample size). The efficiency gains are model-invariant—every model gets cheaper, by 33% to 61%—while quality gains are capability-dependent: the improvement a model extracts from the harness correlates almost perfectly with its baseline strength (r=0.99r=0.99, n=6n=6), a phenomenon we term harness leverage. Quality per dollar rises 82% and task-completions per million tokens rise from 54.9 to 92.0. On this workload, the orchestration layer moved cost per task more than switching between the cheapest and most expensive model did. We formalize token economics at the orchestration layer, including an effective-input-price model under prompt caching; define token maxing; detail the six mechanism families behind the effect, from cache-shape discipline to failure-spend governance; compare six widely used agent systems on the same axes; and argue that the harness is the one component whose efficiency multiplies across every model an organization runs—present and future.智能体 AI 开发中,主流模式是“Token 最大化”:用 Token 买能力。推理过程更长、交互轮次更多、工具负载更重、上下文重复更多。任务价值没涨,Token 消耗却涨得飞快。单价跌了,掩盖了问题,但总支出还是高。我们认为,对付“Token 最大化”的杠杆是 Harness(编排层)。它负责组装上下文、挂载工具、调度轮次、委派任务,也是企业部署的观测与治理界面。我们做了一次受控实验:22 个固定任务,6 个基础模型(Claude Sonnet 4.6, Gemini 3.1, Gemini Flash 3.5, Qwen 3.6, GLM 5.1, Palmyra X6)。只换编排层:一边是常规生产 Agent 循环(基准),一边是 Writer Agent Harness。模型不变,用 Harness 做核心执行,任务平均成本降 41%($0.21 → $0.12),中位数耗时降 44%(48s → 27s),任务 Token 消耗降 38%(14.2k → 8.8k)。任务完成质量持平(0.78 → 0.81)。效率提升与模型无关,每个模型都省了 33% 到 61%。质量提升则看模型底子,模型越强,Harness 带来的红利越大,我们称之为“Harness 杠杆效应”。每美元质量提升 82%,每百万 Token 完成任务数从 54.9 升至 92.0。在此负载下,编排层对成本的影响,超过了更换模型本身。我们确立了编排层的 Token 经济学框架,定义了 Token 最大化,详述了六大机制,对比了六种主流 Agent 系统。Harness 是组织内唯一能让所有模型(无论现在还是未来)效率倍增的组件。

1 Introduction1 引言

An agentic task is not one model call. A single request—“reconcile these two contracts and draft the redline memo”—unfolds into a dozen or more turns: system prompt, tool schemas, retrieval payloads, intermediate reasoning, tool outputs, and, in naive implementations, the full replay of everything above on every subsequent turn. The token bill for the task is the sum over that loop, and the loop is governed not by the model but by the software around it. We call that software the harness: the orchestration layer that decides what enters the context window, which tools are visible, when to retrieve, when to retry, when to delegate, and when to stop.智能体任务不是一次模型调用。一个请求——“整理这两份合同,写份红线备忘录”——会拆成十几次交互:系统提示词、工具定义、检索负载、中间推理、工具输出。笨办法是每一轮都把前面全发一遍。任务的 Token 账单就是这些轮次的总和。决定这账单的不是模型,是外围的软件。我们管这软件叫 Harness:它决定什么进上下文窗口,哪些工具可见,何时检索,何时重试,何时委派,何时停止。

The industry’s default response to rising agent capability requirements has been to spend more tokens. Reasoning models emit thousands of deliberation tokens per answer; agent frameworks replay conversation history quadratically in the number of turns; tool ecosystems inject every schema into every call. Per-token prices have fallen steadily [24], and the falling unit price has financed the habit: teams treat tokens as nearly free at the margin and scale consumption to match. This is a textbook Jevons dynamic—efficiency gains in the resource lower its price and raise total consumption [1]—and it produces a development trajectory we name token maxing: quality is purchased with monotonically growing token intensity, at declining marginal quality per token (Definition 1, Section 3). Token maxing is invisible in benchmark tables, which report quality, and painfully visible in cloud invoices, which report tokens.应对智能体能力需求,业界默认做法是多花 Token。推理模型吐出几千个思考 Token;框架把对话历史平方级堆叠;工具生态把所有定义塞进每次调用。Token 单价在跌,这养成了坏习惯:团队觉得 Token 几乎免费,拼命增加消耗。这是典型的杰文斯悖论——资源效率提升,价格下降,总消费反而大增。我们称此路径为“Token 最大化”:用不断增加的 Token 强度换取质量,边际质量却在递减。基准测试表里看不出这问题,云账单里却触目惊心。

Most published efficiency work attacks this problem inside a single model call—prompt compression [9], budget-constrained reasoning [12], terse decoding [13], speculative decoding [14], serving-side memory management [15]—or between models, by routing and cascading [10, 11]. Both families accept the orchestration layer as given. Yet the harness controls every term of the token bill except the model’s own verbosity: the system prompt is replayed or cached by the harness; history is replayed or compacted by the harness; tool schemas are broadcast or scoped by the harness; retrieval payloads are sized by the harness; retries are triggered by the harness. If the harness is the layer that composes model calls into work, it is also the layer that sets the price of work.现有的效率优化多在模型单次调用内:压缩提示词、限制推理预算、精简解码、推测解码、服务端内存管理;或者在模型间做路由调度。这些方法都默认编排层不可动。但除了模型本身的话痨程度,Token 账单的每一项都由 Harness 控制:系统提示词是重发还是缓存、历史记录是重发还是压缩、工具定义是广播还是按需可见、检索负载大小、重试触发——全由 Harness 定。Harness 编排工作,也定下了工作的价格。

This paper measures that claim directly with a natural experiment the layered architecture makes possible: hold the tasks and the models constant, and swap only the orchestration layer. We evaluate 22 locked, capability-audited enterprise tasks on six foundation models spanning five vendors and three weight classes, under two orchestration layers: a conventional production agent loop, frozen on 2026-06-07 as the baseline, and the Writer Agent Harness, a model-agnostic orchestration layer with sub-agent delegation. Every task, prompt, model identifier, judge, and price table is identical across the two arms; the only variable is the orchestration code.本文通过受控实验量化这一结论:固定任务和模型,只换编排层。我们评估了 22 个企业级任务,涵盖 5 家厂商 6 个模型。对比两套编排:一套是 2026-06-07 冻结的常规生产 Agent 循环,另一套是具备子智能体委派功能的 Writer Agent Harness。任务、提示词、模型 ID、评测器、价格表完全一致。变量只有编排代码。

The headline results (Section 6):核心结果(第 6 节):

  1. 1.

    Efficiency moves a lot, and it moves everywhere. Blended across models, cost per task falls 41% ($0.21 \rightarrow $0.12), median wall-clock falls 44% (48 s \rightarrow 27 s; 1.8×\times faster), and tokens per task fall 38% (14.2k \rightarrow 8.8k). The per-model cost reduction ranges from -33% (Gemini 3.1) to -61% (Flash 3.5): every model gets substantially cheaper. On this workload the harness is a bigger cost lever than model choice: moving from the most to the least expensive model under the baseline saves 36%, while keeping any model and adopting the harness saves 33% to 61%.效率全方位提升。模型加权后,任务成本降 41%($0.21 → $0.12),中位耗时降 44%(48s → 27s,快了 1.8 倍),任务 Token 消耗降 38%(14.2k → 8.8k)。单模型成本降幅从 33%(Gemini 3.1)到 61%(Flash 3.5)。在此负载下,Harness 是比换模型更大的成本杠杆:基准配置下,换最便宜模型省 36%;保持模型不动换上 Harness,省 33% 到 61%。

  2. 2.

    Quality holds—and where it moves, it moves with the model. Headline task-completion is 0.78 \rightarrow 0.81 (a wash at n=22n=22). Across the 48 capability×\timesmodel cells, 30 improve, 11 are flat, and 7 regress; all 7 regressions occur on the three smaller models, concentrated in orchestration-heavy capabilities (tool use via MCP, multi-step Playbooks). The mean quality gain a model extracts from the harness tracks its baseline strength almost perfectly (r=0.99r=0.99): stronger models convert harness structure into quality; weaker models can be overwhelmed by it. We call this harness leverage (Section 6.4).质量保持稳定,且随模型能力联动。任务完成率 0.78 → 0.81(22 个样本下差异不大)。48 个能力×模型组合中,30 个提升,11 个持平,7 个下降。下降的全在三个较小模型上,且集中在编排密集的任务中(如 MCP 工具使用、多步执行)。模型从 Harness 中提取的质量增益与基准强度几乎完全正相关(r=0.99):强模型能把 Harness 结构转化为质量;弱模型容易被复杂编排压垮。我们称此为“Harness 杠杆效应”(第 6.4 节)。

  3. 3.

    Efficiency and capability are not a trade this time. Quality per dollar rises 82%; task-completions per million tokens rise from 54.9 to 92.0 (++68%). The harness also adds one net-new capability—delegated sub-agents—which crosses a usable reliability threshold only on the two strongest models (0.85–0.86), a concrete instance of the capability floor that harness features carry.效率与能力不再是取舍关系。每美元质量提升 82%;每百万 Token 的任务完成数从 54.9 升至 92.0(+68%)。Harness 还新增了一项能力——委派子智能体,这仅在两个最强模型上达到可用可靠性阈值(0.85–0.86),证明了 Harness 功能带来的能力底座。

Contributions. (1) A formal framing of token economics at the orchestration layer, including a decomposition of per-task token spend into harness-controlled terms, an effective-input-price model under prompt caching, and a definition of token maxing as a measurable development trajectory (Section 3). (2) A controlled harness-swap methodology that isolates orchestration effects from model effects, with a locked task set, frozen baseline, and trace-level accounting (Section 5). (3) Empirical findings across six models: model-invariant efficiency gains, capability-dependent quality gains (harness leverage), and a capability floor for advanced orchestration features (Section 6). (4) A mechanism inventory of the harness—cache-shape discipline, structured compaction, context offload, zero-token waiting, failure-spend governance, and a model-agnostic floor—mapped onto the cost decomposition, together with an architectural comparison of six widely used agent systems on the same axes (Sections 4.34.4). (4) An economic analysis of why harness efficiency compounds—it multiplies against every model, every vendor migration, and every unit of volume—and what that implies for the own-versus-rent decision on orchestration infrastructure (Section 7).贡献:(1) 编排层 Token 经济学框架,包括任务 Token 支出分解、提示词缓存下的有效输入价格模型、Token 最大化的定义。(2) 受控 Harness 替换方法论,隔离编排与模型影响。(3) 六个模型的实证发现:模型无关的效率提升、能力依赖的质量增益(Harness 杠杆)、高级编排的能力底座。(4) Harness 机制清单:缓存形状约束、结构化压缩、上下文卸载、零 Token 等待、故障支出治理。(5) 经济学分析:为何 Harness 效率会复合增长——它对每个模型、每个厂商、每单位流量都起效。

2 Related Work2 相关工作

Model-side and serving-side efficiency.模型侧与服务端效率。

A large literature reduces the cost of a given sequence of tokens: speculative decoding accelerates autoregressive generation without changing outputs [14]; PagedAttention and related memory managers raise serving throughput [15]. These techniques lower the price of a token. They do not reduce how many tokens an agent decides to consume, which is the quantity this paper targets.大量文献致力于降低给定 Token 序列的成本:推测解码加速生成,PagedAttention 提高吞吐。这些技术降低了 Token 单价,但不减少智能体决定消耗的 Token 数量。本文针对的是后者。

Prompt-side token reduction.提示词侧 Token 缩减。

LLMLingua and successors compress prompts while preserving task performance [9]; token-budget-aware reasoning constrains deliberation length [12]; Chain-of-Draft shows that terse intermediate reasoning can match verbose chain-of-thought [13, 4]. These methods operate within one call. An agent harness invokes such calls dozens of times per task and adds cross-call structure—history replay, tool schemas, retrieval—that no single-call method sees. Our results indicate the cross-call structure is where the larger savings sit: a 38% reduction in tokens per task achieved with no change to models or prompts.LLMLingua 等方法压缩提示词;预算感知推理限制思考长度;Chain-of-Draft 证明精简推理可匹配详细思维链。这些方法都在单次调用内起效。Agent Harness 每任务调用几十次,并增加跨调用结构——历史重放、工具定义、检索。我们的结果表明,跨调用结构才是节省大头:在不改模型和提示词的情况下,任务 Token 消耗减少 38%。

Routing and cascades.路由与级联。

FrugalGPT and RouteLLM cut cost by sending easy queries to cheap models [10, 11]. Routing chooses which model pays the bill; the harness determines how large the bill is for whichever model is chosen. The two are complementary, and Section 7.4 argues our capability-floor finding sharpens routing: requests should be routed not only by difficulty but by the orchestration features they will exercise.FrugalGPT 和 RouteLLM 通过将简单查询发给廉价模型来省钱。路由决定谁付账,Harness 决定账单多大。两者互补。第 7.4 节指出,我们的“能力底座”发现能优化路由:请求路由不仅要看难度,还要看它将调用的编排功能。

Agent architectures.智能体架构。

ReAct interleaved reasoning and tool calls [6]; Reflexion added verbal self-correction [7]; Toolformer trained tool invocation into the model [8]; SWE-agent showed that the agent–computer interface materially changes task success [18]; MemGPT introduced OS-style context paging [16]; the Model Context Protocol standardized tool connectivity [23]. This line established that scaffolding shapes capability. Long-context studies explain one mechanism: models attend unevenly across long inputs, so bloated contexts can reduce accuracy while raising cost [17]. We extend the scaffolding-shapes-capability result with its economic dual—scaffolding sets cost—and quantify both sides under a controlled swap.ReAct 交叉推理与工具调用;Reflexion 加入自我修正;Toolformer 训练工具调用;SWE-agent 展示了 Agent-计算机界面对成功率的影响;MemGPT 引入 OS 级上下文分页;MCP 标准化工具连接。这些研究确立了脚手架塑造能力。长上下文研究解释了机制:模型对长输入的关注不均,臃肿的上下文会降低精度并增加成本。我们扩展了这一结论,提出了其经济学对偶——脚手架设定成本,并进行了量化。

Evaluation and test-time compute.评测与测试时计算。

LLM-as-judge protocols [20] and locked benchmark suites [19] underpin our measurement design. Work on optimal test-time compute allocation [21] shows quality can be bought with inference tokens; our framing treats that purchase as an economic decision with a declining marginal rate, and the harness as the buyer of record. Reflective prompt-evolution systems such as GEPA [22] optimize the instructions a harness issues; they are natural complements that operate through the layer we measure.LLM-as-judge 和固定基准测试支撑了我们的测量设计。测试时计算分配研究表明,质量可以用推理 Token 换取;我们将此视为经济决策,Harness 是购买方。

Economics of inference.推理经济学。

Scaling laws priced training compute [2, 3]; Epoch AI documents multi-order-of-magnitude declines in inference prices [24]. Jevons observed in 1865 that efficiency in coal use raised total coal consumption [1]; Section 3 argues agentic AI is mid-Jevons, and that the harness is the layer at which consumption discipline can actually be implemented.缩放定律为训练计算定价;Epoch AI 记录了推理价格的量级下跌。杰文斯 1865 年观察到,煤炭利用效率提升反而增加了总消费。第 3 节认为智能体 AI 正处于杰文斯效应中期,Harness 是实施消费纪律的关键层。

The harness as first-class object.Harness 作为一等对象。

Closest to our framing, a recent line of work elevates the harness itself to an object of study: Gu argues agentic progress is now system scaling—context governance, memory, skill routing—as much as model scaling [27], and Harness-Bench measures, across 5,194 trajectories, that completion, efficiency, and failure behavior vary substantially across model–harness configurations [28]. Practitioner evidence points the same way: production context-engineering guidance treats KV-cache hit rate as the first metric of an agent [30]; provider documentation prices cached input at roughly a tenth of list [32]; controlled measurements show model quality degrading as input grows [31]; and Anthropic reports agents at 4×{\sim}4\times and multi-agent systems at 15×{\sim}15\times chat-level token consumption [29]. We add the missing economic layer: a formal token-economics framing at the harness (Section 3), the mechanism inventory that implements it (Section 4.3), and a controlled swap that prices it (Section 6).最近的工作将 Harness 提升为研究对象:Gu 认为智能体进步不仅是模型缩放,更是系统缩放——上下文治理、记忆、技能路由。Harness-Bench 测量显示,完成度、效率和失败行为随模型-Harness 配置大幅波动。生产实践也指向此处:KV 缓存命中率是 Agent 的首要指标;提供商对缓存输入降价;Anthropic 报告称 Agent 的 Token 消耗约为聊天级的 4 倍,多智能体系统约为 15 倍。我们补上了缺失的经济学层:Harness 的 Token 经济学框架、机制清单及受控实验。

3 Token Economics at the Orchestration Layer3 编排层的 Token 经济学

3.1 The bill for one agentic task3.1 单个智能体任务的账单

Let a task execute as a kk-turn agent loop. Turn ii submits input tokens TiinT^{\mathrm{in}}_{i} and receives output tokens TioutT^{\mathrm{out}}_{i}. With input price pinp_{\mathrm{in}} and output price poutp_{\mathrm{out}} (per token), the task costs假设任务执行 k 轮 Agent 循环。第 i 轮提交输入 Token Tiin,接收输出 Token Tiout。输入价格 pinp,输出价格 pout,任务成本为...

C=i=1k(pinTiin+poutTiout).C\;=\;\sum_{i=1}^{k}\Big(p_{\mathrm{in}}\,T^{\mathrm{in}}_{i}+p_{\mathrm{out}}\,T^{\mathrm{out}}_{i}\Big). (1)

The input side decomposes into terms the harness constructs:输入侧可拆解为 Harness 构建的项:

Tiin=Sisystem+Hihistory+Gitool schemas+Riretrieval+Uiuser turn,T^{\mathrm{in}}_{i}\;=\;\underbrace{S_{i}}_{\text{system}}+\underbrace{H_{i}}_{\text{history}}+\underbrace{G_{i}}_{\text{tool schemas}}+\underbrace{R_{i}}_{\text{retrieval}}+\underbrace{U_{i}}_{\text{user turn}}, (2)

and the loop multiplies everything by retries and dead-end branches, which the harness also governs. A naive harness replays the full transcript: Hi=j<i(Uj+Tjout+Oj)H_{i}=\sum_{j<i}(U_{j}+T^{\mathrm{out}}_{j}+O_{j}) where OjO_{j} is tool output at turn jj. Total input tokens then grow quadratically in turn count,循环将一切乘以重试和死胡同分支,这也是 Harness 管辖的。笨拙的 Harness 重放完整记录:Hi=∑j<i(Uj+Tjout+Oj),其中 Oj 是第 j 轮的工具输出。总输入 Token 随轮次平方级增长。

i=1kTiinkS+k(k1)2m¯+kG¯+iRi,\sum_{i=1}^{k}T^{\mathrm{in}}_{i}\;\approx\;kS\;+\;\frac{k(k-1)}{2}\,\bar{m}\;+\;k\bar{G}\;+\;\textstyle\sum_{i}R_{i}, (3)

with m¯\bar{m} the mean per-turn payload. A harness that compacts history, caches the invariant prefix (tool schemas included), offloads bulky tool outputs, and truncates retrieval to the evidential minimum converts the quadratic term to (approximately) linear (Figure 1). Nothing about the model changes; the bill does.m¯ 是平均每轮负载。如果 Harness 压缩历史、缓存不变前缀(含工具定义)、卸载大块工具输出、将检索截断至证据最小值,就能将平方级增长转为线性(图 1)。模型没变,账单变了。

Refer to caption
Figure 1: Where token maxing comes from. Illustrative cumulative input tokens for a kk-turn agent loop. Full-history replay grows as O(k2)O(k^{2}) (Eq. 3); harness-managed context—prefix caching, history compaction, offloaded tool outputs—grows as O(k)O(k). The shaded gap is spend that buys no quality. Schematic; measured aggregates appear in Section 6.图 1:Token 最大化的来源。k 轮循环的累计输入 Token。全历史重放呈 O(k²) 增长;Harness 管理的上下文——前缀缓存、历史压缩、卸载工具输出——呈 O(k) 增长。阴影部分是白花的钱。

Two further facts sharpen Eq. (1). First, agent workloads are input-dominated: because the transcript is re-submitted on every turn, production agents report input-to-output token ratios near 100:1 [30], so the pinp_{\mathrm{in}} term is nearly the whole bill. Second, the price of an input token is not one number. Providers serve tokens that repeat a previously seen prompt prefix from cache at a deep discount—roughly 0.1×0.1\times the base input rate [32, 30]. If a fraction hh of input tokens are cache reads billed at multiplier κ\kappa, the effective input price is两个事实加剧了 Eq.(1)。首先,Agent 负载是输入主导的:因为记录每轮重发,生产 Agent 的输入输出比接近 100:1,所以 pinp 项几乎就是全部账单。其次,输入 Token 价格不统一。提供商对缓存命中部分有深折扣——约为基础输入价的 0.1 倍。如果比例 h 的输入 Token 是缓存读取,有效输入价格为...

pineff=pin(1h(1κ)),κ0.1,p^{\mathrm{eff}}_{\mathrm{in}}\;=\;p_{\mathrm{in}}\big(1-h\,(1-\kappa)\big),\qquad\kappa\approx 0.1, (4)

so a harness that holds hh near 11 pays roughly a tenth of list price for the dominant term of Eq. (1). Crucially, hh is neither a model property nor a provider favor: it is a function of prompt byte-stability across turns, which is set entirely by how the orchestration layer assembles context. The harness therefore controls both factors of the bill—how many tokens are submitted (Eq. (2)) and the price at which the dominant ones are billed (Eq. (4)).所以,将 h 保持在 1 附近的 Harness,其核心项成本仅为定价的十分之一。关键在于,h 既不是模型属性,也不是提供商恩赐,而是由编排层如何组装上下文决定的。Harness 因此控制了账单的两个因子——提交了多少 Token,以及核心项按什么价格结算。

3.2 Token intensity, efficiency, and token maxing3.2 Token 强度、效率与 Token 最大化

Define token intensity τ\tau as total tokens per completed task and let Q[0,1]Q\in[0,1] be task quality under a fixed judge protocol. We use two efficiency metrics:定义 Token 强度 τ 为每完成任务的总 Token 数,Q 为评测协议下的任务质量。我们使用两个效率指标...

η$=QC(quality per dollar),CPM=Q106τ(task-completions per million tokens).\eta_{\mathdollar}\;=\;\frac{Q}{C}\quad\text{(quality per dollar)},\qquad\mathrm{CPM}\;=\;\frac{Q\cdot 10^{6}}{\tau}\quad\text{(task-completions per million tokens)}. (5)

Definition 1 (Token maxing). A development trajectory {(Qt,τt)}t\{(Q_{t},\tau_{t})\}_{t} exhibits token maxing if token intensity grows, τt+1>τt\tau_{t+1}>\tau_{t}, while marginal quality per token declines, Qt+1Qtτt+1τt<Qtτt\frac{Q_{t+1}-Q_{t}}{\tau_{t+1}-\tau_{t}}<\frac{Q_{t}}{\tau_{t}} — i.e., each release buys quality at a worse token exchange rate than the system’s running average.定义 1(Token 最大化):开发轨迹 {(Qt,τt)}t 若满足 τt+1 > τt 且边际质量每 Token 递减,即 (Qt+1-Qt)/(τt+1-τt) < Qt/τt,则表现为 Token 最大化。即每次发布换来的质量提升,不如系统平均水平。

Token maxing is individually rational for a team judged on benchmark quality and collectively expensive for the organization paying per token. It is also self-reinforcing under falling prices: when pp drops, the budget-constrained optimum shifts toward higher τ\tau, and total spend NCN\cdot C can rise even as pτp\cdot\tau per task falls—Jevons’ coal, restated for tokens [1, 24]. The escape is not cheaper tokens but a higher CPM\mathrm{CPM}: doing the same work with fewer tokens. Equation (2) says four of the five input terms, plus retries, are code, not model. That places the escape route squarely in the harness.Token 最大化对追求基准分的团队是理性的,对付钱的组织则是昂贵的。在降价环境下,它会自我强化:当 p 下降,预算约束下的最优解倾向于更高 τ,导致总支出不降反升。逃离路径不是更便宜的 Token,而是更高的 CPM:用更少 Token 做同样的工作。Eq.(2) 表明,五个输入项中的四个,加上重试,全是代码,不是模型。逃离路径就在 Harness 里。

3.3 Why harness savings compound3.3 为何 Harness 节省会复合增长

Let a fleet run NN tasks per month across models mm\in\mathcal{M} with mix weights wmw_{m}. Monthly spend is mwmNCm\sum_{m}w_{m}N\,C_{m}. A model-side optimization improves one CmC_{m}; a routing policy improves the mix ww; a harness improvement multiplies every CmC_{m} by a factor (1sm)(1-s_{m}) simultaneously—and keeps multiplying when the model set changes, because it is implemented above the model API. Empirically (Section 6.2) sm[0.33,0.61]s_{m}\in[0.33,0.61] across all six models with no exceptions, which is what makes the harness the unusual asset in the stack: the one whose returns are indifferent to which vendor wins the model race.设月度任务量为 N,模型组合权重为 wm,月支出为 ∑wm N Cm。模型侧优化只改进一个 Cm;路由策略改进 wm;而 Harness 改进将每个 Cm 乘以系数 (1-sm)。由于它在模型 API 之上实现,当模型集更换时,它依然有效。实证(第 6.2 节)显示 sm 在 0.33 到 0.61 之间,无一例外。这是 Harness 的独特资产属性:无论哪家模型厂商赢,它都赚钱。

4 The Writer Agent Harness4 Writer Agent Harness

4.1 What the harness is4.1 Harness 是什么

In Writer’s architecture the harness is the runtime between the application and any foundation model: it owns (i) context assembly—system prompt, conversation state, retrieval payloads from document and knowledge-graph grounding; (ii) the tool layer—native tools and external connectors via the Model Context Protocol [23], including schema exposure and call mediation; (iii) workflow execution—multi-step Playbooks run end to end; (iv) delegation—spawning scoped sub-agents and merging their results (absent from the baseline loop); and (v) observability—a trace shim that records prompt tokens, completion tokens, tool events, and wall-clock (duration_seconds) for every turn of every run. The trace shim is what makes this paper’s accounting possible: cost is computed at report build by applying a pinned public price table to the recorded token counts, so both arms are priced identically. The same layer is the enterprise control plane: the trace shim that meters tokens is also the audit trail, the progressive tool disclosure that saves tokens is also tool governance, and deterministic workflow execution is what makes agent behavior reviewable—efficiency and control are properties of one component, which is why the harness sits at the core of everything else the platform does.Writer 的架构中,Harness 是应用与基础模型间的运行时:它拥有 (i) 上下文组装;(ii) 工具层(原生工具及 MCP);(iii) 工作流执行;(iv) 委派(生成子智能体);(v) 可观测性(记录 Token、工具事件、耗时)。追踪 shim 让本文的核算成为可能:成本在报告生成时通过固定价格表计算。同一层也是企业控制平面:计量 Token 的 shim 也是审计跟踪,渐进式工具披露既省 Token 又做治理,确定性工作流让行为可审查——效率与控制是同一组件的属性。

The harness is model-agnostic by construction: the executing model is a configuration value (model_name), which is precisely what permits the controlled swap in Section 5—six models, two orchestration layers, one task set, one judge panel. It is the runtime behind Writer’s Action Agent, whose public results (61% on the hardest GAIA level; the top overall score on the CUB computer-use benchmark) situate the production system this evaluation instruments [26].Harness 是模型无关的:执行模型只是一个配置值。这正是第 5 节受控实验的前提——6 个模型、2 种编排、1 套任务。它是 Writer Action Agent 后面的运行时,该系统在 GAIA 和 CUB 基准测试中表现顶尖。

4.2 The harness versus a conventional agent loop4.2 Harness 与常规 Agent 循环

The baseline is the conventional agent loop as previously deployed in production, frozen on 2026-06-07. Its design is the industry default and worth stating concretely, because each element is a token-economics decision made by omission: a monolithic system prompt of roughly 49 KB replayed on every turn; tool invocation parsed by regular expressions from XML emitted in the text stream; destructive middle-truncation when context overflows; per-model prompt tuning; and waits implemented as polling. The Writer Agent Harness, evaluated at its intended general-availability configuration, replaces each with an explicit contract: one execution path for any of the six models; native tool calling only (the XML path was deleted, not ported); non-destructive structured compaction in place of truncation; early streaming with live tool-call status; cancellation and retry as first-class states; and sub-agents—delegating a sub-task to one or more spawned agents with scoped context and merging their results—a capability the baseline loop does not possess at all.基准是 2026-06-07 冻结的常规生产循环。其设计是行业默认做法,每个元素都是通过“省略”做出的经济决策:约 49KB 的单体系统提示词每轮重发;XML 解析工具调用;上下文溢出时破坏性截断;按模型调优提示词;轮询等待。Writer Agent Harness 替换了这些:统一执行路径;仅原生工具调用;非破坏性结构化压缩;早期流式传输与实时状态;取消与重试作为一等状态;子智能体委派——这是基准循环完全不具备的能力。

4.3 Mechanisms: how the harness rewrites the bill4.3 机制:Harness 如何改写账单

Mapping the observed deltas onto Eqs. (2) and (4), the savings must come from the terms the harness rewrote. The design goal can be stated in one sentence: maximize the fraction of tokens that are (a) cached, (b) decision-relevant, and (c) spent inside committed, recoverable work—and enforce all three with structure rather than model behavior. Six mechanism families implement it.节省必须来自 Harness 重写的项。设计目标一句话:最大化 (a) 缓存,(b) 决策相关,(c) 投入在可恢复工作中的 Token 比例——并用结构而非模型行为来强制执行。六大机制实现之。

Refer to caption
Figure 2: The two-zone prompt. A byte-stable prefix (tool-schema catalog, stable system prompt, append-only transcript) carries up to four provider cache breakpoints [C1–C4]; everything that changes per turn is confined to a volatile tail that is rebuilt each turn and structurally excluded from caching. Measured on an identical-prefix call: 99.9% of prompt tokens (7,876 of 7,886) served as cache reads, pricing the dominant input term at \approx0.1×\times list (Eq. 4).图 2:双区提示词。字节稳定的前缀(工具定义、系统提示词、追加记录)承载四个缓存断点;每轮变化的放在易变尾部,结构上排除在缓存外。实测:99.9% 的输入 Token 被缓存读取,核心项成本降至定价的 0.1 倍。

(1) Cache-shape discipline: the two-zone prompt.(1) 缓存形状约束:双区提示词。

Every prompt the harness emits has a deliberate physical shape (Figure 2): a byte-stable prefix—the full tool-schema catalog, the stable system prompt, and the append-only durable transcript—followed by a volatile tail rebuilt every turn (clock, file listings, plan state, one-shot reminders, voice and custom instructions). The split is enforced as a correctness rule, not an optimization: anything that changes per turn is structurally banned from the prefix, and the cache-marker logic refuses to place a breakpoint at or after the first volatile message. Up to four provider breakpoints are pinned—after the schema catalog, after the stable system prompt, and sliding along the two newest durable turns—with one-hour retention latched per session so the policy never flips mid-run. Measured on an identical-prefix call in the harness repository, 7,876 of 7,886 prompt tokens (99.9%) were served as cache reads; by Eq. (4) with κ0.1\kappa\!\approx\!0.1, such a turn pays roughly a tenth of list price for its dominant term. Because agent workloads are input-dominated, cache hit rate is the single highest-leverage cost variable a harness controls—and it lives in the placement of SiS_{i}, GiG_{i}, and HiH_{i}, not in the model.Harness 发出的每个提示词都有物理形状:字节稳定的前缀(工具目录、系统提示词、持久记录)加上每轮重建的易变尾部(时间、文件列表、计划状态)。这是正确性规则而非优化:任何每轮变化的内容严禁进入前缀。缓存标记逻辑拒绝在易变消息后设置断点。四个断点被固定——工具目录后、系统提示词后、两轮持久记录后,会话内一小时保留,防止策略中途翻转。实测中 99.9% 的输入 Token 为缓存读取。因为 Agent 负载是输入主导的,缓存命中率是 Harness 控制的最高杠杆。

(2) Structured, incremental, cache-aware compaction.(2) 结构化、增量、缓存感知压缩。

At 80% of the model’s input budget, older history is folded into a typed checkpoint with four artifacts: durable memory (decisions, constraints, rejected approaches), an eight-section execution summary written for resumability (current state, files touched, errors, next steps), preserved verbatim user requirements, and skill references. A live tail of the 4–12 most recent messages (\leq30% of budget) always survives verbatim; each checkpoint folds the previous one forward, so compaction cost stays incremental; summarization runs on a cheaper helper model, off the paying loop; and an empty or degraded summary aborts the compaction rather than persisting it. If the prompt still does not fit, a deterministic ladder (shrink the tail, then middle-truncate with an explicit restate-what-matters nudge) guarantees a sendable request. Compaction and caching are co-designed: a summarizer that rewrote history every turn would destroy the very prefix stability Eq. (4) prices, so checkpoints are durable rows and the rebuilt prompt becomes the new cacheable prefix. This is the mechanism that converts Eq. (3) from quadratic to linear (Figure 1) without the baseline’s destructive truncation.当输入达到模型预算的 80%,旧历史被折叠为带四个工件的检查点:持久记忆、八段执行摘要、用户需求、技能引用。最近 4-12 条消息的实时尾部始终保留;压缩成本保持增量;摘要在廉价辅助模型上运行,不在主循环计费;若摘要为空或降级则放弃压缩。如果仍超限,确定性阶梯(缩减尾部,再中间截断)保证请求可发送。压缩与缓存协同设计:摘要器若每轮重写历史会破坏前缀稳定性,因此检查点是持久行,重建的提示词成为新的可缓存前缀。此机制将 Eq.(3) 从平方级转为线性。

(3) Context offload: tokens the model never pays for.(3) 上下文卸载:模型无需支付的 Token。

A family of mechanisms keeps information available without keeping it in context, attacking RiR_{i} and the tool-output share of HiH_{i}. Sub-agents act as context firewalls: a child agent performs broad reading or searching in its own context and returns a summary capped at 8 KB, with citations carried on a metadata sidecar the parent model never reads; delegation is depth-capped and idempotent under retries, so delegated exploration cannot inflate the parent loop. Skills use progressive disclosure: the prompt carries only a name-and-description table, the full skill document is installed in the sandbox and read only when invoked (inline seed capped at 20K characters), and compaction later drops skill bodies deterministically while keeping the reference. Bulky tool outputs spill to files: shell output beyond 20K characters is head-and-tail previewed with the complete output written to a workspace file (under a banner forbidding the model to infer success from the preview), web fetches inline 8K characters and spill full pages, and oversized file reads are rejected with guidance rather than silently truncated—the filesystem is the unbounded memory; the context holds pointers. Plan and canvas state are event-sourced and projected once per turn as a compact rendering, with plan-tool results replaced by one-line acknowledgements so state is never duplicated across the transcript; the per-turn re-rendering doubles as objective recitation, countering long-horizon goal drift [30]. Background-task results are delivered exactly once as an ephemeral tail reminder; loaded media is bounded (at most four images or 2 MB in context, older ones evicted with a reload stub); and side work—telemetry, reasoning-trace titles, deliverable materialization—runs off the paying loop on helper models.一系列机制在不占用上下文的情况下保持信息可用。子智能体充当上下文防火墙:子智能体在自身上下文中执行阅读,仅返回 8KB 摘要,引用挂在元数据侧车,父模型不读;委派深度受限。技能使用渐进披露:提示词只带名称描述,完整文档安装在沙箱中,仅在调用时读取(内联种子限制 20K 字符)。大块工具输出溢出到文件:Shell 输出超过 20K 字符时仅预览头尾,其余写文件;网页抓取内联 8K 字符,其余溢出;超大文件读取拒绝而非静默截断——文件系统是无界内存,上下文只存指针。计划与画布状态事件溯源,每轮投影为紧凑渲染,状态从不跨记录重复。背景任务结果仅交付一次;加载媒体有界;辅助工作在辅助模型上运行。

(4) Zero-token waiting; durability as economics.(4) 零 Token 等待;持久性即经济学。

Waiting is a continuation, not a loop: when a run needs a human answer, an approval, or a long background job, it suspends durably at zero token cost and resumes on an ingress event—no polling turns. The same durability layer bounds catastrophic spend: every event is journaled to a write-ahead log before it is streamed, crashed or preempted runs resume under generation fencing at exactly the next sequence number, and tool results are persisted before they are shown. A crash that loses a 40-turn run means re-buying 40 turns of tokens; here it means resuming from durable state.等待是延续而非循环:当运行需要人工确认或长任务时,它在零 Token 成本下持久挂起,并在事件触发时恢复——没有轮询。持久层限制了灾难性支出:每个事件在流式传输前先记入预写日志,崩溃或抢占的运行从持久状态恢复。崩溃意味着不用重买 40 轮 Token,而是从检查点恢复。

(5) Failure-spend governance.(5) 故障支出治理。

Every failure is classified into a typed class (rate limit, stall, timeout, malformed stream, provider outage, permanent) before any decision is made, and only whitelisted classes fall through to the next provider in the route plan. Mid-stream failures become discarded attempts: the partial draft is cleared and no side effects—tool executions, durable writes—can originate from a discarded attempt, which is precisely the piece generic library fallbacks omit (streaming failover in orchestration libraries is documented to work only before the first chunk [34]). A circuit breaker halts a model that re-issues a byte-identical failing tool call three times, with cause-aware steering (change the arguments vs. back off); truncated outputs and output-length caps terminate loudly as typed states, never silently; the loop is capped at 50 iterations and tool parallelism at four. Retries, dead ends, and doom loops are the multiplier on Eq. (1) that no per-token discount fixes; the harness bounds the multiplier.每个故障在决策前被分类,仅白名单类别的故障才触发路由计划中的下一提供商。流中故障成为丢弃尝试:部分草稿被清除,无副作用产生。断路器会停止重复发出三次相同错误工具调用的模型;截断输出和长度上限会明确终止而非静默;循环上限 50 次,工具并行度上限 4 次。重试、死胡同和死循环是 Eq.(1) 的乘数,折扣解决不了,Harness 限制了乘数。

(6) A model-agnostic floor.(6) 模型无关的底座。

Which model runs, over which providers, in what fallback order, is a typed route plan supplied as data—the loop never branches on a model name—and every provider stream is normalized into one chunk contract before the loop sees it. Native tool calling is the only invocation path, backed by schema hygiene for weaker models: $refs are inlined, double-encoded JSON arguments are recovered, framework internals are scrubbed from validation errors, and overloaded tool schemas are split when weaker models misuse them. These floors hold whatever model is driving, which is the structural explanation for the model-invariance of Section 6.2 and for why harness leverage (Section 6.4) appears as a clean function of model capability: the harness fixes the floor; the model sets the ceiling.哪个模型运行、走哪个路由,是作为数据提供的计划——循环从不依赖模型名称分支。每个提供商流在循环可见前被归一化为统一契约。原生工具调用是唯一路径,辅以针对弱模型的模式卫生:$refs 内联,双重编码 JSON 参数恢复,框架内部错误被清洗。这些底座支撑了任何模型,解释了模型无关的效率提升,以及为何“Harness 杠杆”表现为模型能力的函数:Harness 设定底座,模型设定上限。

The through-line of all six families is that token economy and output quality are one lever pulled once. Long, distractor-dense contexts measurably degrade every frontier model tested [31, 17]; a mechanism that removes stale or bulky tokens is simultaneously cutting the bill and cleaning the model’s working set—which is what the grounding results of Section 6.6 show in miniature.六大机制的共同主线是:Token 经济与输出质量是同一杠杆。长、干扰密集的上下文会降低所有前沿模型的效果;移除陈旧或臃肿 Token 的机制,既省钱又清洗了模型的工作集。

Two implementation notes, stated for completeness: sub-agent delegation currently executes as blocking calls, and automatic whole-turn retry on output-length caps is designed but not yet shipped. The mechanisms above describe the configuration under evaluation.两个实现说明:子智能体委派目前执行为阻塞调用,输出长度上限的自动重试已设计但尚未发布。以上机制描述了评估中的配置。

4.4 Token economics in existing harnesses and frameworks4.4 现有 Harness 与框架中的 Token 经济学

Recent work argues the harness should be a first-class object of design and evaluation [27], and cross-harness measurement finds that agent capability is a property of the model–harness configuration rather than of the model alone [28]. Our claim here is narrower and economic: among widely used agent systems, token economics is nowhere a first-class, published contract. Table 1 summarizes the landscape on the mechanism families of Section 4.3, assessed from public documentation and from a source-level study of caching behavior conducted during this harness’s design.最近工作认为 Harness 应成为设计与评估的一等对象。我们的主张更窄且经济:在主流 Agent 系统中,Token 经济学从未成为一等发布的契约。表 1 总结了基于公开文档的机制评估。

Table 1: Token-economic mechanisms across widely used agent systems, assessed from public documentation and a design-time source study—not head-to-head measurement (see Section 8). “app” = left to the application to build and budget.表 1:主流 Agent 系统的 Token 经济机制评估。“app”表示留给应用层去构建和预算。
System Deployment class Model- portable Structural cache policy Compaction contract Firewalled delegation Zero-token waits Per-task accounting
Claude Code single-user client no yes yes partial no no
Claude Cowork single-user client no yes yes partial no no
LangGraph library yes app app app partial no
CrewAI framework yes app partial no no no
AutoGen / AG2 framework yes app app no partial no
Hermes Agent personal agent yes partial yes yes partial no
Writer Harness multi-tenant runtime yes yes yes yes yes yes

Vendor-integrated clients.厂商集成客户端。

Claude Code and Claude Cowork [37] are among the most sophisticated harnesses in wide deployment, and several of their patterns—cache-breakpoint latching, byte-stable prefixes, sub-agent task splitting—were adopted from or validated against them during this harness’s design. The differences are deployment class and contract: both are single-user, client-side tools bound to one model vendor; their internal token management is not exposed as a per-task accounting surface; and their economics do not transfer across models by construction. They optimize a session for a person. An enterprise runtime must meter a fleet.Claude Code 和 Claude Cowork 是部署最广的复杂 Harness 之一,其模式(缓存断点、字节稳定前缀、子智能体拆分)在设计时被参考。差异在于部署类别:它们是绑定单一模型厂商的单用户工具;其内部 Token 管理未暴露为 per-task 核算界面;其经济性不跨模型迁移。它们优化的是个人的会话,而企业运行时必须计量整个机队。

Orchestration libraries.编排库。

LangGraph [34] supplies graph orchestration, checkpointing, and interrupts as excellent low-level primitives—and deliberately leaves prompt-cache policy, compaction, context offload, and failure-spend governance to the application. Token economics thereby becomes the application team’s unbudgeted responsibility: nothing in the framework measures it, enforces it, or reports it, and the framework’s own documentation concedes the streaming-failover gap noted in Section 4.3.LangGraph 提供出色的图编排、检查点和中断原语——但故意将缓存策略、压缩、卸载、故障治理留给应用。Token 经济学因此成为应用团队未预算的责任:框架不测量、不强制、不报告。

Multi-agent conversation frameworks.多智能体对话框架。

CrewAI [35] and the AutoGen lineage [33] (the original framework, now in maintenance mode, continued by the AG2 community fork and succeeded for Microsoft stacks by the Agent Framework) organize work as conversations among role-prompted agents that share transcripts. Shared-transcript multi-agency is a token multiplier by construction: each participating agent re-reads the growing conversation and each carries its own role preamble. The best public measurement is Anthropic’s own: agents consume roughly 4×4\times the tokens of chat and multi-agent systems roughly 15×15\times, with token volume explaining about 80% of performance variance on their research evaluation [29]. That is Definition 1 operating as an architecture—quality purchased through token multiplication. It can be worth paying, as Anthropic is candid that only high-value, parallelizable tasks justify it; but none of these frameworks meters the multiplier per task, and none firewalls delegated context the way a capped-summary sub-agent contract does.CrewAI 和 AutoGen lineage 将工作组织为共享记录的对话。共享记录本身就是 Token 乘数:每个智能体重读不断增长的对话。Anthropic 自己的测量显示,Agent 消耗约为聊天级的 4 倍,多智能体系统约为 15 倍。这是 Token 最大化作为架构的表现——用 Token 换质量。虽然有时值得,但没有框架会按任务核算乘数,也没有框架像子智能体契约那样防火墙化委派上下文。

Open personal harnesses.开源个人 Harness。

Hermes Agent [36] is a capable, genuinely model-agnostic open-source harness with isolated sub-agents. The design-time source study found, however, that it does not place its tool schemas inside a cached prefix—forfeiting the largest single discount available on an input-dominated workload (Eq. (4))—and, as a personal single-user agent, it publishes no per-task token or cost contract.Hermes Agent 是一个模型无关的开源 Harness。但设计研究发现,它未将工具定义放入缓存前缀——放弃了输入主导负载下最大的折扣——且作为个人 Agent,它不发布 per-task Token 或成本契约。

What the comparison shows.比较结论。

The last column of Table 1 is this paper in miniature. Without per-task token accounting built into the orchestration layer, token maxing is unobservable—and what is unobservable is unmanaged. The Writer harness’s trace shim makes cost per task and CPM measurable per release precisely because the meter lives in the same layer that spends the tokens; the release-gate posture of Section 7.3 is only possible on top of it. We stress the epistemic status: Table 1 compares public designs and documentation, not measured runs; configuration-level measurement of the token dimension across harnesses, in the style of Harness-Bench [28], is natural future work.表 1 的最后一列是本文的缩影。没有内置于编排层的 Token 核算,Token 最大化就不可见,不可见即不可管理。Writer Harness 的 trace shim 使成本可测量,因为计量器与消耗 Token 的层级在一起。我们强调认识论状态:表 1 对比的是公开设计,而非实测运行。

5 Experimental Setup5 实验设置

5.1 Design5.1 设计

The experiment is a paired swap. Both arms execute the same 22 tasks on the same six models with the same judges and the same pinned price table; only the orchestration layer differs. The baseline arm is the conventional production loop invoked via model_name override, run once and frozen on 2026-06-07 as the reference. The harness arm is the Writer Agent Harness at its intended general-availability configuration. Because prompts, models, and judges are held fixed, any systematic difference in tokens, cost, latency, or quality is attributable to the harness.实验是配对交换。两组在 6 个模型上执行相同的 22 个任务,使用相同评测器和价格表;仅编排层不同。基准组是 2026-06-07 冻结的常规生产循环。Harness 组是 Writer Agent Harness。由于提示词、模型、评测器固定,任何系统性差异均归因于 Harness。

5.2 Task set5.2 任务集

The evaluation uses n=22n=22{} locked-baseline prompts, audited for capability coverage and for defensible pass/fail criteria before either arm ran. The set spans nine capability areas that mirror production usage of an enterprise agent platform: Model & System Awareness (MSA: identity, scope, safe refusal), Grounding & Retrieval (GDR: document and knowledge-graph grounding, web search, citation), Content Generation (CNG), Playbooks (PLY: predefined multi-step workflows executed end to end), MCP tool use (MCP: discovering and calling external connectors), Presentations (PRN: slide generation, an early-stage capability), Voice (VOX: brand voice and custom instructions), Image Analysis & Generation (IMG), and—under the harness only—Sub-agents (delegated work, net-new). Tasks include multi-turn cases (e.g., a three-turn Medicare grounding dialogue) and long-horizon cases (multi-step research synthesis).评估使用 22 个锁定基准提示词,审计了能力覆盖和通过/失败标准。领域涵盖:模型与系统意识(MSA)、接地与检索(GDR)、内容生成(CNG)、工作流(PLY)、MCP 工具使用(MCP)、演示(PRN)、语音(VOX)、图像分析(IMG)以及仅在 Harness 下的子智能体(SUB)。任务包含多轮对话和长周期研究合成。

5.3 Models5.3 模型

Table 2: Models under evaluation. Every model runs identically under both harness arms.表 2:待测模型。每种模型在两套测试框架下,跑法完全一样。
Model Vendor Class Role in analysis
Claude Sonnet 4.6 Anthropic frontier strong generalist
Gemini 3.1 Google DeepMind frontier strong generalist
Gemini Flash 3.5 Google DeepMind fast tier low-cost generalist
Qwen 3.6 Alibaba open weight candidate model
GLM 5.1 Zhipu AI open weight candidate model
Palmyra X6 Writer enterprise platform-native model

Table 2 lists the six models: two frontier generalists, one fast-tier model, two open-weight candidates, and Writer’s platform-native Palmyra X6. The spread is deliberate—the central claim is about a layer above the model, so it must be tested across weight classes and vendors, not on a single flagship.表 2 列出六种模型:两款前沿通用型,一款快速响应型,两款开源权重型,还有 Writer 自家的 Palmyra X6。选得杂,是有意的。核心论点在于模型之上的那层架构。所以不能只测旗舰款,得看不同量级、不同厂商的表现。

5.4 Metrics5.4 指标

Quality. Headline quality is task-completion scored by an LLM-judge panel against the audited pass/fail criteria [20]. Secondary judges score coherence, communication quality, and hallucination; these are reported per-prompt and did not headline any decision. Tokens. Prompt and completion tokens are recorded per turn by the trace shim; token intensity τ\tau is the per-task total. Cost. Computed at report build as (prompt tokens×pin)+(completion tokens×pout)(\text{prompt tokens}\times p_{\mathrm{in}})+(\text{completion tokens}\times p_{\mathrm{out}}) using a pinned OpenRouter price table, identically for both arms. Latency. Wall-clock from first token to terminal event (duration_seconds), median across runs. UX. A hand-scored qualitative rubric (streaming responsiveness, tool-call transparency, citation rendering, error/cancel handling, multi-turn coherence) reported separately from automated metrics.质量。以 LLM 裁判组根据审计标准 [20] 判定的任务完成度为准。辅以连贯性、沟通质量和幻觉评分,这些只在单次提示词中记录,不作为决策依据。Token。由追踪垫片记录每轮的输入输出 Token。Token 强度 τ 为单任务总量。成本。在报告生成时,按 OpenRouter 冻结价格表计算:(输入 Token × 进价) + (输出 Token × 出价)。两套框架算账方式一致。延迟。从第一个 Token 到结束事件的墙上时间(秒),取多次运行的中位数。UX。人工评分,包含流式响应、工具调用透明度、引用呈现、错误处理和多轮连贯性,与自动化指标分开报告。

5.5 Statistical posture5.5 统计姿态

At n=22n=22{}, quality deltas are directional, not statistically significant, and we treat them as such throughout: the headline +0.03+0.03 is reported as parity, not improvement. Cost, token, and latency deltas are large, uniform in sign across all 22 prompts and all six models, and therefore decisive at this sample size in a way the quality deltas are not. Candidate-model failures on advanced features are expected and scored, not excluded; model-agnosticism is a claim about the execution path, not a promise of uniform capability.样本 n=22。质量差值仅具方向性,统计学上不显著。所以统一视为持平,+0.03 的提升不算改进。成本、Token 和延迟的差值很大,且在 22 个提示词和 6 种模型中表现一致。在这个样本量下,这些指标具有决定性。模型在高级功能上失败是预料之中,照样记分,不予剔除。模型无关性是架构路径的声明,不是对能力的承诺。

6 Results6 结果

6.1 Headline: the same work, 38% fewer tokens6.1 核心:活儿没变,Token 省了 38%

Refer to caption
Figure 3: Blended efficiency across six models and 22 tasks. Replacing the baseline loop with the harness, models held constant: cost per task -41%, median wall-clock -44%, tokens per task -38%.图 3:六种模型、22 项任务的综合效率。模型不变,换上新框架:单任务成本降 41%,中位延迟降 44%,Token 消耗降 38%。
Table 3: Parity scorecard, blended across models. Quality is task-completion under the locked judge protocol; derived rows use Eq. (5).表 3:模型综合表现对比。质量按锁定裁判协议判定;衍生行使用公式 (5) 计算。
Dimension Baseline Harness Δ\Delta Reading
Quality (task-completion) 0.78 0.81 +0.03+0.03 wash at n=22n=22
Cost / task $0.21 $0.12 41%-41\% decisive
Wall-clock / task (median) 48 s 27 s 44%-44\% decisive
Tokens / task 14.2k 8.8k 38%-38\% decisive
Quality per dollar (η$\eta_{\mathdollar}) 3.71 6.75 +82%+82\% derived
Completions per Mtok (CPM) 54.9 92.0 +68%+68\% derived
UI/UX (hand-scored) baseline improved \uparrow qualitative

Table 3 and Figure 3 give the blended result. Replacing the baseline with the harness removes 38% of token intensity and 41% of cost while quality stays at parity—which is the operational definition of escaping token maxing: the token exchange rate improved instead of degrading. In efficiency terms, a dollar buys 82% more completed quality and a million tokens completes 68% more tasks. The hand-scored UX rubric moved the same direction for the mechanical reasons one would expect from the architecture: earlier streaming, live tool status instead of a spinner, clean cancellation.表 3 和图 3 展示了结果。换上新框架,Token 强度减 38%,成本减 41%,质量维持不变。这就是所谓的“逃离 Token 上限”:Token 兑换率提升了,没降级。换算效率,一美元能多买 82% 的质量,一百万 Token 能多做 68% 的任务。人工 UX 评分也同步向好:流式传输更快,工具状态实时显示,取消操作更干脆。

6.2 Model invariance: everyone gets cheaper6.2 模型不变:谁用谁便宜

Refer to caption
Figure 4: Per-model efficiency under the orchestration swap. Every model’s cost and latency fall; reductions range from -33% to -61% (cost) and -33% to -55% (latency). The effect is a property of the orchestration layer, not of any model.图 4:编排切换后的模型效率。所有模型成本和延迟全降。成本降幅 33% 到 61%,延迟降幅 33% 到 55%。这是编排层的功劳,跟模型本身无关。
Table 4: Per-model efficiency, baseline \rightarrow harness. Same tasks, same judges, same price table.表 4:模型效率,基准 → 框架。任务、裁判、价格表全一样。
Cost / task Wall-clock / task (median)
Model Base Harness Δ\Delta Base Harness Δ\Delta
Claude Sonnet 4.6 $0.24 $0.15 39%-39\% 52 s 31 s 41%-41\%
Gemini 3.1 $0.19 $0.13 33%-33\% 49 s 29 s 40%-40\%
Gemini Flash 3.5 $0.18 $0.07 61%-61\% 60 s 27 s 55%-55\%
Qwen 3.6 $0.16 $0.09 44%-44\% 44 s 29 s 33%-33\%
GLM 5.1 $0.21 $0.11 47%-47\% 47 s 29 s 38%-38\%
Palmyra X6 $0.25 $0.12 52%-52\% 50 s 26 s 48%-48\%

Table 4 and Figure 4 break the effect out per model. Two observations. First, uniformity: six models, five vendors, three weight classes, and not one exception—every model’s cost falls by at least a third. This is the signature of a layer-level effect: if the savings came from any model-specific behavior, the spread would show it. Second, magnitude relative to model choice: under the baseline, moving from the most expensive model (Palmyra X6 at $0.25) to the cheapest (Qwen 3.6 at $0.16) saves 36%; keeping any model and adopting the harness saves 33% to 61%. On this workload, the orchestration layer is a larger cost lever than the model menu—which inverts where optimization attention usually goes.表 4 和图 4 展示了单模型表现。两点观察。第一,统一性:六种模型、五家厂商、三个量级,无一例外。成本全降至少三分之一。这是层级效应的特征:如果是因为模型特异性,分布不会这么齐。第二,相对于模型选择的量级:基准下,最贵模型(Palmyra X6,$0.25)换成最便宜(Qwen 3.6,$0.16)省 36%;保持模型不动,换上新框架省 33% 到 61%。在这个工作负载下,编排层比选模型更能省钱。这颠覆了常规的优化重心。

The largest relative gains land on the fast tier: Flash 3.5 drops 61% in cost and 55% in latency. This is consistent with the decomposition in Eq. (2): for small, cheap models, harness overhead (replayed history, broadcast schemas) is a larger share of the total bill, so removing it removes proportionally more.快速 tier 收益最大:Flash 3.5 成本降 61%,延迟降 55%。这符合公式 (2) 的拆解:对于小而便宜的模型,框架开销(历史重放、广播模式)占大头,去掉它,省得就多。

6.3 Quality: parity in aggregate, model-dependent at the edges6.3 质量:整体持平,边缘依赖模型

Refer to caption
Figure 5: Quality parity across 48 capability×\timesmodel cells. Each point is one capability score for one model, baseline (xx) vs. harness (yy). Points above the diagonal improved under the harness: 30 improve, 11 are flat (|Δ|0.02|\Delta|\leq 0.02), 7 regress. All regressions belong to the three smaller models, concentrated in orchestration-heavy capabilities (MCP, Playbooks, Presentations).图 5:48 个能力×模型单元的质量对比。每个点代表一个模型的能力分数,基准 (x) 对比框架 (y)。对角线以上为提升:30 个提升,11 个持平,7 个退步。退步全发生在三个小模型上,集中在编排密集型能力(MCP、Playbooks、演示文稿)。

Figure 5 plots all 48 capability×\timesmodel cells (eight legacy capabilities ×\times six models; the full matrix is Appendix A). The mass sits on or above the diagonal: 30 cells improve, 11 are flat, 7 regress. The regressions are not random. Every one occurs on Flash 3.5, Qwen 3.6, or GLM 5.1, and six of the seven fall in capabilities that exercise orchestration hardest—MCP tool use (Qwen 0.15-0.15, GLM 0.06-0.06, Flash 0.04-0.04), Playbooks, and Presentations. The frontier models and Palmyra improve most in exactly those categories (MCP: Sonnet +0.10+0.10, Palmyra +0.10+0.10; GDR: Sonnet +0.10+0.10, Palmyra +0.12+0.12). The same richer harness that a strong model converts into quality, a weaker model experiences as load.图 5 绘制了 48 个单元。重心在对角线上方:30 升,11 平,7 降。退步不随机。全发生在 Flash 3.5、Qwen 3.6 或 GLM 5.1 上。七个里有六个是编排重头戏:MCP 工具使用、Playbooks 和演示文稿。前沿模型在这些领域提升最明显。框架越强,强模型转化出质量,弱模型却感到吃力。

6.4 Harness leverage6.4 框架杠杆

Refer to caption
Figure 6: Harness leverage scales with baseline capability. Mean quality gain from adopting the harness (Δq¯\Delta\bar{q}, eight capabilities) against the model’s baseline strength. Stronger models extract more quality from the same orchestration upgrade (r=0.99r=0.99; n=6n=6, suggestive). Palmyra X6 +0.079+0.079 and Sonnet 4.6 +0.073+0.073 lead; Qwen 3.6 is net negative (0.031-0.031).图 6:框架杠杆随基准能力增长。采用框架后的平均质量增益与模型基准强度对比。模型越强,从编排升级中提取的质量越多。Palmyra X6 和 Sonnet 4.6 领先;Qwen 3.6 为负。

Collapsing each model to its mean across the eight scored capabilities gives a single number for how much quality it extracted from the swap: Palmyra X6 +0.079+0.079, Sonnet 4.6 +0.073+0.073, Gemini 3.1 +0.050+0.050, GLM 5.1 +0.028+0.028, Flash 3.5 +0.010+0.010, Qwen 3.6 0.031-0.031. Plotted against the baseline (Figure 6), the relationship is nearly linear (r=0.99r=0.99; six points, so suggestive rather than conclusive). We call the slope harness leverage: the rate at which a model converts orchestration structure into quality. The economic consequence is asymmetric and useful: efficiency gains are unconditional, quality gains are earned by capability. A weak model under the harness still gets its 44–61% cost cut; it simply does not also get better. This decouples the two reasons to upgrade a harness and lets each be priced on its own.将每个模型在八项能力上的平均分折算成一个数:Palmyra X6 +0.079,Sonnet 4.6 +0.073,Gemini 3.1 +0.050,GLM 5.1 +0.028,Flash 3.5 +0.010,Qwen 3.6 -0.031。与基准对比(图 6),关系近乎线性。我们称之为框架杠杆:模型将编排结构转化为质量的比率。经济后果不对称:效率提升是无条件的,质量收益靠能力换取。弱模型用新框架,成本照样降 44–61%,只是质量没变好。这让效率和质量的升级理由解耦,可以分别定价。

6.5 The net-new capability and its floor6.5 新增能力及其门槛

Table 5: Sub-agent delegation (net-new under the harness; no baseline arm exists). Scores are task-completion on delegation tasks: spawn one or more scoped sub-agents and merge their results.表 5:子代理委派(框架下的新增能力)。分数为委派任务的完成度:生成一个或多个子代理并合并结果。
Sonnet 4.6 Gemini 3.1 Flash 3.5 Qwen 3.6 GLM 5.1 Palmyra X6
Sub-agents 0.85 0.70 0.45 0.42 0.58 0.86

The harness’s one genuinely new capability—delegating work to spawned sub-agents—lands above a usable reliability threshold only on the two strongest models (Table 5: Palmyra X6 at 0.86, Sonnet 4.6 at 0.85), degrades on Gemini 3.1 (0.70) and GLM 5.1 (0.58), and is not yet dependable on the fast tier (0.42–0.45). This is the harness-leverage result in its sharpest form: an orchestration feature carries a capability floor, below which exposing it produces failures rather than function. Delegation also has a distinctive token-economics profile—sub-agent context is scoped, so delegated tokens are spent in a bounded side-loop rather than inflating the parent context—the context-firewall contract of Section 4.3: a capped summary returns, citations ride a sidecar, and the parent loop never pays for the child’s exploration.框架真正的新能力——委派工作给子代理——只有在两个最强模型上才跨过可用门槛。Gemini 3.1 和 GLM 5.1 退步,快速 tier 尚不可靠。这是框架杠杆最直观的表现:编排功能有能力门槛,低于门槛只会产生故障,而非功能。委派的 Token 经济学也不同:子代理上下文受限,委派的 Token 在侧回路消耗,不膨胀父上下文。这实现了 4.3 节的上下文防火墙:返回汇总,引用走侧车,父循环不为子循环的探索买单。

6.6 Per-prompt texture: where the tokens went and what came back6.6 提示词纹理:Token 去向与产出

Four exemplar prompts show the distribution behind the aggregates. The three-turn Medicare grounding dialogue improves from 0.60 to 0.90—the single largest quality jump in the set—under a harness whose retrieval shaping sends less, better-selected evidence, consistent with the finding that long, noisy contexts degrade attention [17]. Identity/refusal holds at 0.90/0.90 while its cost halves ($0.04 \rightarrow $0.02): safety behavior at half price. Contract Q&A moves +0.07+0.07 (0.75 \rightarrow 0.82). The most expensive task in the set, multi-step research synthesis, drops from $0.61 to $0.33 (46%-46\%) but regresses in quality (0.80 \rightarrow 0.60)—the one place the aggregate parity conceals a real trade, driven by the smaller models, and the reason the release recommendation (Section 7) holds candidates back pending a fix rather than shipping the regression. Secondary judges move with the headline: coherence 0.85 \rightarrow 0.88, communication 0.79 \rightarrow 0.80, hallucination clean in both arms.四个示例展示了总量背后的分布。三轮 Medicare 溯源对话质量从 0.60 升至 0.90,是所有任务中提升最大的。框架优化了检索,送入的证据更少、更准。这印证了长而嘈杂的上下文会分散注意力 [17]。身份/拒绝任务在 0.90/0.90 持平,成本减半:安全行为,半价实现。合同问答升 0.07。最贵的任务——多步研究综合——成本从 $0.61 降至 $0.33,但质量退步(0.80 → 0.60)。这是唯一一处总量持平掩盖的真实折损,由小模型驱动。这就是为什么发布建议(第 7 节)要扣住候选模型,先修补再发布。辅裁判指标同步:连贯性 0.85 → 0.88,沟通 0.79 → 0.80,幻觉两边都干净。

7 Discussion7 讨论

7.1 The harness is the price-setter7.1 框架是定价者

The results separate cleanly along the framing of Section 3. The terms of Eq. (2) that are pure code—history, schemas, retrieval size, retries—moved for every model, because code executes the same way regardless of which model reads its output. The quality terms are mediated by the model’s ability to exploit structure, and they moved with capability. One layer, two currencies: the harness sets the price of work unconditionally, and sets the ceiling of work jointly with the model. The layer that sets the price is also the layer an enterprise governs by; efficiency and control concentrate in the same component.结果按第 3 节的框架清晰分割。公式 (2) 中纯代码的部分——历史、模式、检索大小、重试——对所有模型都变了,因为代码执行方式不随模型而变。质量项由模型利用结构的能力调节,随能力变动。一层架构,两种货币:框架无条件设定工作价格,与模型共同设定工作上限。设定价格的层级,也是企业治理的层级;效率和控制权集中在同一个组件。

This reframes a common procurement instinct. Teams comparing $/Mtok across vendors are comparing pp; the bill is p×τp\times\tau, and τ\tau belongs to the harness. On this workload the harness moved the bill more than the entire spread of the model menu did (Section 6.2). An organization that rents its orchestration layer has outsourced the variable it controls most.这重构了采购逻辑。团队比较厂商间的 $/Mtok,其实是在比 pp;账单是 p×τ,而 τ 在框架手里。这次测试,框架对账单的影响力超过了所有模型差价的总和。租用编排层的组织,等于外包了自己最能掌控的变量。

7.2 Fleet economics: why the savings compound7.2 车队经济学:节省为何叠加

Refer to caption
Figure 7: Harness savings at fleet scale. Blended cost per task applied to monthly task volume. At one million agent tasks per month, the harness is worth $90k/month over the baseline ($1.08M/year); the gap widens linearly with volume and multiplies against every model in the mix.图 7:车队规模下的框架节省。按月任务量计算。每月一百万任务,框架比基准省 $90k/月($1.08M/年);差距随规模线性扩大,且在所有模型组合中叠加。

Per-task deltas understate the object of interest, which is fleet spend. At the blended rates measured here, an organization running one million agent tasks a month pays $210k under the baseline loop and $120k under the harness—$1.08M a year from an orchestration change alone (Figure 7). Three properties make this saving unusual among AI optimizations. It is model-portable: implemented above the API, it survived all six models here and, by construction, applies to models that do not exist yet. It is volume-linear: it grows with exactly the quantity—agentic task volume—that is growing fastest in enterprise deployments. And it stacks: per-token price declines [24], routing [10, 11], and prompt-level compression [9, 13] all multiply against it rather than substituting for it. Latency compounds the same way: 1.8×\times faster per task is also 1.8×\times the throughput per unit of orchestration infrastructure, and shorter loops are cheaper to retry.单任务差值低估了车队支出。按测得的综合费率,每月一百万任务,基准循环需 $210k,框架只需 $120k——仅凭编排变更一年省 $1.08M(图 7)。这种节省在 AI 优化中很少见。它模型可移植:在 API 之上实现,六种模型通用,未来模型也适用。它规模线性:随企业部署中增长最快的代理任务量同步增长。它可叠加:Token 单价下降、路由和提示词压缩,全都能乘上去,而不是替代它。延迟也一样:单任务快 1.8 倍,意味着编排基础设施的吞吐量也高 1.8 倍,且短循环重试成本更低。

7.3 Escaping token maxing: change the KPI7.3 逃离 Token 上限:改变 KPI

Definition 1 suggests the managerial fix is a measurement fix. Teams that report quality alone will token-max, because tokens are someone else’s line item; teams that report CPM\mathrm{CPM} or η$\eta_{\mathdollar} cannot. The swap studied here moved CPM\mathrm{CPM} from 54.9 to 92.0 completions per million tokens—the direction opposite to the industry trajectory—while quality held. We suggest CPM belongs next to quality in every agent release gate, for the same reason performance-per-watt sits next to performance in chip design: it is the number that predicts the bill.定义 1 暗示管理上的修复是测量上的修复。只报质量的团队会触碰 Token 上限,因为 Token 是别人的开支;报 CPM 或 η$ 的团队则不会。本次切换将 CPM 从 54.9 移至 92.0,与行业轨迹背道而驰,同时质量未降。建议在每个代理发布门槛中加入 CPM,就像芯片设计中性能/瓦特必须紧跟性能一样:它是预测账单的数字。

7.4 Harness–model co-design and routing by feature demand7.4 框架-模型协同设计与按需路由

The regression pattern (all seven on smaller models, concentrated in MCP/Playbooks) and the sub-agent floor (Table 5) argue that orchestration features are not free to expose. A harness is a contract the model must be strong enough to honor. Two practical consequences follow. First, harness capabilities should degrade gracefully by model tier—scoping down tool catalogs and disabling delegation below the floor—rather than presenting one interface to all models. Second, routing research [10, 11] should route on feature demand, not just prompt difficulty: a request that will exercise sub-agents belongs on Palmyra X6 or Sonnet 4.6 regardless of how simple its text looks, while a grounded Q&A request can take the 61%-cheaper fast tier with no quality penalty (Figure 5, GDR row: every model improves).退步模式(全在小模型,集中在 MCP/Playbooks)和子代理门槛(表 5)证明编排功能不是免费的。框架是模型必须有足够能力去遵守的契约。两点后果。第一,框架功能应随模型等级优雅降级——低于门槛就缩减工具目录、禁用委派,而不是对所有模型展示同一界面。第二,路由研究应按功能需求路由,不只是按提示词难度:涉及子代理的请求,无论文本多简单,都应去 Palmyra X6 或 Sonnet 4.6;而溯源问答请求可以走便宜 61% 的快速 tier,质量无损(图 5,GDR 行:所有模型都提升)。

7.5 Release posture implied by the data7.5 数据暗示的发布姿态

Read as a deployment decision, the evidence supports: general availability on the two models above the sub-agent floor (Palmyra X6, Sonnet 4.6); holding the open-weight candidates pending the multi-step-research fix; and leading external claims with efficiency—which is uniform and decisive—rather than quality, which is directional at n=22n=22{}. We record this here because it is the honest reading of Table 3: the temptation in industry reporting is to headline +0.03+0.03 quality; the defensible headline is 38%-38\%{} tokens at parity.作为部署决策,证据支持:在跨过子代理门槛的两个模型(Palmyra X6, Sonnet 4.6)上全面发布;开源候选模型暂缓,等待多步研究修复;对外宣传以效率为主——这是统一且决定性的——而不是质量,质量在 n=22 下仅具方向性。这是表 3 的诚实解读:行业报告倾向于标榜 +0.03 的质量;可辩护的标题是:同等质量下,Token 减少 38%。

8 Threats to Validity8 有效性威胁

Sample size. n=22n=22{} prompts is sufficient for the uniform, large efficiency deltas and insufficient for quality inference; all quality claims are directional, and the headline is parity, not improvement. Single-run baseline. The baseline was run once and frozen (2026-06-07); run-to-run variance on the baseline is unmeasured. Judge dependence. Task-completion is LLM-judged; judge bias is partially mitigated by locked criteria and secondary judges but not eliminated [20]. Price-table dependence. Dollar figures inherit one pinned public price table; token and latency results are price-independent and tell the same story. Workload shape. The task set mirrors an enterprise assistant workload (grounding, workflows, tools, content); results may differ on long-horizon coding benchmarks [19, 18]. Six points. The harness-leverage correlation is computed over six models and should be read as a strong pattern awaiting a wider model panel. One pair. We compare one baseline loop and one harness, both from a single vendor; the framing of Section 3 is general, but the magnitudes are specific to this pair. Cross-harness claims are architectural. Section 4.4 characterizes other systems from public documentation and a design-time source study; it contains no cross-harness measurements, and Table 1 should be read as a design comparison, not a benchmark.样本量。n=22 对统一、巨大的效率差值足够,对质量推断不足;所有质量主张均为方向性,结论为持平而非提升。单次基准。基准运行一次并冻结(2026-06-07);未测量运行波动。裁判依赖。任务完成度由 LLM 判定;通过锁定标准和辅裁判部分缓解了偏见,但未完全消除 [20]。价格表依赖。美元数据继承冻结的公共价格表;Token 和延迟结果与价格无关。工作负载形态。任务集模拟企业助理(溯源、流程、工具、内容);长程编码基准可能不同 [19, 18]。六个点。框架杠杆相关性基于六个模型,应视为等待更广泛模型验证的强模式。一对一。对比一个基准循环和一个框架,均来自同一供应商;第 3 节的架构是通用的,但量级仅针对这对组合。跨框架主张属架构层面。第 4.4 节基于公开文档和设计源研究;不含跨框架测量,表 1 应视为设计对比而非基准。

9 Conclusion9 结论

Held to the same 22 tasks and judged by the same panel, six foundation models did the same work for 38% fewer tokens, 41% less money, and in 44% less time—because the software around them changed, not the models. The efficiency was unconditional; the quality gains went to the models strong enough to earn them; the one new capability came with a capability floor. Token maxing is a choice made at the orchestration layer, and it can be unmade there—mechanism by mechanism: cache the stable, compact the old, offload the bulky, suspend the waiting, bound the failing. The harness is not the plumbing of an agent system. On the evidence here, it is the P&L.在同样的 22 项任务和裁判组下,六种基础模型多做了活,Token 少用 38%,钱少花 41%,时间快 44%——因为软件变了,模型没变。效率提升是无条件的;质量收益归于有能力的模型;新能力附带能力门槛。Token 上限是编排层的选择,也可以在那里撤销——机制对机制:缓存稳定的,压缩旧的,卸载笨重的,挂起等待的,限制失败的。框架不是代理系统的管道。证据表明,它是盈亏表。

Disclosure.披露。

The authors are employed by Writer, Inc., which develops the Agent Harness and the Palmyra model family evaluated in this work; the last author is co-founder and CTO. The evaluation design—frozen baseline, locked prompts, identical judges and price tables across arms, candidate-model failures scored rather than excluded—is intended to make the comparison auditable; the release artifacts are enumerated in Appendix D.作者受雇于 Writer, Inc.,该公司开发了本文评估的 Agent Harness 和 Palmyra 模型家族;最后一位作者是联合创始人兼 CTO。评估设计——冻结基准、锁定提示词、跨组统一裁判和价格表、候选模型失败照常记分——旨在使对比可审计;发布产物列于附录 D。

References参考文献

  • [1] W. S. Jevons. The Coal Question: An Inquiry Concerning the Progress of the Nation, and the Probable Exhaustion of Our Coal-Mines. Macmillan, 1865. [1] W. S. Jevons. 煤炭问题:关于国家进步及煤矿枯竭可能性的探讨。Macmillan, 1865.
  • [2] J. Kaplan, S. McCandlish, T. Henighan, et al. Scaling laws for neural language models. arXiv:2001.08361, 2020. [2] J. Kaplan, S. McCandlish, T. Henighan, 等. 神经语言模型的缩放定律。arXiv:2001.08361, 2020.
  • [3] J. Hoffmann, S. Borgeaud, A. Mensch, et al. Training compute-optimal large language models. arXiv:2203.15556, 2022. [3] J. Hoffmann, S. Borgeaud, A. Mensch, 等. 训练计算最优的大语言模型。arXiv:2203.15556, 2022.
  • [4] J. Wei, X. Wang, D. Schuurmans, et al. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022. [4] J. Wei, X. Wang, D. Schuurmans, 等. 思维链提示激发大语言模型的推理能力。NeurIPS, 2022.
  • [5] X. Wang, J. Wei, D. Schuurmans, et al. Self-consistency improves chain of thought reasoning in language models. In ICLR, 2023. [5] X. Wang, J. Wei, D. Schuurmans, 等. 自洽性提升语言模型的思维链推理。ICLR, 2023.
  • [6] S. Yao, J. Zhao, D. Yu, et al. ReAct: Synergizing reasoning and acting in language models. In ICLR, 2023. [6] S. Yao, J. Zhao, D. Yu, 等. ReAct:语言模型中推理与行动的协同。ICLR, 2023.
  • [7] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning. In NeurIPS, 2023. [7] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, S. Yao. Reflexion:带口头强化学习的语言代理。NeurIPS, 2023.
  • [8] T. Schick, J. Dwivedi-Yu, R. Dessì, et al. Toolformer: Language models can teach themselves to use tools. In NeurIPS, 2023. [8] T. Schick, J. Dwivedi-Yu, R. Dessì, 等. Toolformer:语言模型可以自学使用工具。NeurIPS, 2023.
  • [9] H. Jiang, Q. Wu, C.-Y. Lin, Y. Yang, and L. Qiu. LLMLingua: Compressing prompts for accelerated inference of large language models. In EMNLP, 2023. [9] H. Jiang, Q. Wu, C.-Y. Lin, Y. Yang, L. Qiu. LLMLingua:压缩提示词以加速大语言模型推理。EMNLP, 2023.
  • [10] L. Chen, M. Zaharia, and J. Zou. FrugalGPT: How to use large language models while reducing cost and improving performance. arXiv:2305.05176, 2023. [10] L. Chen, M. Zaharia, J. Zou. FrugalGPT:如何在降低成本的同时提升大语言模型性能。arXiv:2305.05176, 2023.
  • [11] I. Ong, A. Almahairi, V. Wu, et al. RouteLLM: Learning to route LLMs with preference data. arXiv:2406.18665, 2024. [11] I. Ong, A. Almahairi, V. Wu, 等. RouteLLM:利用偏好数据学习 LLM 路由。arXiv:2406.18665, 2024.
  • [12] T. Han, Z. Wang, C. Fang, S. Zhao, S. Ma, and Z. Chen. Token-budget-aware LLM reasoning. In Findings of ACL, 2025. arXiv:2412.18547. [12] T. Han, Z. Wang, C. Fang, S. Zhao, S. Ma, Z. Chen. Token 预算感知的 LLM 推理。ACL Findings, 2025. arXiv:2412.18547.
  • [13] S. Xu, W. Xie, L. Zhao, and P. He. Chain of draft: Thinking faster by writing less. arXiv:2502.18600, 2025. [13] S. Xu, W. Xie, L. Zhao, P. He. 草稿链:通过少写来更快思考。arXiv:2502.18600, 2025.
  • [14] Y. Leviathan, M. Kalman, and Y. Matias. Fast inference from transformers via speculative decoding. In ICML, 2023. [14] Y. Leviathan, M. Kalman, Y. Matias. 通过投机解码实现 Transformer 的快速推理。ICML, 2023.
  • [15] W. Kwon, Z. Li, S. Zhuang, et al. Efficient memory management for large language model serving with PagedAttention. In SOSP, 2023. [15] W. Kwon, Z. Li, S. Zhuang, 等. PagedAttention:大语言模型服务的内存高效管理。SOSP, 2023.
  • [16] C. Packer, S. Wooders, K. Lin, et al. MemGPT: Towards LLMs as operating systems. arXiv:2310.08560, 2023. [16] C. Packer, S. Wooders, K. Lin, 等. MemGPT:迈向作为操作系统的 LLM。arXiv:2310.08560, 2023.
  • [17] N. F. Liu, K. Lin, J. Hewitt, et al. Lost in the middle: How language models use long contexts. TACL, 2024. [17] N. F. Liu, K. Lin, J. Hewitt, 等. 迷失在中间:语言模型如何利用长上下文。TACL, 2024.
  • [18] J. Yang, C. E. Jimenez, A. Wettig, et al. SWE-agent: Agent–computer interfaces enable automated software engineering. In NeurIPS, 2024. [18] J. Yang, C. E. Jimenez, A. Wettig, 等. SWE-agent:代理-计算机接口实现自动化软件工程。NeurIPS, 2024.
  • [19] C. E. Jimenez, J. Yang, A. Wettig, et al. SWE-bench: Can language models resolve real-world GitHub issues? In ICLR, 2024. [19] C. E. Jimenez, J. Yang, A. Wettig, 等. SWE-bench:语言模型能解决现实世界的 GitHub 问题吗?ICLR, 2024.
  • [20] L. Zheng, W.-L. Chiang, Y. Sheng, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In NeurIPS, 2023. [20] L. Zheng, W.-L. Chiang, Y. Sheng, 等. 用 MT-Bench 和 Chatbot Arena 评判 LLM 裁判。NeurIPS, 2023.
  • [21] C. Snell, J. Lee, K. Xu, and A. Kumar. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. In ICLR, 2025. arXiv:2408.03314. [21] C. Snell, J. Lee, K. Xu, A. Kumar. 优化扩展 LLM 测试时计算比扩展模型参数更有效。ICLR, 2025. arXiv:2408.03314.
  • [22] L. A. Agrawal, S. Tan, D. Soylu, et al. GEPA: Reflective prompt evolution can outperform reinforcement learning. In ICLR, 2026. arXiv:2507.19457. [22] L. A. Agrawal, S. Tan, D. Soylu, 等. GEPA:反射式提示进化优于强化学习。ICLR, 2026. arXiv:2507.19457.
  • [23] Anthropic. Introducing the Model Context Protocol. Technical note, November 2024. https://www.anthropic.com/news/model-context-protocol. [23] Anthropic. 介绍模型上下文协议。技术笔记,2024 年 11 月。
  • [24] B. Cottier, B. Snodin, D. Owen, and T. Adamczewski. LLM inference prices have fallen rapidly but unequally across tasks. Epoch AI data insight, 2025. https://epoch.ai/data-insights/llm-inference-price-trends. [24] B. Cottier, B. Snodin, D. Owen, T. Adamczewski. LLM 推理价格下降迅速,但各任务间不均衡。Epoch AI 数据洞察,2025。
  • [25] Writer, Inc. The Palmyra model family: technical overview. Company technical documentation, 2025. https://writer.com. [25] Writer, Inc. Palmyra 模型家族:技术概述。公司技术文档,2025。
  • [26] Writer, Inc. Say hello to Action Agent. Engineering blog, 2026. https://writer.com/engineering/writer-action-agent/. [26] Writer, Inc. 向 Action Agent 打个招呼。工程博客,2026。
  • [27] S. Gu. From model scaling to system scaling: Scaling the harness in agentic AI. arXiv:2605.26112, 2026. [27] S. Gu. 从模型缩放扩展到系统缩放:代理 AI 中框架的缩放。arXiv:2605.26112, 2026.
  • [28] Y. Yao, X. Tan, C.-H. Liu, et al. Harness-Bench: Measuring harness effects across models in realistic agent workflows. arXiv:2605.27922, 2026. [28] Y. Yao, X. Tan, C.-H. Liu, 等. Harness-Bench:测量真实代理工作流中跨模型的框架效应。arXiv:2605.27922, 2026.
  • [29] Anthropic. How we built our multi-agent research system. Engineering blog, June 2025. https://www.anthropic.com/engineering/multi-agent-research-system. [29] Anthropic. 我们如何构建多代理研究系统。工程博客,2025 年 6 月。
  • [30] Y. Ji. Context engineering for AI agents: Lessons from building Manus. Manus blog, 2025. https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus. [30] Y. Ji. AI 代理的上下文工程:构建 Manus 的经验教训。Manus 博客,2025。
  • [31] K. Hong, et al. Context rot: How increasing input tokens impacts LLM performance. Chroma Research technical report, 2025. https://www.trychroma.com/research/context-rot. [31] K. Hong, 等. 上下文腐烂:增加输入 Token 如何影响 LLM 性能。Chroma Research 技术报告,2025。
  • [32] Anthropic. Prompt caching. Claude platform documentation. https://platform.claude.com/docs/en/build-with-claude/prompt-caching. [32] Anthropic. 提示词缓存。Claude 平台文档。
  • [33] Q. Wu, G. Bansal, J. Zhang, et al. AutoGen: Enabling next-gen LLM applications via multi-agent conversation. In COLM, 2024. arXiv:2308.08155. [33] Q. Wu, G. Bansal, J. Zhang, 等. AutoGen:通过多代理对话实现下一代 LLM 应用。COLM, 2024. arXiv:2308.08155.
  • [34] LangChain, Inc. LangGraph documentation. 2026. https://docs.langchain.com. [34] LangChain, Inc. LangGraph 文档。2026。
  • [35] CrewAI, Inc. CrewAI framework documentation. 2026. https://docs.crewai.com. [35] CrewAI, Inc. CrewAI 框架文档。2026。
  • [36] Nous Research. Hermes Agent. Open-source project and documentation, 2026. https://github.com/NousResearch/hermes-agent. [36] Nous Research. Hermes Agent。开源项目及文档,2026。
  • [37] Anthropic. Claude Code and Claude Cowork. Product documentation, 2026. https://docs.claude.com. [37] Anthropic. Claude Code 和 Claude Cowork。产品文档,2026。

Appendix A Full Capability ×\times Model Matrix附录 A 完整能力×模型矩阵

Table 6: Capability scores, baseline \rightarrow harness, model held constant per cell. Codes: MSA = Model & System Awareness; GDR = Grounding & Retrieval; CNG = Content Generation; PLY = Playbooks; MCP = tool use via Model Context Protocol; PRN = Presentations (early-stage); VOX = Voice; IMG = Image Analysis & Generation. Regressions (Δ<0.02\Delta<-0.02) in bold.表 6:能力评分,基准 → 框架。代码:MSA = 模型与系统意识;GDR = 溯源与检索;CNG = 内容生成;PLY = Playbooks;MCP = 模型上下文协议工具使用;PRN = 演示文稿(早期);VOX = 语音;IMG = 图像分析与生成。退步 (Δ<-0.02) 加粗。
Capability Sonnet 4.6 Gemini 3.1 Flash 3.5 Qwen 3.6 GLM 5.1 Palmyra X6
MSA .95\to.97 .94\to.95 .93\to.94 .92\to.93 .93\to.95 .95\to.97
GDR .80\to.90 .78\to.85 .75\to.82 .74\to.80 .76\to.83 .79\to.91
CNG .80\to.87 .78\to.84 .76\to.80 .72\to.68 .77\to.83 .80\to.88
PLY .75\to.82 .72\to.77 .70\to.69 .66\to.62 .71\to.76 .76\to.84
MCP .78\to.88 .75\to.82 .70\to.66 .65\to.50 .72\to.66 .78\to.88
PRN .62\to.72 .60\to.64 .56\to.54 .52\to.45 .58\to.57 .64\to.74
VOX .88\to.92 .87\to.90 .86\to.89 .85\to.87 .86\to.89 .88\to.92
IMG .70\to.78 .68\to.75 .66\to.66 .62\to.58 .69\to.75 .71\to.80
Mean .785\to.858 .765\to.815 .740\to.750 .710\to.679 .752\to.780 .789\to.867
Δq¯\Delta\bar{q} +.073+.073 +.050+.050 +.010+.010 .031-.031 +.028+.028 +.079+.079
Sub-agents (new) .85 .70 .45 .42 .58 .86

Appendix B Per-Prompt Exemplars and Secondary Judges附录 B 示例提示词与辅裁判

Table 7: Exemplar prompts (headline task-completion) and cost extremes. Secondary judges, blended: coherence .85\to.88; communication .79\to.80; hallucination clean in both arms. The remaining 18 prompts are reported in the per-prompt grid of the engineering evaluation report.表 7:示例提示词与成本极值。辅裁判综合:连贯性 .85→.88;沟通 .79→.80;幻觉两边都干净。其余 18 个提示词在工程评估报告的网格中报告。
Prompt Baseline Harness Note
Medicare grounding (3-turn) 0.60 0.90 largest gain; retrieval shaping
Identity / refusal 0.90 0.90 parity; cost $0.04\to$0.02 (50%-50\%)
Contract Q&A 0.75 0.82 gain
Multi-step research synthesis 0.80 0.60 regression; cost $0.61\to$0.33 (46%-46\%)

Appendix C Derived-Metric Definitions附录 C 衍生指标定义

Quality per dollar η$=Q/C\eta_{\mathdollar}=Q/C uses headline task-completion and blended cost per task: 0.78/$0.21=3.710.81/$0.12=6.750.78/\mathdollar 0.21=3.71\to 0.81/\mathdollar 0.12=6.75 (+82%+82\%). Completions per million tokens CPM=Q106/τ\mathrm{CPM}=Q\cdot 10^{6}/\tau: 0.78106/14,200=54.90.81106/8,800=92.00.78\cdot 10^{6}/14{,}200=54.9{}\to 0.81\cdot 10^{6}/8{,}800=92.0{} (+68%+68\%). Per-model harness leverage Δq¯\Delta\bar{q} averages the eight legacy capabilities of Appendix A; the sub-agent capability is excluded from means because it has no baseline arm.单位美元质量 η$=Q/C 使用任务完成度和单任务综合成本:0.78/$0.21=3.71 → 0.81/$0.12=6.75 (+82%)。每百万 Token 完成数 CPM=Q·10^6/τ:0.78·10^6/14,200=54.9 → 0.81·10^6/8,800=92.0 (+68%)。模型框架杠杆 Δq¯ 平均了附录 A 的八项传统能力;子代理能力因无基准组而被排除。

Appendix D Reproducibility and Release Checklist附录 D 可复现性与发布检查清单

Artifacts to accompany the public release of this paper: (1) the locked prompt set with pass/fail criteria; (2) full per-turn traces (prompt tokens, completion tokens, tool events, duration_seconds) for both arms and all six models; (3) the pinned price-table snapshot used at report build; (4) judge prompts and panel configuration; (5) baseline and harness configuration manifests sufficient to identify the orchestration deltas of Section 4.3.随本文发布的产物:(1) 锁定提示词集及通过/失败标准;(2) 所有模型两套框架下的完整追踪记录(输入/输出 Token、工具事件、秒数);(3) 报告生成时的冻结价格表快照;(4) 裁判提示词及配置;(5) 足以识别 4.3 节编排差值的基准与框架配置文件。