JetBrains AIJetBrains AI
Supercharge your tools with AI-powered features inside many JetBrains products在 JetBrains 的多款产品中，利用 AI 功能为你的工具赋能。
Does Speaking to Agents Like Cavemen Really Save 65% of Tokens? We Test像原始人一样跟 AI 对话，真能节省 65% 的 Token 吗？我们测了一下。
Advertised saving: 65%. Measured saving: 8.5%.宣传节省 65%，实测节省 8.5%。
Output-token saving on real agentic tasks, with the skill forcibly activated. This is the ceiling, not the usual-case result.这是在强制开启该功能的情况下，针对实际代理任务测得的输出 Token 节省量。这是上限，不是常规情况。
Why we ran this为什么要测这个
We at JetBrains are investing more and more into proper testing of the tooling around coding agents, and one skill got our attention: “Caveman”. Its pitch is best described in its own dialect:我们在 JetBrains 投入了大量精力，专门测试编程代理工具。其中有个叫“原始人（Caveman）”的技能引起了我们的注意。它的宣传语用它自己的方言说最合适：
Skill make agent talk like caveman. Why use many token when few do trick. Filler die. Code, commands stay byte-exact. 65% output token saved. Every reply. Forever. Work with 30+ agents. Many GitHub star.技能让代理像原始人说话。能少用 Token 为什么要多用。废话死。代码、命令保持原样。输出 Token 节省 65%。每次回复都这样。一直这样。支持 30 多种代理。GitHub 上星很多。
We think:我们的看法是：
Claim cheap to make. Verify expensive. Agent not chat window. Agent output mostly tool call, file edit, code: skill promise not touch those. So we measure two things README not measure: real saving on multi-step agent work, and whether squeezing agent think-out-loud hurt task outcome.吹牛成本低，验证成本高。代理不是聊天窗口。代理输出的大头是工具调用、文件编辑和代码，而这个技能承诺不动这些。所以我们测了两个 README 里没写的东西：在多步代理任务中的真实节省，以及压缩代理的“思考过程”是否会损害任务结果。
Setup设置
| Harness | Harbor 0.17: Docker-sandboxed trials, task-level verifiers, paired runs. |
| Agent | Claude Code 2.1.200, headless, bypassPermissions. |
| Model | claude-sonnet-5, reasoning effort low (--effort low). |
| Benchmark | SkillsBench (benchflow/skillsbench): 86 of 87 tasks. Each task is auto-graded by its own tests on a 0-1 scale, where 1 means solved and partial credit is possible. |
| Arm A | no-skill: stock Claude Code. |
| Arm B | with-skill-forced: Caveman installed via Harbor --skill plus one instruction line forcing activation: “Use caveman mode…” |
| Pairing | Same tasks, same model, same settings, same budget per arm; excluded tasks excluded from both arms. |
| Volume | 3 runs, about 240 billed trials, about USD 106 total. |
Why “forced” matters: Caveman is user-activated. It triggers on phrases like “caveman mode” or “be brief”. We forced it on in every reply, which means every number below is the skill’s best case. In normal use, where the agent must decide to activate it on its own, the realized saving can only be equal or lower than the roughly 10% ceiling measured here.为什么要强调“强制”：原始人模式通常由用户触发，比如输入“原始人模式”或“简短点”。我们强制在每次回复中都开启它，这意味着下文的所有数据都是该技能的最好情况。在正常使用中，代理得自己决定是否开启，节省量只会等于或低于这里测得的 10% 左右的上限。
Finding 1: the saving is about 8.5%, not 65%发现 1：节省量约为 8.5%，而不是 65%
Advertised savings come from chat-style prose answers. Agentic output is different: code, diffs, tool invocations, and exact error strings dominate the token stream, and Caveman correctly leaves all of it verbatim. Only the narration between tool calls gets compressed, and there is not much of it.宣传的节省量来自聊天式的问答。但代理的输出不一样：代码、差异比对（diffs）、工具调用和报错信息占了数据流的大头，而“原始人”技能很聪明，把这些都原样保留了。它压缩的只是工具调用之间的叙述性文字，而那部分本来就没多少。
Finding 2: no detectable quality degradation发现 2：没发现质量下降
The question we actually cared about: does making the agent terse make it worse? Across 82 paired tasks in the full run, the answer is no: the arms are statistically indistinguishable.我们真正关心的是：让代理说话变简短，会不会让它变笨？在全量运行的 82 组任务中，答案是否定的：两者的表现没有统计学上的显著差异。
Style transfer itself works exactly as designed: forced-arm transcripts are unmistakably caveman, while code artifacts stay untouched and normal.风格转换本身如设计般生效：强制开启模式后的对话确实是原始人味儿，而代码产物保持原样，不受影响。
Finding 3: the cost saving is real but fragile发现 3：成本节省是真实的，但很脆弱
Cost tracks the roughly 8.5% token saving, so the skill arm should come out roughly 10% cheaper, and per task, it does. But the raw arm totals in our full run showed the skill arm 11.6% more expensive: USD 40.60 vs. USD 36.39. The entire inversion came from a single trial: one dependency-audit task ballooned past the 200k long-context pricing tier in the skill arm and billed USD 8.29 vs. USD 0.33. In an earlier run the same task threw a USD 3.25 outlier in the baseline arm. It is a property of the task, not the skill.成本与 8.5% 的 Token 节省量挂钩，所以理论上技能组应该便宜 10% 左右，单次任务确实如此。但在全量运行的总计中，技能组反而贵了 11.6%：40.60 美元对 36.39 美元。这种反转完全是因为单次试验：某个依赖审计任务在技能组中触发了超过 200k 长上下文的定价档位，花了 8.29 美元，而对比组只花了 0.33 美元。在早先的一次测试中，同一个任务在基准组里也出现过 3.25 美元的异常值。这是任务本身的特性，不是技能的问题。
Outcome结论
Safe, honest about style, oversold on savings. Forced on, Caveman reliably changes how the agent talks without any measurable damage to what the agent produces: 82 paired tasks, sign test p = 0.82. But on real agentic work it trims about 8.5% of output tokens and about 10% of cost at absolute best, because the tokens that dominate agent sessions are code and tool calls, which the skill deliberately preserves. The advertised 65% belongs to chat-style Q&A, not to coding agents.它很安全，对风格的改变很诚实，但在节省量上吹过头了。强制开启后，它确实能改变代理的说话方式，且不会对输出结果造成可测量的损害（82 组任务，符号检验 p=0.82）。但在实际的代理工作中，它最多只能节省约 8.5% 的输出 Token 和约 10% 的成本，因为代理会话中占比最大的是代码和工具调用，而这些是该技能特意保留的。那 65% 的宣传只适用于聊天式问答，不适用于编程代理。
Recommendation: use it if you like it. It is fun, and it costs you nothing measurable in quality. Just do not expect huge savings on daily agentic tasks: a high-single-digit percentage is the realistic ceiling.建议：喜欢就用。它挺有趣的，而且在质量上没造成可测量的损失。只是别指望在日常代理任务中能省下大钱：个位数的百分比已经是现实中的天花板了。
- Quality: no detectable degradation: 8 tasks better, 10 worse, 64 tied; average task score differs by 0.015 on a 0-1 scale (p = 0.82).质量：无明显下降：8 个任务更好，10 个更差，64 个持平；平均任务得分在 0-1 量表上仅差 0.015（p=0.82）。
- Tokens: -8.5% output tokens with activation forced, meaning this is the ceiling; auto-triggered usage saves less or nothing.Token：强制开启后输出 Token 减少 8.5%，这意味着这是上限；自动触发的使用方式节省得更少，甚至不节省。
- Cost: roughly -10% in expectation, routinely erased by single-trial variance.成本：预期节省约 10%，但经常会被单次试验的随机波动抹平。
- Methodology bonus: our first 10-task run “showed” a -30% token saving. It dissolved as sample size grew. Never trust a k=1 eval.方法论小贴士：我们最初 10 个任务的测试“显示”节省了 30%。随着样本量增加，这个数字就消失了。永远别相信 k=1 的评估。
You want next skill tested? Drop name in comments. Few word enough. We test.想测下一个技能？评论区留名。话不用多。我们测。
Run details: Harbor 0.17; claude-sonnet-5 with reasoning effort low; SkillsBench 86/87 tasks; about 240 trials; about USD 106 total spend.运行细节：Harbor 0.17；claude-sonnet-5（推理强度设为低）；SkillsBench 86/87 个任务；约 240 次试验；总花费约 106 美元。
