Simon Willison’s Weblog

Kimi K3, and what we can still learn from the pelican benchmarkKimi K3，以及我们从“鹈鹕基准测试”中还能学到什么

16th July 2026

Chinese AI lab Moonshot AI announced Kimi K3 this morning, describing it as their “most capable model to date, with 2.8 trillion parameters”. It’s currently available via their website and API, but an open weight release is promised “by July 27, 2026”.中国 AI 初创公司月之暗面（Moonshot AI）今早发布了 Kimi K3，称其为“迄今为止能力最强的模型，拥有 2.8 万亿参数”。该模型目前已通过其官网和 API 提供服务，并承诺于“2026 年 7 月 27 日前”发布开源权重版本。

Moonshot are calling this the first “open 3T-class model” (I guess they’re rounding 2.8 trillion up to 3 trillion), taking the crown from DeepSeek’s 1.6T v4 Pro. Their self-reported benchmarks have K3 mostly beating Claude Opus 4.8 max and GPT-5.5 high, while losing out to Claude Fable 5 and GPT-5.6 Sol.月之暗面将其称为首个“开源 3T 级模型”（我猜他们是将 2.8 万亿向上取整到了 3 万亿），从 DeepSeek 的 1.6T v4 Pro 手中夺过了桂冠。根据他们自行公布的基准测试，K3 在大多数指标上超越了 Claude Opus 4.8 max 和 GPT-5.5 high，但稍逊于 Claude Fable 5 和 GPT-5.6 Sol。

A few highlights from the Artificial Analysis report on the model:以下是 Artificial Analysis 报告中关于该模型的一些亮点：

• “On our private long-horizon knowledge work evaluation, Kimi K3 reaches an overall Elo of 1547, +732 points from Kimi K2.6 and behind only Claude Fable 5.”“在我们私有的长跨度知识工作评估中，Kimi K3 的综合 Elo 分数为 1547 分，较 Kimi K2.6 提升了 732 分，仅次于 Claude Fable 5。”
• “Cost per task ($0.94) is similar to GPT-5.6 Sol ($1.04), ~1/2 the price of Opus 4.8 ($1.80) and higher than open weights peers”“单任务成本（0.94 美元）与 GPT-5.6 Sol（1.04 美元）相当，约为 Opus 4.8（1.80 美元）价格的一半，高于开源权重同类产品。”
• “Kimi K3’s token usage on the Artificial Analysis Intelligence Index decreased significantly, using 21% fewer output tokens than K2.6.”“Kimi K3 在 Artificial Analysis 智能指数上的 Token 使用量显著下降，输出 Token 比 K2.6 少了 21%。”

The model is also now the leading model on Arena.ai’s Frontend Code arena, surpassing even Claude Fable 5.该模型目前也是 Arena.ai 前端代码竞技场（Frontend Code arena）中的领先模型，甚至超越了 Claude Fable 5。

The new model is notable for the pricing: $3/million input tokens and $15/million output tokens, putting it at the same level as Anthropic’s Claude Sonnet series and making it the most expensive model released by a Chinese AI lab to date. This is a significant increase on their earlier models such as Kimi K2.6 at $0.95/$4. 2.8 trillion parameters is also more than twice the size of that 1T model.这款新模型的定价值得关注：输入 Token 每百万 3 美元，输出 Token 每百万 15 美元，这使其与 Anthropic 的 Claude Sonnet 系列处于同一水平，并成为中国 AI 实验室迄今为止发布的最昂贵的模型。相比他们早期的模型（如 Kimi K2.6 的 0.95 美元/4 美元）有了显著增长。2.8 万亿的参数量也超过了那款 1T 模型两倍以上。

But how does it pelican? #但是，它“鹈鹕”得怎么样？#

I used OpenRouter (to avoid signing up for a Moonshot API key) with the llm-openrouter plugin to generate an SVG of a pelican riding a bicycle:我使用了 OpenRouter（为了避免注册 Moonshot API 密钥）配合 llm-openrouter 插件，生成了一张鹈鹕骑自行车的 SVG 图片：

llm -m openrouter/moonshotai/kimi-k3 'Generate an SVG of a pelican riding a bicycle'

Here’s the transcript. It looks like this:以下是记录。看起来是这样的：

That pelican took 95 input tokens and 16,658 output tokens (13,241 were reasoning tokens), for a total cost of 25 cents!那只鹈鹕消耗了 95 个输入 Token 和 16,658 个输出 Token（其中 13,241 个是推理 Token），总成本为 25 美分！

Since K3 accepts image input I ran it against that rendered SVG above (with my alt text prompt) and got back (for 0.6 cents):由于 K3 支持图像输入，我用上面渲染出的 SVG（配合我的 Alt 文本提示词）对其进行了测试，得到了以下反馈（花费 0.6 美分）：

Cartoon illustration of a white pelican wearing a red scarf, riding a red bicycle along a gray road with white dashed lines; the pelican has a large orange beak and webbed orange feet pedaling, with white motion lines behind it; the background shows a light blue sky with white clouds, a yellow sun, two small black birds in flight, and green grass with tiny white flowers in the foreground卡通插画：一只戴着红围巾的白鹈鹕，正沿着带有白色虚线的灰色道路骑着一辆红色自行车；鹈鹕有着巨大的橙色喙和正在蹬踏的橙色蹼足，身后有白色的运动轨迹线；背景是带有白云的浅蓝色天空、一个黄色的太阳、两只正在飞行的小黑鸟，前景是带有微小白色花朵的绿草地。

What can we learn from the pelican? #我们能从鹈鹕身上学到什么？#

My Generate an SVG of a pelican riding a bicycle test is 21 months old now. It was never a particularly great benchmark. It started out as a joke on how absurdly difficult it is to compare these models, but then for the first year it turned out to have a surprising correlation to how good the models actually were.我那个“生成一张鹈鹕骑自行车的 SVG”的测试距今已经 21 个月了。它从来都不是一个特别好的基准测试。最初这只是一个玩笑，用来讽刺比较这些模型有多么荒谬和困难，但后来在第一年里，它竟然与模型的实际表现有着惊人的相关性。

That connection has been mostly severed now. The GPT-5.6 and Claude Fable 5 pelicans are outclassed by GLM-5.2, and much as I love GLM I don’t think that’s a Fable-class model.这种联系现在基本断开了。GPT-5.6 和 Claude Fable 5 的鹈鹕表现被 GLM-5.2 超越了，尽管我很喜欢 GLM，但我不认为它属于 Fable 级别的模型。

(I’m still not convinced that labs are training for the benchmark—if they were, I’d expect much better results. There’s a chance that Gemini has optimized for any combination of an animal on a vehicle though!)（我仍然不相信实验室会专门针对这个基准进行训练——如果真的有，我预期的结果会好得多。不过，Gemini 确实有可能针对任何“动物骑载具”的组合进行了优化！）

The biggest limitation of the pelican is that it doesn’t touch at all on the thing that matters most for today’s model: agentic tool calling and the ability to operate tools reliably as conversations grow in length.鹈鹕测试最大的局限性在于，它完全没有触及当今模型最核心的能力：代理式工具调用（agentic tool calling），以及在对话长度增加时可靠操作工具的能力。

So don’t go using pelicans to compare models!所以，别再用鹈鹕来比较模型了！

All of that said, I still get a decent amount of value out of running the benchmark myself.话虽如此，我自己运行这个基准测试仍然能获得不少价值。

Firstly, it’s a forcing function for actually trying the model. If I show you a pelican, that means I’ve managed to run a prompt through it. If the model has an official API I’ll use that, if it’s open weight (and small enough to fit a 128GB M5 MacBook Pro) I’ll try running it on my own machine, usually via llama.cpp or LM Studio or Ollama. I’ll frequently use OpenRouter since that usually provides a proxy to an official API without me needing a new API key.首先，这是一个强迫自己去实际使用模型的手段。如果我能向你展示一只鹈鹕，就意味着我成功地让模型运行了一个提示词。如果该模型有官方 API，我会使用它；如果是开源权重模型（且足够小，能塞进 128GB 的 M5 MacBook Pro），我会尝试在自己的机器上运行，通常是通过 llama.cpp、LM Studio 或 Ollama。我经常使用 OpenRouter，因为它通常能提供官方 API 的代理，而无需我申请新的 API 密钥。

Most of my pelicans are generated using my LLM CLI tool, which helps encourage me to ensure the latest models are supported by that (via one of its plugins).我的大多数鹈鹕都是使用我的 LLM CLI 工具生成的，这促使我确保最新的模型能通过其插件得到支持。

More importantly though, even the act of a single prompt to “Generate an SVG of a pelican riding a bicycle” can reveal interesting model characteristics.更重要的是，即使只是执行“生成一张鹈鹕骑自行车的 SVG”这样一个简单的提示词，也能揭示出模型有趣的特性。

Consider the result for Kimi K3 today. Running those simple prompts helped emphasize several points about the model.看看今天 Kimi K3 的结果。运行这些简单的提示词有助于强调该模型的几个要点。

1. It only has one reasoning effort right now, “max”—and it shows. The model consumed 13,241 reasoning tokens to output 3,417 tokens of response. This is expensive—the pelican cost 25 cents!它目前只有一种推理强度，即“max”——效果很明显。该模型消耗了 13,241 个推理 Token 才输出了 3,417 个响应 Token。这很昂贵——那只鹈鹕花了 25 美分！
2. How does the prompt “Generate an SVG of a pelican riding a bicycle” add up to 95 input tokens? OpenAI’s tokenizer counts 10, Anthropic’s counts 10 for Opus 4.6, 30 for Opus 4.7 and 25 for Sonnet 5/Fable 5. Prompting “hi” to Kimi K3 counted 86 tokens, suggesting there may be an 85 token hidden system prompt. It refused to leak it though.“生成一张鹈鹕骑自行车的 SVG”这个提示词是怎么算成 95 个输入 Token 的？OpenAI 的分词器计算结果是 10 个，Anthropic 的 Opus 4.6 计算为 10 个，Opus 4.7 为 30 个，Sonnet 5/Fable 5 为 25 个。对 Kimi K3 输入“hi”计算为 86 个 Token，这表明可能存在一个 85 个 Token 的隐藏系统提示词。不过它拒绝泄露该内容。
3. Vision works well: the alt text it generated is very good.视觉能力表现良好：它生成的 Alt 文本非常出色。

K3 currently only has one thinking effort level, but I’ve been deriving quite a bit of value recently from running the same pelican prompt through different effort levels to get a quick idea for what impact those have. Here’s my matrix for the GPT-5.6 model family, for example.K3 目前只有一个思维强度级别，但我最近通过在不同强度级别下运行相同的鹈鹕提示词，从中获益良多，可以快速了解它们的影响。例如，这是我针对 GPT-5.6 模型系列的矩阵。

Really though the main things I gain from the pelican test are:说真的，我从鹈鹕测试中获得的主要收获是：

1. It’s a “hello world” exercise for prompting a model这是一个用于提示模型测试的“Hello World”练习。
2. A rough cost and reasoning estimate for a simple task针对简单任务的粗略成本和推理估算。
3. Confirmation that the model can output valid SVG and has a basic idea of geometry and spatial awareness. This is a much bigger deal for the smaller models that run on my laptop.确认模型能够输出有效的 SVG，并具备基本的几何和空间感知概念。对于在我笔记本电脑上运行的小型模型来说，这一点意义重大。
4. It’s still interesting to compare pelicans between releases in the same model family. K3’s pelican is a notable improvement from Kimi 2.5.比较同一模型系列不同版本之间的鹈鹕作品依然很有趣。K3 的鹈鹕相较于 Kimi 2.5 有了显著提升。
5. It’s something I can share that demonstrates I’ve tried it. Plus a comment with a pelican in it is kind of a tradition on Hacker News at this point, any time I’m late I get comments asking where it is!这是一个我可以分享的内容，证明我已经尝试过该模型。此外，在 Hacker News 上，评论里带上一只鹈鹕已经成为一种传统，每次我发晚了，都会有人评论问鹈鹕去哪了！