Controlling Reasoning Effort in LLMs控制大语言模型（LLM）的推理投入

How LLMs Learn Low-, Medium-, and High-Effort Reasoning Modes大语言模型如何学习低、中、高三种推理模式

It has been almost two years since OpenAI released o1, a model that popularized the idea of LLM-based reasoning models. DeepSeek-R1 followed about four months later, together with details of a reinforcement learning with verifiable rewards (RLVR) recipe to train such reasoning models.距离 OpenAI 发布 o1 已过去近两年，该模型普及了基于大语言模型的推理模型概念。DeepSeek-R1 在四个月后紧随其后，并详细介绍了用于训练此类推理模型的“可验证奖励强化学习”（RLVR）方案。

Last week, OpenAI released the GPT-5.6 model family. It comes in three sizes, each with roughly five or six reasoning-effort settings.上周，OpenAI 发布了 GPT-5.6 模型系列。该系列包含三种尺寸，每种尺寸均配有大约五到六种推理投入设置。

Figure 1: The GPT 5.6 Sol model with different reasoning effort settings. (Benchmark numbers for Ultra are currently not available but should be relatively similar to Max, since it uses a similar effort level but accelerates the work with four subagents.)图 1：具有不同推理投入设置的 GPT 5.6 Sol 模型。（Ultra 版本的基准测试数据目前尚不可用，但应与 Max 版本相对接近，因为它使用了相似的投入水平，但通过四个子智能体加速了工作。）

So yes, reasoning models are here to stay. They have become a standard part of modern model releases.所以，是的，推理模型已成定局。它们已成为现代模型发布的标准组成部分。

In the past, I covered the methodology of reasoning models (Understanding Reasoning LLMs) as well as relevant research papers (The State of Reinforcement Learning for LLM Reasoning and The State of LLM Reasoning Model Inference). And I even wrote a whole new 440-page book on how to develop reasoning models, Build A Reasoning Model (From Scratch).过去，我曾介绍过推理模型的方法论（《理解推理型大语言模型》）以及相关的研究论文（《大语言模型推理强化学习现状》和《大语言模型推理模型推理现状》）。我甚至写了一本全新的 440 页书籍，专门讲解如何开发推理模型，书名为《从零构建推理模型》（Build A Reasoning Model (From Scratch)）。

Figure 2: My new Build A Reasoning Model (From Scratch) book. In color!图 2：我的新书《从零构建推理模型》。彩色版！

These resources have focused on turning a conventional LLM into a reasoning model. Now, in this article, I want to focus on and explain how to develop a reasoning model that has multiple effort modes, similar to what’s shown in the figure at the beginning of this article.这些资源主要侧重于将传统大语言模型转化为推理模型。现在，在本文中，我想重点解释如何开发一种具有多种投入模式的推理模型，类似于本文开头图示中的效果。

No worries, this article can be read as a standalone article. However, the aforementioned resources may be interesting and useful.不用担心，本文可以作为独立文章阅读。当然，上述提到的资源可能也会让你感兴趣并有所收获。

1. A brief definition of reasoning models

1. 推理模型的简要定义

When talking about pretty much any machine learning or AI technique or subfield, the one lesson is that we usually shouldn’t take technical terms “literally”. For example, an (artificial) neural network in machine learning and AI doesn’t literally work like a biological neural network like the human brain.在谈论几乎任何机器学习或人工智能技术或子领域时，我们要学会的一个经验是：通常不应“字面理解”技术术语。例如，机器学习和人工智能中的（人工）神经网络并不真的像人脑那样的生物神经网络那样工作。

Similarly, when talking about “reasoning models”, we shouldn’t expect that these models literally reason like us humans. In the context of AI and LLM research, “reasoning model” means a model that outputs an intermediate reasoning trace, which is like an intermediate response that works through a question or task step by step.同样，在谈论“推理模型”时，我们也不应期望这些模型真的像人类一样思考。在人工智能和大语言模型研究的语境下，“推理模型”是指能够输出中间推理轨迹的模型，这就像是一个逐步解决问题或任务的中间响应过程。

It’s probably easiest to explain this by showing an example.通过一个例子来解释可能最容易。

Figure 3: Illustration of a conventional LLM answer (left) and an answer by a reasoning model (right).图 3：传统大语言模型回答（左）与推理模型回答（右）的对比示意图。

2. A brief overview of training and inference scaling reasoning models

2. 推理模型训练与推理扩展简述

There are essentially two ways to improve (reasoning) task performance: training scaling and inference scaling.提高（推理）任务性能本质上有两种方法：训练扩展和推理扩展。

Figure 4: Training and inference-scaling are two ways to improve LLM and reasoning model problem-solving capabilities. Plot based on Learning to reason with LLMs图 4：训练扩展和推理扩展是提高大语言模型和推理模型问题解决能力的两种途径。图表基于《学习使用大语言模型进行推理》（Learning to reason with LLMs）。

Let’s briefly talk about training first.让我们先简要谈谈训练。

2.1 Training reasoning models

2.1 训练推理模型

In a nutshell, DeepSeek-R1 proposed training an LLM using reinforcement learning with verifiable rewards (RLVR) to turn it into a reasoning model. RLVR is a technique to provide a reward signal (0=incorrect and 1=correct) for verifiable data domains. These verifiable data domains here are math (we can use a symbolic math checker like SymPy or WolframAlpha to check results) and code (we can use a compiler or unit tests, or integrated platforms like LeetCode) to check for correctness.简而言之，DeepSeek-R1 提出使用可验证奖励强化学习（RLVR）来训练大语言模型，将其转化为推理模型。RLVR 是一种为可验证数据领域提供奖励信号（0=错误，1=正确）的技术。这里的可验证数据领域包括数学（我们可以使用 SymPy 或 WolframAlpha 等符号数学检查器来验证结果）和代码（我们可以使用编译器、单元测试或 LeetCode 等集成平台来检查正确性）。

Figure 5: Illustration of accuracy and format rewards during RLVR training.图 5：RLVR 训练期间准确性和格式奖励的示意图。

Notably, the reasoning trace itself was not used for training or updating the model. Although they tried to use this intermediate response information for training, the DeepSeek-R1 paper reported that it wasn’t helpful for the model training, so it was ultimately not used. (Whether and how to incorporate intermediate reasoning traces in the training signal via process reward models is an active area of research.)值得注意的是，推理轨迹本身并未用于模型的训练或更新。尽管研究人员曾尝试将这些中间响应信息用于训练，但 DeepSeek-R1 论文报告称这对于模型训练并无帮助，因此最终未被采用。（是否以及如何通过过程奖励模型将中间推理轨迹纳入训练信号，是当前的一个活跃研究领域。）

Figure 6: The intermediate reasoning trace is ignored during RLVR; only the final answer and response format determine the reward.图 6：在 RLVR 期间，中间推理轨迹被忽略；只有最终答案和响应格式决定了奖励。

2.2 “Aha” moments

2.2 “顿悟”时刻

Anyway, just training on the output rewards alone, as Figure 7 shows, turned out to be sufficient for the model to learn how to reason through a problem, meaning that it would learn to write intermediate explanations, backtrack, and self-correct itself. These moments when the model realizes that it made a mistake and self-corrects itself are called “Aha” moments.无论如何，正如图 7 所示，仅根据输出奖励进行训练，已足以让模型学会如何通过推理解决问题，这意味着它会学会撰写中间解释、回溯并进行自我修正。模型意识到自己犯错并自我修正的这些时刻被称为“顿悟”（Aha）时刻。

Figure 7: An example of an aha moment, where a reasoning model notices an error in its intermediate reasoning and corrects it before producing the final answer.图 7：顿悟时刻的示例，推理模型在其中间推理中发现了错误，并在产生最终答案之前进行了修正。

By the way, while DeepSeek-R1 is inarguably the more popular paper, and the paper that created excitement around reinforcement learning with verifiable rewards and the development of reasoning models, there is another paper, Kimi K1.5, published on exactly the same day on arXiv (22 Jan 2025). Also, the term RLVR was already coined two months earlier in Tülu 3: Pushing Frontiers in Open Language Model Post-Training.顺便提一下，虽然 DeepSeek-R1 无疑是更受欢迎的论文，也是引发人们对可验证奖励强化学习和推理模型开发热情的论文，但还有另一篇论文 Kimi K1.5，于同一天（2025 年 1 月 22 日）在 arXiv 上发布。此外，RLVR 一词早在两个月前就在《Tülu 3：推动开放语言模型后训练的前沿》一文中被提出。

​

One reason why the DeepSeek R1 is ultimately the more popular paper is that it demonstrated that reasoning behavior can be achieved with pure reinforcement learning (RL).DeepSeek R1 最终成为更受欢迎论文的原因之一，在于它证明了推理行为可以通过纯强化学习（RL）实现。

Figure 8: DeepSeek-R1-Zero applies RLVR directly to the pretrained base model without supervised fine-tuning.图 8：DeepSeek-R1-Zero 将 RLVR 直接应用于预训练的基础模型，而无需监督微调（SFT）。

For instance, Tülu 3 and Kimi K1.5 applied reinforcement learning on top of a supervised fine-tuned (SFT) model. The DeepSeek-R1 model was also trained from an SFT checkpoint of the DeepSeek-V3 base model, and it included a DeepSeek-R1-Zero variant trained with pure RLVR. R1 Zero is a weaker model than R1, but it showed that RLVR is sufficient for teaching the model to generate and use reasoning traces.例如，Tülu 3 和 Kimi K1.5 是在监督微调（SFT）模型的基础上应用强化学习的。DeepSeek-R1 模型也是从 DeepSeek-V3 基础模型的 SFT 检查点训练而来的，它还包含了一个使用纯 RLVR 训练的 DeepSeek-R1-Zero 变体。R1 Zero 虽然是一个较弱的模型，但它证明了 RLVR 足以教会模型生成和使用推理轨迹。

​While R1-Zero was more of a proof-of-concept model, note that the full DeepSeek-R1 reasoning model training pipeline is usually multi-stage and a bit more complicated, as mentioned above.虽然 R1-Zero 更像是一个概念验证模型，但请注意，完整的 DeepSeek-R1 推理模型训练流程通常是多阶段的，并且稍微复杂一些，如上所述。

Figure 9: More detailed reasoning model training pipeline. This one depicts the various DeepSeek-R1 models. For more details, see my other article: Understanding Reasoning LLMs图 9：更详细的推理模型训练流程。此图描绘了各种 DeepSeek-R1 模型。更多详情，请参阅我的另一篇文章：《理解推理型大语言模型》。

​

By the way, most of today’s LLMs are effectively reasoning models, meaning they have been trained in a similar fashion to DeepSeek-R1 using a form of RLVR.顺便说一下，当今大多数大语言模型实际上都是推理模型，这意味着它们是以类似于 DeepSeek-R1 的方式，使用某种形式的 RLVR 进行训练的。

2.3 Inference scaling in a nutshell

2.3 推理扩展简述

Next to improving reasoning behavior through training, another lever for improving model performance is inference compute scaling. In short, this means that we are spending more compute after training the model, during usage, to get better answers.除了通过训练改进推理行为外，提高模型性能的另一个杠杆是推理计算扩展。简而言之，这意味着我们在模型训练完成后，在使用过程中投入更多的计算资源，以获得更好的答案。

This is a whole topic by itself, and you could read through my The State of LLM Reasoning Model Inference for a more detailed rundown:这是一个完整的话题，你可以阅读我的《大语言模型推理模型推理现状》来获取更详细的概述：

The State of Reinforcement Learning for LLM Reasoning大语言模型推理强化学习现状

I will try to summarize what’s most essential to mention as background info below.我将在下面尝试总结作为背景信息最必要的内容。

First, training a model with RLVR is already implicitly leading to a form of inference scaling, since reasoning models usually output more tokens during inference compared to conventional LLMs, and that means we are spending more compute during inference.首先，使用 RLVR 训练模型已经隐含地导致了一种推理扩展形式，因为推理模型在推理过程中通常比传统大语言模型输出更多的 Token，这意味着我们在推理过程中投入了更多的计算量。

Second, we can further adjust this output length via reasoning effort levels, but more on that later.其次，我们可以通过推理投入水平进一步调整这种输出长度，稍后会详细介绍。

​Third, there are many additional inference scaling techniques. A popular one is self-consistency, which is often implemented as a form of majority voting where the model is queried multiple times, and the final answer is selected via majority vote.第三，还有许多额外的推理扩展技术。一种流行的方法是自洽性（Self-consistency），通常实现为一种多数投票形式，即模型被多次查询，最终答案通过多数投票选出。

Figure 10: An example of self-consistency, a popular inference scaling technique.图 10：自洽性示例，这是一种流行的推理扩展技术。

This can be applied to conventional LLMs as well as reasoning models. Also, this method can be used on demand and in addition to reasoning training. A good example of that is DeepSeekMath-V2, where the researchers applied extreme inference-scaling on top of a reasoning model (specialized for math) to achieve state-of-the-art performance on challenging math olympiad-type problems.这既适用于传统大语言模型，也适用于推理模型。此外，该方法可以按需使用，并作为推理训练的补充。一个很好的例子是 DeepSeekMath-V2，研究人员在推理模型（专门针对数学）之上应用了极端的推理扩展，从而在具有挑战性的数学奥赛类问题上实现了最先进的性能。

Figure 11: Two types of inference scaling (self-consistency and self-refinement) used together to improve math performance. Figure adapted from DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning 图 11：两种推理扩展类型（自洽性和自我修正）结合使用以提高数学性能。图表改编自《DeepSeekMath-V2：迈向可自我验证的数学推理》。

But again, I will refer to my other article, The State of LLM Reasoning Model Inference for an overview of other techniques:但同样，我建议参考我的另一篇文章《大语言模型推理模型推理现状》以获取其他技术的概述：

The State of LLM Reasoning Model Inference大语言模型推理模型推理现状

3. Think tokens

3. 思维 Token

You may have seen the <think></think> tokens in the earlier “Aha moments” figure. I also included the corresponding figure below so you don’t have to scroll all the way up.你可能在前面的“顿悟时刻”图中看到过 Token。我也在下方附上了相应的图表，这样你就不必向上滚动了。

​

Figure 12: Common formatting tokens in reasoning models.图 12：推理模型中常见的格式化 Token。

These <think> and </think> tags are cosmetic with respect to reasoning ability. They do not make the model reason, and they are not required to achieve good reasoning performance. One could train the same model without these delimiters and likely reach similar benchmark performance.这些 标签在推理能力方面仅是修饰性的。它们不会让模型进行推理，也不是实现良好推理性能的必要条件。人们可以在没有这些分隔符的情况下训练同一个模型，并可能达到相似的基准测试性能。

The purpose of these <think> tags or tokens is mainly to mark where the reasoning trace begins and ends so that the training pipeline or user interface can separate it from the final answer and optionally hide it from the user. (UIs like ChatGPT or Codex usually do this.)这些 本身也没有什么特别之处。另一对分隔符也可以达到同样的目的。

The point here is that the <think> tokens are not giving the model the ability to “think” or reason or reason better. One could train the same models without such <think> tokens and reach similar benchmark performance.这里的关键在于， 本身并没有什么特殊之处。换成其他任何一对分隔符也能达到同样的目的。

There is also nothing special about the literal strings <think> and </think>. Another pair of delimiters could serve the same purpose.

By the way, the way this is implemented is typically by adding a formatting reward during the RLVR stage. So instead of just rewarding the model based on answer correctness, one would provide additional reward for the use of <think> tokens, which in turn encourages the model to use those.顺便说一下，这种实现方式通常是在 RLVR 阶段添加格式化奖励。因此，与其仅仅根据答案的正确性奖励模型，不如为 。

In DeepSeek-R1, for example, the overall reward was calculated as以 DeepSeek-R1 为例，其总奖励的计算公式为：

R_total = R_accuracy + R_formatR_total = R_accuracy + R_format

where the format reward was a simple rule-based check that encouraged the model to place its reasoning inside:其中，格式奖励（format reward）是通过简单的基于规则的检查来实现的，旨在鼓励模型将其推理过程置于以下标记内：

<think>。

reasoning trace

</think>.

4. Reasoning mode on and off switches

4. 推理模式的开启与关闭开关

The first generation of reasoning models was dedicated reasoning models. With that, I mean that there was a DeepSeek-V3 base model and a separate DeepSeek-R1 reasoning model.第一代推理模型是专用推理模型。我的意思是，当时存在一个 DeepSeek-V3 基础模型和一个独立的 DeepSeek-R1 推理模型。

No matter what the prompt is, R1 generally outputs very verbose responses using lots of tokens, even for simple prompts. It also lacks a built-in option to turn off the reasoning mode.无论提示词是什么，R1 通常都会使用大量 Token 输出非常冗长的回答，即使对于简单的提示词也是如此。它也缺乏内置的关闭推理模式的选项。

Figure 13: Reasoning models are very verbose, even for the simplest prompts.图 13：推理模型非常冗长，即使对于最简单的提示词也是如此。

Later models, like Qwen3 and others, experimented with hybrid approaches, where the same model can behave like a regular instruction fine-tuned model or a reasoning model on demand.后来的模型，如 Qwen3 等，尝试了混合方法，即同一个模型可以根据需要表现得像普通的指令微调模型或推理模型。

Note: Some model developers call this “thinking mode,” while others call it “reasoning mode.” Both terms refer to the same behavior.注：一些模型开发者将其称为“思考模式”，而另一些则称为“推理模式”。这两个术语指的是相同的行为。

In Qwen3, this is handled via the tokenizer using enable_thinking=True or enable_thinking=False. Under the hood, setting enable_thinking=False essentially adds an empty <think></think> section to the beginning of the assistant response to turn off Qwen3’s reasoning (”thinking”) mode.在 Qwen3 中，这是通过 Tokenizer 使用 enable_thinking=True 或 enable_thinking=False 来处理的。在底层，设置 enable_thinking=False 本质上是在助手响应的开头添加一个空的 部分，以关闭 Qwen3 的推理（“思考”）模式。

Figure 14: Response of Qwen3 0.6B reasoning model with thinking=False and thinking=True. (The empty <think></think> tags are hidden in the interface on the left as they are part of the modified input prompt, not the generated answer.)图 14：Qwen3 0.6B 推理模型在 thinking=False 和 thinking=True 下的响应。（左侧界面中隐藏了空的 标签，因为它们是修改后的输入提示词的一部分，而不是生成的答案。）

How is this implemented during training, such that the model supports this toggle during inference time, as shown in the figure above?这是如何在训练期间实现的，使得模型在推理时支持这种切换，如上图所示？

In short, as explained in the Qwen3 technical report, this on/off behavior is introduced primarily through supervised fine-tuning (SFT) and then reinforced during general RL in their largest flagship models.简而言之，正如 Qwen3 技术报告中所述，这种开/关行为主要是通过监督微调（SFT）引入的，然后在他们最大的旗舰模型中通过通用强化学习（RL）得到强化。

For instance, after the initial reasoning model is trained via long-chain-of-thought SFT and reasoning RL, they add a “Thinking Mode Fusion” stage. During this additional SFT stage, the model sees both thinking and non-thinking examples:例如，在通过长思维链 SFT 和推理 RL 训练出初始推理模型后，他们增加了一个“思维模式融合”（Thinking Mode Fusion）阶段。在此额外的 SFT 阶段，模型会看到思维和非思维示例：

• /think: <think>{reasoning}</think>{answer}/think: {answer}

• /no_think: <think></think>{answer}/no_think: {answer}

Thinking is the default behavior, so /think can also be omitted. The subsequent general RL stage further reinforces this mode and format following.思考是默认行为，因此 /think 也可以省略。随后的通用 RL 阶段进一步强化了这种模式和格式的遵循。

These /think and /no_think flags are a “soft” switch. However, the enable_thinking=False setting mentioned earlier, which force-adds the empty <think></think> in the False case, acts then as a “hard” switch.这些 /think 和 /no_think 标志是一个“软”开关。然而，前面提到的 enable_thinking=False 设置（在 False 的情况下强制添加空的 ）则充当了“硬”开关。

Figure 15: “Thinking Mode Fusion” in Qwen3’s training pipeline to enable the reasoning mode on and off switch.图 15：Qwen3 训练流程中的“思维模式融合”，以启用推理模式的开/关切换。

In other words, the tokenizer does not add /no_think to the query. It directly fills in the empty <think></think> section at the beginning of the assistant response. The model only sees the resulting tokens and continues directly with the answer.换句话说，Tokenizer 不会将 /no_think 添加到查询中。它直接在助手响应的开头填充空的 部分。模型只看到生成的 Token，并直接继续输出答案。

Anyway, this on-and-off toggle is essentially a simplified version of the reasoning effort levels in GPT-5.6 and others, which I cover in the next section.无论如何，这种开/关切换本质上是 GPT-5.6 等模型中推理投入水平的简化版本，我将在下一节中介绍。

5. How “reasoning effort” settings work

5. “推理投入”设置的工作原理

In this section, I want to provide a brief overview of how the different reasoning effort toggles may be implemented, which have been introduced in models like GPT 5 and are present in pretty much any flagship model today.在本节中，我想简要概述一下不同的推理投入开关是如何实现的，这些开关已在 GPT 5 等模型中引入，并且存在于当今几乎所有的旗舰模型中。

Concretely, at the beginning of this article, I showed a figure from the Codex GPT 5.6 interface that lets users select multiple reasoning “effort” settings.具体来说，在本文开头，我展示了来自 Codex GPT 5.6 界面的一张图，允许用户选择多种推理“投入”设置。

Figure 16: GPT-5.6 exposes six reasoning effort settings, ranging from Light to Ultra.图 16：GPT-5.6 提供了六种推理投入设置，从“轻量”到“超强”不等。

The following subsection will illustrate how these settings may be implemented. Then, in the next section, I will go over some of the more interesting research papers related to this topic.接下来的小节将说明这些设置是如何实现的。然后，在下一节中，我将讨论一些与此主题相关的更有趣的研究论文。

5.1 Reasoning effort and response length and quality

5.1 推理投入与响应长度及质量

Unfortunately, the implementation details of their effort settings are not shared by OpenAI, but there is some evidence out there that can be used for educated guesses.遗憾的是，OpenAI 没有分享其投入设置的实现细节，但确实有一些证据可用于进行有根据的推测。

For instance, via their open-source gpt-oss models from last year (I wrote about them in From GPT-2 to gpt-oss: Analyzing the Architectural Advances), we know that OpenAI allows us to toggle the reasoning effort setting via the system prompt (”Reasoning effort: low/medium/high”) that is prepended to each prompt.例如，通过他们去年开源的 gpt-oss 模型（我在《从 GPT-2 到 gpt-oss：分析架构进步》中写过相关内容），我们知道 OpenAI 允许我们通过预置在每个提示词之前的系统提示词（“Reasoning effort: low/medium/high”）来切换推理投入设置。

Figure 17: The gpt-oss chat template inserts the selected reasoning effort into the system message before sending the prompt to the same model.图 17：gpt-oss 聊天模板在将提示词发送给同一个模型之前，会将选定的推理投入插入到系统消息中。

As expected, the reasoning effort directly affects the response length and accuracy, as shown below.正如预期的那样，推理投入直接影响响应长度和准确性，如下所示。

Figure 18: Response length and quality of gpt-oss models under different reasoning efforts (annotated figure from the model card)图 18：gpt-oss 模型在不同推理投入下的响应长度和质量（模型卡中的注释图）。

Presumably, their GPT 5 models, including the recent GPT 5.6 models, use a similar approach.据推测，他们的 GPT 5 模型，包括最近的 GPT 5.6 模型，也使用了类似的方法。

By the way, note how different effort settings scale the response length in the figure above. The effort level seems directly correlated to token usage, which in turn seems correlated to accuracy. It might be possible to come up with effort settings beyond the “high” one, but I assume performance would saturate at some point. This saturation can be seen more clearly for the GPT 5.6 Sol model, which also shows that increasing reasoning budgets can become uneconomical at some point.顺便提一下，请注意上图中不同的投入设置是如何缩放响应长度的。投入水平似乎与 Token 使用量直接相关，而这又似乎与准确性相关。可能会有超出“高”投入水平的设置，但我认为性能会在某一点达到饱和。这种饱和在 GPT 5.6 Sol 模型中可以更清楚地看到，这同时也表明增加推理预算在某一点之后可能会变得不经济。

Figure 19: Reasoning effort increases both API cost and coding-agent performance, with diminishing returns at the highest GPT-5.6 settings. Figure based on the Artificial Analysis Coding Agent Index v1.1.图 19：推理投入增加了 API 成本和编码智能体性能，但在 GPT-5.6 的最高设置下收益递减。图表基于 Artificial Analysis 编码智能体指数 v1.1。

Another good, very recent data point that shows the relationship between reasoning effort, token usage, and benchmark performance is this week’s new open-weight Inkling release by Thinking Machine Labs.另一个很好的、非常新的数据点展示了推理投入、Token 使用量和基准性能之间的关系，这就是本周 Thinking Machine Labs 发布的新型开放权重模型 Inkling。

Figure 20: Increasing the Inkling effort level generally increases generated tokens and benchmark performance, with diminishing or uneven gains at higher effort. Figure from the Inkling announcement blog.图 20：提高 Inkling 的投入水平通常会增加生成的 Token 和基准测试性能，但在更高投入下收益递减或不均匀。图表来自 Inkling 发布博客。

As discussed in this section, during inference, the reasoning effort level can simply be controlled via a system prompt. (The ChatGPT UI presumably simply maps the menu choice to a system prompt.) However, this would not work for an arbitrary model and requires certain modifications to the training pipeline, which will be discussed next.正如本节所讨论的，在推理期间，推理投入水平可以通过系统提示词简单地进行控制。（ChatGPT 界面大概只是将菜单选择映射到系统提示词。）然而，这对于任意模型并不适用，且需要对训练流程进行某些修改，这将在接下来讨论。

5.2 Possible effort level implementations

5.2 可能的投入水平实现方式

While the training details are not public, neither for GPT 5.6 nor the open-source gpt-oss models, typically, the reasoning effort label is included in prompts during post-training.虽然训练细节尚未公开（无论是 GPT 5.6 还是开源的 gpt-oss 模型），但通常情况下，推理投入标签会在后训练期间包含在提示词中。

There are typically two ways to implement this.通常有两种实现方式。

First, we can implement it as part of the RLVR process and apply a different length penalty when different system prompts are used. For example, a high length penalty when “Reasoning effort: low” and a mild or no penalty when “Reasoning effort: high”.首先，我们可以将其作为 RLVR 过程的一部分来实现，并在使用不同的系统提示词时应用不同的长度惩罚。例如，当“Reasoning effort: low”时应用高长度惩罚，而当“Reasoning effort: high”时应用轻微惩罚或无惩罚。

Second, we can fine-tune the model after RLVR to follow different effort instructions via supervised fine-tuning (SFT).其次，我们可以在 RLVR 之后对模型进行微调，通过监督微调（SFT）来遵循不同的投入指令。

For instance, after the core RLVR stage and during SFT, the prompts in the training dataset are paired with target responses that exhibit the desired amount of reasoning. (The targets may be written by humans, generated by another model, or generated and then filtered.)例如，在核心 RLVR 阶段之后和 SFT 期间，训练数据集中的提示词会与表现出所需推理量的目标响应配对。（目标可以由人类编写、由另一个模型生成，或者生成后经过过滤。）

Figure 21: Illustration of effort-conditioned RLVR and SFT. (This is a possible implementation, not a confirmed description of OpenAI’s training pipeline.)图 21：投入条件化 RLVR 和 SFT 的示意图。（这是一种可能的实现方式，而非 OpenAI 训练流程的确认描述。）

During this SFT stage, the model learns the association between the effort label and the target reasoning length directly from the training examples. An RL-based implementation would instead place the effort labels and budget-aware reward inside the RLVR stage. The two approaches could also be combined, which I suspect was done for both gpt-oss and GPT 5.6 (note that effort settings in GPT 5.6 are likely just changing the system prompt for a given user query).在此 SFT 阶段，模型直接从训练示例中学习投入标签与目标推理长度之间的关联。基于 RL 的实现则会将投入标签和预算感知奖励放置在 RLVR 阶段内。这两种方法也可以结合使用，我怀疑 gpt-oss 和 GPT 5.6 都是这样做的（请注意，GPT 5.6 中的投入设置很可能只是针对给定的用户查询更改系统提示词）。

5.3 Inkling case study

5.3 Inkling 案例研究

The just-released Inkling technical report gives a small but somewhat concrete example of effort-level training.刚刚发布的 Inkling 技术报告提供了一个关于投入水平训练的小型但相当具体的示例。

Figure 22: Inkling sweeps a continuous effort value between 0.2 and 0.99; higher effort generally produces longer responses and higher benchmark scores.图 22：Inkling 在 0.2 到 0.99 之间扫描连续的投入值；较高的投入通常会产生更长的响应和更高的基准测试分数。

During large-scale RL, they did two things for each sample:在大规模 RL 期间，他们对每个样本做了两件事：

1. Specified the desired effort level in the system message.在系统消息中指定所需的投入水平。

2. Adjusted the cost assigned to each generated token.调整分配给每个生成 Token 的成本。

Conceptually, the reward likely looked something like this:从概念上讲，奖励可能看起来像这样：

$R(e)=R_{\text{task}}-\lambda (e)N_{\text{tokens}}$

Here, e is the requested effort level and λ(e) controls the token penalty.这里，e 是请求的投入水平，λ(e) 控制 Token 惩罚。

• Low effort uses a larger per-token cost, encouraging shorter reasoning traces.低投入使用较高的单 Token 成本，鼓励更短的推理轨迹。

• High effort uses a smaller per-token cost, allowing the model to spend more tokens.高投入使用较低的单 Token 成本，允许模型花费更多的 Token。

Then, at inference time, Inkling receives a system message such as Thinking effort level: 0.8, and adjusts its token usage accordingly. The difference between Inkling and models such as gpt-oss and GPT-5.6 is that the effort label is a continuous number between 0 and 1 instead of ordinal labels such as low, medium, and high.然后，在推理时，Inkling 接收到诸如“Thinking effort level: 0.8”之类的系统消息，并相应地调整其 Token 使用量。Inkling 与 gpt-oss 和 GPT-5.6 等模型的区别在于，投入标签是一个 0 到 1 之间的连续数字，而不是像低、中、高这样的序数标签。

This places Inkling’s effort-level conditioning primarily in the Reasoning RL stage, not only in the later SFT stage.这使得 Inkling 的投入水平条件化主要位于推理 RL 阶段，而不仅仅是在后期的 SFT 阶段。

They do not disclose the exact reward formula, token-cost coefficients, or whether effort conditioning was also included in SFT, though.不过，他们没有披露确切的奖励公式、Token 成本系数，也没有透露是否在 SFT 中也包含了投入条件化。

5.4 A short note about inference scaling versus training scaling

5.4 关于推理扩展与训练扩展的简短说明

Before moving on to the reasoning effort papers, I want to briefly connect this section back to the earlier “2.3 Inference scaling in a nutshell” section.在进入推理投入论文之前，我想简要地将本节与前面的“2.3 推理扩展简述”一节联系起来。

Earlier, I separated scaling into training compute scaling and inference-time scaling. The GPT-5.6 interface provides a nice way to illustrate the difference, as shown below.早些时候，我将扩展分为训练计算扩展和推理时间扩展。GPT-5.6 界面提供了一种很好的方式来说明这种差异，如下所示。

On the left, selecting Luna, Terra, or Sol changes the model itself. As a rough analogy, this corresponds to training compute scaling. These are separate trained models. At a fixed training recipe and dataset size, a larger model requires more training compute. It also generally requires more compute per generated token.在左侧，选择 Luna、Terra 或 Sol 会改变模型本身。作为一个粗略的类比，这对应于训练计算扩展。这些是单独训练的模型。在固定的训练方案和数据集大小下，更大的模型需要更多的训练计算。它通常也需要在每个生成的 Token 上消耗更多的计算量。

On the right, we keep the model fixed and only change the reasoning effort. This is inference-time scaling. The model weights stay the same, but the model is allowed to spend fewer or more tokens working on the answer.在右侧，我们保持模型固定，只改变推理投入。这是推理时间扩展。模型权重保持不变，但模型被允许花费更少或更多的 Token 来处理答案。

Figure 23: The model selection and reasoning effort menus correspond to two different scaling axes. Selecting Luna, Terra, or Sol changes the model, whereas changing the reasoning effort adjusts the inference-time compute for a fixed model.图 23：模型选择和推理投入菜单对应于两个不同的扩展轴。选择 Luna、Terra 或 Sol 会改变模型，而更改推理投入则会调整固定模型的推理时间计算量。

One small terminology caveat is that selecting a different model from the menu is not training scaling at that moment. The training has already happened. It is better to think of the model menu as selecting among models that were produced at different training scales.一个小小的术语警告是，从菜单中选择不同的模型并不是那一刻的训练扩展。训练已经发生。将模型菜单视为在不同训练规模下产生的模型之间进行选择会更好。

The Artificial Analysis results below show how these two axes interact in practice.下方的 Artificial Analysis 结果显示了这两个轴在实践中是如何相互作用的。

Each blue curve corresponds to one model, Luna, Terra, or Sol. Moving along a curve by increasing the reasoning effort is inference scaling. Moving from one model curve to another corresponds to model scaling, which I use here as a practical proxy for training scaling.每条蓝色曲线对应一个模型：Luna、Terra 或 Sol。通过增加推理投入沿着曲线移动是推理扩展。从一条模型曲线移动到另一条曲线对应于模型扩展，我在这里将其用作训练扩展的实际代理。

As expected, both approaches can improve the benchmark score, but they also increase the cost. More interestingly, the curves overlap. For instance, a smaller model at a higher reasoning effort can sometimes reach a similar score as a larger model at a lower reasoning effort.正如预期的那样，两种方法都可以提高基准测试分数，但它们也会增加成本。更有趣的是，曲线会重叠。例如，在较高推理投入下的较小模型有时可以达到与较低推理投入下的较大模型相似的分数。

Figure 24: Training scaling and inference scaling for the GPT-5.6 model family on the Artificial Analysis Coding Agent Index. Moving along each model curve corresponds to increasing the reasoning effort. Moving across the Luna, Terra, and Sol curves corresponds to selecting a different model.图 24：GPT-5.6 模型系列在 Artificial Analysis 编码智能体指数上的训练扩展和推理扩展。沿着每条模型曲线移动对应于增加推理投入。跨越 Luna、Terra 和 Sol 曲线移动对应于选择不同的模型。

By the way, the x-axis in this figure shows API cost rather than raw compute. The API cost is a useful practical measure, but it also depends on the provider’s pricing and the number of generated tokens. Also, the exact shape of these curves is benchmark-specific.顺便提一下，此图中的 X 轴显示的是 API 成本而不是原始计算量。API 成本是一个有用的实际度量，但它也取决于提供商的定价和生成的 Token 数量。此外，这些曲线的确切形状是基准测试特定的。

So, the model size and reasoning effort form two separate knobs. We can use a larger model, increase the reasoning effort, or combine both. Which combination is best depends on the desired accuracy, cost, and latency.因此，模型大小和推理投入形成了两个独立的调节旋钮。我们可以使用更大的模型、增加推理投入，或两者结合。哪种组合最好取决于所需的准确性、成本和延迟。

So far, the article should give you a pretty solid understanding of how reasoning effort modes work and how they are implemented. This is a fine point to wrap the article if you are short on time. Otherwise, if you want to look into some of the nitty-gritty details of some of the recent open-weight models, please read on!到目前为止，本文应该能让你对推理投入模式的工作原理及其实现方式有一个相当扎实的了解。如果你时间紧迫，这可以作为文章的一个很好的结尾。否则，如果你想深入了解最近一些开放权重模型的细节，请继续阅读！

6. Bonus: Different ways to implement reasoning efforts (in flagship open-weight LLMs)

6. 奖励：实现推理投入的不同方式（在旗舰开放权重大语言模型中）

[This section is fine to skip unless you are interested in some additional details][除非你对一些额外细节感兴趣，否则可以跳过本节]

Section 5 described two possible ways to train reasoning-effort controls, namely effort-conditioned supervised fine-tuning and reinforcement learning with different token costs. Originally, I wanted to cover research articles on alternative ways to implement reasoning budgets. However, reading through most of these articles, they seemed more like proofs-of-concept that may or may not work well in practice.第 5 节描述了两种训练推理投入控制的可能方法，即投入条件化监督微调和具有不同 Token 成本的强化学习。最初，我想涵盖有关实现推理预算替代方法的各种研究文章。然而，在阅读了大部分文章后，它们看起来更像是可能在实践中有效也可能无效的概念验证。

So, instead of covering those, I decided to pivot a bit and cover those recipes used by state-of-the-art and notable open-weight (flagship) LLMs. For these models, there is at least evidence that the methods work in practice.因此，我决定稍微调整一下，转而介绍最先进和著名的开放权重（旗舰）大语言模型所使用的方案。对于这些模型，至少有证据表明这些方法在实践中是有效的。

This leaves six examples. DeepSeek V4, Nemotron 3 Ultra, Kimi K2.5, GLM-5, Qwen3, and Inkling. They have different levels of detail in their reporting, but each contributes a useful variation. (I exclude models whose reports only show an effort setting in the user interface without explaining how that behavior was trained.)这留下了六个例子：DeepSeek V4、Nemotron 3 Ultra、Kimi K2.5、GLM-5、Qwen3 和 Inkling。它们在报告中的详细程度各不相同，但每种模型都贡献了一种有用的变体。（我排除了那些报告仅在用户界面中显示投入设置，而未解释该行为是如何训练的模型。）

6.1 DeepSeek V4 trains separate effort specialists

6.1 DeepSeek V4 训练独立的投入专家模型

Let’s start with the DeepSeek V4 technical report, which describes the use of three modes:让我们从 DeepSeek V4 技术报告开始，它描述了三种模式的使用：

• Non-think produces a direct response without a reasoning trace.Non-think（非思考）：直接产生响应，无需推理轨迹。

• Think High is the classic approach where the model places the reasoning trace between <think> and </think> tags. This is similar to what was discussed in the DeepSeek R1 section (section 2) at the beginning of this article.Think High（高强度思考）：这是经典方法，模型将推理轨迹置于 标签之间。这类似于本文开头在 DeepSeek R1 部分（第 2 节）中讨论的内容。

• Think Max is the same as above but adds a special system instruction. (More on that below.)Think Max（极致思考）：与上述相同，但添加了特殊的系统指令。（下面会有更多说明。）

The additional system prompt instruction for Think Max starts with “Reasoning Effort: Absolute maximum with no shortcuts permitted.”Think Max 的额外系统提示词指令以“Reasoning Effort: Absolute maximum with no shortcuts permitted”（推理投入：绝对最大，不允许任何捷径）开头。

Figure 25: Reasoning effort control overview from the DeepSeek V4 documentation图 25：DeepSeek V4 文档中的推理投入控制概述。

At first, this sounds like a simple prompt engineering trick, but this prompt is actually backed by a different training setup. That is, each mode uses its own context window and length penalty (unfortunately, the exact length penalty implementation is not detailed in this report). Think Max receives a longer context window and a smaller length penalty than Think High, which gives it more room to continue reasoning.起初，这听起来像是一个简单的提示词工程技巧，但这个提示词实际上是由不同的训练设置支持的。也就是说，每种模式都使用自己的上下文窗口和长度惩罚（遗憾的是，报告中没有详细说明确切的长度惩罚实现）。Think Max 接收到比 Think High 更长的上下文窗口和更小的长度惩罚，这给了它更多继续推理的空间。

So, the system instruction selects a behavior that was created during post-training. Adding the same instruction to an arbitrary model would not have the same effect.因此，系统指令选择的是在后训练期间创建的行为。将相同的指令添加到任意模型中不会产生相同的效果。

Figure 26: DeepSeek V4 describes the three effort modes and the larger teacher pool in separate parts of the report. The teacher pool contains more than ten domain specialists. The report does not disclose how these teachers map to Non-think, Think High, and Think Max.图 26：DeepSeek V4 在报告的不同部分描述了三种投入模式和更大的教师池。教师池包含十多个领域专家。报告没有披露这些教师如何映射到 Non-think、Think High 和 Think Max。

Unfortunately, the public, and otherwise very detailed DeepSeek V4 report does not connect the descriptions of the reasoning mode and domain specialists in enough detail to reconstruct the exact teacher assignment.遗憾的是，公开且内容非常详尽的 DeepSeek V4 报告并没有以足够的细节将推理模式和领域专家的描述联系起来，以重构确切的教师分配。

However, the report states that the final model, which supports different reasoning effort levels, was created via on-policy distillation from said teachers.然而，报告指出，支持不同推理投入水平的最终模型是通过从上述教师模型进行策略内蒸馏（on-policy distillation）创建的。

To summarize, DeepSeek V4 develops the three reasoning specialists during post-training. Starting from the base model, it applies supervised fine-tuning followed by RLVR via GRPO. The RL configuration differs for each mode. In particular, each specialist uses its own context window and length penalty, while Think Max additionally receives a special system instruction.总结一下，DeepSeek V4 在后训练期间开发了三个推理专家模型。从基础模型开始，它应用监督微调，随后通过 GRPO 进行 RLVR。每种模式的 RL 配置各不相同。特别是，每个专家模型都使用自己的上下文窗口和长度惩罚，而 Think Max 额外接收一条特殊的系统指令。

Then, including domain specialists, the different reasoning mode specialists are distilled into a single checkpoint that supports all three effort modes.然后，包括领域专家在内，不同的推理模式专家被蒸馏到一个支持所有三种投入模式的单一检查点中。

6.2 Nemotron 3 Ultra combines learned modes with hard budgets

6.2 Nemotron 3 Ultra 将学习到的模式与硬预算相结合

The Nemotron 3 Ultra technical report describes three settings called reasoning-off, regular, and medium-effort, analogous to DeepSeek V4 in the previous section. Medium-effort is the cheaper reasoning mode compared with regular. NVIDIA introduces this mode during SFT using examples generated by GPT-OSS-120B in its medium-effort mode, and then further optimizes it during RLVR. About 2.5% of the RLVR prompts use medium-effort (this corresponds to length-based adjustments applied to their rewards).Nemotron 3 Ultra 技术报告描述了三种设置，分别称为 reasoning-off（关闭推理）、regular（常规）和 medium-effort（中等投入），类似于上一节中的 DeepSeek V4。与常规模式相比，中等投入是一种更便宜的推理模式。NVIDIA 在 SFT 期间使用由 GPT-OSS-120B 在其中等投入模式下生成的示例引入了此模式，然后在 RLVR 期间对其进行了进一步优化。大约 2.5% 的 RLVR 提示词使用中等投入（这对应于应用于其奖励的基于长度的调整）。

6.2.1 Using Nemotron reasoning budgets during inference

6.2.1 在推理过程中使用 Nemotron 推理预算

At inference time, all three modes are selected through the chat template.在推理时，所有三种模式均通过聊天模板（chat template）进行选择。

Figure 27: Nemotron 3 Ultra reasoning settings via the chat template (examples from the official model card)图 27：通过聊天模板设置 Nemotron 3 Ultra 的推理参数（示例来自官方模型卡）

1) Regular is the default and uses enable_thinking=True, which starts the assistant response with an opening <think> tag.1) Regular（常规）模式是默认设置，使用 enable_thinking=True，这会使助手的回复以 ，客户端将关闭推理块并继续生成以产生最终答案。学习到的努力模式决定了模型如何使用其推理 token，而预算则限制了推理轨迹的持续长度。这使得根据所需的成本和准确性，将任一模式与更严格或更宽松的预算配对成为可能。

2) Medium-effort uses enable_thinking=True together with medium_effort=True where the latter setting also appends {reasoning effort: efficient} to the latest user message.2) 中等努力模式（Medium-effort）需同时设置 enable_thinking=True 和 medium_effort=True，后者会在最新的用户消息中附加 {reasoning effort: efficient}。

By the way, to further complicate things, the regular and medium-effort modes can also be combined with a separate inference-time reasoning budget. This budget acts as an external stopping mechanism. In the released implementation, the chat client asks the model to end the reasoning trace near the chosen token limit. If the model has not emitted </think>, the client closes the reasoning block and continues generation to produce the final answer. The learned effort mode determines how the model uses its reasoning tokens, while the budget constrains how long the reasoning trace can continue. This makes it possible to pair either mode with a tighter or looser budget depending on the desired cost and accuracy.顺便提一下，情况会更复杂一些：常规模式和中等努力模式还可以与独立的推理时间预算（inference-time reasoning budget）结合使用。该预算充当一种外部停止机制。在已发布的实现中，聊天客户端会要求模型在选定的 Token 限制附近结束推理过程。如果模型尚未输出 </think>，客户端将强制关闭推理区块并继续生成最终答案。学习到的“努力模式”决定了模型如何使用其推理 Token，而“预算”则限制了推理过程的持续长度。这使得我们可以根据对成本和准确性的需求，将任一模式与更严格或更宽松的预算进行搭配。

3) Reasoning-off uses enable_thinking=False, which prefills an empty <think></think> block (similar to Qwen3 discussed in section 4) so that the model proceeds directly to the final response. Thus, these are chat-template controls rather than system prompts.3) Reasoning-off（关闭推理）模式使用 enable_thinking=False，它会预填充一个空的 块（类似于第 4 节中讨论的 Qwen3），以便模型直接进行最终回复。因此，这些是聊天模板控件，而不是系统提示词。

6.2.2 Reasoning budget-aware training in Nemotron

6.2.2 Nemotron 中的推理预算感知训练

The inference controls described above are backed by two related SFT components. The first introduces medium-effort behavior using GPT-OSS-120B traces, as discussed earlier. The second prepares the model for hard reasoning budgets.上述推理控件由两个相关的 SFT（监督微调）组件支持。第一个组件如前所述，使用 GPT-OSS-120B 轨迹引入中等努力行为。第二个组件则让模型为硬性推理预算做好准备。

To construct this training data, the authors take regular reasoning traces, truncate them at randomly selected token budgets, and keep the original final answers. The inserted </think> token is masked from the SFT loss. As a result, the model sees examples where it has to move from an incomplete reasoning trace to the answer after the reasoning block has been closed externally.为了构建这些训练数据，作者采用常规的推理轨迹，在随机选择的 token 预算处截断它们，并保留原始的最终答案。插入的 </think> token 在 SFT 损失计算中会被掩盖。因此，模型看到的示例需要它在推理块被外部关闭后，从不完整的推理轨迹过渡到答案。

Medium-effort training then continues during RLVR. About 2.5% of the RL prompts use the medium-effort setting across math, STEM, and coding tasks. The report notes that the mode can be calibrated through reward hyperparameters where length-based reward adjustments provide additional control over the cost-quality trade-off.中等努力训练随后在 RLVR（基于验证的强化学习）期间继续进行。在数学、STEM 和编码任务中，约 2.5% 的 RL 提示词使用了中等努力设置。报告指出，可以通过奖励超参数对该模式进行校准，其中基于长度的奖励调整为成本与质量的权衡提供了额外的控制。

Figure 28: Nemotron 3 Ultra introduces medium effort with teacher-generated SFT data, random-budget truncation, and a small medium-effort subset during RLVR.图 28：Nemotron 3 Ultra 通过教师生成的 SFT 数据、随机预算截断以及 RLVR 期间的一小部分中等努力子集，引入了中等努力模式。

6.3 Kimi K2.5 alternates budgeted and unconstrained RL

6.3 Kimi K2.5 交替使用预算限制和无约束强化学习

The Kimi K2.5 technical report discusses a training method called Token Efficient RL for lower reasoning effort. (While there was a K3 announcement this week, the reasoning-effort methodology of K3 is not publicly disclosed, but it could be similar or related to K2.5.)Kimi K2.5 技术报告讨论了一种称为“Token 高效强化学习”（Token Efficient RL）的训练方法，旨在降低推理努力。（虽然本周有 K3 的发布公告，但 K3 的推理努力方法尚未公开，不过它可能与 K2.5 相似或相关。）

6.3.1 Kimi’s Toggle method

6.3.1 Kimi 的 Toggle 方法

The report mentions that a fixed token budget can make a reasoning model overfit to short solutions. That means the model becomes more concise (i.e., faster and cheaper), but it may lose the ability to benefit from additional inference-time compute and can thus perform poorly.报告提到，固定的 token 预算可能会导致推理模型对简短的解决方案产生过拟合。这意味着模型变得更加简洁（即更快、更便宜），但可能会失去从额外的推理时计算中获益的能力，从而导致性能下降。

Figure 29: The proposed Toggle method makes Kimi K2.5 much more token-efficient while keeping the overall benchmark performance similar. Annotated figure from https://arxiv.org/abs/2602.02276图 29：所提出的 Toggle 方法使 Kimi K2.5 在保持整体基准性能相似的同时，显著提高了 token 使用效率。图片标注来自 https://arxiv.org/abs/2602.02276

Kimi K2.5’s method, called Toggle, alternates between two RL phases every fixed number of training iterations:Kimi K2.5 的 Toggle 方法每隔固定的训练迭代次数，在两个强化学习阶段之间交替：

1. In the budgeted phase, correct solutions are encouraged to stay within a problem-specific token budget.1. 在预算限制阶段，鼓励正确的解决方案保持在特定问题的 token 预算内。

2. In the unconstrained phase, the usual maximum generation length is restored so that the model can still learn from longer solutions.2. 在无约束阶段，恢复通常的最大生成长度，以便模型仍能从更长的解决方案中学习。

For each problem, the budget is estimated from a selected percentile of response lengths among correct rollouts in RLVR. The budget constraint is then only activated once the mean accuracy on that problem exceeds a threshold. This avoids forcing the model to shorten its reasoning before it can solve the problem reliably.对于每个问题，预算是根据 RLVR 中正确输出的响应长度的选定百分位数估算的。只有当该问题的平均准确率超过阈值时，才会激活预算限制。这避免了在模型能够可靠地解决问题之前强迫其缩短推理过程。

Figure 30: Overview of the two phases of the Toggle method.图 30：Toggle 方法的两个阶段概述。

The report evaluates Toggle on K2 Thinking and finds that it reduces generated tokens by about 25 to 30% with little change in benchmark performance. The same behavior also transfers from math and coding RL tasks to GPQA and MMLU-Pro.报告在 K2 Thinking 上评估了 Toggle，发现它在基准性能几乎没有变化的情况下，减少了约 25% 到 30% 的生成 token。这种行为也从数学和编码 RL 任务转移到了 GPQA 和 MMLU-Pro 上。

Toggle supplies a concrete flagship-model recipe for training a more token-efficient reasoning policy while preserving its ability to scale at test time.Toggle 为训练更具 token 效率的推理策略提供了一个具体的旗舰模型方案，同时保留了其在测试时扩展的能力。

6.3.2 What Toggle changes at inference

6.3.2 Toggle 在推理时的变化

Toggle operates entirely during RL training. Both alternating phases update the same policy (i.e., LLM), and the final (unified) checkpoint has no budgeted-versus-unconstrained selector. At inference, the resulting model then runs in thinking mode by default.Toggle 完全在强化学习训练期间运行。两个交替阶段更新同一个策略（即 LLM），最终的（统一）检查点没有预算限制与无约束的选择器。在推理时，生成的模型默认以思考模式运行。

Interestingly, though, Kimi K2.5 itself exposes a separate binary choice between thinking and instant modes in some APIs I checked (like vLLM or SGLang). Thinking mode is enabled by default. Instant mode disables the reasoning trace through thinking: {”type”: “disabled”} in the official API or chat_template_kwargs={”thinking”: False} when serving the model through vLLM or SGLang. However, these settings are separate from Toggle.有趣的是，Kimi K2.5 本身在我检查的一些 API（如 vLLM 或 SGLang）中，在思考模式和即时模式之间提供了一个单独的二进制选择。思考模式是默认启用的。即时模式通过 thinking: {”type”: “disabled”}（官方 API）或在使用 vLLM 或 SGLang 服务模型时设置 chat_template_kwargs={”thinking”: False} 来禁用推理轨迹。然而，这些设置与 Toggle 是分开的。

Also, the official Kimi report does not provide a separate training recipe for instant mode. However, K2.5’s SFT data were generated using both the earlier K2 model, which produces direct responses without long reasoning, and K2 Thinking, which produces extended reasoning traces. This likely exposes the unified checkpoint to both response formats similar to what’s done in Nemotron 3 above. At inference time, the chat template selects between them by prefilling either an open <think> tag for thinking mode or an empty <think></think> block for instant mode. But again, unfortunately, the report does not disclose the exact data mixture or whether additional mode-specific RL was used.此外，官方的 Kimi 报告没有为即时模式提供单独的训练方案。然而，K2.5 的 SFT 数据是使用早期的 K2 模型（产生无需长推理的直接响应）和 K2 Thinking（产生扩展的推理轨迹）生成的。这很可能使统一检查点接触到了两种响应格式，类似于上面提到的 Nemotron 3。在推理时，聊天模板通过预填充一个开放的 块（即时模式）来进行选择。但遗憾的是，报告没有披露确切的数据混合比例，也没有说明是否使用了额外的模式特定强化学习。

The newer Kimi K3 provides a more direct inference-time effort interface. The current Kimi Code documentation lists three settings called low, high, and max, with max as the default. These are passed through the reasoning_effort parameter. However, Moonshot has not yet explained how the three effort levels were created during training. Its launch post says that these details will appear in a future K3 technical report, so I’ll stay tuned for that.较新的 Kimi K3 提供了更直接的推理时努力接口。目前的 Kimi Code 文档列出了三种设置，分别为 low、high 和 max，其中 max 为默认值。这些通过 reasoning_effort 参数传递。然而，月之暗面（Moonshot）尚未解释这三种努力级别是如何在训练过程中创建的。其发布文章称，这些细节将出现在未来的 K3 技术报告中，因此我将拭目以待。

6.4 GLM-5 introduces turn-level and interleaved thinking through SFT

6.4 GLM-5 通过 SFT 引入了轮次级和交错式思考

The GLM-5 technical report extends the binary on/off thinking switch introduced with GLM-4.5 to multi-turn and tool-using scenarios. It describes three related behaviors (rather than three effort levels):GLM-5 技术报告将 GLM-4.5 中引入的二进制思考开关扩展到了多轮和工具使用场景。它描述了三种相关的行为（而不是三个努力级别）：

• Interleaved thinking: this inserts a reasoning block before each response and tool call.交错式思考（Interleaved thinking）：在每次响应和工具调用之前插入一个推理块。

• Preserved thinking: here, the chat retains earlier reasoning blocks across turns so that the model can reuse them later.保留式思考（Preserved thinking）：聊天在轮次之间保留早期的推理块，以便模型稍后可以重用它们。

• Turn-level thinking: this enables or disables reasoning separately for each request in a conversation.轮次级思考（Turn-level thinking）：为对话中的每个请求单独启用或禁用推理。

At inference time, turn-level thinking is the actual on-off switch. In the Z.ai API, thinking is enabled by default and can be disabled for an individual request with thinking: {”type”: “disabled”}. The hosted implementation is not disclosed but the open GLM-5 chat template shows the equivalent mechanism when self-hosting with Transformers, vLLM, or SGLang.在推理时，轮次级思考是实际的开关。在 Z.ai API 中，思考默认启用，可以通过 thinking: {”type”: “disabled”} 为单个请求禁用。托管实现的细节未公开，但开源的 GLM-5 聊天模板展示了在使用 Transformers、vLLM 或 SGLang 自托管时的等效机制。

It starts the assistant response with <|assistant|><think> when thinking is enabled and <|assistant|></think> when it is disabled. The latter closes the reasoning block immediately, so generation proceeds directly to the final answer.当启用思考时，它以 <|assistant|> 开头。后者会立即关闭推理块，因此生成过程直接进入最终答案。

The report says that these behaviors are introduced during multi-task SFT together with an updated chat template.报告称，这些行为是在多任务 SFT 期间与更新后的聊天模板一起引入的。

After SFT, GLM-5 goes through reasoning RL, agentic RL, and general RL. And a final on-policy distillation step uses checkpoints from the preceding stages as teachers. This helps the final model recover capabilities that may have weakened during the sequential RL stages.在 SFT 之后，GLM-5 经历了推理强化学习、智能体强化学习和通用强化学习。最终的策略蒸馏步骤使用前几个阶段的检查点作为教师。这有助于最终模型恢复在顺序强化学习阶段可能减弱的能力。

Figure 31: GLM-5 training pipeline.图 31：GLM-5 训练流程。

6.5 Qwen3 uses mode fusion and inference-time truncation

6.5 Qwen3 使用模式融合和推理时截断

Qwen3 was already covered in Section 4, so I will only summarize the parts that matter for this comparison. According to the Qwen3 technical report, its post-training pipeline has four stages. These are long-chain-of-thought SFT, reasoning RL, Thinking Mode Fusion, and general RL.Qwen3 已在第 4 节中介绍过，因此我只总结对本次比较重要的部分。根据 Qwen3 技术报告，其训练后流程有四个阶段：长思维链 SFT、推理强化学习、思考模式融合（Thinking Mode Fusion）和通用强化学习。

Thinking Mode Fusion is the key stage for the effort on-off switch. Here the model is trained via SFT on a mixture of thinking and non-thinking examples. The /think examples contain a reasoning trace, while /no_think examples begin with an empty <think></think> block that is accompanied by a short answer. The following general RL stage reinforces instruction and format following for both behaviors.思考模式融合是努力开关的关键阶段。在这里，模型通过 SFT 在思考和非思考示例的混合数据上进行训练。/think 示例包含推理轨迹，而 /no_think 示例以一个空的 块开头，并附带简短的答案。随后的通用强化学习阶段加强了两种行为的指令和格式遵循。

Qwen3 also supports a hard thinking budget. At the requested threshold, the reasoning span is stopped and a stop-thinking instruction is inserted before the model continues with its final answer. The report says that this partial-reasoning behavior was not trained explicitly. It emerged after Thinking Mode Fusion.Qwen3 还支持硬性思考预算。在达到请求的阈值时，推理范围会停止，并插入一条“停止思考”指令，然后模型继续给出最终答案。报告称，这种部分推理行为并非显式训练的，而是在思考模式融合后自然涌现的。

This gives Qwen3 a learned on-off switch plus an inference-time budget. It is similar but simpler than the DeepSeek V4 and Nemotron recipes.这为 Qwen3 提供了一个学习到的开关加上推理时预算。它与 DeepSeek V4 和 Nemotron 的方案相似但更简单。

6.6 Inkling conditions RL on a continuous effort value

6.6 Inkling 根据连续努力值调节强化学习

Inkling was already discussed in Section 5.3. The short version is that its technical report mentions that they use continuous effort conditioning (values between 0.0 and 1.0) rather than fixed effort labels.Inkling 已在第 5.3 节中讨论过。简而言之，其技术报告提到他们使用连续的努力调节（0.0 到 1.0 之间的值）而不是固定的努力标签。

After a relatively small initial SFT stage, most of Inkling’s post-training comes from asynchronous RL with more than 30 million rollouts. The desired effort is included in the system message, and the token length penalty is adjusted according to that value during RL. As previously discussed, a higher token cost encourages a shorter response. A lower token cost gives the model more room to reason.在相对较小的初始 SFT 阶段之后，Inkling 的大部分训练后处理来自具有超过 3000 万次 rollout 的异步强化学习。所需的努力程度包含在系统消息中，并且在强化学习期间根据该值调整 token 长度惩罚。如前所述，更高的 token 成本鼓励更短的响应，而更低的 token 成本则为模型提供了更多的推理空间。

6.7 Overview of the known recipes

6.7 已知方案概述

The table below summarizes what is actually documented in the six technical reports.下表总结了六份技术报告中实际记录的内容。

Figure 32: Comparison of the disclosed training mechanisms and inference controls for six open-weight models with reasoning-effort settings.图 32：六种具有推理努力设置的开源权重模型的训练机制和推理控件对比。

So, looking at the six different open-weight models, they have a shared framework. First, they introduce effort mode control through SFT and the chat template. Qwen3 explicitly mixes thinking and non-thinking examples, while GLM-5 adds interleaved, preserved, and turn-level thinking patterns.综上所述，这六种不同的开源权重模型拥有一个共享框架。首先，它们通过 SFT 和聊天模板引入努力模式控制。Qwen3 显式混合了思考和非思考示例，而 GLM-5 增加了交错式、保留式和轮次级思考模式。

The second shared component is a mode-conditioned RL stage, where context windows and length penalties change with the requested effort. DeepSeek V4, Nemotron 3 Ultra, and Inkling use this approach.第二个共享组件是模式调节强化学习阶段，其中上下文窗口和长度惩罚随请求的努力程度而变化。DeepSeek V4、Nemotron 3 Ultra 和 Inkling 使用了这种方法。

A third ingredient improves robustness under explicit budgets. Nemotron trains on randomly truncated traces, Qwen3 can continue from a forcibly stopped reasoning span, and Kimi alternates budgeted with unconstrained RL. These methods help preserve answer quality when the available reasoning length changes and is even cut short.第三个要素提高了在显式预算下的鲁棒性。Nemotron 在随机截断的轨迹上进行训练，Qwen3 可以从被强制停止的推理范围继续，而 Kimi 则交替使用预算限制和无约束强化学习。这些方法有助于在推理长度发生变化甚至被截断时保持答案质量。

7. Conclusion

7. 结论

The open-weight examples in this article implement reasoning effort through several different mechanisms. Similar labels can be backed by separate specialists, mixed SFT data, mode-conditioned rewards, hard token budgets, or combinations of these methods.本文中的开源权重示例通过几种不同的机制实现了推理努力。相似的标签背后可能由不同的专家模型、混合 SFT 数据、模式调节奖励、硬性 token 预算或这些方法的组合所支持。

It is difficult to say which approach is best. The models differ in their base checkpoints, training data, post-training compute, benchmarks, and serving goals. Their reports also omit many details needed for a controlled comparison. (Also, there may not be a one-size-fits-all, and a method that works well for an interactive assistant may be a poor fit for a long-running coding agent.)很难说哪种方法最好。这些模型在基础检查点、训练数据、训练后算力、基准测试和部署目标上各不相同。它们的报告也省略了受控比较所需的许多细节。（此外，可能不存在“一刀切”的方法，适用于交互式助手的方案可能并不适合长期运行的编码代理。）

The holy grail is of course automatic effort selection. We saw this a while back with GPT 5’s Auto mode. It’s a tricky problem to solve, and in the end, the implementation was probably more miss than hit, which is why it got removed from the UI (at least, I can’t find it anymore).圣杯当然是自动努力选择。我们之前在 GPT 5 的 Auto 模式中看到过这一点。这是一个棘手的问题，最终实现的效果可能并不理想，这就是为什么它从 UI 中被移除的原因（至少，我再也找不到了）。

In the near future, I think reasoning effort will remain an explicit model input, which will most often be delivered through the system prompt. However agent wrapper/harness around the LLM, or an internal router may increasingly infer the appropriate mode and budget from the task state and available resources automatically (while of course still allowing a user override).在不久的将来，我认为推理努力将保持为显式的模型输入，最常见的是通过系统提示词传递。然而，围绕 LLM 的智能体封装/框架，或内部路由器可能会越来越多地根据任务状态和可用资源自动推断适当的模式和预算（当然，同时仍允许用户覆盖）。

I still hope that effort selection will become more automatic. Similar to GPT 5’s auto mode, a cheap model or router could choose the mode from the request, tool state, and remaining time or token budget while still allowing a user override. The override is useful if you want to optimize for latency or cost, or maximum performance.我仍然希望努力选择能变得更加自动化。类似于 GPT 5 的自动模式，轻量级模型或路由器可以根据请求、工具状态以及剩余时间和 token 预算来选择模式，同时仍然允许用户覆盖。如果您想针对延迟、成本或最高性能进行优化，覆盖功能非常有用。

I realize that this was a long article, and it was perhaps not the flashiest topic. But I thought that given all the talk about LLMs, reasoning models, and agents, a look at reasoning models was something not covered before, and I hope it was a unique and somewhat useful overview!我知道这是一篇很长的文章，而且可能不是最引人注目的主题。但考虑到关于 LLM、推理模型和智能体的所有讨论，我认为对推理模型进行审视是之前未曾涵盖的内容，希望这是一个独特且有一定参考价值的概述！

Further resources

更多资源

If you want a hands-on implementation of the core training methods behind reasoning models, my Build a Reasoning Model (From Scratch) book walks through reinforcement learning with verifiable rewards and inference-time scaling step by step, with code.如果您想要动手实现推理模型背后的核心训练方法，我的《Build a Reasoning Model (From Scratch)》一书通过代码一步步讲解了基于可验证奖励的强化学习和推理时缩放。

This article focused on how a trained reasoning model can support different effort modes. The book takes a step back and shows how to turn a conventional LLM into a reasoning model in the first place. It is a sequel to Build a Large Language Model (From Scratch) and starts where that book leaves off.本文重点介绍了训练好的推理模型如何支持不同的努力模式。而该书则退后一步，展示了如何首先将传统的 LLM 转变为推理模型。它是《Build a Large Language Model (From Scratch)》的续篇，从前一本书结束的地方开始。

The print edition has now started shipping印刷版现已开始发货

Build a Reasoning Model (From Scratch) [Manning] [Amazon]Build a Reasoning Model (From Scratch) [Manning] [Amazon]

If you liked my previous Build a Large Language Model (From Scratch) book, this is essentially a sequel implementing inference-time scaling techniques and reinforcement learning algorithms from scratch.如果您喜欢我之前的书《Build a Large Language Model (From Scratch)》，这本质上是一本续作，从零开始实现了推理时缩放技术和强化学习算法。

And if you want to support future long-form articles like this one, consider becoming a paid subscriber. It helps me keep writing these independent deep dives and sharing the accompanying code, figures, and experiments.如果您想支持未来像这样长篇幅的文章，请考虑成为付费订阅者。这有助于我继续撰写这些独立的深度解析，并分享随附的代码、图表和实验。