This is the first article in a series about post-training. Follow along on Radar.这是关于训练后阶段系列文章的第一篇。请在“雷达”（Radar）上关注后续内容。
Before post-training, there was a major problem with LLMs: Almost nobody could use them. The story of post-training is also the story of how AI went from a research curiosity to a product used by about a billion people.在训练后阶段出现之前，大语言模型（LLM）存在一个重大问题：几乎没人能用。训练后阶段的发展史，也是 AI 从一项研究领域的好奇心，演变为被约十亿人使用的产品的历程。
Post-training is the reason why a model behaves a certain way. This set of training techniques makes LLMs useful (e.g., able to chat with people and interact with AI agents), safe (e.g., aligned with human intentions), and more capable (e.g., through “reasoning” to tackle difficult tasks). Behavior is powerful, and doesn’t just mean holding a conversation or following a user’s instructions. Behavior includes making it possible for the model to use tools, like a calculator tool, a search API, or any application through an MCP. Behavior can even elevate a model’s intelligence, for example by teaching the model to use “reasoning”: that is, working through problems before giving a final answer rather than “guessing” or “memorizing.”训练后阶段决定了模型的行为方式。这一系列训练技术使 LLM 变得实用（例如，能够与人聊天并与 AI 智能体互动）、安全（例如，符合人类意图）且能力更强（例如，通过“推理”解决复杂任务）。行为的力量非常强大，它不仅仅意味着进行对话或遵循用户指令。行为还包括使模型能够使用工具，如计算器、搜索 API，或通过 MCP 使用任何应用程序。行为甚至可以提升模型的智能，例如通过教导模型进行“推理”：即在给出最终答案之前先思考问题，而不是单纯地“猜测”或“背诵”。
From GPT-3 to ChatGPT: The post-training revolution从 GPT-3 到 ChatGPT：训练后阶段的革命
GPT-3 showed up in June 2020. A completion engine, it followed patterns it had seen from its pretraining data, which were not predominantly chat conversations. Imagine scraping data on the internet: that pretraining data had a lot of questions that were followed by other questions—for example, on an exam template. GPT-3 was 175B parameters, large for its time, and it had a wide, general range of abilities, although many of them were latent.GPT-3 于 2020 年 6 月问世。作为一个补全引擎，它遵循的是从预训练数据中习得的模式，而这些数据并非主要是聊天对话。想象一下从互联网上抓取数据：预训练数据中包含许多问题，后面紧跟着其他问题——例如在考试模板中。GPT-3 拥有 1750 亿个参数，在当时规模巨大，并具备广泛的通用能力，尽管其中许多能力处于潜伏状态。
If you gave GPT-3 a prompt like “Why do people like golden retrievers?” it might say something nonsensical:如果你给 GPT-3 一个类似于“为什么人们喜欢金毛寻回犬？”的提示，它可能会说出一些毫无意义的话：
Why do people like labrador retrievers?
Why do people like poodles?
10 Reasons You Should Adopt a Dog Today为什么人们喜欢拉布拉多寻回犬？为什么人们喜欢贵宾犬？今天你应该收养狗的 10 个理由
These answers look absurd in isolation, but if you imagine a web page with a list of FAQ links, this is a perfectly reasonable next chunk of text. GPT-3 might have just been completing a listicle on a website, because it had seen millions of websites in its pretraining data.这些答案单独看显得很荒谬，但如果你想象一个包含常见问题链接列表的网页，这完全是一段合理的后续文本。GPT-3 可能只是在完成网站上的列表文章，因为它在预训练数据中见过数百万个网站。
The common way to nudge GPT-3 to answer a question back then was by prompt engineering with a Q&A template and few-shot examples.当时引导 GPT-3 回答问题的常用方法是通过问答模板和少样本（few-shot）示例进行提示工程。
Q: Why do people like labrador retrievers? A: Because they are friendly, loyal, and easy to train.
Q: Why do people like beagles? A: Because they are curious, great with kids, and have a gentle temperament.
Q: Why do people like golden retrievers? A:问：为什么人们喜欢拉布拉多寻回犬？ 答：因为它们友好、忠诚且易于训练。问：为什么人们喜欢比格犬？ 答：因为它们好奇心强，与孩子相处融洽，性情温和。问：为什么人们喜欢金毛寻回犬？ 答：
Then, GPT-3 might say:这时，GPT-3 可能会说：
Because they are affectionate, patient, and make excellent family pets.因为它们深情、耐心，是非常优秀的家庭宠物。
While this technique worked, it was brittle. If you forgot the few-shot examples, rephrased the question, or even added a space after “A:,” you’d get something completely different (possibly unhinged) that was far from a reasonable response.虽然这种技术有效，但它很脆弱。如果你忘记了少样本示例、改写了问题，甚至只是在“答：”后面加了一个空格，你就会得到完全不同（甚至可能是胡言乱语）的回答，远非合理的响应。
In fact, if you were a researcher working with GPT-3 at the time, you probably at some point found the space at the beginning of the response ” Because they are gentle dogs.” annoying and would try to end your prompt with a space “A: ” instead of “A:”. In those cases, it was common for GPT-3 to go off a cliff and produce a drastically different response, sometimes completely off like “dogs dogs dogs dogs…” repeating indefinitely.事实上，如果你是当时研究 GPT-3 的研究人员，你很可能在某个时候发现响应开头带有的空格（“ 因为它们是温顺的狗。”）很烦人，并尝试将提示结尾改为“答：”而不是“答： ”。在这种情况下，GPT-3 经常会“脱轨”并产生截然不同的响应，有时甚至完全离谱，比如无限重复“狗狗狗狗……”之类的词。
The reason behind the differing responses to “A:” and “A: ” is because “A:” might tokenize to one token while “A: ” tokenizes to two different tokens. The model literally sees different input sequences, each with different statistical completions in its training data. It’s like asking two completely different questions. While a space is a tiny syntactic change that is meaningless to a person, it becomes extremely meaningful to the model that now sees two different prompts (the tokens change!) with two very different statistical futures to complete.“答：”和“答： ”产生不同响应的原因在于，“答：”可能被标记为一个 token，而“答： ”则被标记为两个不同的 token。模型实际上看到的是不同的输入序列，每个序列在训练数据中都有不同的统计补全方式。这就像是在问两个完全不同的问题。虽然空格只是一个微小的句法变化，对人类来说毫无意义，但对模型来说却意义重大，因为它现在看到了两个不同的提示（token 变了！），并且每个提示都有截然不同的统计学补全路径。
You still encounter the modern equivalent of this when working with chat templates. If you forget to apply the model’s chat template and instead just concatenate
'User: ' + prompt + '\nAssistant: ', you’re sending the model a token sequence that it was not robustly trained on. The tokens are wrong, not the model. Post-training teaches the model to respond to specific token patterns (like<|im_start|>user\nin Qwen models). Not using them is like speaking to someone in a language they half-understand. However, most open source models will be trained to be at least somewhat robust without their templates too.在使用聊天模板时，你仍然会遇到现代版的此类问题。如果你忘记应用模型的聊天模板，而是简单地拼接 'User: ' + prompt + '\nAssistant: '，你发送给模型的是它并未经过稳健训练的 token 序列。问题在于 token，而不是模型。训练后阶段会教导模型响应特定的 token 模式（例如 Qwen 模型中的 <|im_start|>user\n）。不使用它们就像用对方只懂一半的语言与人交谈。不过，大多数开源模型也会被训练得在没有模板的情况下也具备一定的稳健性。
Under those circumstances, most people would assume AI still didn’t work. The model wasn’t trained to answer questions; its data wasn’t primarily conversation transcripts. Instead, it was trained to predict the next token in downloaded websites, articles, and documents.在这种情况下，大多数人会认为 AI 还是没用。模型并没有被训练去回答问题；它的数据主要不是对话文本。相反，它被训练去预测下载的网站、文章和文档中的下一个 token。
Thankfully, this can all be fixed with post-training. And that’s when most people started to believe that AI had undergone a paradigm shift and just might work.值得庆幸的是，这一切都可以通过训练后阶段来修复。也就是在那时，大多数人开始相信 AI 已经经历了一场范式转移，并且可能真的能派上用场。
Post-training versus pretraining训练后阶段与预训练的对比
Pretraining heavily influences the model’s knowledge capacity prior to post-training. The model gets raw intelligence during pretraining. Then, during post-training, that intelligence is made useful through behaviors like dialogue and reasoning. In a frontier lab, these two phases are such different processes that very different teams work on them.预训练在很大程度上决定了模型在训练后阶段之前的知识容量。模型在预训练期间获得原始智能。然后，在训练后阶段，通过对话和推理等行为使这种智能变得有用。在前沿实验室中，这两个阶段是截然不同的过程，由完全不同的团队负责。
A model’s factual knowledge about the French Revolution, its understanding of Python syntax, and its grasp of calculus all come from pretraining. Post-training primarily shapes which knowledge the model reaches for, how it presents that knowledge, what tone it uses, whether it declines certain requests, and whether it thinks step-by-step before answering, though targeted SFT on new domains can introduce information the model didn’t encounter in pretraining.模型关于法国大革命的事实知识、对 Python 语法的理解以及对微积分的掌握，都来自预训练。训练后阶段主要塑造模型会调用哪些知识、如何呈现这些知识、使用什么语气、是否拒绝某些请求，以及是否在回答前进行逐步思考，尽管针对新领域的定向监督微调（SFT）也可以引入模型在预训练中未曾接触过的信息。
If a model gives a wrong answer about history, the root cause is likely in pretraining data, but the practical fix might still come through post-training—for example, teaching the model to use search tools, express uncertainty, or chain-of-thought verify its own claims. But if a model gives correct information in a condescending way or refuses to help with a reasonable request or fails to use tools when it should, those are squarely post-training problems.如果模型在历史问题上给出了错误答案，根本原因很可能在于预训练数据，但实际的修复方案可能仍来自训练后阶段——例如，教导模型使用搜索工具、表达不确定性，或通过思维链验证自己的主张。但如果模型给出的信息正确，但语气傲慢，或者拒绝协助合理的请求，又或者在应该使用工具时没有使用，这些就完全是训练后阶段的问题。
Pretraining预训练
The work of pretraining is centered around cleaning and curating large-scale data, optimizing the model toward relatively clear loss signals, and working with scaling laws given bounded compute.预训练的工作核心在于清理和整理大规模数据，针对相对清晰的损失信号优化模型，以及在有限计算资源下遵循缩放定律（scaling laws）。
In pretraining, the model learns to predict the next token across a large curated dataset, typically for one or a small number of passes over the training data, though some models train for multiple epochs, especially as high-quality data becomes scarce relative to compute budgets. This is where you’ll hear how a model is fed the entire internet’s worth of data to gain intelligence, although in practice nearly all of the data (often 90% or more) may be thrown out because it’s unsuitable for training.在预训练中，模型学习预测大型整理数据集中的下一个 token，通常会对训练数据进行一次或少量几次遍历，尽管有些模型会进行多次 epoch 训练，尤其是在高质量数据相对于计算预算变得稀缺时。这就是人们常说的模型如何通过吸收整个互联网的数据来获得智能，尽管实际上几乎所有数据（通常 90% 或更多）都可能因为不适合训练而被丢弃。
Pretraining is an unsupervised process that runs at increasingly larger scales to match the size of the model. While scaling, thousands of experiments are used to understand what data mix, what architecture considerations, what compute optimizations, what hyperparameters can lead to the best results. There’s variance in each run due to stochasticity found in both software and hardware, so multiple experiments are needed to verify results. Because compute is limited and needs to be used sparingly, researchers will scale iteratively, expanding to the next, say, 10x compute budget, when they gain confidence in the right configuration. A full run isn’t possible to iterate on due to the compute cost and time it would take: The final run, often called the “god run,” can take over a month on thousands of GPUs.预训练是一个无监督过程，其规模随着模型大小的增加而不断扩大。在缩放过程中，研究人员会进行数千次实验，以了解什么样的数据组合、架构考量、计算优化和超参数能带来最佳结果。由于软件和硬件中存在的随机性，每次运行都会有差异，因此需要多次实验来验证结果。由于计算资源有限且需谨慎使用，研究人员会进行迭代式缩放，在对配置有信心后，将计算预算扩展到下一个数量级（例如 10 倍）。由于计算成本和时间昂贵，无法对完整的预训练运行进行迭代：最终的运行（通常被称为“上帝视角运行”或“god run”）可能需要在数千个 GPU 上运行超过一个月。
Pretraining progress is typically very clearly measurable, using a metric like perplexity, which measures, roughly, the model’s average uncertainty per token. Lower is better, where 1 means the model knows with absolute certainty what token comes next. Meanwhile, a perplexity of 50 means the model’s predictions are, on average, as uncertain as if it were choosing uniformly among 50 equally likely tokens—though in practice, the distribution is peaked, not uniform.预训练的进展通常非常清晰可测，使用困惑度（perplexity）等指标，该指标大致衡量了模型对每个 token 的平均不确定性。数值越低越好，1 表示模型能绝对确定下一个 token 是什么。同时，困惑度为 50 意味着模型的预测平均而言，就像在 50 个等概率 token 中进行均匀选择一样不确定——尽管实际上分布是峰值分布而非均匀分布。
Post-training训练后阶段
Rather than consuming hundreds of millions of tokens of internet data, post-training operates on far more intentional datasets for downstream tasks. These datasets include human-written demonstrations of ideal responses, human judgments about which responses from the model are better, and carefully designed functions that score the model’s outputs programmatically. They shape what “good” looks like.与消耗数亿个 token 的互联网数据不同，训练后阶段针对下游任务使用更为精心的数据集。这些数据集包括人类编写的理想响应示范、人类对模型响应优劣的判断，以及精心设计的用于程序化评分模型输出的函数。它们定义了什么是“好”。
Like pretraining, post-training can also be more effective with scaling data and compute. Specifically, massive compute budgets have been dedicated to post-training to learn reasoning capabilities (or the ability for models to “think step-by-step” to arrive at more logically sound answers), matching the scale of pretraining compute.与预训练一样，训练后阶段也可以通过扩展数据和计算来提高效果。具体而言，为了学习推理能力（即模型“逐步思考”以得出逻辑上更合理的答案的能力），已经投入了巨大的计算预算用于训练后阶段，其规模足以媲美预训练的计算量。
Post-training is messier than petraining, which has an elegant, clear optimization objective to minimize the loss over the next token prediction across a huge corpus. The goals of post-training are things like “be more helpful” or “don’t say harmful things.” Many of these objectives are inherently subjective and require human judgment, proxy models that approximate human judgment, or programmatic verifiers that can become elaborate or inefficient. The loss curves are noisier. The quality of the data and feedback matter even more.训练后阶段比预训练更混乱，预训练有一个优雅、清晰的优化目标，即最小化整个语料库中下一个 token 预测的损失。训练后阶段的目标则是“更有帮助”或“不说有害的话”。许多此类目标本质上是主观的，需要人类判断、模拟人类判断的代理模型，或者可能变得复杂或低效的程序化验证器。损失曲线更嘈杂。数据和反馈的质量在这里显得尤为重要。
The scale of post-training is also more complicated than in pretraining. Standard post-training remains relatively modest in compute: tens to hundreds of GPUs for days rather than thousands of GPUs for months needed in pretraining. This makes post-training for alignment highly amenable to rapid iteration; researchers can try something, observe results, form a hypothesis, and run again on a timescale of days.训练后阶段的规模也比预训练复杂。标准的训练后阶段在计算上相对适中：只需数十到数百个 GPU 运行几天，而不是预训练所需的数千个 GPU 运行数月。这使得用于对齐（alignment）的训练后阶段非常适合快速迭代；研究人员可以在几天的时间尺度内尝试某项改进、观察结果、形成假设并再次运行。
The picture changes dramatically when post-training is used to develop reasoning capabilities. For reasoning models, the compute dedicated to post-training can easily account for half of the overall compute of the model. The gap between a standard instruct model and a reasoning model is increasingly a gap in post-training compute, not pretraining scale. This means post-training now spans a wide spectrum from fast, cheap, highly iterable fine-tuning runs to massive RL campaigns that rival pretraining in both cost and engineering complexity.当训练后阶段用于开发推理能力时，情况会发生巨大变化。对于推理模型，投入到训练后阶段的计算量很容易占到模型总计算量的一半。标准指令模型与推理模型之间的差距，日益表现为训练后阶段计算量的差距，而非预训练规模的差距。这意味着训练后阶段现在涵盖了广泛的范围，从快速、廉价、高度可迭代的微调运行，到在成本和工程复杂性上均可与预训练媲美的大规模强化学习（RL）活动。
Why post-training matters为什么训练后阶段很重要
So why can’t we just stick with pretraining? It comes down to three main pieces: usability, safety, and capability.那么，为什么我们不能只停留在预训练阶段？这归结为三个主要方面：可用性、安全性和能力。
Usability可用性
A pretrained model is like if someone gave you a large download of Wikipedia in a single PDF. It’s a ton of knowledge that you can sift through, but there’s no way to easily understand what is going on in the data. Post-training gives the model the ability to integrate this information for you and respond to your request naturally. This extends to having longer multiturn conversations and following instructions. Without it, every user would need to be a prompt engineer. With it, anyone who can type a sentence can use the model.预训练模型就像有人给了你一份维基百科的大型 PDF 下载包。虽然知识量巨大，可以供你筛选，但很难轻易理解数据中的内容。训练后阶段赋予模型为你整合这些信息并自然地响应你请求的能力。这延伸到进行更长的多轮对话和遵循指令。没有它，每个用户都必须成为提示工程师。有了它，任何能打字的人都能使用模型。
Safety安全性
A lot of data in pretraining can be toxic, biased, misleading, or outright dangerous. Or it might not be dangerous on its own, but when a model can integrate knowledge from different fields, it can create something novel that is dangerous.预训练中的大量数据可能是毒性的、有偏见的、误导性的或完全危险的。或者数据本身并不危险，但当模型能够整合来自不同领域的知识时，它可能会创造出危险的新事物。
The model has no inherent sense of what content is good or bad. It will follow any request, based on its pretraining data. To prevent that, you can add safety guardrails to the model in post-training, to refuse harmful requests like asking the model to build a bioweapon and avoid accidentally generating toxic content such as inappropriate sexual content (even if it wasn’t in the user’s request). This is also the place to teach the model to express uncertainty, when it doesn’t know something, whether that’s “I don’t know” or “that’s beyond my knowledge cutoff” or “as a large language model, I’m limited in my knowledge so please consult a healthcare professional.”模型本身没有什么是好是坏的内在意识。基于预训练数据，它会遵循任何请求。为了防止这种情况，你可以在训练后阶段为模型添加安全护栏，以拒绝有害请求（例如要求模型制造生物武器），并避免意外生成毒性内容（例如不适当的性内容，即使这不在用户的请求中）。这也是教导模型在不知道某事时表达不确定性的地方，无论是说“我不知道”、“这超出了我的知识范围”，还是“作为大语言模型，我的知识有限，请咨询医疗专业人士”。
Making a model safe is part of a broader area in the AI research community called “alignment,”1 where the goal is to align the model with human values and preferences. Post-training is typically the main way to achieve that.使模型安全是 AI 研究社区中一个更广泛领域的一部分，称为“对齐”（alignment）1，其目标是将模型与人类的价值观和偏好相一致。训练后阶段通常是实现这一目标的主要方式。
Model companies will usually have additional safeguards beyond post-training, including lightweight models that check whether the user’s request was safe, as a second layer of protection against responding to harmful requests.模型公司通常会在训练后阶段之外采取额外的保障措施，包括使用轻量级模型来检查用户的请求是否安全，作为防止响应有害请求的第二层保护。
Capability能力
Post-training doesn’t just make a model nicer or safer; it can make the model smarter at hard tasks. The clearest example is reasoning. A pretrained model might have all the mathematical knowledge needed to solve a complex word problem, but it might jump to an incorrect answer because it’s pattern-matching from pretraining data or pattern-matching from how to answer questions (e.g., with succinct immediate answers).训练后阶段不仅使模型更友好或更安全；它还能使模型在处理困难任务时更聪明。最明显的例子是推理。预训练模型可能拥有解决复杂应用题所需的所有数学知识，但它可能会直接给出一个错误的答案，因为它是在根据预训练数据进行模式匹配，或者根据回答问题的方式（例如简洁的即时回答）进行模式匹配。
It turns out that making the model output more tokens before giving an answer (or “think longer”), results in better answers. This process is known as reasoning, and post-training can teach the model to reason more effectively. A more capable pretrained model is a more dangerous model if it’s not properly aligned. A more intelligent model is a less useful model to humans if it can’t communicate clearly. And, every point of improvement in a reasoning benchmark now maps to real revenue for companies deploying these models.事实证明，让模型在给出答案之前输出更多的 token（或“思考更久”），会产生更好的答案。这个过程被称为推理，训练后阶段可以教导模型更有效地进行推理。如果未经过适当对齐，更强大的预训练模型就是一个更危险的模型。如果一个更智能的模型无法与人类进行清晰的沟通，那么它对人类的用处就会降低。而且，推理基准测试中每一点的提升，现在都直接转化为部署这些模型的公司的实际收入。
Superhuman performance超越人类的表现
Can post-training push models beyond human-level performance? Yes, in specific domains.训练后阶段能将模型推向超越人类水平的表现吗？可以，在特定领域。
In competitive programming, top reasoning models can now solve problems at a level that exceeds the vast majority of human competitive programmers. In math, models have achieved scores on Math Olympiad-level competitions that would place them among the top competitors in the world. In certain scientific domains, models have generated novel hypotheses and solutions that human experts found valuable.在竞技编程中，顶尖的推理模型现在可以解决超过绝大多数人类竞技程序员水平的问题。在数学领域，模型在数学奥林匹克级别的竞赛中取得了足以使其跻身世界顶尖选手的成绩。在某些科学领域，模型生成的创新假设和解决方案已被人类专家认为具有价值。
This might seem paradoxical. If the model’s knowledge comes from human-generated data (in pretraining), and its behavior is shaped by human feedback (in post-training), how can it exceed human performance?这看起来可能很矛盾。如果模型的知识来自人类生成的数据（在预训练中），且其行为由人类反馈塑造（在训练后阶段），它怎么能超越人类的表现？
Two things make this possible. First, integration across domains. Research is about combining or mixing fields. Imagine mixing every possible field. The pretraining data aggregates knowledge from millions of sources, and no single human has read all of it. Second, post-training, particularly RL with reasoning, teaches the model to explore many approaches to a problem, far more than a human would try in a single sitting. A human might try one or two approaches to a hard math problem.两件事使这成为可能。首先，跨领域整合。研究的本质在于结合或混合不同领域。想象一下混合所有可能的领域。预训练数据汇集了来自数百万个来源的知识，没有任何一个人阅读过所有内容。其次，训练后阶段，特别是结合推理的强化学习，教导模型探索解决问题的多种途径，这远超人类在单次尝试中所能做到的。人类在面对一道难题时可能只会尝试一两种方法。
This means post-training is not just about making models mimic human behavior. It’s about pushing beyond it. This is especially possible to scale with verifier-based RL. In those scenarios, you can expect models to achieve superhuman performance in an expanding set of domains. And that starts with verifiers that are very well-defined, easy to access, efficient, and cheap relative to the ROI of the model learning it. The limitation is no longer the model’s intelligence, but our ability to specify what “good” means through reward signals.这意味着训练后阶段不仅仅是为了让模型模仿人类行为。它旨在超越人类行为。通过基于验证器的强化学习进行扩展，这尤其可行。在这些场景中，你可以预期模型在不断扩大的领域中实现超越人类的表现。而这一切始于定义明确、易于获取、高效且相对于模型学习收益而言成本低廉的验证器。限制因素不再是模型的智能，而是我们通过奖励信号指定“好”意味着什么的能力。
Footnote脚注
- See Richard Ngo, Lawrence Chan, and Sören Mindermann’s “The Alignment Problem from a Deep Learning Perspective” and Iason Gabriel’s “Artificial Intelligence, Values, and Alignment.” ↩︎参见 Richard Ngo、Lawrence Chan 和 Sören Mindermann 的《从深度学习视角看对齐问题》（The Alignment Problem from a Deep Learning Perspective）以及 Iason Gabriel 的《人工智能、价值观与对齐》（Artificial Intelligence, Values, and Alignment）。 ↩︎
