Inside the Transformer: The Life of a TokenTransformer内部：一个Token的生命历程

In this post, I'll do a deep dive into the internals of a modern dense transformer [1]. I'll focus exclusively on the forward pass on a single GPU, as if we were about to perform a training step, while ignoring the backward pass and distributed systems details (in practice, large Transformers are sharded across multiple devices during both training and inference).在这篇文章中，我将深入剖析现代密集Transformer[1]的内部机制。我将专注于单GPU上的前向传播，就像我们即将执行一个训练步骤一样，同时忽略反向传播和分布式系统的细节（实际上，大型Transformer在训练和推理时都会跨多个设备分片）。

As a running example, I'll use the exact architecture of Rnj 1.5 - a model I worked on with my team at Ashish Vaswani's AI Lab (Essential AI Labs).作为运行示例，我将使用Rnj 1.5的确切架构——这是我和团队在Ashish Vaswani的AI实验室（Essential AI Labs）共同开发的模型。

We announced it this week, with weights released on Hugging Face.我们本周发布了它，权重已在Hugging Face上公开。

It's a long-context follow-up to Rnj 1.0 [2] that extends the context window from 32k to 160k, scoring 79% on RULER on a 128k context window. This release also offers stronger coding abilities on a wider range of harnesses. See our model card for more details.这是Rnj 1.0[2]的长上下文后续版本，将上下文窗口从32k扩展到160k，在128k上下文窗口的RULER上得分79%。此版本还在更广泛的测试集上提供了更强的编码能力。更多详情请参见我们的模型卡。

This post is structured into seven parts:本文分为七个部分：

1. Transformer forward pass: high-level flow of a tokenTransformer前向传播：Token的高级流程
2. RMSNorm: the normalization layerRMSNorm：归一化层
3. GeGLU MLP: GELU-gated feedforward blockGeGLU MLP：GELU门控前馈块
4. MHA: multi-head self-attentionMHA：多头自注意力
5. YaRN: positional embeddings for long contextYaRN：长上下文的位置嵌入
6. Core Attention: global + block local核心注意力：全局+块局部
7. Transformer math: FLOPs/token, cluster sizing, and moreTransformer数学：FLOPs/Token、集群规模等

In a follow-up post, I'll dive into conditional computation, focusing on sparse transformers (MoE).在后续文章中，我将深入探讨条件计算，重点关注稀疏Transformer（MoE）。

Transformer forward passTransformer前向传播

As a running example, assume we sample 2 "documents" from a dataset, with:作为运行示例，假设我们从数据集中采样2个“文档”，其中：

• batch size = 1批量大小 = 1
• sequence length = 16序列长度 = 16
• document packing enabled启用文档打包

We'll trace how a token flows through the transformer and, along the way, unpack each component.我们将追踪一个Token如何流经Transformer，并在此过程中逐一解析每个组件。

Let's start. Spend some time analyzing the following:让我们开始吧。花些时间分析以下内容：

Figure 1: Tokenization stage图1：分词阶段

We tokenize the documents into sequences of integers, then pack the two documents into a single sequence.我们将文档分词为整数序列，然后将两个文档打包成一个序列。

Alongside the tokens, we construct two supporting structures:除了Token之外，我们还构建了两个辅助结构：

• inputs positions - used by the positional embedding module (YaRN)输入位置——由位置嵌入模块（YaRN）使用
• segmentation mask - used in attention for masking分段掩码——用于注意力中的掩码操作

This is the preprocessing stage.这是预处理阶段。

Next, we use the input tokens to index into the embedding table.接下来，我们使用输入Token索引嵌入表。

You can think of the embedding table as the vocabulary of the LLM.你可以将嵌入表视为LLM的词汇表。

This indexing operation converts our sequence of integers into a sequence of 16 4096-dim bf16 vectors:这个索引操作将我们的整数序列转换为16个4096维的bf16向量序列：

Figure 2: Embedding stage图2：嵌入阶段

Now we're ready to enter the first transformer layer.现在我们准备进入第一个Transformer层。

Note that all transformer layers have (almost) the same structure, so I'll explain just one. In practice, we pass through 32 such layers - you can think of it as a for loop, but in Rnj 1.5 each layer has its own learnable weights.请注意，所有Transformer层具有（几乎）相同的结构，因此我将只解释一层。实际上，我们经过32个这样的层——你可以将其视为一个for循环，但在Rnj 1.5中，每一层都有自己的可学习权重。

Let's do a forward pass through transformer blocks. Analyze the following carefully:让我们通过Transformer块进行前向传播。仔细分析以下内容：

Figure 3: Forward pass through transformer blocks图3：通过Transformer块的前向传播

At a high level, the block consists of four RMSNorm submodules, an MLP, an attention module, two residual connections, and two sum operations. The residual connections simply carry forward copies of vectors from earlier in the block.在高层，该块由四个RMSNorm子模块、一个MLP、一个注意力模块、两个残差连接和两个求和操作组成。残差连接只是将块中较早的向量副本向前传递。

Importantly, all submodules operate on individual vectors, except for attention.重要的是，除了注意力之外，所有子模块都作用于单个向量。

Once the vectors exit the final transformer block, they're projected into a 128,256-dimensional space via a matrix multiplication. This produces logits, which are converted into a probability distribution via softmax. We sample from it during inference and use it in the cross-entropy loss during training.一旦向量离开最后一个Transformer块，它们通过矩阵乘法投影到128,256维空间。这产生logits，再通过softmax转换为概率分布。我们在推理时从中采样，并在训练时用于交叉熵损失。

Figure 4: 图4：

Next, let's dive into the individual sublayers. I'll go in reverse order this time, which conveniently takes us from the simplest to the most complex:接下来，让我们深入各个子层。这次我将按相反顺序进行，这恰好从最简单到最复杂：

1. RMSNorm (Root Mean Square Layer Normalization)RMSNorm（均方根层归一化）
2. GeGLU MLP (Multi-Layer Perceptron)GeGLU MLP（多层感知器）
3. Attention (Scaled Dot-Product Attention)注意力（缩放点积注意力）

RMSNorm (Root Mean Square Layer Normalization)

RMSNorm [7] is a normalization technique used to stabilize the training of deep neural networks.RMSNorm[7]是一种用于稳定深度神经网络训练的归一化技术。

As mentioned earlier, RMSNorm operates on individual vectors, so we'll focus on a single bf16 4096-dim vector (all others are processed in parallel in the same way). The output has the same shape and dtype:如前所述，RMSNorm作用于单个向量，因此我们将专注于一个bf16 4096维向量（所有其他向量以相同方式并行处理）。输出具有相同的形状和数据类型：

Figure 5: RMSNorm图5：RMSNorm

GeGLU MLP (Multi-Layer Perceptron)

The MLP is a simple, pointwise feedforward neural network that is used to learn the non-linear relationships between the input and output vectors.MLP是一个简单的逐点前馈神经网络，用于学习输入和输出向量之间的非线性关系。

Our variant is GeGLU (GELU-gated linear unit [8]), where the gating mechanism uses GELU and takes the form W2 @ GELU(W0@X)*(W1@X):我们的变体是GeGLU（GELU门控线性单元[8]），其中门控机制使用GELU，形式为W2 @ GELU(W0@X)*(W1@X)：

Figure 6: GeGLU MLP图6：GeGLU MLP

With ReLU, “gate” is more literal because the gating vector is nonnegative, so it only suppresses or scales features. With GELU, gating values can be negative, so the gate can also invert a feature's sign, which makes “gate” a looser historical term.对于ReLU，“门”更字面化，因为门控向量是非负的，所以它只抑制或缩放特征。对于GELU，门控值可以为负，因此门也可以反转特征的符号，这使得“门”成为一个更宽松的历史术语。

MHA (Multi-Head Attention)MHA（多头注意力）

MHA is a self-attention mechanism used to model relationships between different tokens in a sequence. We use a special variant of MHA, called GQA, short for group query attention (the number of K/V heads is reduced compared to Q heads, hence multiple queries (group) attend to the same key).MHA是一种自注意力机制，用于建模序列中不同Token之间的关系。我们使用MHA的一个特殊变体，称为GQA，即分组查询注意力（K/V头的数量相对于Q头减少，因此多个查询（组）关注同一个键）。

First, I'll give the high level overview - then we'll dig into the two most interesting components: YaRN and core attention.首先，我将给出高层概述——然后我们将深入两个最有趣的组件：YaRN和核心注意力。

We start by mapping each vector independently into query, key, and value vectors. We then reshape them, normalize queries and keys, and apply YaRN (which injects positional information through rotation). Next comes core attention, which mixes information across positions. Finally, we apply a linear projection to produce the output.我们首先将每个向量独立映射为查询、键和值向量。然后我们重塑它们，对查询和键进行归一化，并应用YaRN（通过旋转注入位置信息）。接下来是核心注意力，它跨位置混合信息。最后，我们应用线性投影产生输出。

Figure 7: MHA - multi head attention图7：MHA - 多头注意力

Let's now focus on YaRN (Yet another RoPE extensioN).现在让我们聚焦于YaRN（Yet another RoPE extensioN）。

YaRNYaRN

YaRN [9] modifies RoPE [10] (rotary position embeddings) in a clever way, that leads to better extrapolation to longer context lengths.YaRN[9]以巧妙的方式修改了RoPE[10]（旋转位置嵌入），从而更好地外推到更长的上下文长度。

But why do we need positional embeddings in the first place?但为什么我们首先需要位置嵌入呢？

Figure 8: The WHY behind positional embeddings图8：位置嵌入的“为什么”

Now that we understand the why, let's see how does RoPE work:现在我们已经理解了原因，让我们看看RoPE是如何工作的：

Figure 9: YaRN frequency table图9：YaRN频率表

Here is a visualization showing how different YaRN frequencies behave. Notice that our slowest frequency does 1 cycle every 1.088M positions!这是一个可视化，展示了不同YaRN频率的行为。注意，我们最慢的频率每1.088M个位置才完成一个周期！

Figure 10: YaRN frequencies图10：YaRN频率

With this we're ready to see how positional embeddings are injected during the forward pass:至此，我们准备了解位置嵌入在前向传播过程中是如何注入的：

Figure 11: YaRN - forward pass图11：YaRN - 前向传播

That wraps up the YaRN forward pass.这就结束了YaRN的前向传播。

Now that we understand the mechanics of it you might still be wondering: how does YaRN encode relative positional information via pairwise coordinate rotations of the query and key vectors, followed by a dot product?现在我们已经理解了其机制，你可能仍然想知道：YaRN如何通过查询和键向量的成对坐标旋转，然后进行点积，来编码相对位置信息？

Figure 12: How does RoPE encode relative positional information?图12：RoPE如何编码相对位置信息？

And that's all there is to RoPE/YaRN! :)这就是RoPE/YaRN的全部内容！:)

Core Attention核心注意力

Finally, let’s analyze the core attention mechanism. In practice, we use FlashAttention, which deserves a separate blog post (I actually wrote one back in ’23, check it out [11]). Here, I’ll walk through vanilla attention.最后，让我们分析核心注意力机制。实际上，我们使用FlashAttention，这值得单独一篇博客文章（我实际上在23年写过一篇，可以看看[11]）。在这里，我将讲解标准注意力。

Core attention is the mechanism that models relationships between tokens in a sequence. Take some time to analyze this:核心注意力是建模序列中Token之间关系的机制。花些时间分析这个：

Figure 13: Computing (seqlen, seqlen) matrix of attention scores图13：计算注意力分数的(seqlen, seqlen)矩阵

Now if we just stopped here we'd have a situation where:如果我们停在这里，会出现以下情况：

1. tokens from document 1 could attend to tokens from document 2 (and vice versa)文档1中的Token可以关注文档2中的Token（反之亦然）
2. token i could attend to token i+1 (future token) which breaks the causalityToken i可以关注Token i+1（未来Token），这破坏了因果性

In order to prevent this we need to introduce masking!为了防止这种情况，我们需要引入掩码！

Figure 14: Attention masking & value vector aggregation图14：注意力掩码与值向量聚合

Now imagine our sequence length is 32,768 instead of 16. For simplicity, assume a single document with no padding. What would the mask look like?现在想象我们的序列长度是32,768而不是16。为简单起见，假设单个文档且无填充。掩码会是什么样子？

Figure 15: Hybrid attention: block local + global图15：混合注意力：块局部 + 全局

Here’s another way to visualize the layout, focusing on tokens at positions 9,000 and 10,000:这是另一种可视化布局的方式，聚焦于位置9,000和10,000处的Token：

Figure 16: Hybrid attention layout图16：混合注意力布局

You can see that in most layers (block-local), these two tokens cannot attend to positions beyond 4,096. In the remaining eight layers (global), they can attend all the way back to position 0.你可以看到，在大多数层（块局部）中，这两个Token无法关注超过4,096的位置。在剩余的八层（全局）中，它们可以一直关注到位置0。

Transformer mathTransformer数学

Finally, I want to briefly touch on the KV cache, as it’s an extremely important concept for understanding inference. So far, we’ve looked at the forward pass during training.最后，我想简要谈谈KV缓存，因为它是理解推理的极其重要的概念。到目前为止，我们一直在看训练期间的前向传播。

During inference, transformers are autoregressive - we generate one token at a time. It would be extremely inefficient to recompute the keys and values for all previous tokens at every step. Fortunately, there’s no need: in a causal transformer, they remain unchanged. Instead, we compute them once and store them in a cache.在推理期间，Transformer是自回归的——我们一次生成一个Token。如果每一步都重新计算所有先前Token的键和值，效率会极低。幸运的是，没有必要：在因果Transformer中，它们保持不变。相反，我们只计算一次并将它们存储在缓存中。

Let’s go through the basic KV cache storage requirements:让我们了解基本的KV缓存存储需求：

Figure 17: KV cache calculation图17：KV缓存计算

Let's now also calculate how many learnable parameters Rnj 1.5 has. We just need to go through the architecture and account for all the learnable weights.现在让我们也计算一下Rnj 1.5有多少可学习参数。我们只需要遍历架构并统计所有可学习权重。

Figure 18: Number of learnable parameters calculation图18：可学习参数数量计算

For quick mental math notice that you only need to account for 3 matrices inside MLP and 4 matrices inside attention and you can ignore everything else.为了快速心算，注意你只需要考虑MLP中的3个矩阵和注意力中的4个矩阵，其他都可以忽略。

Let's now calculate how much compute (FLOPs) we need per token. This is extremely valuable when it comes to planning cluster sizing - more on that after this section.现在让我们计算每个Token需要多少计算量（FLOPs）。这在规划集群规模时非常有价值——更多内容将在本节之后介绍。

Figure 19: FLOPs/token calculation图19：FLOPs/Token计算

It's worth remembering the 6N formula. It's also worth remembering the setting under which it holds (i.e. seqlen << inner model dimension).值得记住6N公式。也值得记住它成立的条件（即seqlen远小于内部模型维度）。

Finally let's see how we can use the above formula for cluster sizing:最后，让我们看看如何将上述公式用于集群规模规划：

Figure 20: Cluster sizing calculation图20：集群规模计算

You now go to Masayoshi Son and ask for a $1B seed round.你现在可以去找孙正义，要求10亿美元的种子轮融资。

Figure 21: Profit图21：利润

Epilogue后记

We've seen how a single token flows through the transformer and how all the subcomponents work together.我们已经看到了单个Token如何流经Transformer，以及所有子组件如何协同工作。

We've explored YaRN and attention in depth, and derived some of the most important transformer formulas.我们深入探讨了YaRN和注意力，并推导出了一些最重要的Transformer公式。

In upcoming posts, I'll dig deeper into MoE, Muon (the optimizer [12]), and several architectural innovations such as MLA (DeepSeek), MTP (multi-token prediction), and DSA (sparse attention [13]).在接下来的文章中，我将更深入地探讨MoE、Muon（优化器[12]）以及若干架构创新，如MLA（DeepSeek）、MTP（多Token预测）和DSA（稀疏注意力[13]）。

References参考文献

1. "Attention Is All You Need", https://arxiv.org/abs/1706.03762"Attention Is All You Need", https://arxiv.org/abs/1706.03762
2. RNJ 1.0, https://essential.ai/research/rnj-1RNJ 1.0, https://essential.ai/research/rnj-1
3. "Efficient Training of Language Models to Fill in the Middle", https://arxiv.org/abs/2207.14255"Efficient Training of Language Models to Fill in the Middle", https://arxiv.org/abs/2207.14255
4. "Attention Residual Learning", https://arxiv.org/abs/2603.15031"Attention Residual Learning", https://arxiv.org/abs/2603.15031
5. "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model", https://arxiv.org/abs/2405.04434"DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model", https://arxiv.org/abs/2405.04434
6. "Kimi Linear: An Expressive, Efficient Attention Architecture", https://arxiv.org/abs/2510.26692"Kimi Linear: An Expressive, Efficient Attention Architecture", https://arxiv.org/abs/2510.26692
7. "Root Mean Square Layer Normalization", https://arxiv.org/abs/1910.07467"Root Mean Square Layer Normalization", https://arxiv.org/abs/1910.07467
8. "GLU Variants Improve Transformer", https://arxiv.org/abs/2002.05202"GLU Variants Improve Transformer", https://arxiv.org/abs/2002.05202
9. "YaRN: Efficient Context Window Extension of Large Language Models", https://arxiv.org/abs/2309.00071"YaRN: Efficient Context Window Extension of Large Language Models", https://arxiv.org/abs/2309.00071
10. "RoFormer: Enhanced Transformer with Rotary Position Embedding", https://arxiv.org/abs/2104.09864"RoFormer: Enhanced Transformer with Rotary Position Embedding", https://arxiv.org/abs/2104.09864
11. "Eli5 Flash Attention", https://gordicaleksa.medium.com/eli5-flash-attention-5c44017022ad"Eli5 Flash Attention", https://gordicaleksa.medium.com/eli5-flash-attention-5c44017022ad
12. Muon, https://kellerjordan.github.io/posts/muon/Muon, https://kellerjordan.github.io/posts/muon/
13. "Dissecting Sparsity in Large Language Models: Intrinsic Data-Aware Sparse Attention", https://arxiv.org/abs/2512.02556"Dissecting Sparsity in Large Language Models: Intrinsic Data-Aware Sparse Attention", https://arxiv.org/abs/2512.02556