Before Q, K, and V: Reconstructing the Transformer在 Q、K 和 V 之前：重构 Transformer

Do we really need keys, queries, values, and dot product attention?我们真的需要键（keys）、查询（queries）、值（values）和点积注意力机制吗？

“You need keys and queries for tokens to talk to each other,” says the popular Internet analogy. But why? There’s a lot of great analogies for how they work, but a lot less material about why we truly need them. Are there any alternatives or are these abstract concepts inevitable?互联网上流行的类比说：“你需要键和查询，好让标记（tokens）之间能够对话。”但为什么呢？关于它们如何工作，有很多精彩的类比，但关于我们为什么真正需要它们的内容却少得多。是否存在替代方案，还是这些抽象概念是不可避免的？

This might seem like a silly question given the utter success of the Transformer architecture in 2026. But if history is any indication, Transformers will eventually be replaced by something better. The more we understand why they work, the faster we can move beyond them.考虑到 Transformer 架构在 2026 年取得的巨大成功，这似乎是个愚蠢的问题。但历史经验表明，Transformer 终将被更好的东西所取代。我们越了解它们为何有效，就能越快地超越它们。

Incredibly, we can show that the general shape of the Transformer is hard to avoid! Start with a few key design pressures, and the alphabet soup of matrices begins to look much less arbitrary. Without relying on any “token asks a question” analogies, we’ll see that queries arise from a symmetry problem, while values and attention heads appear when we replace an unwieldy dynamic weight matrix with a small set of reusable transformations.令人难以置信的是，我们可以证明 Transformer 的总体形态是难以避免的！从几个关键的设计压力出发，那些令人眼花缭乱的矩阵组合就开始变得不再那么随意了。无需依赖任何“标记提出问题”的类比，我们将看到查询产生于对称性问题，而当我们将笨重的动态权重矩阵替换为一组小型可重用变换时，值和注意力头便应运而生。

And as we reconcile our toy model with the original (autoregressive) Transformer, we’ll end with one last fascinating connection: the MLP—the often-overlooked feedforward block—can be viewed as a key-value store of its own.当我们把这个玩具模型与原始的（自回归）Transformer 进行调和时，我们将以一个迷人的结论结束：MLP（多层感知机）——那个常被忽视的前馈模块——本身就可以被视为一个键值存储器。

Ready to invent the Transformer for ourselves? First, we need to travel back to 2014 and ask why recurrent neural networks aren’t enough.准备好亲自发明 Transformer 了吗？首先，我们需要回到 2014 年，问问为什么循环神经网络（RNN）还不够好。

Why Fixed Memory Fails为什么固定内存会失效

There are many issues with standard recurrent neural networks (RNNs.) Some issues, like the “gradient vanishing” issue, were solved by the extremely popular LSTM (Long Short-Term Memory network) by Hochreiter and Schmidhuber [1] which is a more advanced flavor of RNN. But one core issue remains for all flavors: RNNs “squish” past inputs together into fixed memory.标准的循环神经网络（RNN）存在许多问题。一些问题，比如“梯度消失”，通过 Hochreiter 和 Schmidhuber [1] 提出的极为流行的 LSTM（长短期记忆网络）得到了解决，这是一种更高级的 RNN。但所有变体都面临一个核心问题：RNN 将过去的输入“挤压”进固定的内存中。

To see why, recall that an RNN has a memory state that is “written to” by the current input and the previous state. This connection between states in time is where the “recurrence” comes from. Consider the following two layer RNN unrolled in time:要明白原因，请回想一下，RNN 有一个内存状态，由当前输入和前一个状态“写入”。这种时间上的状态连接正是“循环”的由来。考虑以下按时间展开的两层 RNN：

Here the inputs are green, the outputs are blue, and the two intermediate layers of the network that compute standard neural network activations are gray. All of the components are vectors, and the directed arrows are matrix weights that multiply their inputs. To see the network at one point in time, simply look at a single vertical slice of the diagram; in that slice, there are two fixed gray units that collectively represent the “memory” of the network at that point in time.图中输入为绿色，输出为蓝色，计算标准神经网络激活值的两个中间层为灰色。所有组件都是向量，有向箭头是乘以输入的矩阵权重。要观察某一时刻的网络，只需看图中的垂直切片；在该切片中，有两个固定的灰色单元，它们共同代表了网络在该时刻的“内存”。

Let’s use a crude analogy to show why this “fixed memory” is an issue. Say you encode the sentence “I have five dollars” into the RNN’s memory, which we’ll represent with the grey rectangle below:让我们用一个粗略的类比来说明为什么这种“固定内存”是个问题。假设你将句子“I have five dollars”编码到 RNN 的内存中，我们用下方的灰色矩形表示：

Now let’s extend that sentence to “I have five dollars and forty cents in my pocket”:现在让我们把句子扩展为“I have five dollars and forty cents in my pocket”：

Yikes. After adding more information into finite memory, there is greater “competition” for real estate and some memory is overwritten. This is disastrous when you need to recall specific facts or follow very specific instructions.糟糕。在有限的内存中加入更多信息后，内存空间的“竞争”加剧，部分内存被覆盖。当你需要回忆特定事实或遵循非常具体的指令时，这是灾难性的。

You might be thinking, “Why not use dynamic memory that grows with the input sequence?”你可能会想：“为什么不使用随输入序列增长的动态内存呢？”

Great intuition! This is exactly what Bahdanau et al. [2] tried in 2014 when they popularized the idea of “attention” within the RNN (yes, RNNs used attention before Transformers did!)直觉很棒！这正是 Bahdanau 等人 [2] 在 2014 年尝试的方法，当时他们推广了 RNN 内的“注意力”概念（没错，RNN 在 Transformer 之前就使用过注意力机制！）

The idea, at a high level, is to keep the entire previous history of the RNN states as our “expanding memory”, which naturally has the property of growing with the input sequence. Consider the following updated diagram (this is not the exact architecture in the Bahdanau paper, but an analogy):从宏观上看，其思路是将 RNN 状态的整个历史记录作为我们的“扩展内存”，它天然具有随输入序列增长的特性。考虑以下更新后的图示（这不是 Bahdanau 论文中的精确架构，而是一个类比）：

Note the new connections in red; these connect the gray states in the RNN layers to every previous input in time. Before adding these connections, each state was forced to compress all of the historical information and pass it along in the left-to-right “recurrent” connections. But this compression is no longer needed since every state now has direct access to the entire history of inputs, aka our growing memory!注意红色的新连接；它们将 RNN 层中的灰色状态连接到时间上的每一个历史输入。在添加这些连接之前，每个状态被迫压缩所有历史信息，并通过从左到右的“循环”连接传递。但这种压缩不再需要了，因为每个状态现在都可以直接访问整个输入历史，即我们不断增长的内存！

There is one big challenge that remains: training speed. At training time, we have each input sequence available up front. But generating the final Nth output requires N sequential steps in time given the recurrent dependencies; with long input sequences, we have long sequential computations that cannot be parallelized by GPUs.但还存在一个巨大的挑战：训练速度。在训练时，我们可以预先获得每个输入序列。但生成最终的第 N 个输出需要 N 个按时间顺序的步骤，因为存在循环依赖；对于长输入序列，我们有无法通过 GPU 并行化的长序列计算。

The idea in the landmark paper by Vaswani et al. [3] is this: What if we can remove those recurrent left-to-right connections? What if the red connections are all you need? See the following diagram that represents this idea at a high level:Vaswani 等人 [3] 的里程碑式论文中的想法是：如果我们能移除那些从左到右的循环连接会怎样？如果红色的连接就是你所需要的一切呢？请看下方的示意图，它从宏观上表达了这个想法：

Let’s stack the recurrent and non-recurrent architectures side by side, and compare their total number of compute steps by putting ordered labels for each step:让我们将循环和非循环架构并排对比，通过为每一步标记顺序标签，比较它们的总计算步骤：

Note how the non-recurrent model needed only 2 compute steps, since it can compute each layer entirely in parallel once the previous layer is computed. On the other hand, the recurrent model needed 5 compute steps due to the recurrent dependencies within each layer. As the sequence gets longer, the non-recurrent model would stay at 2 steps while the recurrent model’s steps would grow forever with the sequence. It’s not looking good for recurrence!注意非循环模型仅需 2 个计算步骤，因为一旦计算出前一层，它就可以完全并行计算每一层。另一方面，由于每层内的循环依赖，循环模型需要 5 个计算步骤。随着序列变长，非循环模型的步骤保持在 2 步，而循环模型的步骤会随序列无限增长。这对循环架构来说不是好消息！

Let’s pivot to the non-recurrent model on the right. Now, we run into our next challenge: How on Earth should we pick the weights for these red connections?让我们转向右侧的非循环模型。现在，我们遇到了下一个挑战：我们到底该如何为这些红色连接选择权重？

Transformers and Dynamic WeightsTransformer 与动态权重

If you look at the diagram of our non-recurrent network, it looks just like an ordinary neural network with two layers and four units per layer. With such a network, we might ask: Why not learn fixed weights like we do with any other network?如果你看我们非循环网络的图，它看起来就像一个普通的神经网络，有两层，每层四个单元。对于这样的网络，我们可能会问：为什么不像其他网络那样学习固定权重呢？

But unlike this fixed diagram, sequences are not fixed during training or prediction. The network could encounter an input sequence of size 2 or it could encounter an input sequence of size 2000, and our diagram could grow indefinitely to the right with ever more units and weights. So how do we set those incoming new weights?但与这个固定图表不同，序列在训练或预测过程中是不固定的。网络可能遇到长度为 2 的输入序列，也可能遇到长度为 2000 的输入序列，我们的图表可能会随着更多的单元和权重向右无限增长。那么，我们如何设置那些传入的新权重呢？

What we need is a function to generate new weights on the fly, with parameters that we can set during training. We need dynamic weights!我们需要一个函数来即时生成新权重，并带有可以在训练期间设置的参数。我们需要动态权重！

To identify a good function, let’s zoom into one particular unit with a length 3 sequence:为了确定一个好的函数，让我们放大一个特定单元，假设序列长度为 3：

First, an important note on a new diagram addition: To make the diagram complete, we needed to add skip connections. Through these, the inputs x1, x2, and x3 are added back to the outputs of O1, O2, and O3 respectively (and this process repeats again at the next layer when O1, O2, and O3 become inputs themselves.) These “skips” free intermediate transformations from having to preserve the inputs and let them focus on the much easier task of additively adjusting those inputs. Skip connections are a critical performance optimization that were pioneered in the famous ResNet architecture by He et al. [4].首先，关于新图示添加的一个重要说明：为了使图示完整，我们需要添加残差连接（skip connections）。通过这些连接，输入 x1、x2 和 x3 分别被加回到 O1、O2 和 O3 的输出中（当 O1、O2 和 O3 本身成为下一层的输入时，这个过程会再次重复）。这些“残差”使中间变换无需保留输入，从而专注于更容易的任务——对这些输入进行加法调整。残差连接是 He 等人 [4] 在著名的 ResNet 架构中开创的关键性能优化。

Now back to our function to generate weights dynamically. First, we need to decide what the weights should be a function of.现在回到我们动态生成权重的函数。首先，我们需要决定权重应该是什么的函数。

We could start by defining a given weight as a function of the output of the weight’s source unit as well as the position of that source unit in the left-right sequence. To make things simple, I’ll combine “input” and “position” together and just say “input” going forward (this merging can be implemented by encoding position into the input directly; to see how this might be done, read about sinusoidal encodings in the original Transformer architecture.) Then, our diagram looks something like this:我们可以先将给定的权重定义为权重源单元输出及其在序列中位置的函数。为了简化起见，我将“输入”和“位置”合并，后续直接称为“输入”（这种合并可以通过将位置直接编码到输入中来实现；要了解如何实现，请阅读原始 Transformer 架构中的正弦位置编码）。那么，我们的图表看起来就像这样：

There’s one obvious downside here: If x1 needs to be “important” to O3 via a “large” weight, then it is forced to also be important to O2 and O1 since they all share the same value for their x1 weights. This means that nearby blue units within a layer will compute very similar things, defeating the flexibility of this architecture to model unique concepts. To fix this, we need to break the “symmetry” and make each of the weights of O1, O2, and O3 unique from one another.这里有一个明显的缺点：如果 x1 需要通过一个“大”权重对 O3 变得“重要”，那么它也被迫对 O2 和 O1 变得重要，因为它们共享相同的 x1 权重值。这意味着同一层内相邻的蓝色单元将计算出非常相似的内容，从而削弱了该架构建模独特概念的灵活性。为了解决这个问题，我们需要打破“对称性”，使 O1、O2 和 O3 的每个权重彼此唯一。

We could break the symmetry by making a given weight a function of both the source unit’s output and the end unit’s output; however, using the end unit’s output directly is circular since we first need the weight to compute the end output.我们可以通过使给定权重成为源单元输出和目标单元输出的函数来打破对称性；然而，直接使用目标单元的输出是循环的，因为我们需要权重来计算目标输出。

To get around this circularity, note that each unit’s unique purpose is to modify its input stream; for example, O3 is uniquely responsible for modifying x3 via the skip connection. Since x3 and O3 are uniquely intertwined, x3 is a natural candidate for the second, symmetry-breaking argument to generate O3’s weights!为了绕过这种循环性，请注意每个单元的独特目的是修改其输入流；例如，O3 通过残差连接唯一负责修改 x3。由于 x3 和 O3 紧密交织，x3 是生成 O3 权重的第二个打破对称性的参数的自然候选者！

When we update the diagram it is clear that all weights are now unique:当我们更新图表时，很明显现在所有的权重都是唯一的：

If you have a good eye, you might start to spot the “key” and “query” of the Transformer architecture already! But if not, no worries; we’ll build up to that more formally.如果你眼光敏锐，可能已经开始发现 Transformer 架构中的“键”和“查询”了！如果没有，别担心；我们将更正式地构建它。

One last call out: We could also break symmetry by making the weights zero for every input except the one that a unit modifies, but then you get the following diagram that shows this is effectively a state-less network:最后提一点：我们也可以通过将除单元修改的输入之外的所有输入的权重设为零来打破对称性，但这样你会得到下图，这表明它实际上是一个无状态网络：

In other words, symmetry breaking is a necessary but not sufficient condition. We also need some non-zero interactivity between units and other units from different time steps.换句话说，打破对称性是必要条件，但不是充分条件。我们还需要单元之间以及来自不同时间步的其他单元之间存在某种非零的交互。

Keys, Queries, and Values Emerge Naturally键、查询和值的自然涌现

Now that we’ve decided to generate each weight via a function of two arguments, we need to decide what this function actually looks like.既然我们已经决定通过两个参数的函数来生成每个权重，我们需要决定这个函数到底是什么样子。

Remember that each weight is a matrix that multiplies input vectors into output vectors, so our function actually needs to be matrix-valued. But that introduces a new problem: How do we make these matrices dynamic without blowing up the number of parameters in our model?记住，每个权重都是一个将输入向量乘以输出向量的矩阵，所以我们的函数实际上需要是矩阵值的。但这引入了一个新问题：如何在不使模型参数数量爆炸的情况下使这些矩阵动态化？

To see why parameter explosion is real, let’s write our function in matrix form, using the weight between x1 and O3 as an example:为了明白参数爆炸为何真实存在，让我们以 x1 和 O3 之间的权重为例，用矩阵形式写出我们的函数：

Here, each position (i, j) of the (d x d) matrix has its own function. If we parametrize each function separately, that’s d-squared separate sets of parameters that need to be fit. When the dimension d is in the range of hundreds, that’s >100K sets of parameters that have to be learned. Hard pass!在这里，(d x d) 矩阵的每个位置 (i, j) 都有自己的函数。如果我们分别参数化每个函数，那就是 d 平方个需要拟合的独立参数集。当维度 d 在数百量级时，就有超过 10 万个需要学习的参数集。绝对不行！

We could vastly reduce that number by only fitting a diagonal matrix:我们可以通过只拟合一个对角矩阵来大大减少这个数字：

However, this is still hundreds of sets of function parameters to fit, and we haven’t even gotten into the size of those sets. Also, note that diagonal matrices perform element-wise multiplication on their inputs; if parts of the final vector need to be zero for reasons like sparsity, then functions in those diagonal elements need to be very close to zero or zero exactly—which puts a lot of burden on the functions themselves.然而，这仍然有数百个需要拟合的函数参数集，而且我们还没考虑这些集的大小。此外，请注意对角矩阵对其输入执行逐元素乘法；如果最终向量的部分内容由于稀疏性等原因需要为零，那么对角线元素中的函数需要非常接近零或精确为零——这给函数本身带来了很大负担。

To see an alternative, let’s rewrite that diagonal matrix as a linear combination of one-hot matrices:为了看一个替代方案，让我们将那个对角矩阵重写为单热（one-hot）矩阵的线性组合：

What if instead of using those one-hot matrices in the sum, we use any matrices we wanted? We could make each matrix a parameter to be learned, and then pick the final number of learnable matrices so that the total nested parameter count is reasonable. Then we could rewrite our weight between x1 and O3 as a new matrix sum:如果我们在总和中使用的不是那些单热矩阵，而是我们想要的任何矩阵会怎样？我们可以使每个矩阵成为一个待学习的参数，然后选择最终可学习矩阵的数量，使得总嵌套参数计数合理。然后我们可以将 x1 和 O3 之间的权重重写为一个新的矩阵和：

Awesome; we just replaced a painful dynamic matrix with a small number of static matrices and dynamic coefficients! Here, the V’s are the static matrices that we learn as free parameters and the functions in the sum are our dynamic scalar-valued coefficients. Keep the V’s in mind; they’ll show up in the attention “value” calculations later.太棒了；我们刚刚用少量静态矩阵和动态系数替换了一个痛苦的动态矩阵！在这里，V 是我们作为自由参数学习的静态矩阵，总和中的函数是我们的动态标量值系数。记住 V；它们稍后会出现在注意力“值”计算中。

Finally, we need to define the scalar-valued functions inside that sum above. To do that, let’s additively decompose one of the functions into two single argument “non-interaction” functions sandwiching a pure interaction function:最后，我们需要定义上述总和内的标量值函数。为此，让我们将其中一个函数加性分解为两个夹着纯交互函数的单参数“非交互”函数：

The downside to having non-interaction functions is the same downside we encountered with weight symmetries from the last section: If the left term is large, then it’ll be large for all units connected to x1 since x1 is the only dependency in the term. We want to break this symmetry so let’s keep only the middle interaction function v, which I’ll call the “attention” function from now on.拥有非交互函数的缺点与我们在上一节中遇到的权重对称性缺点相同：如果左项很大，那么对于所有连接到 x1 的单元来说它都会很大，因为 x1 是该项中唯一的依赖项。我们想要打破这种对称性，所以从现在起，我只保留中间的交互函数 v，并将其称为“注意力”函数。

One candidate for the attention function was introduced in Bahdanau et al. [2]:Bahdanau 等人 [2] 引入了注意力函数的一个候选者：

You may recognize this as a simple one layer neural network with hyperbolic tangent activation, where the final result is reduced to a scalar value via a dot product.你可能认出这是一个简单的单层神经网络，带有双曲正切激活函数，最终结果通过点积简化为标量值。

There is one downside to using the tanh function here: It is only weakly interacting through its non-linearity. The function’s contours get squashed non-linearly, but their basic structure still looks like a linear sum. To see this visually, compare the similarity in contour plots for tanh of x1 plus x3 vs. the straight sum of x1 and x3 (where we’ll make x1 and x3 1D inputs for ease of visualization):在这里使用 tanh 函数有一个缺点：它仅通过非线性进行微弱的交互。函数的轮廓被非线性地压缩，但其基本结构看起来仍然像一个线性总和。为了直观地看到这一点，比较 x1 加 x3 的 tanh 与 x1 加 x3 的直接总和的轮廓图相似度（为了方便可视化，我们将 x1 和 x3 设为一维输入）：

Let’s really highlight the issue with an example: Suppose all the units in our network use one of the functions above, and one unit has inputs x1 and x3.让我们通过一个例子真正突出这个问题：假设我们网络中的所有单元都使用上述函数之一，并且一个单元有输入 x1 和 x3。

If this unit needs to output a large positive value, then there’s only one region that satisfies this on either plot: the top right half. But if another unit shares the same x1 input and requires x1 to be on the left side (i.e. negative) to achieve its goal, then you’re stuck on the first unit’s requirement unless x3 is a huge positive number to compensate. Both units’ requirements are, to a degree, incompatible. This network is less flexible in what its different units can model when they share some inputs.如果该单元需要输出一个大的正值，那么在两个图上只有一个区域满足此条件：右上半部分。但如果另一个单元共享相同的 x1 输入，并要求 x1 在左侧（即负数）才能实现其目标，那么除非 x3 是一个巨大的正数来补偿，否则你就会被困在第一个单元的要求中。两个单元的要求在一定程度上是不兼容的。当这些单元共享一些输入时，该网络在不同单元能够建模的内容上灵活性较差。

On the other hand, look at the contour plot of the product function below:另一方面，看看下面乘积函数的轮廓图：

Now, there are two distinct regions where the final output is large, not just one! If another unit shares the same x1 input and needs x1 to be negative, not a problem; the first unit can still output a large positive value if x3 is negative. Both units’ requirements are no longer as incompatible as before. This is a subtle mathematical argument for how “interactivity” (for example, via a product) matters in allowing units to model unique things despite having overlap in their inputs.现在，最终输出较大的有两个不同的区域，而不仅仅是一个！如果另一个单元共享相同的 x1 输入并需要 x1 为负，没问题；如果 x3 为负，第一个单元仍然可以输出一个大的正值。两个单元的要求不再像以前那样不兼容。这是一个微妙的数学论证，说明了“交互性”（例如，通过乘积）如何对于允许单元在输入重叠的情况下建模独特事物至关重要。

You might ask, “Why not use a deeper neural network to model a more interactive attention function?” This is quite costly in practice because the number of attention calculations scales quadratically with sequence length. And consider that we don’t make each layer of a traditional neural network complicated either; often we use simple non-linear functions like ReLU (rectified linear units) and let additional layers iteratively build up complexity.你可能会问：“为什么不使用更深层的神经网络来建模更具交互性的注意力函数？”这在实践中成本很高，因为注意力计算的数量随序列长度呈二次方增长。而且考虑到我们也不会让传统神经网络的每一层都变得复杂；通常我们使用 ReLU（修正线性单元）等简单的非线性函数，并让额外的层迭代地建立复杂性。

Could we do something similar here, i.e. go with a simple yet interactive attention function and build up complexity over layers? If so, the product function is a great candidate—and GPUs love it too!我们能在这里做类似的事情吗，即使用一个简单但具有交互性的注意力函数，并随着层数增加复杂性？如果是这样，乘积函数是一个很好的候选者——而且 GPU 也喜欢它！

There is one tweak we need to make since our attention inputs are multi-dimensional vectors, not 1D inputs. Instead of a scalar product, we need a dot product. More generally, you might want to compute the dot product in a particular sub-space, which results in a bilinear form that looks like this:我们需要做一个调整，因为我们的注意力输入是多维向量，而不是一维输入。我们需要点积，而不是标量积。更一般地说，你可能希望在特定的子空间中计算点积，这会导致双线性形式，如下所示：

Note that this reduces to a plain dot product when A is the identity matrix, so this form is more general.注意，当 A 是单位矩阵时，这会简化为普通点积，因此这种形式更通用。

There is another consideration that we need to start discussing here: the computation “cache” (what you may have heard as the “key-value cache” in modern architectures.) We can avoid a significant number of attention computations by caching matrix multiplications (e.g. A times x1 above) for past inputs, since we reuse these past calculations for every new time step at every layer.我们需要在这里开始讨论另一个考量：计算“缓存”（你可能在现代架构中听说过的“键值缓存”）。我们可以通过缓存过去输入的矩阵乘法（例如上面的 A 乘以 x1）来避免大量的注意力计算，因为我们在每一层的每一个新时间步都会重用这些过去的计算。

However, the size of this cache becomes a pretty big pain point. Example: If we have a sequence length of 5,000 (quite a bit smaller than many standard LLM conversations), 50 layers, 20 different V matrices –> 20 different dot product functions (see our matrix sum definition earlier), cached matrix-vector products of dimension 1000, and 2 bytes per floating point number, then the final size of our cache is: 5000 x 50 x 20 x 1000 x 2 = 10 GB. Ouch.然而，这个缓存的大小成了一个相当大的痛点。例如：如果我们有 5000 的序列长度（比许多标准 LLM 对话要小得多）、50 层、20 个不同的 V 矩阵 -> 20 个不同的点积函数（参见我们之前的矩阵总和定义）、维度为 1000 的缓存矩阵向量积，以及每个浮点数 2 字节，那么我们缓存的最终大小为：5000 x 50 x 20 x 1000 x 2 = 10 GB。哎哟。

Larger caches mean more GPU cost and memory overhead. One optimization is to simply reduce the dimension of the cached products from 1000 to a lower dimension r, as long as it doesn’t significantly impact model accuracy. But this requires the final dot product to now be in an r-dimensional space rather than a 1000-dimensional one.更大的缓存意味着更多的 GPU 成本和内存开销。一种优化方法是简单地将缓存乘积的维度从 1000 降低到更低的维度 r，只要它不会显著影响模型精度。但这要求最终的点积现在是在 r 维空间而不是 1000 维空间中进行。

We can lower that dot product dimension by factorizing the matrix A into two matrices that “project” the vectors x3 and x1 into the r-dimensional space:我们可以通过将矩阵 A 分解为两个将向量 x3 和 x1 “投影”到 r 维空间的矩阵来降低点积维度：

Here, Wq and Wk are (r x 1000) matrices, and the right-most expression is the desired dot product between two r-dimensional vectors. If r is only 200, then the final cache size goes from 10 GB to 2 GB—a 5x reduction!这里，Wq 和 Wk 是 (r x 1000) 矩阵，最右侧的表达式是两个 r 维向量之间所需的点积。如果 r 仅为 200，则最终缓存大小从 10 GB 变为 2 GB——减少了 5 倍！

Now for the grand reveal: The left and right terms in the new dot product are nothing but the “query” and “key” in the Transformer architecture, and the projection matrices are the same learnable matrices in the original paper! (One caveat is that the Transformer architecture adds scaling for computational stability, hence the term “scaled dot product attention”. But for the rest of this article, I’ll focus on the shape of the architecture rather than on training optimizations like scaling.)现在进行隆重揭秘：新点积中的左右项正是 Transformer 架构中的“查询”和“键”，而投影矩阵就是原始论文中相同的可学习矩阵！（一个注意事项是 Transformer 架构为了计算稳定性增加了缩放，因此得名“缩放点积注意力”。但在本文的其余部分，我将专注于架构的形态，而不是像缩放这样的训练优化。）

We can now substitute this attention function into the coefficients of our earlier matrix sum, with different parameters for each j:我们现在可以将此注意力函数代入我们之前矩阵总和的系数中，每个 j 具有不同的参数：

Quick summary of where we are: The function “f” on the left spits out the weight matrix between input x1 and unit O3; this matrix is equal to a sum involving H scalar attention functions as coefficients, and based on our discussion, we’ve chosen to use dot product attention for each function. Then we can represent the sum total of all weighted inputs into O3 with a new sum s3:快速总结我们所处的位置：左侧的函数“f”吐出输入 x1 和单元 O3 之间的权重矩阵；该矩阵等于一个涉及 H 个标量注意力函数作为系数的总和，根据我们的讨论，我们选择对每个函数使用点积注意力。然后，我们可以用一个新的总和 s3 来表示所有加权输入到 O3 的总和：

The term WQ * x3 is the query for x3, the term WK * xi is the key for xi, and the term V * xi is the value for xi. The subscript j on the matrices denotes a particular attention “head” among the H heads; each head has a unique learnable query, key, and value projection matrix. This is starting to look just like a Transformer!项 WQ * x3 是 x3 的查询，项 WK * xi 是 xi 的键，项 V * xi 是 xi 的值。矩阵上的下标 j 表示 H 个头中的一个特定的注意力“头”；每个头都有一个独特的可学习查询、键和值投影矩阵。这看起来越来越像 Transformer 了！

From Our Attention to Transformer Attention从我们的注意力到 Transformer 注意力

In our earlier example, the outer sum is over a paltry sequence of size 3. But this sum blows up in size when the sequence gets really long.在我们之前的例子中，外层总和是在一个区区 3 的序列长度上进行的。但当序列变得非常长时，这个总和的大小会爆炸。

One way to address this is a standard machine learning trick called normalization: Take the existing coefficients in the weighted sum and transform them so that the transformed coefficients sum to 1 regardless of the previous number and size of coefficients.解决这个问题的一种方法是标准的机器学习技巧，称为归一化：取加权总和中现有的系数并进行变换，使得变换后的系数之和为 1，而不管之前系数的数量和大小如何。

We might also want sparsity in the transformed coefficients; in the domain of language, there are hundreds of irrelevant words that can add up weight quickly, so we want to aggressively squash everything but a few coefficients.我们可能还希望变换后的系数具有稀疏性；在语言领域，有数百个不相关的词会迅速增加权重，所以我们想要积极地压缩除少数系数外的一切。

To see how we should add normalization to our formula, note that we need to do this normalization in the sum over the sequence length, not in the sum over the heads. In that case, we need to flip the order of the sums as follows:要了解我们应该如何将归一化添加到我们的公式中，请注意我们需要在序列长度的总和中进行此归一化，而不是在头的总和中。在这种情况下，我们需要按如下方式翻转总和的顺序：

Now, the term inside the outer parentheses looks like a sum over the sequence length L with coefficient weights (key-query dot products) on the final value vectors—just as we wanted. We then apply a normalization transformation to those dot product coefficients. The natural choice of transformation for achieving soft sparsity is the softmax transformation, which exponentiates each coefficient and divides that result by the sum of all exponentiated coefficients:现在，外括号内的项看起来像是对序列长度 L 的求和，其中包含最终值向量上的系数权重（键-查询点积）——正如我们所愿。然后，我们将归一化变换应用于那些点积系数。实现软稀疏性的自然变换选择是 softmax 变换，它对每个系数进行指数化，并将该结果除以所有指数化系数的总和：

This is called the “softmax” because it aggressively pushes the largest (max) coefficient to 1 and squashes all other coefficients towards 0. The tau parameter in the exponent controls how aggressive this transformation is.这被称为“softmax”，因为它积极地将最大系数推向 1，并将所有其他系数压缩向 0。指数中的 tau 参数控制此变换的积极程度。

Okay, all of this is great—but where are the matrices Q, K, and V that the article title promised us?好的，这一切都很棒——但文章标题承诺给我们的矩阵 Q、K 和 V 在哪里？

To match our expression with the iconic Transformer equation, let’s temporarily ignore the outer sum over heads by picking a particular head j and rewrite the normalized sum at position L for that head:为了将我们的表达式与标志性的 Transformer 方程匹配，让我们通过选择一个特定的头 j 暂时忽略关于头的外部总和，并为该头重写位置 L 处的归一化总和：

Here, q is the query vector for the Lth unit, and the rows of matrices K and V are the key and value vectors respectively. The subscript 1:L indicates that we only use the keys and values for positions 1 through L, which reflects the fact that the Lth unit is only connected to previous units in time. The product between q and K^T creates a vector containing every dot product between q and a key in K, and the softmax on top normalizes the final dot product scores. Afterwards, we do a weighted sum of value vectors in V using the resulting softmax scores.这里，q 是第 L 个单元的查询向量，矩阵 K 和 V 的行分别是键和值向量。下标 1:L 表示我们仅使用位置 1 到 L 的键和值，这反映了第 L 个单元仅连接到时间上以前的单元这一事实。q 和 K^T 之间的乘积创建了一个向量，其中包含 q 和 K 中每个键之间的所有点积，顶部的 softmax 对最终点积得分进行归一化。之后，我们使用生成的 softmax 分数对 V 中的值向量进行加权求和。

We can extend this into a single consolidated matrix expression for the sum at every position, not just position L:我们可以将其扩展为每个位置（不仅仅是位置 L）的总和的单个合并矩阵表达式：

This is exactly the iconic Transformer equation (minus dimensional scaling) but for an autoregressive Transformer! Here, each row of S is the final sum at that position, and each row of Q, K, and V are the respective query, key, and value vectors at that position. Since we are using the full matrices Q, K, and V, we need a masking matrix M to zero out invalid dot products (such as the dot product between a query at position p and a key at a later position) by adding in negative infinity to those entries; the softmax then squashes those results to zero.这正是标志性的 Transformer 方程（减去维度缩放），但针对的是自回归 Transformer！在这里，S 的每一行是该位置的最终总和，Q、K 和 V 的每一行分别是该位置各自的查询、键和值向量。由于我们使用的是完整的矩阵 Q、K 和 V，我们需要一个掩码矩阵 M 来将无效点积（例如位置 p 的查询和后续位置的键之间的点积）置零，方法是在这些条目中加入负无穷大；softmax 然后将这些结果压缩为零。

Why write this with matrices instead of our original sum if the two expressions are identical? The answer is simple: GPUs love matrix multiplication. It is often more efficient to rewrite a computation as a matrix multiplication, even if that means doing some unnecessary work like computing dot products which will be masked out anyway. That being said, I find the sum notation easier to use for explanations, so I’ll stick with it for the rest of the article.如果这两个表达式完全相同，为什么要用矩阵而不是我们最初的总和来写这个？答案很简单：GPU 喜欢矩阵乘法。将计算重写为矩阵乘法通常更有效，即使这意味着要做一些不必要的工作，例如计算无论如何都会被掩盖的点积。话虽如此，我发现总和符号在解释时更容易使用，所以我将在本文的其余部分坚持使用它。

Let’s close out the section by returning to the sum over attention heads. You may have noticed that while we sum over the H attention heads, Vaswani et al. [3] and implementations like PyTorch’s “MultiheadAttention” concatenate the outputs across each head instead—and multiply the concatenation by a “mixing matrix” Wo that collapses the concatenation back to the previous model dimension.让我们通过回到注意力头的总和来结束本节。你可能已经注意到，当我们对 H 个注意力头求和时，Vaswani 等人 [3] 和像 PyTorch 的“MultiheadAttention”这样的实现改为连接每个头的输出——并将连接乘以一个“混合矩阵”Wo，将连接折叠回之前的模型维度。

This might look like a cosmetic difference; why concatenate if you are just going to collapse back down?这看起来可能是一个表面上的差异；如果你只是要折叠回来，为什么要连接？

The trick is caching, but now with a focus on value vectors: We need the mixing matrix to allow us to compress those value vectors (V * x’s) so our cache doesn’t explode. Let’s see what happens to our double sum expression if we concatenate the output of each head j into a block vector instead of summing over the heads:诀窍在于缓存，但现在重点在于值向量：我们需要混合矩阵来允许我们压缩那些值向量（V * x），这样我们的缓存就不会爆炸。让我们看看如果我们连接每个头 j 的输出到一个块向量而不是对头求和，我们的双重总和表达式会发生什么：

For the sake of readability, I’ve substituted in alphas for the previous normalized coefficients so there aren’t a bunch of exponentials everywhere.为了可读性，我用 alpha 替换了之前的归一化系数，这样就不会到处都是指数了。

Now let’s left-multiply this long block vector by the mixing matrix Wo. Let’s first re-write Wo into a matching block matrix form and then perform the multiplication:现在让我们用混合矩阵 Wo 左乘这个长块向量。让我们首先将 Wo 重写为匹配的块矩阵形式，然后执行乘法：

Note how similar the final result is to our original double sum! The only difference is the block matrix left multiplying V; when each block matrix is the identity matrix, the new expression is totally identical to our original expression.注意最终结果与我们的原始双重总和有多么相似！唯一的区别是左乘 V 的块矩阵；当每个块矩阵是单位矩阵时，新表达式与我们的原始表达式完全相同。

You may ask, “Why not just absorb the Wo block matrices into the V matrices?” At first glance, the two matrices seem redundant if you can collapse them into a single one. But the reason we’d want to keep them separate is the same reason we didn’t use a single matrix A for our attention dot product: lowering our cache size.你可能会问：“为什么不直接将 Wo 块矩阵吸收到 V 矩阵中？”乍一看，如果你能将它们折叠成一个，这两个矩阵似乎是多余的。但我们想要保持它们分离的原因与我们没有为注意力点积使用单个矩阵 A 的原因相同：减小缓存大小。

Let’s say we did absorb everything into a single matrix V. If our model dimension is 1000 and we use the same conditions from our key-based cache example earlier, then the cumulative cache size for our value vectors is an unwieldy 10 GB. But if V has the shape (r x 1000) where r < 1000, then the final dimension of our value vectors will be r < 1000 and our cache size can be considerably smaller. 假设我们确实将所有内容吸收到单个矩阵 V 中。如果我们的模型维度是 1000，并且我们使用之前基于键的缓存示例中的相同条件，那么我们值向量的累积缓存大小是一个难以处理的 10 GB。但如果 V 的形状为 (r x 1000) 且 r < 1000，那么我们值向量的最终维度将是 r < 1000，我们的缓存大小可以小得多。

Herein lies the need for the Wo matrices; after we’re done having our fun in lower dimensions, we need to multiply the final values by a matrix of shape (1000 x r) to “up project” back to the larger model dimension of 1000.这就是 Wo 矩阵的需求所在；在我们完成低维度的乐趣后，我们需要将最终值乘以一个形状为 (1000 x r) 的矩阵，以“上投影”回 1000 的较大模型维度。

The Transformer MLP: The Other Key-Value StoreTransformer MLP：另一个键值存储器

Everything we did earlier was to define the weights in our network. If we isolate one input xi in the sum of weighted inputs from our last expression, then its weight into unit L is equal to:我们之前所做的一切都是为了定义网络中的权重。如果我们从最后一个表达式中分离出加权输入总和中的一个输入 xi，那么它进入单元 L 的权重等于：

Here, each alpha coefficient is unit L’s softmax normalized scalar for input i and a given head j.这里，每个 alpha 系数是单元 L 对于输入 i 和给定头 j 的 softmax 归一化标量。

Once you have all the weights, all you need to do is to sum up the weighted inputs into each unit, compute the position-wise non-linearity, and add back to the residual stream via the skip connection to update the positional representation. Voila, there’s your neural network!一旦你有了所有的权重，你所需要做的就是对进入每个单元的加权输入求和，计算逐位置非线性，并通过残差连接加回残差流以更新位置表示。瞧，这就是你的神经网络！

These steps can be visually represented by the following recap diagram, using the same length 3 sequence for simplicity:这些步骤可以通过以下回顾图直观地表示，为简单起见，使用相同的长度为 3 的序列：

You may have noticed that Vaswani et al. [3] introduce two new weight matrices (W1 and W2) that sandwich the non-linearity in the blue units. At first glance, these additions feel like arbitrary add-ons. If we’ve already done all this work to define dynamic weights via attention, then do we really need two new matrices around the non-linearity?你可能已经注意到 Vaswani 等人 [3] 引入了两个新的权重矩阵（W1 和 W2），它们夹住了蓝色单元中的非线性。乍一看，这些添加感觉像是随意的附加物。如果我们已经完成了所有这些工作来通过注意力定义动态权重，那么我们真的需要在非线性周围增加两个新矩阵吗？

It turns out that W2 isn’t there just for the sake of it. Geva et al. [5] give us important perspective for understanding why: The Transformer MLP (feed-forward block) acts like a key-value store in its own right.事实证明，W2 的存在并非毫无意义。Geva 等人 [5] 为我们理解原因提供了重要的视角：Transformer MLP（前馈模块）本身就像一个键值存储器。

To see this, let’s actually show where the “keys” and “values” would live in an example MLP network:为了看到这一点，让我们实际展示“键”和“值”在 MLP 网络示例中的位置：

In this network, the middle layer has red, green, and blue units that each model different patterns. The incoming (first layer) weights of the same color represent the “key” vectors for each pattern, and the outgoing (last layer) weights of the same color represent the “value” vectors for each pattern. Geva et al. [5] demonstrate what some of these “patterns” could look like; for the model they trained, one example of a learned pattern was whenever the input sentence ended in the word “substitutes”, and another was whenever the input sentence contained a “part of” relation.在这个网络中，中间层有红色、绿色和蓝色单元，每个单元建模不同的模式。相同颜色的传入（第一层）权重代表每个模式的“键”向量，相同颜色的传出（最后一层）权重代表每个模式的“值”向量。Geva 等人 [5] 展示了这些“模式”可能是什么样子的；对于他们训练的模型，学习到的一个模式示例是每当输入句子以单词“substitutes”结尾时，另一个是每当输入句子包含“part of”关系时。

Now let’s walk through what the “keys” and “values” actually do here. When the input aligns closely with the key vector (incoming weights) for a given pattern, the dot product between the key and input is high, and the unit associated with that pattern outputs a large number. That number then multiplies the value vector (outgoing weights) for the pattern and this weighted value vector is finally added to the output layer.现在让我们看看“键”和“值”在这里实际上做了什么。当输入与给定模式的键向量（传入权重）紧密对齐时，键和输入之间的点积很高，与该模式关联的单元输出一个大数字。该数字然后乘以该模式的值向量（传出权重），这个加权值向量最终被加到输出层。

So back to our extra parameters: What does this tell us about why we need W2?所以回到我们额外的参数：这告诉我们关于为什么我们需要 W2 的什么信息？

Recall that W2 is just the matrix multiplied to the output of the non-linearity in the original Transformer. In our MLP example, W2 is exactly equal to the matrix of weights in the final layer, i.e. a matrix where each column is one value vector.回想一下，W2 只是原始 Transformer 中乘以非线性输出的矩阵。在我们的 MLP 示例中，W2 完全等于最后一层的权重矩阵，即每一列都是一个值向量的矩阵。

If we get rid of W2, our diagram would instead look something like this:如果我们摆脱 W2，我们的图表看起来就像这样：

Each unit in this network is writing to a single coordinate of the residual stream (since we add the output layer directly back to the residual stream.) But there are two reasons why this is bad:该网络中的每个单元都在写入残差流的单个坐标（因为我们将输出层直接加回残差流）。但有两个原因说明这是糟糕的：

1. If we want our units to learn arbitrary patterns, then each of those arbitrary patterns might not map cleanly to single coordinates of the “semantic space” at that layer. In that case, it is far more expressive to allow each unit to write arbitrary vectors (values) into the residual stream rather than coupling them to individual output coordinates.如果我们希望我们的单元学习任意模式，那么这些任意模式中的每一个可能无法清晰地映射到该层“语义空间”的单个坐标上。在这种情况下，允许每个单元将任意向量（值）写入残差流，而不是将它们耦合到单个输出坐标，要表达力得多。
2. We want our MLPs to model as many patterns as possible, which pushes us to have more units in the MLP network relative to the model dimension in the residual stream (where the latter is kept smaller to avoid gnarly attention computations that don’t benefit as much from high dimensions.) In that case, a 1:1 correspondence of coordinates isn’t even possible; we need a matrix to “down project” back to the smaller model dimension anyways.我们希望我们的 MLP 建模尽可能多的模式，这促使我们在 MLP 网络中拥有比残差流中模型维度更多的单元（后者保持较小以避免无法从高维度中受益太多的棘手注意力计算）。在这种情况下，坐标的 1:1 对应甚至是不可能的；无论如何，我们需要一个矩阵来“下投影”回较小的模型维度。

We’ve spent all this time discussing W2; what about W1, the extra matrix multiplying inside the non-linearity?我们花了这么多时间讨论 W2；那 W1 呢，即在非线性内部相乘的额外矩阵？

This may actually be the more arbitrary matrix. In our setup, W1 could indeed be absorbed into our dynamic weight matrices (specifically, Wo from the previous section) and dropped as an extra parameter. But there is a reason this can’t be done in the original Transformer: In that setup, there are two intermediate steps between the attention output and the non-linearity that prevent consolidation of matrices.这可能实际上是更随意的矩阵。在我们的设置中，W1 确实可以被吸收到我们的动态权重矩阵中（特别是上一节中的 Wo），并作为额外参数丢弃。但有一个原因导致这在原始 Transformer 中无法做到：在该设置中，注意力输出和非线性之间有两个中间步骤，阻止了矩阵的合并。

First, the Transformer splits our single big skip connection into a skip connection between the input and the sum output as well as a second skip between the sum output and the non-linearity output. See the modified diagram with the split skip connection:首先，Transformer 将我们的单个大残差连接拆分为输入和总和输出之间的残差连接，以及总和输出和非线性输出之间的第二个残差连接。请看带有拆分残差连接的修改后图示：

Once you have this additional skip connection, you need a separate weight matrix applied on the sums of the output from the attention (capital sigma) blocks and the residuals from the first skips. The weight matrix applied to that sum of outputs can no longer be cleanly absorbed, so you need a separate parameter.一旦你有了这个额外的残差连接，你就需要一个单独的权重矩阵应用于注意力（大 sigma）块输出的总和以及来自第一个残差的残差。应用于该输出总和的权重矩阵无法再被干净地吸收，所以你需要一个单独的参数。

Secondly, there is an additional scaling transformation between the sum of attention outputs and the non-linearity that prevents absorption. While these scaling transformations are important, they are training optimizations, and so I’ve omitted them to focus purely on architectural shape here.其次，注意力输出的总和与非线性之间存在额外的缩放变换，阻止了吸收。虽然这些缩放变换很重要，但它们是训练优化，所以我在这里省略了它们，以专注于架构形态。

Recapping the Logic总结逻辑

Pat yourself on the back; you just invented the Transformer! Here’s a recap of the steps we took to get here:给自己鼓个掌吧，你刚刚发明了 Transformer！下面回顾一下我们实现这一目标所采取的步骤：

1. We needed to access every past state of the sequence without compression loss. This required direct connections to past states, aka “attention”.我们需要在不损失压缩信息的情况下访问序列的所有过去状态。这要求与过去的状态建立直接连接，即所谓的“注意力”（attention）。
2. We needed parallelism for fast GPU training. This required dropping recurrence.为了实现快速的 GPU 训练，我们需要并行化。这就要求放弃循环结构。
3. We needed a way to set new weights for future inputs without introducing weight symmetries across the network. This required making the weights a function of the source unit and the end unit (both unique per weight), where we used the residual stream’s value as a stand-in for the “end unit”. These two function arguments are the “key” and “query” prior to projection.我们需要一种方法来为未来的输入设置新的权重，且不能在整个网络中引入权重对称性。这要求使权重成为源单元和目标单元（每个权重都有唯一对应）的函数，在此我们利用残差流的值作为“目标单元”的替代。这两个函数参数在投影前就是“键”（key）和“查询”（query）。
4. We needed to make the weight-generating function avoid symmetries by encoding pure interactions between the arguments. A GPU-efficient choice for this interaction function was the dot product between the key and query arguments.我们需要让权重生成函数通过编码参数之间的纯交互来避免对称性。对于这种交互函数，一个 GPU 高效的选择是键参数和查询参数之间的点积。
5. We needed to reduce our cache size for the reusable matrix-vector multiplies in the dot product, which required computing the dot product in lower dimensions. This required projection matrices Wk and Wq to project the key and query into that lower dimensional space.我们需要减小点积中可重用矩阵向量乘法的缓存大小，这要求在更低的维度上计算点积。因此需要投影矩阵 Wk 和 Wq 将键和查询投影到该低维空间中。
6. We needed to reduce our search over the entire d^2 space of linear operations (matrices) into a search over a smaller number H of possible operations that we can select from via our dot product weights. This is represented by H different value projections V representing our H attention heads.我们需要将对线性运算（矩阵）整个 d^2 空间的搜索，缩减为对 H 个可能运算的搜索，并能通过点积权重进行选择。这由代表 H 个注意力头的 H 个不同值投影 V 来表示。
7. We needed to make the final weights on the value projections sparse so noise doesn’t blow up over larger sequences. This required softmax normalization of the dot product coefficients over sequence length, per head.我们需要使值投影上的最终权重稀疏化，以防止噪声在长序列中被放大。这要求对每个头的序列长度进行 softmax 归一化处理。
8. We needed to reduce our cache size for the reusable matrix-vector multiplies between the value projections V and the inputs. This required the Vs to “down project” the final value vectors into a lower dimensional space, and a separate mixing matrix Wo to “up project” back into the model dimension.我们需要减小值投影 V 与输入之间可重用矩阵向量乘法的缓存大小。这要求 V 将最终的值向量“向下投影”到低维空间，并使用一个独立的混合矩阵 Wo 将其“向上投影”回模型维度。
9. We needed a larger “feature space” for our non-linearities to learn lots of patterns, while allowing these features to write arbitrary vectors to the lower dimensional residual stream. This required a new matrix W2 that multiplies the output of the non-linearity.我们需要更大的“特征空间”供非线性函数学习大量模式，同时允许这些特征将任意向量写入低维残差流。这需要一个新的矩阵 W2 来乘以非线性函数的输出。

Why Transformers Aren’t Inevitable为什么 Transformer 并非不可替代

The only thing inevitable in AI/ML is one architecture replacing another, and Transformers are no exception to the rule.在 AI/ML 领域，唯一不可避免的就是旧架构被新架构取代，Transformer 也不例外。

But why will Transformers be replaced if they work so well today?但既然 Transformer 目前表现如此出色，为什么还会被取代呢？

For starters, Transformers have one massive downside: Computations scale quadratically with sequence length. You could implement sliding window attention to try and get around this, but then you lose the ability to recall pin-point facts or instructions hidden within longer context.首先，Transformer 有一个巨大的缺点：计算量随序列长度呈二次方增长。你可以尝试通过实现滑动窗口注意力来规避这个问题，但这样会失去召回隐藏在长上下文中的特定事实或指令的能力。

You may be wondering, “But most of the attention scores are basically zero due to the softmax normalization. Do we need to calculate all n^2 scores?”你可能会问：“但由于 softmax 归一化，大部分注意力得分基本上都是零。我们真的需要计算所有 n^2 个得分吗？”

That’s a great insight, and attempts at sparse attention mechanisms have indeed been made. However, despite the major reduction in calculations, these mechanisms counterintuitively run slower than full quadratic attention due to GPU memory bottlenecks becoming the bigger pain point, as discussed by Dao et al. [6].这是一个非常深刻的见解，实际上也确实有人尝试过稀疏注意力机制。然而，正如 Dao 等人 [6] 所讨论的那样，尽管计算量大幅减少，但由于 GPU 内存瓶颈成为更大的痛点，这些机制运行起来反而比完整的二次方注意力更慢，这有些违背直觉。

This brings us to another point: Transformers are benefitting enormously from being tailor-made for GPUs. Many of their design decisions revolve around the quirky memory and compute patterns of GPUs, and anything that could dethrone Transformers has to ultimately pass the “GPU smell test”. In other words, we are trapped in a hardware local minima and are missing out on much better algorithms because of it. Human intelligence doesn’t run on GPUs, so why should we settle for Transformers?这引出了另一点：Transformer 受益于为 GPU 量身定制的特性。它们的许多设计决策都围绕着 GPU 独特的内存和计算模式，任何想要取代 Transformer 的技术最终都必须通过“GPU 气味测试”（GPU smell test）。换句话说，我们被困在了硬件的局部极小值中，并因此错过了更好的算法。人类智能并非运行在 GPU 上，所以我们为什么要满足于 Transformer 呢？

References参考文献

[1] S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory,” Neural Computation, vol. 9, no. 8, pp. 1735-1780, 1997.[1] S. Hochreiter 和 J. Schmidhuber，“Long Short-Term Memory，” Neural Computation, 第 9 卷, 第 8 期, 第 1735-1780 页, 1997。

[2] D. Bahdanau, K. Cho, and Y. Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate,” International Conference on Learning Representations (ICLR), 2015.[2] D. Bahdanau, K. Cho 和 Y. Bengio，“Neural Machine Translation by Jointly Learning to Align and Translate，” International Conference on Learning Representations (ICLR), 2015。

[3] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention Is All You Need,” Advances in Neural Information Processing Systems 30 (NIPS), pp. 5998-6008, 2017.[3] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser 和 I. Polosukhin，“Attention Is All You Need，” Advances in Neural Information Processing Systems 30 (NIPS), 第 5998-6008 页, 2017。

[4] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.[4] K. He, X. Zhang, S. Ren 和 J. Sun，“Deep Residual Learning for Image Recognition，” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 第 770-778 页, 2016。

[5] M. Geva, R. Schuster, J. Berant, and O. Levy, “Transformer Feed-Forward Layers Are Key-Value Memories,” Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5484-5495, 2021.[5] M. Geva, R. Schuster, J. Berant 和 O. Levy，“Transformer Feed-Forward Layers Are Key-Value Memories，” Conference on Empirical Methods in Natural Language Processing (EMNLP), 第 5484-5495 页, 2021。

[6] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré, “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness,” Advances in Neural Information Processing Systems 35 (NeurIPS), pp. 16344-16359, 2022.[6] T. Dao, D. Y. Fu, S. Ermon, A. Rudra 和 C. Ré，“FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness，” Advances in Neural Information Processing Systems 35 (NeurIPS), 第 16344-16359 页, 2022。