Discussions: Hacker News (65 points, 4 comments), Reddit r/MachineLearning (29 points, 3 comments)
Translations: Arabic, Chinese (Simplified) 1, Chinese (Simplified) 2, French 1, French 2, Italian, Japanese, Korean, Persian, Russian, Spanish 1, Spanish 2, Vietnamese
Watch: MIT’s Deep Learning State of the Art lecture referencing this post
Featured in courses at Stanford, Harvard, MIT, Princeton, CMU and others讨论： Hacker News（65 分，4 条评论），Reddit r/MachineLearning（29 分，3 条评论） 翻译版本：阿拉伯语、中文（简体）1、中文（简体）2、法语 1、法语 2、意大利语、日语、韩语、波斯语、俄语、西班牙语 1、西班牙语 2、越南语 观看：麻省理工学院（MIT）深度学习前沿讲座（引用了本文） 被斯坦福大学、哈佛大学、麻省理工学院、普林斯顿大学、卡内基梅隆大学等高校课程收录

Update: This post has now become a book! Check out LLM-book.com which contains (Chapter 3) an updated and expanded version of this post speaking about the latest Transformer models and how they've evolved in the seven years since the original Transformer (like Multi-Query Attention and RoPE Positional embeddings).

In the previous post, we looked at Attention – a ubiquitous method in modern deep learning models. Attention is a concept that helped improve the performance of neural machine translation applications. In this post, we will look at The Transformer – a model that uses attention to boost the speed with which these models can be trained. The Transformer outperforms the Google Neural Machine Translation model in specific tasks. The biggest benefit, however, comes from how The Transformer lends itself to parallelization. It is in fact Google Cloud’s recommendation to use The Transformer as a reference model to use their Cloud TPU offering. So let’s try to break the model apart and look at how it functions.在上一篇文章中，我们探讨了“注意力机制”（Attention）——一种在现代深度学习模型中无处不在的方法。注意力机制是一个有助于提升神经机器翻译应用性能的概念。在本文中，我们将探讨 Transformer——一种利用注意力机制来提高模型训练速度的模型。Transformer 在特定任务上的表现优于 Google 神经机器翻译模型。然而，它最大的优势在于其易于并行化。事实上，Google Cloud 建议将 Transformer 作为使用其 Cloud TPU 服务的参考模型。那么，让我们尝试剖析该模型，看看它是如何运作的。

The Transformer was proposed in the paper Attention is All You Need. A TensorFlow implementation of it is available as a part of the Tensor2Tensor package. Harvard’s NLP group created a guide annotating the paper with PyTorch implementation. In this post, we will attempt to oversimplify things a bit and introduce the concepts one by one to hopefully make it easier to understand to people without in-depth knowledge of the subject matter.Transformer 是在论文《Attention is All You Need》中提出的。其 TensorFlow 实现版本作为 Tensor2Tensor 软件包的一部分提供。哈佛大学的 NLP 小组创建了一份指南，对该论文进行了注释，并提供了 PyTorch 实现。在本文中，我们将尝试简化一些内容，逐一介绍相关概念，希望能够让没有深入专业知识背景的读者更容易理解。

2025 Update: We’ve built a free short course that brings the contents of this post up-to-date with animations:2025 年更新：我们制作了一门免费的简短课程，通过动画更新了本文的内容：

A High-Level Look宏观视角

Let’s begin by looking at the model as a single black box. In a machine translation application, it would take a sentence in one language, and output its translation in another.让我们先将该模型视为一个黑盒。在机器翻译应用中，它会接收一种语言的句子，并输出另一种语言的翻译。

Popping open that Optimus Prime goodness, we see an encoding component, a decoding component, and connections between them.打开这个“擎天柱”般的黑盒，我们可以看到一个编码组件、一个解码组件以及它们之间的连接。

The encoding component is a stack of encoders (the paper stacks six of them on top of each other – there’s nothing magical about the number six, one can definitely experiment with other arrangements). The decoding component is a stack of decoders of the same number.编码组件是由一堆编码器组成的（论文中将六个编码器堆叠在一起——数字六并没有什么神奇之处，完全可以尝试其他结构）。解码组件则是相同数量的解码器堆叠而成。

The encoders are all identical in structure (yet they do not share weights). Each one is broken down into two sub-layers:编码器的结构完全相同（但它们不共享权重）。每个编码器都分为两个子层：

The encoder’s inputs first flow through a self-attention layer – a layer that helps the encoder look at other words in the input sentence as it encodes a specific word. We’ll look closer at self-attention later in the post.编码器的输入首先流经一个自注意力层（self-attention layer）——该层有助于编码器在编码特定单词时关注输入句子中的其他单词。我们将在本文后面详细探讨自注意力机制。

The outputs of the self-attention layer are fed to a feed-forward neural network. The exact same feed-forward network is independently applied to each position.自注意力层的输出被馈送到前馈神经网络（feed-forward neural network）。完全相同的前馈网络被独立应用于每个位置。

The decoder has both those layers, but between them is an attention layer that helps the decoder focus on relevant parts of the input sentence (similar what attention does in seq2seq models).解码器也包含这两个层，但它们之间有一个注意力层，有助于解码器关注输入句子中相关的部分（类似于 seq2seq 模型中的注意力机制）。

Bringing The Tensors Into The Picture引入张量（Tensors）

Now that we’ve seen the major components of the model, let’s start to look at the various vectors/tensors and how they flow between these components to turn the input of a trained model into an output.既然我们已经了解了模型的主要组件，现在让我们来看看各种向量/张量，以及它们如何在这些组件之间流动，从而将已训练模型的输入转化为输出。

As is the case in NLP applications in general, we begin by turning each input word into a vector using an embedding algorithm.与一般的 NLP 应用一样，我们首先使用嵌入算法（embedding algorithm）将每个输入单词转换为向量。

The embedding only happens in the bottom-most encoder. The abstraction that is common to all the encoders is that they receive a list of vectors each of the size 512 – In the bottom encoder that would be the word embeddings, but in other encoders, it would be the output of the encoder that’s directly below. The size of this list is hyperparameter we can set – basically it would be the length of the longest sentence in our training dataset.嵌入过程仅发生在最底层的编码器中。所有编码器共有的抽象概念是：它们接收一个向量列表，每个向量的大小为 512。在底层编码器中，这些是词嵌入；但在其他编码器中，它们是直接位于其下方的编码器的输出。列表的大小是一个我们可以设置的超参数——基本上，它就是训练数据集中最长句子的长度。

After embedding the words in our input sequence, each of them flows through each of the two layers of the encoder.在对输入序列中的单词进行嵌入后，每个单词都会流经编码器的两个子层。

Here we begin to see one key property of the Transformer, which is that the word in each position flows through its own path in the encoder. There are dependencies between these paths in the self-attention layer. The feed-forward layer does not have those dependencies, however, and thus the various paths can be executed in parallel while flowing through the feed-forward layer.在这里，我们开始看到 Transformer 的一个关键属性，即每个位置的单词在编码器中都有自己的流动路径。在自注意力层中，这些路径之间存在依赖关系。然而，前馈层没有这些依赖关系，因此在流经前馈层时，不同的路径可以并行执行。

Next, we’ll switch up the example to a shorter sentence and we’ll look at what happens in each sub-layer of the encoder.接下来，我们将把示例换成一个较短的句子，并看看编码器每个子层中发生了什么。

Now We’re Encoding!开始编码！

As we’ve mentioned already, an encoder receives a list of vectors as input. It processes this list by passing these vectors into a ‘self-attention’ layer, then into a feed-forward neural network, then sends out the output upwards to the next encoder.正如我们已经提到的，编码器接收一个向量列表作为输入。它通过将这些向量传入“自注意力”层，然后传入前馈神经网络，最后将输出向上传递给下一个编码器来处理这个列表。

Self-Attention at a High Level自注意力机制概览

Don’t be fooled by me throwing around the word “self-attention” like it’s a concept everyone should be familiar with. I had personally never came across the concept until reading the Attention is All You Need paper. Let us distill how it works.不要被我随口提到的“自注意力”这个词所迷惑，好像这是一个每个人都应该熟悉的概念。在阅读《Attention is All You Need》论文之前，我个人从未接触过这个概念。让我们来提炼一下它是如何工作的。

Say the following sentence is an input sentence we want to translate:假设以下句子是我们想要翻译的输入句子：

”The animal didn't cross the street because it was too tired”“The animal didn't cross the street because it was too tired”（动物没有穿过街道，因为它太累了）

What does “it” in this sentence refer to? Is it referring to the street or to the animal? It’s a simple question to a human, but not as simple to an algorithm.这句话中的“it”指的是什么？是指街道还是动物？对于人类来说，这是一个简单的问题，但对于算法来说却并非如此。

When the model is processing the word “it”, self-attention allows it to associate “it” with “animal”.当模型处理单词“it”时，自注意力机制允许它将“it”与“animal”关联起来。

As the model processes each word (each position in the input sequence), self attention allows it to look at other positions in the input sequence for clues that can help lead to a better encoding for this word.当模型处理每个单词（输入序列中的每个位置）时，自注意力机制允许它查看输入序列中的其他位置，寻找有助于为该单词生成更好编码的线索。

If you’re familiar with RNNs, think of how maintaining a hidden state allows an RNN to incorporate its representation of previous words/vectors it has processed with the current one it’s processing. Self-attention is the method the Transformer uses to bake the “understanding” of other relevant words into the one we’re currently processing.如果你熟悉 RNN，可以回想一下，维护隐藏状态如何让 RNN 将其对之前处理过的单词/向量的表征与当前处理的向量结合起来。自注意力是 Transformer 用来将对其他相关单词的“理解”融入到我们当前正在处理的单词中的方法。

Be sure to check out the Tensor2Tensor notebook where you can load a Transformer model, and examine it using this interactive visualization.请务必查看 Tensor2Tensor 笔记本，在那里你可以加载 Transformer 模型，并使用此交互式可视化工具进行检查。

Self-Attention in Detail自注意力机制详解

Let’s first look at how to calculate self-attention using vectors, then proceed to look at how it’s actually implemented – using matrices.让我们先看看如何使用向量计算自注意力，然后继续看看它是如何通过矩阵实际实现的。

The first step in calculating self-attention is to create three vectors from each of the encoder’s input vectors (in this case, the embedding of each word). So for each word, we create a Query vector, a Key vector, and a Value vector. These vectors are created by multiplying the embedding by three matrices that we trained during the training process.计算自注意力的第一步是根据每个编码器的输入向量（在本例中，是每个单词的嵌入）创建三个向量。因此，对于每个单词，我们创建一个查询向量（Query）、一个键向量（Key）和一个值向量（Value）。这些向量是通过将嵌入向量乘以我们在训练过程中训练出的三个矩阵而创建的。

Notice that these new vectors are smaller in dimension than the embedding vector. Their dimensionality is 64, while the embedding and encoder input/output vectors have dimensionality of 512. They don’t HAVE to be smaller, this is an architecture choice to make the computation of multiheaded attention (mostly) constant.注意，这些新向量的维度比嵌入向量小。它们的维度是 64，而嵌入向量和编码器输入/输出向量的维度是 512。它们不一定非要更小，这是一种架构选择，目的是使多头注意力（multiheaded attention）的计算（大部分）保持恒定。

What are the “query”, “key”, and “value” vectors?

They’re abstractions that are useful for calculating and thinking about attention. Once you proceed with reading how attention is calculated below, you’ll know pretty much all you need to know about the role each of these vectors plays.什么是“查询”、“键”和“值”向量？ 它们是用于计算和思考注意力机制的抽象概念。一旦你阅读了下面关于如何计算注意力的内容，你就会了解这些向量各自所扮演的角色。

The second step in calculating self-attention is to calculate a score. Say we’re calculating the self-attention for the first word in this example, “Thinking”. We need to score each word of the input sentence against this word. The score determines how much focus to place on other parts of the input sentence as we encode a word at a certain position.计算自注意力的第二步是计算分数。假设我们正在计算本例中第一个单词“Thinking”的自注意力。我们需要根据这个单词对输入句子的每个单词进行评分。该分数决定了我们在编码特定位置的单词时，应该对输入句子的其他部分投入多少关注。

The score is calculated by taking the dot product of the query vector with the key vector of the respective word we’re scoring. So if we’re processing the self-attention for the word in position #1, the first score would be the dot product of q1 and k1. The second score would be the dot product of q1 and k2.分数是通过将查询向量与我们正在评分的相应单词的键向量进行点积计算得出的。因此，如果我们正在处理位置 #1 的单词的自注意力，第一个分数将是 q1 和 k1 的点积。第二个分数将是 q1 和 k2 的点积。

The third and fourth steps are to divide the scores by 8 (the square root of the dimension of the key vectors used in the paper – 64. This leads to having more stable gradients. There could be other possible values here, but this is the default), then pass the result through a softmax operation. Softmax normalizes the scores so they’re all positive and add up to 1.第三步和第四步是将分数除以 8（这是论文中使用的键向量维度的平方根——64。这有助于获得更稳定的梯度。这里可能有其他可能的值，但这是默认值），然后将结果通过 Softmax 操作。Softmax 对分数进行归一化，使它们都为正数且总和为 1。

This softmax score determines how much each word will be expressed at this position. Clearly the word at this position will have the highest softmax score, but sometimes it’s useful to attend to another word that is relevant to the current word.这个 Softmax 分数决定了每个单词在当前位置的表达程度。显然，当前位置的单词将具有最高的 Softmax 分数，但有时关注与当前单词相关的另一个单词也很有用。

The fifth step is to multiply each value vector by the softmax score (in preparation to sum them up). The intuition here is to keep intact the values of the word(s) we want to focus on, and drown-out irrelevant words (by multiplying them by tiny numbers like 0.001, for example).第五步是将每个值向量乘以 Softmax 分数（为求和做准备）。这里的直觉是保持我们想要关注的单词的值完整，并抑制不相关的单词（例如，通过将它们乘以 0.001 这样的小数字）。

The sixth step is to sum up the weighted value vectors. This produces the output of the self-attention layer at this position (for the first word).第六步是将加权后的值向量相加。这产生了该位置（对于第一个单词）自注意力层的输出。

That concludes the self-attention calculation. The resulting vector is one we can send along to the feed-forward neural network. In the actual implementation, however, this calculation is done in matrix form for faster processing. So let’s look at that now that we’ve seen the intuition of the calculation on the word level.自注意力计算到此结束。生成的向量可以发送到前馈神经网络。然而，在实际实现中，为了更快的处理速度，该计算是以矩阵形式完成的。既然我们已经了解了单词层面的计算直觉，现在让我们来看看矩阵形式。

Matrix Calculation of Self-Attention自注意力的矩阵计算

The first step is to calculate the Query, Key, and Value matrices. We do that by packing our embeddings into a matrix X, and multiplying it by the weight matrices we’ve trained (WQ, WK, WV).第一步是计算查询、键和值矩阵。我们通过将嵌入向量打包成矩阵 X，并将其乘以我们训练好的权重矩阵（WQ, WK, WV）来实现这一点。

Finally, since we’re dealing with matrices, we can condense steps two through six in one formula to calculate the outputs of the self-attention layer.最后，由于我们处理的是矩阵，我们可以将第二步到第六步浓缩为一个公式，来计算自注意力层的输出。

The Beast With Many Heads多头注意力机制

The paper further refined the self-attention layer by adding a mechanism called “multi-headed” attention. This improves the performance of the attention layer in two ways:论文通过增加一种称为“多头”（multi-headed）注意力的机制进一步优化了自注意力层。这在两个方面提高了注意力层的性能：

1. It expands the model’s ability to focus on different positions. Yes, in the example above, z1 contains a little bit of every other encoding, but it could be dominated by the actual word itself. If we’re translating a sentence like “The animal didn’t cross the street because it was too tired”, it would be useful to know which word “it” refers to.它扩展了模型关注不同位置的能力。是的，在上面的例子中，z1 包含了一点点其他所有编码的信息，但它可能被单词本身所主导。如果我们正在翻译像“The animal didn’t cross the street because it was too tired”这样的句子，知道“it”指的是哪个单词将非常有用。

2. It gives the attention layer multiple “representation subspaces”. As we’ll see next, with multi-headed attention we have not only one, but multiple sets of Query/Key/Value weight matrices (the Transformer uses eight attention heads, so we end up with eight sets for each encoder/decoder). Each of these sets is randomly initialized. Then, after training, each set is used to project the input embeddings (or vectors from lower encoders/decoders) into a different representation subspace.它为注意力层提供了多个“表征子空间”。正如我们接下来将看到的，通过多头注意力，我们不仅拥有一组，而是多组查询/键/值权重矩阵（Transformer 使用八个注意力头，所以每个编码器/解码器都有八组）。这些集合中的每一组都是随机初始化的。然后，在训练之后，每一组都被用来将输入嵌入（或来自较低层编码器/解码器的向量）投影到不同的表征子空间中。

If we do the same self-attention calculation we outlined above, just eight different times with different weight matrices, we end up with eight different Z matrices如果我们执行上述相同的自注意力计算，只是使用不同的权重矩阵重复八次，最终会得到八个不同的 Z 矩阵。

This leaves us with a bit of a challenge. The feed-forward layer is not expecting eight matrices – it’s expecting a single matrix (a vector for each word). So we need a way to condense these eight down into a single matrix.这给我们带来了一个挑战。前馈层并不期望接收八个矩阵——它期望接收一个单一的矩阵（每个单词对应一个向量）。因此，我们需要一种方法将这八个矩阵浓缩为一个矩阵。

How do we do that? We concat the matrices then multiply them by an additional weights matrix WO.我们该怎么做呢？我们将矩阵拼接起来，然后将它们乘以一个额外的权重矩阵 WO。

That’s pretty much all there is to multi-headed self-attention. It’s quite a handful of matrices, I realize. Let me try to put them all in one visual so we can look at them in one place多头自注意力基本上就是这样。我意识到这涉及相当多的矩阵。让我试着把它们放在一张图中，这样我们就可以在一个地方查看它们。

Now that we have touched upon attention heads, let’s revisit our example from before to see where the different attention heads are focusing as we encode the word “it” in our example sentence:既然我们已经接触了注意力头，让我们回顾一下之前的例子，看看当我们编码示例句子中的单词“it”时，不同的注意力头都在关注哪里：

If we add all the attention heads to the picture, however, things can be harder to interpret:然而，如果我们把所有的注意力头都加到图中，事情可能会变得更难解释：

Representing The Order of The Sequence Using Positional Encoding使用位置编码表示序列顺序

One thing that’s missing from the model as we have described it so far is a way to account for the order of the words in the input sequence.到目前为止，我们所描述的模型中缺少的一点是考虑输入序列中单词顺序的方法。

To address this, the transformer adds a vector to each input embedding. These vectors follow a specific pattern that the model learns, which helps it determine the position of each word, or the distance between different words in the sequence. The intuition here is that adding these values to the embeddings provides meaningful distances between the embedding vectors once they’re projected into Q/K/V vectors and during dot-product attention.为了解决这个问题，Transformer 在每个输入嵌入中添加了一个向量。这些向量遵循模型学习到的特定模式，这有助于它确定每个单词的位置，或者序列中不同单词之间的距离。这里的直觉是，将这些值添加到嵌入中，在它们被投影到 Q/K/V 向量以及在点积注意力计算过程中，提供了嵌入向量之间有意义的距离。

If we assumed the embedding has a dimensionality of 4, the actual positional encodings would look like this:如果我们假设嵌入的维度为 4，那么实际的位置编码看起来会是这样的：

What might this pattern look like?这种模式看起来是什么样的？

In the following figure, each row corresponds to a positional encoding of a vector. So the first row would be the vector we’d add to the embedding of the first word in an input sequence. Each row contains 512 values – each with a value between 1 and -1. We’ve color-coded them so the pattern is visible.在下图中，每一行对应一个向量的位置编码。因此，第一行就是我们将添加到输入序列中第一个单词的嵌入向量。每一行包含 512 个值——每个值都在 1 和 -1 之间。我们对它们进行了颜色编码，以便可以看到模式。

The formula for positional encoding is described in the paper (section 3.5). You can see the code for generating positional encodings in get_timing_signal_1d(). This is not the only possible method for positional encoding. It, however, gives the advantage of being able to scale to unseen lengths of sequences (e.g. if our trained model is asked to translate a sentence longer than any of those in our training set).位置编码的公式在论文中（第 3.5 节）有描述。你可以在 get_timing_signal_1d() 中看到生成位置编码的代码。这不是位置编码的唯一可能方法。然而，它的优势在于能够扩展到未见过的序列长度（例如，如果我们的训练模型被要求翻译一个比训练集中任何句子都长的句子）。

July 2020 Update: The positional encoding shown above is from the Tensor2Tensor implementation of the Transformer. The method shown in the paper is slightly different in that it doesn’t directly concatenate, but interweaves the two signals. The following figure shows what that looks like. Here’s the code to generate it:2020 年 7 月更新： 上面显示的位置编码来自 Transformer 的 Tensor2Tensor 实现。论文中显示的方法略有不同，它不是直接拼接，而是将两个信号交织在一起。下图显示了它的外观。这是生成它的代码：

The Residuals残差连接

One detail in the architecture of the encoder that we need to mention before moving on, is that each sub-layer (self-attention, ffnn) in each encoder has a residual connection around it, and is followed by a layer-normalization step.在继续之前，我们需要提到的编码器架构的一个细节是，每个编码器中的每个子层（自注意力、前馈网络）周围都有一个残差连接，并跟随一个层归一化（layer-normalization）步骤。

If we’re to visualize the vectors and the layer-norm operation associated with self attention, it would look like this:如果我们可视化与自注意力相关的向量和层归一化操作，它看起来会是这样的：

This goes for the sub-layers of the decoder as well. If we’re to think of a Transformer of 2 stacked encoders and decoders, it would look something like this:这也适用于解码器的子层。如果我们考虑一个由 2 个堆叠的编码器和解码器组成的 Transformer，它看起来会是这样的：

The Decoder Side解码器端

Now that we’ve covered most of the concepts on the encoder side, we basically know how the components of decoders work as well. But let’s take a look at how they work together.既然我们已经涵盖了编码器端的大部分概念，我们基本上也知道了解码器的组件是如何工作的。但让我们看看它们是如何协同工作的。

The encoder start by processing the input sequence. The output of the top encoder is then transformed into a set of attention vectors K and V. These are to be used by each decoder in its “encoder-decoder attention” layer which helps the decoder focus on appropriate places in the input sequence:编码器首先处理输入序列。最顶层编码器的输出随后被转换为一组注意力向量 K 和 V。这些向量将由每个解码器在其“编码器-解码器注意力”层中使用，这有助于解码器聚焦于输入序列中的适当位置：

The following steps repeat the process until a special symbol is reached indicating the transformer decoder has completed its output. The output of each step is fed to the bottom decoder in the next time step, and the decoders bubble up their decoding results just like the encoders did. And just like we did with the encoder inputs, we embed and add positional encoding to those decoder inputs to indicate the position of each word.以下步骤重复该过程，直到达到一个特殊的 <eos> 符号，表示 Transformer 解码器已完成其输出。每一步的输出都会在下一个时间步被馈送到最底层的解码器，解码器会像编码器一样向上冒泡它们的解码结果。正如我们对编码器输入所做的那样，我们对这些解码器输入进行嵌入并添加位置编码，以指示每个单词的位置。

The self attention layers in the decoder operate in a slightly different way than the one in the encoder:解码器中的自注意力层与编码器中的自注意力层操作方式略有不同：

In the decoder, the self-attention layer is only allowed to attend to earlier positions in the output sequence. This is done by masking future positions (setting them to -inf) before the softmax step in the self-attention calculation.在解码器中，自注意力层只允许关注输出序列中较早的位置。这是通过在自注意力计算的 Softmax 步骤之前屏蔽未来位置（将它们设置为 -inf）来实现的。

The “Encoder-Decoder Attention” layer works just like multiheaded self-attention, except it creates its Queries matrix from the layer below it, and takes the Keys and Values matrix from the output of the encoder stack.“编码器-解码器注意力”层的工作方式与多头自注意力完全相同，只是它从其下方的层创建查询矩阵（Queries），并从编码器堆栈的输出中获取键（Keys）和值（Values）矩阵。

The Final Linear and Softmax Layer最终的线性层和 Softmax 层

The decoder stack outputs a vector of floats. How do we turn that into a word? That’s the job of the final Linear layer which is followed by a Softmax Layer.解码器堆栈输出一个浮点数向量。我们如何将其转化为单词？这就是最终的线性层（Linear layer）的工作，其后跟随一个 Softmax 层。

The Linear layer is a simple fully connected neural network that projects the vector produced by the stack of decoders, into a much, much larger vector called a logits vector.线性层是一个简单的全连接神经网络，它将解码器堆栈产生的向量投影到一个更大得多的向量中，称为 logits 向量。

Let’s assume that our model knows 10,000 unique English words (our model’s “output vocabulary”) that it’s learned from its training dataset. This would make the logits vector 10,000 cells wide – each cell corresponding to the score of a unique word. That is how we interpret the output of the model followed by the Linear layer.假设我们的模型知道从训练数据集中学到的 10,000 个独特的英语单词（我们模型的“输出词汇表”）。这将使 logits 向量有 10,000 个单元宽——每个单元对应一个独特单词的分数。这就是我们如何解释模型经过线性层后的输出。

The softmax layer then turns those scores into probabilities (all positive, all add up to 1.0). The cell with the highest probability is chosen, and the word associated with it is produced as the output for this time step.然后，Softmax 层将这些分数转化为概率（全部为正，总和为 1.0）。选择概率最高的单元，并将与之关联的单词作为该时间步的输出。

Recap Of Training训练回顾

Now that we’ve covered the entire forward-pass process through a trained Transformer, it would be useful to glance at the intuition of training the model.既然我们已经涵盖了通过训练好的 Transformer 的整个前向传播过程，那么了解一下训练模型的直觉将会很有用。

During training, an untrained model would go through the exact same forward pass. But since we are training it on a labeled training dataset, we can compare its output with the actual correct output.在训练期间，一个未经训练的模型会经历完全相同的前向传播。但由于我们是在标记的训练数据集上训练它，我们可以将其输出与实际的正确输出进行比较。

To visualize this, let’s assume our output vocabulary only contains six words(“a”, “am”, “i”, “thanks”, “student”, and “<eos>” (short for ‘end of sentence’)).为了可视化这一点，假设我们的输出词汇表只包含六个单词（“a”、“am”、“i”、“thanks”、“student”和“<eos>”（“句子结束”的缩写））。

Once we define our output vocabulary, we can use a vector of the same width to indicate each word in our vocabulary. This also known as one-hot encoding. So for example, we can indicate the word “am” using the following vector:一旦我们定义了输出词汇表，我们就可以使用相同宽度的向量来表示词汇表中的每个单词。这也被称为独热编码（one-hot encoding）。例如，我们可以使用以下向量来表示单词“am”：

Following this recap, let’s discuss the model’s loss function – the metric we are optimizing during the training phase to lead up to a trained and hopefully amazingly accurate model.在这次回顾之后，让我们讨论模型的损失函数（loss function）——这是我们在训练阶段优化的指标，旨在引导模型成为一个训练有素且希望非常准确的模型。

The Loss Function损失函数

Say we are training our model. Say it’s our first step in the training phase, and we’re training it on a simple example – translating “merci” into “thanks”.假设我们正在训练我们的模型。假设这是我们训练阶段的第一步，我们正在用一个简单的例子来训练它——将“merci”翻译成“thanks”。

What this means, is that we want the output to be a probability distribution indicating the word “thanks”. But since this model is not yet trained, that’s unlikely to happen just yet.这意味着我们希望输出是一个表示单词“thanks”的概率分布。但由于该模型尚未训练，这种情况不太可能发生。

How do you compare two probability distributions? We simply subtract one from the other. For more details, look at cross-entropy and Kullback–Leibler divergence.你如何比较两个概率分布？我们只需将一个减去另一个。有关更多详细信息，请查看交叉熵（cross-entropy）和 Kullback–Leibler 散度。

But note that this is an oversimplified example. More realistically, we’ll use a sentence longer than one word. For example – input: “je suis étudiant” and expected output: “i am a student”. What this really means, is that we want our model to successively output probability distributions where:但请注意，这是一个过度简化的例子。更现实的情况是，我们将使用比一个单词更长的句子。例如——输入：“je suis étudiant”，预期输出：“i am a student”。这真正意味着，我们希望我们的模型连续输出概率分布，其中：

• Each probability distribution is represented by a vector of width vocab_size (6 in our toy example, but more realistically a number like 30,000 or 50,000)每个概率分布由一个宽度为 vocab_size 的向量表示（在我们的玩具示例中为 6，但更现实的是 30,000 或 50,000 这样的数字）
• The first probability distribution has the highest probability at the cell associated with the word “i”第一个概率分布在与单词“i”关联的单元处具有最高概率
• The second probability distribution has the highest probability at the cell associated with the word “am”第二个概率分布在与单词“am”关联的单元处具有最高概率
• And so on, until the fifth output distribution indicates ‘<end of sentence>’ symbol, which also has a cell associated with it from the 10,000 element vocabulary.依此类推，直到第五个输出分布指示“<句子结束>”符号，该符号也具有来自 10,000 个元素词汇表的关联单元。

After training the model for enough time on a large enough dataset, we would hope the produced probability distributions would look like this:在大型数据集上对模型进行足够长时间的训练后，我们希望生成的概率分布看起来像这样：

Now, because the model produces the outputs one at a time, we can assume that the model is selecting the word with the highest probability from that probability distribution and throwing away the rest. That’s one way to do it (called greedy decoding). Another way to do it would be to hold on to, say, the top two words (say, ‘I’ and ‘a’ for example), then in the next step, run the model twice: once assuming the first output position was the word ‘I’, and another time assuming the first output position was the word ‘a’, and whichever version produced less error considering both positions #1 and #2 is kept. We repeat this for positions #2 and #3…etc. This method is called “beam search”, where in our example, beam_size was two (meaning that at all times, two partial hypotheses (unfinished translations) are kept in memory), and top_beams is also two (meaning we’ll return two translations). These are both hyperparameters that you can experiment with.现在，由于模型一次产生一个输出，我们可以假设模型选择了该概率分布中概率最高的单词，并丢弃了其余部分。这是一种方法（称为贪婪解码）。另一种方法是保留，比如说，前两个单词（例如“I”和“a”），然后在下一步中，运行模型两次：一次假设第一个输出位置是单词“I”，另一次假设第一个输出位置是单词“a”，然后保留在考虑位置 #1 和 #2 后产生较小误差的版本。我们对位置 #2 和 #3 等重复此操作。这种方法称为“束搜索”（beam search），在我们的例子中，beam_size 为 2（意味着始终在内存中保留两个部分假设（未完成的翻译）），top_beams 也为 2（意味着我们将返回两个翻译）。这些都是你可以尝试调整的超参数。

Go Forth And Transform继续前行并进行转换

I hope you’ve found this a useful place to start to break the ice with the major concepts of the Transformer. If you want to go deeper, I’d suggest these next steps:希望你发现这是一个了解 Transformer 主要概念的有用起点。如果你想深入研究，我建议采取以下步骤：

• Read the Attention Is All You Need paper, the Transformer blog post (Transformer: A Novel Neural Network Architecture for Language Understanding), and the Tensor2Tensor announcement.阅读《Attention Is All You Need》论文、Transformer 博客文章（《Transformer: A Novel Neural Network Architecture for Language Understanding》）以及 Tensor2Tensor 的发布公告。
• Watch Łukasz Kaiser’s talk walking through the model and its details观看 Łukasz Kaiser 关于模型及其细节的演讲
• Play with the Jupyter Notebook provided as part of the Tensor2Tensor repo使用作为 Tensor2Tensor 仓库一部分提供的 Jupyter Notebook 进行操作
• Explore the Tensor2Tensor repo.探索 Tensor2Tensor 仓库。

Follow-up works:后续工作：

• Depthwise Separable Convolutions for Neural Machine Translation深度可分离卷积在神经机器翻译中的应用（Depthwise Separable Convolutions for Neural Machine Translation）
• One Model To Learn Them All一个学习所有任务的模型（One Model To Learn Them All）
• Discrete Autoencoders for Sequence Models序列模型的离散自编码器（Discrete Autoencoders for Sequence Models）
• Generating Wikipedia by Summarizing Long Sequences通过总结长序列生成维基百科（Generating Wikipedia by Summarizing Long Sequences）
• Image Transformer图像 Transformer（Image Transformer）
• Training Tips for the Transformer ModelTransformer 模型的训练技巧（Training Tips for the Transformer Model）
• Self-Attention with Relative Position Representations具有相对位置表征的自注意力机制（Self-Attention with Relative Position Representations）
• Fast Decoding in Sequence Models using Discrete Latent Variables使用离散潜在变量的序列模型快速解码（Fast Decoding in Sequence Models using Discrete Latent Variables）
• Adafactor: Adaptive Learning Rates with Sublinear Memory CostAdafactor：具有亚线性内存成本的自适应学习率（Adafactor: Adaptive Learning Rates with Sublinear Memory Cost）

Acknowledgements致谢

Thanks to Illia Polosukhin, Jakob Uszkoreit, Llion Jones , Lukasz Kaiser, Niki Parmar, and Noam Shazeer for providing feedback on earlier versions of this post.感谢 Illia Polosukhin、Jakob Uszkoreit、Llion Jones、Lukasz Kaiser、Niki Parmar 和 Noam Shazeer 为本文的早期版本提供反馈。

Please hit me up on Twitter for any corrections or feedback.如有任何更正或反馈，请在 Twitter 上联系我。