Update Nov 2025 nanoGPT has a new and improved cousin called nanochat. It is very likely you meant to use/find nanochat instead. nanoGPT (this repo) is now very old and deprecated but I will leave it up for posterity.2025年11月更新：nanoGPT有一个名为nanochat的新改进版本。您很可能想使用/查找nanochat。nanoGPT（此仓库）现已非常陈旧且已弃用，但我会保留它以作纪念。
The simplest, fastest repository for training/finetuning medium-sized GPTs. It is a rewrite of minGPT that prioritizes teeth over education. Still under active development, but currently the file train.py reproduces GPT-2 (124M) on OpenWebText, running on a single 8XA100 40GB node in about 4 days of training. The code itself is plain and readable: train.py is a ~300-line boilerplate training loop and model.py a ~300-line GPT model definition, which can optionally load the GPT-2 weights from OpenAI. That's it.训练/微调中型GPT的最简单、最快的仓库。它是对minGPT的重写，优先考虑实用性而非教育性。仍在积极开发中，但目前train.py文件在OpenWebText上复现了GPT-2（124M），在单个8XA100 40GB节点上运行约4天。代码本身简洁易读：train.py是一个约300行的训练循环样板，model.py是一个约300行的GPT模型定义，可以选择加载OpenAI的GPT-2权重。仅此而已。
Because the code is so simple, it is very easy to hack to your needs, train new models from scratch, or finetune pretrained checkpoints (e.g. biggest one currently available as a starting point would be the GPT-2 1.3B model from OpenAI).由于代码非常简单，很容易根据您的需求进行修改，从头训练新模型，或微调预训练检查点（例如，目前可用的最大起点是OpenAI的GPT-2 1.3B模型）。
pip install torch numpy transformers datasets tiktoken wandb tqdm
Dependencies:依赖项：
- pytorch <3pytorch <3
- numpy <3numpy <3
transformersfor huggingface transformers <3 (to load GPT-2 checkpoints)transformers 用于 huggingface transformers <3（加载GPT-2检查点）datasetsfor huggingface datasets <3 (if you want to download + preprocess OpenWebText)datasets 用于 huggingface datasets <3（如果您想下载并预处理OpenWebText）tiktokenfor OpenAI's fast BPE code <3tiktoken 用于OpenAI的快速BPE代码 <3wandbfor optional logging <3wandb 用于可选日志记录 <3tqdmfor progress bars <3tqdm 用于进度条 <3
If you are not a deep learning professional and you just want to feel the magic and get your feet wet, the fastest way to get started is to train a character-level GPT on the works of Shakespeare. First, we download it as a single (1MB) file and turn it from raw text into one large stream of integers:如果您不是深度学习专业人士，只想感受魔力并入门，最快的方法是训练一个字符级GPT在莎士比亚作品上。首先，我们将其下载为单个（1MB）文件，并将其从原始文本转换为一个大的整数流：
python data/shakespeare_char/prepare.pyThis creates a train.bin and val.bin in that data directory. Now it is time to train your GPT. The size of it very much depends on the computational resources of your system:这会在该数据目录中创建train.bin和val.bin。现在是训练您的GPT的时候了。其大小很大程度上取决于您系统的计算资源：
I have a GPU. Great, we can quickly train a baby GPT with the settings provided in the config/train_shakespeare_char.py config file:我有GPU。太好了，我们可以使用config/train_shakespeare_char.py配置文件中提供的设置快速训练一个婴儿GPT：
python train.py config/train_shakespeare_char.pyIf you peek inside it, you'll see that we're training a GPT with a context size of up to 256 characters, 384 feature channels, and it is a 6-layer Transformer with 6 heads in each layer. On one A100 GPU this training run takes about 3 minutes and the best validation loss is 1.4697. Based on the configuration, the model checkpoints are being written into the --out_dir directory out-shakespeare-char. So once the training finishes we can sample from the best model by pointing the sampling script at this directory:如果您查看内部，您会看到我们正在训练一个上下文大小最多256个字符、384个特征通道的GPT，它是一个6层Transformer，每层有6个头。在一台A100 GPU上，此训练运行大约需要3分钟，最佳验证损失为1.4697。根据配置，模型检查点被写入--out_dir目录out-shakespeare-char。因此，一旦训练完成，我们可以通过将采样脚本指向此目录来从最佳模型采样：
python sample.py --out_dir=out-shakespeare-charThis generates a few samples, for example:这会生成一些样本，例如：
ANGELO:
And cowards it be strawn to my bed,
And thrust the gates of my threats,
Because he that ale away, and hang'd
An one with him.
DUKE VINCENTIO:
I thank your eyes against it.
DUKE VINCENTIO:
Then will answer him to save the malm:
And what have you tyrannous shall do this?
DUKE VINCENTIO:
If you have done evils of all disposition
To end his power, the day of thrust for a common men
That I leave, to fight with over-liking
Hasting in a roseman.
lol ¯\_(ツ)_/¯. Not bad for a character-level model after 3 minutes of training on a GPU. Better results are quite likely obtainable by instead finetuning a pretrained GPT-2 model on this dataset (see finetuning section later).lol ¯\_(ツ)_/¯。对于一个在GPU上训练3分钟的字符级模型来说还不错。通过在此数据集上微调预训练的GPT-2模型（参见后面的微调部分），很可能获得更好的结果。
I only have a macbook (or other cheap computer). No worries, we can still train a GPT but we want to dial things down a notch. I recommend getting the bleeding edge PyTorch nightly (select it here when installing) as it is currently quite likely to make your code more efficient. But even without it, a simple train run could look as follows:我只有一台MacBook（或其他廉价电脑）。别担心，我们仍然可以训练GPT，但需要降低要求。我建议获取最新的PyTorch nightly版本（安装时在此处选择），因为它目前很可能使您的代码更高效。但即使没有它，一个简单的训练运行可能如下所示：
python train.py config/train_shakespeare_char.py --device=cpu --compile=False --eval_iters=20 --log_interval=1 --block_size=64 --batch_size=12 --n_layer=4 --n_head=4 --n_embd=128 --max_iters=2000 --lr_decay_iters=2000 --dropout=0.0Here, since we are running on CPU instead of GPU we must set both --device=cpu and also turn off PyTorch 2.0 compile with --compile=False. Then when we evaluate we get a bit more noisy but faster estimate (--eval_iters=20, down from 200), our context size is only 64 characters instead of 256, and the batch size only 12 examples per iteration, not 64. We'll also use a much smaller Transformer (4 layers, 4 heads, 128 embedding size), and decrease the number of iterations to 2000 (and correspondingly usually decay the learning rate to around max_iters with --lr_decay_iters). Because our network is so small we also ease down on regularization (--dropout=0.0). This still runs in about ~3 minutes, but gets us a loss of only 1.88 and therefore also worse samples, but it's still good fun:这里，由于我们在CPU而不是GPU上运行，我们必须同时设置--device=cpu并关闭PyTorch 2.0编译，使用--compile=False。然后，当我们评估时，我们得到更嘈杂但更快的估计（--eval_iters=20，从200降低），我们的上下文大小只有64个字符而不是256，批量大小仅为每次迭代12个示例，而不是64。我们还将使用更小的Transformer（4层，4个头，128嵌入大小），并将迭代次数减少到2000（并相应地通常将学习率衰减到max_iters，使用--lr_decay_iters）。由于我们的网络非常小，我们还降低了正则化（--dropout=0.0）。这仍然在大约3分钟内运行，但损失仅为1.88，因此样本也更差，但仍然很有趣：
python sample.py --out_dir=out-shakespeare-char --device=cpuGenerates samples like this:生成如下样本：
GLEORKEN VINGHARD III:
Whell's the couse, the came light gacks,
And the for mought you in Aut fries the not high shee
bot thou the sought bechive in that to doth groan you,
No relving thee post mose the wear
Not bad for ~3 minutes on a CPU, for a hint of the right character gestalt. If you're willing to wait longer, feel free to tune the hyperparameters, increase the size of the network, the context length (--block_size), the length of training, etc.在CPU上大约3分钟，对于正确的字符形态提示来说还不错。如果您愿意等待更长时间，请随意调整超参数，增加网络大小、上下文长度（--block_size）、训练时长等。
Finally, on Apple Silicon Macbooks and with a recent PyTorch version make sure to add --device=mps (short for "Metal Performance Shaders"); PyTorch then uses the on-chip GPU that can significantly accelerate training (2-3X) and allow you to use larger networks. See Issue 28 for more.最后，在Apple Silicon MacBook上，使用最新的PyTorch版本，请确保添加--device=mps（“Metal Performance Shaders”的缩写）；PyTorch随后使用片上GPU，可以显著加速训练（2-3倍），并允许您使用更大的网络。更多信息请参见Issue 28。
A more serious deep learning professional may be more interested in reproducing GPT-2 results. So here we go - we first tokenize the dataset, in this case the OpenWebText, an open reproduction of OpenAI's (private) WebText:更严肃的深度学习专业人士可能对复现GPT-2结果更感兴趣。那么开始吧——我们首先对数据集进行分词，这里是OpenWebText，OpenAI（私有）WebText的开放复现：
python data/openwebtext/prepare.pyThis downloads and tokenizes the OpenWebText dataset. It will create a train.bin and val.bin which holds the GPT2 BPE token ids in one sequence, stored as raw uint16 bytes. Then we're ready to kick off training. To reproduce GPT-2 (124M) you'll want at least an 8X A100 40GB node and run:这会下载并对OpenWebText数据集进行分词。它将创建一个train.bin和val.bin，其中包含一个序列中的GPT2 BPE token ID，存储为原始uint16字节。然后我们准备开始训练。要复现GPT-2（124M），您至少需要一个8X A100 40GB节点并运行：
torchrun --standalone --nproc_per_node=8 train.py config/train_gpt2.pyThis will run for about 4 days using PyTorch Distributed Data Parallel (DDP) and go down to loss of ~2.85. Now, a GPT-2 model just evaluated on OWT gets a val loss of about 3.11, but if you finetune it it will come down to ~2.85 territory (due to an apparent domain gap), making the two models ~match.这将使用PyTorch分布式数据并行（DDP）运行大约4天，损失降至约2.85。现在，仅在OWT上评估的GPT-2模型验证损失约为3.11，但如果您微调它，它会降至约2.85区域（由于明显的领域差距），使两个模型大致匹配。
If you're in a cluster environment and you are blessed with multiple GPU nodes you can make GPU go brrrr e.g. across 2 nodes like:如果您在集群环境中，并且拥有多个GPU节点，您可以让GPU飞速运行，例如跨2个节点：
# Run on the first (master) node with example IP 123.456.123.456:
torchrun --nproc_per_node=8 --nnodes=2 --node_rank=0 --master_addr=123.456.123.456 --master_port=1234 train.py
# Run on the worker node:
torchrun --nproc_per_node=8 --nnodes=2 --node_rank=1 --master_addr=123.456.123.456 --master_port=1234 train.pyIt is a good idea to benchmark your interconnect (e.g. iperf3). In particular, if you don't have Infiniband then also prepend NCCL_IB_DISABLE=1 to the above launches. Your multinode training will work, but most likely crawl. By default checkpoints are periodically written to the --out_dir. We can sample from the model by simply python sample.py.对您的互连进行基准测试（例如iperf3）是个好主意。特别是，如果您没有Infiniband，那么还要在上述启动命令前加上NCCL_IB_DISABLE=1。您的多节点训练可以工作，但很可能很慢。默认情况下，检查点会定期写入--out_dir。我们可以通过简单的python sample.py从模型采样。
Finally, to train on a single GPU simply run the python train.py script. Have a look at all of its args, the script tries to be very readable, hackable and transparent. You'll most likely want to tune a number of those variables depending on your needs.最后，要在单个GPU上训练，只需运行python train.py脚本。查看其所有参数，该脚本力求非常易读、可修改和透明。您很可能需要根据需求调整其中许多变量。
OpenAI GPT-2 checkpoints allow us to get some baselines in place for openwebtext. We can get the numbers as follows:OpenAI GPT-2检查点使我们能够在openwebtext上建立一些基线。我们可以按如下方式获取数字：
$ python train.py config/eval_gpt2.py
$ python train.py config/eval_gpt2_medium.py
$ python train.py config/eval_gpt2_large.py
$ python train.py config/eval_gpt2_xl.pyand observe the following losses on train and val:并观察训练和验证上的以下损失：
| model | params | train loss | val loss |
|---|---|---|---|
| gpt2 | 124M | 3.11 | 3.12 |
| gpt2-medium | 350M | 2.85 | 2.84 |
| gpt2-large | 774M | 2.66 | 2.67 |
| gpt2-xl | 1558M | 2.56 | 2.54 |
However, we have to note that GPT-2 was trained on (closed, never released) WebText, while OpenWebText is just a best-effort open reproduction of this dataset. This means there is a dataset domain gap. Indeed, taking the GPT-2 (124M) checkpoint and finetuning on OWT directly for a while reaches loss down to ~2.85. This then becomes the more appropriate baseline w.r.t. reproduction.然而，我们必须注意，GPT-2是在（封闭的、从未发布的）WebText上训练的，而OpenWebText只是该数据集的最佳努力开放复现。这意味着存在数据集领域差距。实际上，获取GPT-2（124M）检查点并在OWT上直接微调一段时间，损失降至约2.85。这成为关于复现的更合适基线。
Finetuning is no different than training, we just make sure to initialize from a pretrained model and train with a smaller learning rate. For an example of how to finetune a GPT on new text go to data/shakespeare and run prepare.py to download the tiny shakespeare dataset and render it into a train.bin and val.bin, using the OpenAI BPE tokenizer from GPT-2. Unlike OpenWebText this will run in seconds. Finetuning can take very little time, e.g. on a single GPU just a few minutes. Run an example finetuning like:微调与训练没有区别，我们只需确保从预训练模型初始化，并使用较小的学习率进行训练。有关如何在新文本上微调GPT的示例，请转到data/shakespeare并运行prepare.py以下载小型莎士比亚数据集，并使用GPT-2的OpenAI BPE分词器将其渲染为train.bin和val.bin。与OpenWebText不同，这将在几秒钟内运行。微调所需时间非常短，例如在单个GPU上只需几分钟。运行一个微调示例，如下所示：
python train.py config/finetune_shakespeare.pyThis will load the config parameter overrides in config/finetune_shakespeare.py (I didn't tune them much though). Basically, we initialize from a GPT2 checkpoint with init_from and train as normal, except shorter and with a small learning rate. If you're running out of memory try decreasing the model size (they are {'gpt2', 'gpt2-medium', 'gpt2-large', 'gpt2-xl'}) or possibly decreasing the block_size (context length). The best checkpoint (lowest validation loss) will be in the out_dir directory, e.g. in out-shakespeare by default, per the config file. You can then run the code in sample.py --out_dir=out-shakespeare:这将加载config/finetune_shakespeare.py中的配置参数覆盖（不过我没有过多调整它们）。基本上，我们使用init_from从GPT2检查点初始化，并像往常一样训练，只是时间更短且学习率较小。如果内存不足，请尝试减小模型大小（它们是{'gpt2', 'gpt2-medium', 'gpt2-large', 'gpt2-xl'}）或可能减小block_size（上下文长度）。最佳检查点（最低验证损失）将位于out_dir目录中，例如默认情况下在out-shakespeare中，根据配置文件。然后您可以运行sample.py --out_dir=out-shakespeare中的代码：
THEODORE:
Thou shalt sell me to the highest bidder: if I die,
I sell thee to the first; if I go mad,
I sell thee to the second; if I
lie, I sell thee to the third; if I slay,
I sell thee to the fourth: so buy or sell,
I tell thee again, thou shalt not sell my
possession.
JULIET:
And if thou steal, thou shalt not sell thyself.
THEODORE:
I do not steal; I sell the stolen goods.
THEODORE:
Thou know'st not what thou sell'st; thou, a woman,
Thou art ever a victim, a thing of no worth:
Thou hast no right, no right, but to be sold.
Whoa there, GPT, entering some dark place over there. I didn't really tune the hyperparameters in the config too much, feel free to try!哇，GPT，进入了某个黑暗的地方。我没有过多调整配置中的超参数，请随意尝试！
Use the script sample.py to sample either from pre-trained GPT-2 models released by OpenAI, or from a model you trained yourself. For example, here is a way to sample from the largest available gpt2-xl model:使用脚本sample.py从OpenAI发布的预训练GPT-2模型或您自己训练的模型进行采样。例如，以下是从最大的可用gpt2-xl模型采样的方法：
python sample.py \
--init_from=gpt2-xl \
--start="What is the answer to life, the universe, and everything?" \
--num_samples=5 --max_new_tokens=100If you'd like to sample from a model you trained, use the --out_dir to point the code appropriately. You can also prompt the model with some text from a file, e.g. python sample.py --start=FILE:prompt.txt.如果您想从自己训练的模型采样，请使用--out_dir适当指向代码。您还可以使用文件中的文本提示模型，例如python sample.py --start=FILE:prompt.txt。
For simple model benchmarking and profiling, bench.py might be useful. It's identical to what happens in the meat of the training loop of train.py, but omits much of the other complexities.对于简单的模型基准测试和分析，bench.py可能有用。它与train.py训练循环核心部分发生的情况相同，但省略了许多其他复杂性。
Note that the code by default uses PyTorch 2.0. At the time of writing (Dec 29, 2022) this makes torch.compile() available in the nightly release. The improvement from the one line of code is noticeable, e.g. cutting down iteration time from ~250ms / iter to 135ms / iter. Nice work PyTorch team!请注意，代码默认使用PyTorch 2.0。在撰写本文时（2022年12月29日），这使得torch.compile()在nightly版本中可用。这一行代码的改进是显著的，例如将迭代时间从约250ms/iter减少到135ms/iter。PyTorch团队干得好！
- Investigate and add FSDP instead of DDP研究并添加FSDP替代DDP
- Eval zero-shot perplexities on standard evals (e.g. LAMBADA? HELM? etc.)在标准评估（例如LAMBADA？HELM？等）上评估零样本困惑度
- Finetune the finetuning script, I think the hyperparams are not great微调微调脚本，我认为超参数不太好
- Schedule for linear batch size increase during training训练期间线性批量大小增加的计划
- Incorporate other embeddings (rotary, alibi)整合其他嵌入（旋转位置编码、ALiBi）
- Separate out the optim buffers from model params in checkpoints I think我认为在检查点中将优化器缓冲区与模型参数分开
- Additional logging around network health (e.g. gradient clip events, magnitudes)关于网络健康状况的额外日志记录（例如梯度裁剪事件、幅度）
- Few more investigations around better init etc.关于更好初始化等的更多研究
Note that by default this repo uses PyTorch 2.0 (i.e. torch.compile). This is fairly new and experimental, and not yet available on all platforms (e.g. Windows). If you're running into related error messages try to disable this by adding --compile=False flag. This will slow down the code but at least it will run.请注意，默认情况下此仓库使用PyTorch 2.0（即torch.compile）。这是相当新且实验性的，尚未在所有平台上可用（例如Windows）。如果您遇到相关错误消息，请尝试通过添加--compile=False标志禁用它。这会减慢代码速度，但至少可以运行。
For some context on this repository, GPT, and language modeling it might be helpful to watch my Zero To Hero series. Specifically, the GPT video is popular if you have some prior language modeling context.关于此仓库、GPT和语言建模的一些背景知识，观看我的Zero To Hero系列可能会有所帮助。具体来说，如果您有一些先前的语言建模背景，GPT视频很受欢迎。
For more questions/discussions feel free to stop by #nanoGPT on Discord:更多问题/讨论，请随时加入Discord上的#nanoGPT：
All nanoGPT experiments are powered by GPUs on Lambda labs, my favorite Cloud GPU provider. Thank you Lambda labs for sponsoring nanoGPT!所有nanoGPT实验均由Lambda labs的GPU提供支持，Lambda labs是我最喜欢的云GPU提供商。感谢Lambda labs赞助nanoGPT！
