Scaling laws are one of the most critical empirical findings in deep learning. The observation is simple in form: the training loss $L$ decreases predictably as we scale up model size $N$, dataset size $D$, and compute $C$, following a power-law curve, which appears as a straight line on a log-log plot. We can view scaling laws as a framework for describing the relationship between compute, loss, model size and data; at its core, it is about how to allocate precious compute optimally between $N$ and $D$.缩放定律是深度学习中最关键的实证发现之一。其观察形式简单：训练损失L随着模型规模N、数据集大小D和计算量C的扩大而可预测地降低，遵循幂律曲线，在对数-对数图上表现为一条直线。我们可以将缩放定律视为描述计算量、损失、模型大小和数据之间关系的框架；其核心在于如何在N和D之间最优地分配宝贵的计算资源。

This predictability makes scaling laws highly valuable in practice. A common workflow is to fit scaling laws on a handful of small runs and then extrapolate to estimate the token and compute requirements for larger models.这种可预测性使得缩放定律在实践中极具价值。常见的工作流程是在少量小规模运行上拟合缩放定律，然后外推以估计更大模型的token和计算需求。

Symbol	Note
$N$	Model size, measured in parameter count.
$D$	Training dataset size, usually measured in token count.
$C$	Training compute in FLOPs. As a useful approximation, $C\approx 6ND$ (Kaplan et al. 2020), where $2ND$ accounts for the forward pass and $4ND$ for backpropagation.
$E$	Irreducible loss
$L,\hat{L}(.)$	Test loss / test loss prediction function; can also refer to training loss, since they are strongly correlated.
$ϵ$	Generalization error.

Early days: ML loss predictability早期：机器学习损失的可预测性#

The predictability of generalization error with scale had already been investigated before scaling laws became a mainstream concept.在缩放定律成为主流概念之前，泛化误差随规模的可预测性已被研究。

Amari et al. (1992) derived four types of learning curves using a Bayesian approach and the annealed approximation.Amari等人（1992）使用贝叶斯方法和退火近似推导了四种学习曲线。

1. Deterministic learning algorithm, noiseless data, one unique solution: $ϵ\sim c\cdot D^{-1}$, where $c$ is some constant.确定性学习算法，无噪声数据，唯一解：ϵ∼c⋅D−1，其中c为常数。
2. Deterministic learning algorithm, noiseless data, multiple equivalent solutions: $ϵ\sim c\cdot D^{-2}$; the learning is faster with each new data point, because the model only learns the optimal manifold of parameters, instead of finding the single solution point.确定性学习算法，无噪声数据，多个等价解：ϵ∼c⋅D−2；每个新数据点学习更快，因为模型只学习参数的最优流形，而非寻找单一解点。
3. Deterministic learning algorithm, noisy data: $ϵ\sim c\cdot D^{-1/2}$; noises in data make learning harder.确定性学习算法，有噪声数据：ϵ∼c⋅D−1/2；数据中的噪声使学习更困难。
4. Stochastic learning algorithm, noisy data: $ϵ\sim c\cdot D^{-1}+E$; here the irreducible loss $E$ is the residual error that a stochastic learner cannot reduce further, for example when the model runs out of capacity on large data. All four types of learning curves follow a power law:随机学习算法，有噪声数据：ϵ∼c⋅D−1+E；这里不可约损失E是随机学习器无法进一步降低的残差误差，例如当模型在大数据上耗尽容量时。 所有四种学习曲线都遵循幂律：

where $E$ can be 0 and $\alpha =-2,-1,-1/2$. Although their theoretical setup is based on a simplified binary classification task, it points in a useful direction for building empirical ML loss prediction models.其中E可以为0，α=−2,−1,−1/2。尽管其理论设置基于简化的二分类任务，但为构建实证机器学习损失预测模型指明了有用方向。

One of the earliest empirical studies by Hestness et al. (2017) explained the relationship between generalization error, model size and data. For a given training data size, they identified the best-fit model size via grid search and then plotted loss against training dataset size. Across four different domains in deep learning (neural machine translation, image classification, language modeling, and speech recognition), a recurring pattern was observed where:Hestness等人（2017）最早进行的实证研究之一解释了泛化误差、模型大小和数据之间的关系。对于给定的训练数据大小，他们通过网格搜索确定最佳拟合模型大小，然后绘制损失与训练数据集大小的关系图。在深度学习的四个不同领域（神经机器翻译、图像分类、语言建模和语音识别）中，观察到一种重复出现的模式：

• Generalization error scales as a power law across a set of factors (e.g. data size).泛化误差随一组因素（如数据大小）呈幂律缩放。
• Model improvements shift the error curve but do not seem to affect the power-law exponent.模型改进会移动误差曲线，但似乎不影响幂律指数。
• Interestingly, architecture changes the offset ($E$) of the power-law fit but does not change the exponent ($\alpha$). The slope of the power law appears to be a property of the problem domain rather than the model architecture.有趣的是，架构改变了幂律拟合的偏移量（E），但不改变指数（α）。幂律的斜率似乎是问题域的特性，而非模型架构的特性。
• The number of model parameters $N$ needed to fit a dataset of size $D$ also scales as a power law.拟合大小为D的数据集所需的模型参数数量N也呈幂律缩放。

A conceptual illustration breaks the learning curve into three stages. In the small-data region, when there are not enough learning signals, the model performs only slightly better than random guessing. In the middle (“power-law region”), we observe a power-law relationship between loss, data, and model size. The final irreducible-error region can be attributed to factors such as noise in the data.概念性图示将学习曲线分为三个阶段。在小数据区域，当学习信号不足时，模型性能仅略优于随机猜测。在中间区域（“幂律区域”），我们观察到损失、数据和模型大小之间的幂律关系。最终不可约误差区域可归因于数据中的噪声等因素。

Rosenfeld et al. (2020) pushed this further by trying to model error as a joint function of both model size $N$ and data size $D$, across a diverse set of architectures (ResNet, WRN, LSTM, Transformer) and optimizers (Adam, SGD variants). Empirically they observed that, holding one axis fixed, the error decays as a power law in the other:Rosenfeld等人（2020）进一步推进，尝试将误差建模为模型大小N和数据大小D的联合函数，涵盖多种架构（ResNet、WRN、LSTM、Transformer）和优化器（Adam、SGD变体）。实证观察到，固定一个轴，误差随另一个轴呈幂律衰减：

which can be combined into a joint form:可合并为联合形式：

where $A>0,B>0,\alpha \ge 0,\beta \ge 0$ are scalar constants and $E$ is not dependent on either $N$ or $D$.其中A>0，B>0，α≥0，β≥0为标量常数，E不依赖于N或D。

Thus, they can build a prediction model in the form of a simple parametric function with $\mathit{\theta }=⟨A,B,E,\alpha ,\beta ⟩$ to predict the expected loss for $(D,N)$ > certain thresholds by only training on a set of smaller training configs, $(D,N)$ < certain thresholds.因此，他们可以构建一个简单参数函数形式的预测模型，参数θ=⟨A,B,E,α,β⟩，仅通过在一组较小的训练配置（D,N）<某些阈值上训练，来预测（D,N）>某些阈值的预期损失。

Side note: These early works lean on classical learning-theory intuition like the VC dimension (the cardinality of the largest set of points a model can shatter) as a proxy for capacity, but in modern deep learning work the VC dimension is often too coarse to explain the behavior and the empirical power laws turned out to be much cleaner and more practical than the worst-case bounds that theory provides.附注：这些早期工作依赖于经典学习理论直觉，如VC维（模型能打散的最大点集基数）作为容量的代理，但在现代深度学习中，VC维通常过于粗糙，无法解释行为，而实证幂律比理论提供的最坏情况界限更清晰、更实用。

Scaling Laws in Data-Infinite Region数据无限区域的缩放定律#

Kaplan et al.’s Scaling LawsKaplan等人的缩放定律#

Kaplan et al. (2020) popularized the concept of scaling laws in the language modeling community. They found that the cross-entropy test loss $L$ scales as a power law with each of model size $N$ (excluding embedding layers), dataset size $D$, and training compute $C$ across many orders of magnitude. The findings are aligned with early work in the last section, but Kaplan et al. formalized the concept with a focus on Transformer language models and empirical experimentation at a larger scale, with model size ranging from 768M to 1.5B non-embedding parameters and dataset size from 22M to 23B tokens. All training runs in the paper used a learning rate schedule with a 3000 step linear warmup, followed by a cosine decay to zero.Kaplan等人（2020）在语言建模社区推广了缩放定律的概念。他们发现交叉熵测试损失L随模型大小N（不包括嵌入层）、数据集大小D和训练计算量C分别呈幂律缩放，跨越多个数量级。这些发现与上一节早期工作一致，但Kaplan等人通过关注Transformer语言模型和更大规模的实证实验（模型大小从768M到1.5B非嵌入参数，数据集大小从22M到23B token）形式化了这一概念。论文中所有训练运行使用学习率调度：3000步线性预热，随后余弦衰减至零。

List of key findings:关键发现列表：

• The loss $L$ scales as a power law with $N$, $D$, and $C$ individually; for optimal performance all three must scale in tandem.损失L分别随N、D和C呈幂律缩放；为获得最佳性能，三者必须同步缩放。
• Training curves follow predictable power laws whose parameters are roughly independent of model size.训练曲线遵循可预测的幂律，其参数大致独立于模型大小。
• Larger models are more sample-efficient, meaning that they reach a given loss with fewer optimization steps and fewer data points than small models.更大的模型样本效率更高，即它们以更少的优化步骤和数据点达到给定的损失。
• Architectural details (width, aspect ratio, etc.) matter less than sheer scale.架构细节（宽度、纵横比等）不如纯粹规模重要。
• Train loss and test loss are positively correlated. (Sounds trivial but this is the foundation for pretraining work. On the other hand, whether pretraining loss improvement transfers to posttraining evaluation needs separate studies.)训练损失和测试损失正相关。（听起来平凡，但这是预训练工作的基础。另一方面，预训练损失的改善是否转移到后训练评估需要单独研究。）
• Given a fixed compute budget, it is more efficient to train a very large model and stop before convergence than to train a smaller model all the way to convergence. This finding is where the Chinchilla scaling laws (the next section) disagree: Kaplan et al. overestimated the optimal model size as their fitted exponent was larger.在固定计算预算下，训练一个非常大的模型并在收敛前停止，比训练一个较小的模型直至收敛更高效。这一发现与Chinchilla缩放定律（下一节）存在分歧：Kaplan等人高估了最优模型大小，因为其拟合指数更大。

They summarize the joint dependence on $N$ and $D$ in a single equation:他们将N和D的联合依赖关系总结为一个方程：

A nice consequence of this form is that the extent of overfitting (i.e. model is complex or data is small) depends predominantly on the ratio $N^{\alpha /\beta }/D$, which indicates that the data needs to grow in a specific proportion to the growth of the model size to avoid training being data-limited.这种形式的一个良好结果是，过拟合程度（即模型复杂或数据小）主要取决于比率Nα/β/D，这表明数据需要以特定比例随模型大小增长，以避免训练受数据限制。

The most influential and, in hindsight, most contested conclusion was the compute-optimal allocation. Kaplan et al. found $N_{\text{opt}}\propto C^{0.73}$ and concluded that model size should grow faster than dataset size. Concretely, for a 10x increase in compute they suggested scaling the model size by ~5.5x but the training tokens by only ~1.8x. The Chinchilla paper would later overturn this recommendation, arguing that it leaves large models badly undertrained.最具影响力且事后看来最具争议的结论是计算最优分配。Kaplan等人发现Nopt∝C0.73，并得出结论：模型大小应比数据集大小增长更快。具体来说，对于10倍的计算量增加，他们建议将模型大小扩大约5.5倍，但训练token仅增加约1.8倍。Chinchilla论文后来推翻了这一建议，认为这导致大型模型严重训练不足。

Another useful analysis in Kaplan et al. approximates the number of training FLOPs needed based on $D$ and $N$. Each multiply-add is counted as ~2 FLOPs.Kaplan等人中的另一个有用分析基于D和N近似了所需的训练FLOPs数量。每次乘法-加法计数约为2 FLOPs。

Given a standard config where $d_{\text{attn}}=d_{\text{model}}=d_{\text{ff}}/4$, and excluding embedding layers from $N$ and the per-token forward compute:给定标准配置dattn=dmodel=dff/4，并排除嵌入层N和每token前向计算：

Then we count backward-pass FLOPs as twice the forward-pass FLOPs, because backpropagation runs two matrix multiplications, for gradients with respect to the input activations and the weights, respectively. Thus, in total, the training FLOPs per token are approximately $6N$, and the total FLOPs for training over $D$ tokens are $C\approx 6ND$.然后我们将反向传播FLOPs计为前向传播FLOPs的两倍，因为反向传播运行两个矩阵乘法，分别用于输入激活和权重的梯度。因此，每token的训练FLOPs约为6N，在D个token上训练的总FLOPs约为C≈6ND。

Chinchilla Scaling LawsChinchilla缩放定律#

The Chinchilla paper (Hoffmann et al. 2022) studied the relationship between the optimal model size $N$ (total parameters, including embeddings) and the number of tokens $D$ under a fixed compute budget $C$ with a more careful experimental design and arrived at a somewhat different answer from Kaplan et al..Chinchilla论文（Hoffmann等人，2022）研究了在固定计算预算C下，最优模型大小N（总参数，包括嵌入）与token数量D之间的关系，采用了更仔细的实验设计，得出了与Kaplan等人略有不同的答案。

The central question is on the best strategy to allocate resources given a constraint $\text{FLOPs}(N,D)=C\approx 6ND$. In other words, when we have only limited FLOPs (a given number of GPUs running for a given period of time), how should we choose between more data tokens and more model parameters?核心问题是在约束FLOPs(N,D)=C≈6ND下分配资源的最佳策略。换句话说，当我们只有有限的FLOPs（给定数量的GPU运行给定时间）时，如何在更多数据token和更多模型参数之间选择？

The Chinchilla paper presented three neatly designed methods for scaling laws fitting.Chinchilla论文提出了三种精心设计的缩放定律拟合方法。

The empirical experiments scanned over 400 models, with sizes from 70M to over 16B parameters and training tokens from 5B to 500B. The experiments were under the assumption that every training token is unique (the infinite-data regime). All runs used a cosine learning-rate schedule decaying by 10x over the training horizon. Sweeping over model sizes traces out the compute-optimal frontier.实证实验扫描了400多个模型，参数大小从70M到超过16B，训练token从5B到500B。实验假设每个训练token都是唯一的（无限数据机制）。所有运行使用余弦学习率调度，在训练周期内衰减10倍。扫描模型大小描绘出计算最优前沿。

Method 1: Fix model sizes, vary the token budget方法1：固定模型大小，改变token预算#

For each parameter count $N$, train several runs with different token budgets, and record the minimal loss achieved per FLOP budget $C$.对于每个参数数量N，使用不同的token预算训练多次运行，并记录每个FLOP预算C下达到的最小损失。

Method 2: IsoFLOP profiles方法2：等FLOP曲线#

Fix a compute budget $C$ and plot the final loss against parameter count $N$. Each iso-FLOP curve is roughly a parabola in log-space, and its minimum flags the optimal model size for that compute budget. Then repeating across budgets traces a power-law line in the plot.固定计算预算C，绘制最终损失与参数数量N的关系图。每条等FLOP曲线在对数空间中大致呈抛物线，其最小值指示该计算预算下的最优模型大小。然后在不同预算上重复，在图中描绘出一条幂律线。

Method 3: Parametric fit方法3：参数化拟合#

Fit the same parametric function as in Rosenfeld et al. (2020) directly,直接拟合与Rosenfeld等人（2020）相同的参数化函数，

We can actually get a closed form approximation of optimal $N_{\text{opt}}(C),D_{\text{opt}}(C)$ by minimizing $\hat{L}(N,D)$ under the constraint $\text{FLOPs}(N,D)=C\approx 6ND$.我们实际上可以通过在约束FLOPs(N,D)=C≈6ND下最小化L^(N,D)来获得最优Nopt(C),Dopt(C)的闭式近似。

First let’s reduce the expression to contain only $N$:首先将表达式简化为仅包含N：

When $\alpha \approx \beta$, model size and training tokens should scale at equal rates.当α≈β时，模型大小和训练token应以相同速率缩放。

To find the optimal $\mathit{\theta }=⟨A,B,E,\alpha ,\beta ⟩$, the Chinchilla paper adopts a Huber loss (robust to outliers; $\delta ={10}^{-3}$) and the L-BFGS algorithm (good for curve fitting with a small number of parameters).为找到最优θ=⟨A,B,E,α,β⟩，Chinchilla论文采用Huber损失（对异常值鲁棒；δ=10−3）和L-BFGS算法（适用于参数较少的曲线拟合）。

Chinchilla arrives at its answer through three complementary methods whose final results agree with each other, and this is part of why the result was quite convincing.Chinchilla通过三种互补方法得出答案，最终结果相互一致，这也是结果相当有说服力的部分原因。

The claim in the Chinchilla paper that most large models (at the time, ~2022) were undertrained is supported by a famous demonstration: under the same compute budget as Gopher (Rae et al. 2021; 280B parameter count, 300B token budget), they trained Chinchilla (70B parameter count, 1.4T token budget), a model 4x smaller but trained on roughly 4x more tokens and it outperformed Gopher across the board.Chinchilla论文声称大多数大型模型（当时约2022年）训练不足，这一说法得到了一个著名演示的支持：在与Gopher相同的计算预算下（Rae等人，2021；280B参数，300B token预算），他们训练了Chinchilla（70B参数，1.4T token预算），一个模型小4倍但训练token多约4倍，并在所有方面优于Gopher。

Reconciling Kaplan and Chinchilla调和Kaplan和Chinchilla#

The Chinchilla scaling laws disagree with Kaplan et al. as follows:Chinchilla缩放定律与Kaplan等人的分歧如下：

• Instead of “grow the model faster than the data” ($N_{\text{opt}}\propto C^{0.73}$), for every doubling of model size, you should also double the number of training tokens ($N_{\text{opt}}\propto C^{0.5}$).不是“模型增长快于数据”（Nopt∝C0.73），而是模型大小每翻倍，训练token数量也应翻倍（Nopt∝C0.5）。
• Instead of “train a big model and stop before convergence,” you should train a smaller model on more data.不是“训练大模型并在收敛前停止”，而是训练较小的模型在更多数据上。

Both papers still agree on the same underlying principle, but they disagree on where the optimal size-vs-token tradeoff lies. Why do they disagree so much?两篇论文仍同意相同的基本原则，但在最优规模与token权衡的位置上存在分歧。为什么分歧如此之大？

Difference 1: Kaplan et al. experimented mostly on small models. Kaplan et al. experimented mostly on smaller models, while the Chinchilla paper’s experiments reached more than 10x larger scales. When we extrapolate in log-log space, a small difference in the fit can result in large differences (See toy simulation).差异1：Kaplan等人主要在小模型上实验。 Kaplan等人主要在小模型上实验，而Chinchilla论文的实验规模达到10倍以上。在对数-对数空间中外推时，拟合中的微小差异可能导致巨大差异（参见玩具模拟）。

Difference 2: Embedding parameter count matters for small models. In the small-parameter regime, embedding parameters are a non-negligible fraction of the total and thus counting them or not matters. Pearce & Song (2024) did a thorough analysis along this line. Let’s use $N_{\setminus E},C_{\setminus E}$ to denote model size and compute when embedding is excluded and use $N,C$ to count total parameters.差异2：嵌入参数计数对小模型很重要。 在小参数范围内，嵌入参数占总参数的比例不可忽略，因此是否计数它们很重要。Pearce & Song（2024）对此进行了彻底分析。我们用N∖E,C∖E表示排除嵌入时的模型大小和计算量，用N,C表示总参数。

• Kaplan et al.: $N_{\setminus E}^{\ast }\propto C_{\setminus E}^{0.73}$ (non-embedding)Kaplan等人：N∖E∗∝C∖E0.73（非嵌入）
• Chinchilla: $N^{\ast }\propto C^{0.50}$ (total)Chinchilla：N∗∝C0.50（总参数）

To bridge them, they fit a relationship between total parameters $N_T$ and non-embedding parameters $N_{\setminus E}$, for some constant $\omega$:为弥合差异，他们拟合了总参数NT与非嵌入参数N∖E之间的关系，对于某个常数ω：

This form has nice properties of being strictly increasing and $\underset{N\to \mathrm{\infty }}{lim}N=N_{\setminus E}$ (because $\frac{N}{N_{\setminus E}}=1+\omega {N_{\setminus E}}^{-\frac{2}{3}},\underset{N_{\setminus E}\to \mathrm{\infty }}{lim}\frac{N}{N_{\setminus E}}=1$.这种形式具有严格递增的良好性质，且limN→∞N=N∖E（因为NN∖E=1+ωN∖E−23,limN∖E→∞NN∖E=1）。

Plugging this into the Chinchilla laws equation,将其代入Chinchilla定律方程，

The relationship between $C_{\setminus E}$ and $N_{\setminus E}$ in the above equation is no longer a clean power law. We can only approximate it locally as $N_{\setminus E}^{\ast }\stackrel{\propto }{\sim }{C}_{\setminus E}^g$, where $g$ is a local exponent based on a first-order derivative ($\stackrel{\propto }{\sim }$) rather than a global power-law exponent, resulting in $g=\frac{\mathrm{d}\log C_{\setminus E}}{\mathrm{d}\log N_{\setminus E}}$. See the full details of how the exponent $g$ is approximated in Appendix A.1 in Pearce & Song (2024).上述方程中C∖E与N∖E之间的关系不再是纯粹的幂律。我们只能局部近似为N∖E∗∼∝C∖Eg，其中g是基于一阶导数的局部指数（∼∝），而非全局幂律指数，得到g=dlog⁡C∖Edlog⁡N∖E。有关指数g如何近似的完整细节，请参见Pearce & Song（2024）附录A.1。

As shown in the visualization above, as $C_{\setminus E}$ gets larger, $g$ converges to the Chinchilla estimate. By generating synthetic training curves using above equation, in the range of model size from 768M to 1.5B (as in Kaplan et al.), they estimated that $g$ is close to the Kaplan coefficient of 0.73 in that region.如上图所示，随着C∖E增大，g收敛到Chinchilla估计值。通过使用上述方程生成合成训练曲线，在模型大小从768M到1.5B的范围内（如Kaplan等人），他们估计在该区域g接近Kaplan系数0.73。

Why power law?为什么是幂律？#

Power laws are widely observed across many domains outside AI, such as in Zipf’s law, scale-free networks, urban scaling laws, and many other complex systems. The recurring pattern is that large events are rare, small events are common and the relationship between size and frequency often follows a straight line at log-log scale.幂律在AI之外的许多领域广泛观察到，例如齐普夫定律、无标度网络、城市缩放定律和许多其他复杂系统。重复出现的模式是，大事件罕见，小事件常见，大小与频率之间的关系通常在对数-对数尺度上呈直线。

Why do LLM scaling laws also have the shape of a power law?为什么LLM缩放定律也具有幂律形状？

Inspired partly by different domains displaying different exponents (Hestness et al. 2017), one early explanation by Sharma & Kaplan (2020) hypothesizes that language modeling can be viewed as doing regression on a low-dimensional manifold of data. More model parameters can induce a finer partition of the data manifold and therefore smaller generalization error. In the simplest terms, if a model of effective size $N$ partitions a $d$-dimensional manifold into $O(N)$ regions, the typical linear resolution scales like $\sim N^{-1/d}$. This has a similar power-law form to the scaling laws above. This theory applies most cleanly in the infinite-data, underfitting regime, but in reality estimating the intrinsic dimension of a data manifold is quite hard.部分受不同领域显示不同指数（Hestness等人，2017）的启发，Sharma & Kaplan（2020）的一个早期解释假设语言建模可以视为在数据的低维流形上进行回归。更多模型参数可以诱导数据流形的更精细划分，从而降低泛化误差。简单来说，如果有效大小为N的模型将d维流形划分为O(N)个区域，则典型线性分辨率约为∼N−1/d。这与上述缩放定律具有类似的幂律形式。该理论在无限数据、欠拟合机制下最清晰，但实际中估计数据流形的内在维度相当困难。

A later hypothesis (Michaud et al. 2023, Brill 2024) assumes that knowledge or skills are learned in discrete chunks (“quantized”) and that the frequency distribution of these skills follows a power law. The model learns common skills first and rare skills later, resulting in a smooth power-law decay in loss.后来的假设（Michaud等人，2023；Brill，2024）假设知识或技能以离散块（“量化”）学习，且这些技能的频率分布遵循幂律。模型先学习常见技能，后学习罕见技能，导致损失平滑的幂律衰减。

I only listed two hypotheses here, but there are more studies on explaining the shape of power-law scaling through spectral tails of data, kernel eigenvalues, natural-language statistics, or phase transitions in training dynamics.我在这里只列出了两个假设，但还有更多研究通过数据谱尾、核特征值、自然语言统计或训练动态中的相变来解释幂律缩放形状。

Scaling Laws in Data-Limited Region数据有限区域的缩放定律#

Classic scaling laws assume effectively unlimited unique data, no repetition, and no multi-epoch training. As the model size grows significantly, we are running out of enough high-quality unique tokens. In fact, some arguments about how long scaling in AI can continue are centered on whether we are hitting a “data wall”.经典缩放定律假设有效无限的唯一数据、无重复、无多轮训练。随着模型大小显著增长，我们正在耗尽足够高质量的唯一token。事实上，关于AI缩放能持续多久的一些争论集中在是否正在触及“数据墙”。

It is also worth emphasizing that the dataset behind $D$ is expected to be already cleaned. The pretraining data pipeline is often a large part of an effective pretraining pipeline, with common steps like deduplication (exact and fuzzy), quality filtering, boilerplate removal, safety filtering, PII/copyright masking, benchmark decontamination and careful reweighting of data mix components based on language, quality, content type, etc. Even when two datasets contain the same token count $D$, a high-quality dataset and a dataset of Internet slop can yield drastically different compute efficiency.同样值得强调的是，D背后的数据集预期已经过清洗。预训练数据管道通常是有效预训练管道的重要组成部分，常见步骤包括去重（精确和模糊）、质量过滤、样板移除、安全过滤、PII/版权屏蔽、基准去污染以及基于语言、质量、内容类型等对数据混合组件进行仔细重新加权。即使两个数据集包含相同的token计数D，高质量数据集和互联网垃圾数据集也可能产生截然不同的计算效率。

The study by Hernandez et al. (2022) focused on a controlled version: a mostly-unique dataset with a small fraction of repeated data. Starting from a large dataset, the data mix keeps 90% non-repeated but replaces the remaining 10% with repeats of a tiny portion of the original. By training a Transformer model for 100B tokens, they observed a double-descent phenomenon, that is, the test loss can actually get worse and then better again as a function of how much the repeated data is emphasized, an effect that becomes more pronounced as the repeated fraction grows.Hernandez等人（2022）的研究关注了一个受控版本：一个几乎唯一的数据集，其中一小部分重复数据。从一个大数据集开始，数据混合保持90%非重复，但将剩余10%替换为原始数据的一小部分的重复。通过训练一个Transformer模型处理100B token，他们观察到双下降现象，即测试损失实际上可能随着重复数据强调程度的增加而先变差再变好，这种效应随着重复比例的增加而变得更加明显。

The flat or increasing trend in the middle of training is possibly due to memorization of repeated data. Learning curves with such shapes make scaling law fitting less accurate. They also concluded repeated data hurts some OOD evaluation and downstream fine-tuning. However, their data mix is constructed in a more lab-like setup, and repetition in real-world data is often more nuanced (e.g. different data has different levels of repetition, semantic repetition, etc.).训练中期的平坦或上升趋势可能归因于对重复数据的记忆。具有这种形状的学习曲线使缩放定律拟合不太准确。他们还得出结论，重复数据损害了一些OOD评估和下游微调。然而，他们的数据混合是在更类似实验室的设置中构建的，现实世界数据中的重复通常更微妙（例如，不同数据具有不同级别的重复、语义重复等）。

Rather than saying data repetition hurts training, we are more interested in how to fit scaling laws, given that the unique high-quality data is not infinite and we likely have to repeat data during training.与其说数据重复损害训练，我们更感兴趣的是如何拟合缩放定律，因为高质量唯一数据并非无限，我们可能在训练中不得不重复数据。

Muennighoff et al. (2023) took on the research question of how compute should be allocated optimally when model training is data-constrained. Specifically, they empirically studied the impact of data repetition across roughly 400 experiments, 10M–9B parameters, data sizes up to 900B tokens, and up to 1500 epochs. The exact same dataset is repeated each epoch, shuffled between epochs, and evaluated on a held-out test set.Muennighoff等人（2023）研究了当模型训练受数据约束时如何最优分配计算的研究问题。具体来说，他们实证研究了数据重复的影响，涉及约400次实验，参数从10M到9B，数据大小高达900B token，最多1500个epoch。每个epoch重复完全相同的数据集，在epoch之间打乱，并在保留测试集上评估。

The key modeling adjustment is to decompose the total token count $D$ into two parts: (i) the number of unique tokens $U_D$ and (ii) the number of repeats $R_D$ (i.e. num. epochs - 1). Thus we have $D=U_D(1+R_D)$. With a unique-data budget $D_{\text{uniq}}$, by definition $U_D=min\{D_{\text{uniq}},D\}$ and $R_D=(D/U_D)-1$. They use the Chinchilla scaling laws to find the optimal model size $U_N$ for fitting $U_D$, and define excess model size via repeats $R_N=(N/U_N)-1$.关键建模调整是将总token计数D分解为两部分：（i）唯一token数量UD和（ii）重复次数RD（即epoch数-1）。因此我们有D=UD(1+RD)。在唯一数据预算Duniq下，根据定义UD=min{Duniq,D}且RD=(D/UD)−1。他们使用Chinchilla缩放定律找到拟合UD的最优模型大小UN，并通过重复定义过剩模型大小RN=(N/UN)−1。

They then update the Chinchilla parametric fit (method 3) to use effective (discounted) data $D^{\prime }$ and model size $N^{\prime }$ in place of the raw quantities:然后他们更新Chinchilla参数化拟合（方法3），使用有效（折扣）数据D'和模型大小N'代替原始量：

The intuition is that a token’s value decays exponentially as it is repeated. In their modeling, each repetition costs the token a $(1-1/r_D)$ fraction of its remaining value, where $r_D$ is a learnable “half-life” parameter. When $R_D=0$ or $R_D\ll r_D$, we recover $D^{\prime }\approx D$.直觉是token的价值随着重复而指数衰减。在他们的建模中，每次重复消耗token剩余价值的(1−1/rD)部分，其中rD是可学习的“半衰期”参数。当RD=0或RD≪rD时，我们恢复D'≈D。

A symmetric formulation handles excess model size, $N^{\prime }=U_N+U_N{r}_N(1-\exp (-R_N/r_N))$, capturing the idea that “larger models overfit more quickly on repeated data” and that “a model can be too large for its dataset.” This component is less intuitive, and I could not find a satisfactory explanation for why model size needs to appear in such a symmetric form as repeated data. Later work by Lovelace et al. (2026) changed this assumption.对称公式处理过剩模型大小，N′=UN+UNrN(1−exp⁡(−RN/rN))，捕捉“更大模型在重复数据上过拟合更快”和“模型可能对其数据集过大”的想法。这个组成部分不太直观，我找不到令人满意的解释为什么模型大小需要以这种对称形式出现，类似于重复数据。后来的工作Lovelace等人（2026）改变了这一假设。

Their empirical fit finds that excess parameters decay faster in value than repeated data, $r_N<r_D$, so we should allocate more resources on more epochs rather than more model parameters. One weakness of this modeling, as the authors also pointed out, is that it significantly underestimates the final test loss of failing models (i.e. models whose loss increases midway through training), such as models trained for 44 epochs.他们的实证拟合发现，过剩参数的价值衰减比重复数据更快，rN<rD，因此我们应该将更多资源分配给更多epoch而不是更多模型参数。这种建模的一个弱点，正如作者也指出的，是它显著低估了失败模型（即训练中途损失增加的模型）的最终测试损失，例如训练了44个epoch的模型。

Most recently, Lovelace et al. (2026) revisited the same problem with a different approach. Rather than modeling overparameterization as a diminishing return on effective model size, Lovelace et al. model the interaction between model size $\times$ data repetition explicitly. Empirically, they trained about 300 models, spanning 15M to 1B parameters and 50M to 6B unique tokens.最近，Lovelace等人（2026）使用不同方法重新审视了同一问题。他们没有将过参数化建模为有效模型大小的递减回报，而是显式建模模型大小×数据重复之间的交互。实证上，他们训练了约300个模型，参数范围从15M到1B，唯一token从50M到6B。

When they plot the fit residual for a fixed model size across a range of data-repetition levels, the observation is intuitive: more epochs cause more damage, and interestingly larger models are more sensitive to repetition. This hints that the loss penalty is likely a function of both model size and data size.当他们绘制固定模型大小在不同数据重复水平下的拟合残差时，观察结果直观：更多epoch造成更多损害，有趣的是，更大的模型对重复更敏感。这暗示损失惩罚可能是模型大小和数据大小的函数。

An explicit overfitting penalty term was introduced and built around the capacity ratio $N/U_D$ (parameter count relative to unique tokens):引入了一个显式的过拟合惩罚项，围绕容量比N/UD（参数数量相对于唯一token）构建：

where:其中：

• $R_D$ is the repetition count;RD是重复次数；
• the scalar $P$ is a learnable parameter;标量P是可学习参数；
• the exponent $\kappa$ (the 2nd learnable parameter) lets the penalty scale nonlinearly with the capacity ratio $N/U_D$;指数κ（第二个可学习参数）使惩罚随容量比N/UD非线性缩放；
• the separate exponent $\delta$ (the 3rd learnable parameter) on the repetition count decouples repetition nonlinearity from $\kappa$.重复次数上的单独指数δ（第三个可学习参数）将重复非线性与κ解耦。

The added term (in red) is a direct overfitting penalty that grows with both how many times you repeat the data and how over-parameterized the model is relative to the unique data available.添加的项（红色）是直接的过拟合惩罚，随数据重复次数和模型相对于可用唯一数据的过参数化程度增加而增长。

They also did a case study on how weight decay impacts training with the limited-data constraint and found that strong weight decay reduces the overfitting penalty caused by data repetition.他们还进行了案例研究，探讨权重衰减如何影响有限数据约束下的训练，发现强权重衰减减少了数据重复引起的过拟合惩罚。

Both modeling approaches by Muennighoff et al. and Lovelace et al. are constructed from empirical curve fitting, so it is still unclear why data-constrained scaling laws should have exactly these forms and why each free parameter is needed. Curious about more theoretical work along this line.Muennighoff等人和Lovelace等人的两种建模方法都基于实证曲线拟合，因此仍不清楚为什么数据约束缩放定律应具有这些确切形式，以及为什么需要每个自由参数。期待更多沿此方向的理论工作。

Trickiness of Fitting Scaling Laws in Reality现实中拟合缩放定律的棘手性#

Despite its clean form, in practice, scaling law fitting can be surprisingly sensitive to seemingly trivial procedural choices, like how you count parameters, how you round the precision, how you sum or average the loss, etc.尽管形式简洁，但在实践中，缩放定律拟合可能对看似微不足道的程序选择异常敏感，例如如何计数参数、如何舍入精度、如何求和或平均损失等。

Because a scaling law is only fit on the (relatively small, relatively cheap) models that we can afford to train, and the prediction is extrapolated for a model orders of magnitude larger. In such a setup, choices that look like rounding error may lead to wild differences in prediction.因为缩放定律仅在我们能负担得起的（相对较小、相对便宜）模型上拟合，而预测外推至大几个数量级的模型。在这种设置下，看似舍入误差的选择可能导致预测的巨大差异。

Meanwhile, scaling-law fitting assumes the only changing factor is scale, which means that the model architecture, optimizer, learning rate schedule, batch ramp, data mix, tokenizer, and other design choices should remain the same. Another underlying assumption is that all these settings should have been carefully tuned, as cases like undertrained models can lead to a different conclusion.同时，缩放定律拟合假设唯一变化的因素是规模，这意味着模型架构、优化器、学习率调度、批量提升、数据混合、分词器和其他设计选择应保持不变。另一个潜在假设是所有这些设置都应经过仔细调整，因为训练不足等情况可能导致不同结论。

The disagreement between results by Kaplan et al. and Chinchilla is one example to showcase the trickiness of scaling laws fitting.Kaplan等人和Chinchilla结果之间的分歧是展示缩放定律拟合棘手性的一个例子。

A second example is a follow-up analysis investigating why Chinchilla method 3 is slightly off from the other two methods. Besiroglu et al. (2024) extracted the raw $(N,D,L)$ data points from Figure 4 of Hoffmann et al. (2022) and re-ran the method 3 parametric fitting. They found a couple of concrete issues:第二个例子是后续分析，调查为什么Chinchilla方法3与其他两种方法略有偏差。Besiroglu等人（2024）从Hoffmann等人（2022）的图4中提取了原始(N,D,L)数据点，并重新运行了方法3的参数化拟合。他们发现了一些具体问题：

• A high loss scale in the L-BFGS-B minimizer, caused by averaging Huber-loss values over examples instead of summing them, which led to premature termination of the optimization. The early stopping of loss minimization during both the original fit and bootstrapping produced inconsistent estimates and implausibly narrow confidence intervals.L-BFGS-B最小化器中的高损失尺度，由对示例平均Huber损失值而非求和引起，导致优化过早终止。原始拟合和自举过程中损失最小化的提前停止产生了不一致的估计和难以置信的窄置信区间。
• The reported $\alpha$ and $\beta$ were rounded to 2 digits of precision, which made the derived $A,B$ look more off than they really were.报告的α和β四舍五入到2位精度，使得导出的A,B看起来比实际更偏离。

Toy simulation玩具模拟#

Here is a toy simulation widget, created by ChatGPT, designed to demonstrate three specific failure modes.以下是由ChatGPT创建的玩具模拟小部件，旨在演示三种特定的失败模式。

We assume the ground truth function is:我们假设真实函数为：

and thus $N_{\text{opt}}\propto C^{0.5126},D_{\text{opt}}\propto C^{0.4874}$. This is the estimate from Besiroglu et al. (2024).因此Nopt∝C0.5126，Dopt∝C0.4874。这是Besiroglu等人（2024）的估计。

The simulation plots the loss prediction $\hat{L}$ vs dataset size $D$, while providing a set of sliders to show case:模拟绘制损失预测L^与数据集大小D的关系图，同时提供一组滑块以展示：

• Loss precision: rounding losses from high to low decimal points can change the fitted parameter values.损失精度：将损失从高到低小数位舍入可以改变拟合参数值。
• Loss noise: perturbing loss values by only a multiplier of milli-loss (0.001) units leads to different fit.损失噪声：仅以毫损失（0.001）单位的乘数扰动损失值会导致不同的拟合。
• Fit-region sensitivity: fitting only small models, only medium models, or all models gives different apparent scaling laws.拟合区域敏感性：仅拟合小模型、仅拟合中等模型或拟合所有模型会给出不同的表观缩放定律。

Citation引用#

Please cite this work as:请引用本作品为：

Weng, Lilian. "Scaling Laws, Carefully". Lil'Log (Jun 2026). https://lilianweng.github.io/posts/2026-06-24-scaling-laws/
copy

Or use the BibTex citation:或使用BibTex引用：

@article{weng2026scaling,
 title = {Scaling Laws, Carefully},
 author = {Weng, Lilian},
 journal = {lilianweng.github.io},
 year = {2026},
 month = {June},
 url = "https://lilianweng.github.io/posts/2026-06-24-scaling-laws/"
}
copy

References参考文献#

[1] S. Amari, N. Fujita, and S. Shinomoto. “Four Types of Learning Curves. Neural Computation.” 4(4):605–618, 1992.[1] S. Amari, N. Fujita, and S. Shinomoto. “Four Types of Learning Curves. Neural Computation.” 4(4):605–618, 1992.

[2] Hestness et al. “Deep Learning Scaling is Predictable, Empirically.” arXiv preprint arXiv:1712.00409, 2017.[2] Hestness et al. “Deep Learning Scaling is Predictable, Empirically.” arXiv preprint arXiv:1712.00409, 2017.

[3] Rosenfeld et al. “A Constructive Prediction of the Generalization Error Across Scales.” ICLR 2020.[3] Rosenfeld et al. “A Constructive Prediction of the Generalization Error Across Scales.” ICLR 2020.

[4] Kaplan et al. “Scaling Laws for Neural Language Models.” arXiv preprint arXiv:2001.08361, 2020.[4] Kaplan et al. “Scaling Laws for Neural Language Models.” arXiv preprint arXiv:2001.08361, 2020.

[5] Hoffmann et al. “Training Compute-Optimal Large Language Models.” NeurIPS 2022.[5] Hoffmann et al. “Training Compute-Optimal Large Language Models.” NeurIPS 2022.

[6] Pearce and Song. “Reconciling Kaplan and Chinchilla Scaling Laws.” TMLR 2024.[6] Pearce and Song. “Reconciling Kaplan and Chinchilla Scaling Laws.” TMLR 2024.

[7] Bahri et al. “Explaining Neural Scaling Laws.” arXiv preprint arXiv:2102.06701, 2021.[7] Bahri et al. “Explaining Neural Scaling Laws.” arXiv preprint arXiv:2102.06701, 2021.

[8] Sharma and Kaplan. “A Neural Scaling Law from the Dimension of the Data Manifold.” arXiv preprint arXiv:2004.10802, 2020.[8] Sharma and Kaplan. “A Neural Scaling Law from the Dimension of the Data Manifold.” arXiv preprint arXiv:2004.10802, 2020.

[9] Hernandez et al. “Scaling Laws and Interpretability of Learning from Repeated Data.” arXiv preprint arXiv:2205.10487, 2022.[9] Hernandez et al. “Scaling Laws and Interpretability of Learning from Repeated Data.” arXiv preprint arXiv:2205.10487, 2022.

[10] Muennighoff et al. “Scaling Data-Constrained Language Models.” NeurIPS 2023.[10] Muennighoff et al. “Scaling Data-Constrained Language Models.” NeurIPS 2023.

[11] Lovelace et al. “Prescriptive Scaling Laws for Data Constrained Training.” arXiv preprint arXiv:2605.01640, 2026.[11] Lovelace et al. “Prescriptive Scaling Laws for Data Constrained Training.” arXiv preprint arXiv:2605.01640, 2026.

[12] Besiroglu et al. “Chinchilla Scaling: A Replication Attempt.” arXiv preprint arXiv:2404.10102, 2024.[12] Besiroglu et al. “Chinchilla Scaling: A Replication Attempt.” arXiv preprint arXiv:2404.10102, 2024.

[13] Michaud et al. “The Quantization Model of Neural Scaling” NeurIPS 2023.[13] Michaud et al. “The Quantization Model of Neural Scaling” NeurIPS 2023.

[14] Brill. “Neural Scaling Laws Rooted in the Data Distribution.” arXiv preprint arXiv:2412.07942, 2024.[14] Brill. “Neural Scaling Laws Rooted in the Data Distribution.” arXiv preprint arXiv:2412.07942, 2024.

[15] Rae et al. “Scaling Language Models: Methods, Analysis & Insights from Training Gopher.” arXiv preprint arXiv:2112.11446, 2021.[15] Rae et al. “Scaling Language Models: Methods, Analysis & Insights from Training Gopher.” arXiv preprint arXiv:2112.11446, 2021.