Scaling laws are one of the most critical empirical findings in deep learning. The observation is simple in form: the training loss $L$ decreases predictably as we scale up model size $N$, dataset size $D$, and compute $C$, following a power-law curve, which appears as a straight line on a log-log plot. We can view scaling laws as a framework for describing the relationship between compute, loss, model size and data; at its core, it is about how to allocate precious compute optimally between $N$ and $D$.深度学习里，缩放定律是极重要的经验发现。道理简单：模型参数 N、数据量 D、算力 C 往上加，训练损失 L 就往下掉。画在双对数坐标系里，是一条直线，符合幂律。这定律其实是在算账：怎么把有限的算力，在模型和数据之间分得最划算。

This predictability makes scaling laws highly valuable in practice. A common workflow is to fit scaling laws on a handful of small runs and then extrapolate to estimate the token and compute requirements for larger models.有了这定律，心里有底。先跑几个小规模实验，拟合出曲线。往后要多大的模型，要多少算力，算一算就出来了。

Symbol	Note
$N$	Model size, measured in parameter count.
$D$	Training dataset size, usually measured in token count.
$C$	Training compute in FLOPs. As a useful approximation, $C\approx 6ND$ (Kaplan et al. 2020), where $2ND$ accounts for the forward pass and $4ND$ for backpropagation.
$E$	Irreducible loss
$L,\hat{L}(.)$	Test loss / test loss prediction function; can also refer to training loss, since they are strongly correlated.
$ϵ$	Generalization error.

Early days: ML loss predictability早年间：机器学习损失的预测#

The predictability of generalization error with scale had already been investigated before scaling laws became a mainstream concept.缩放定律成主流前，早有人琢磨过规模与泛化误差的关系。

Amari et al. (1992) derived four types of learning curves using a Bayesian approach and the annealed approximation.1992 年，Amari 等人用了贝叶斯方法和退火近似，推导出四种学习曲线。

1. Deterministic learning algorithm, noiseless data, one unique solution: $ϵ\sim c\cdot D^{-1}$, where $c$ is some constant.确定性算法，数据无噪，解唯一：ϵ∼c⋅D−1，c 是常数。
2. Deterministic learning algorithm, noiseless data, multiple equivalent solutions: $ϵ\sim c\cdot D^{-2}$; the learning is faster with each new data point, because the model only learns the optimal manifold of parameters, instead of finding the single solution point.确定性算法，数据无噪，多解等价：ϵ∼c⋅D−2。新数据越多，学得越快。模型只管最优参数流形，不用死磕某一个点。
3. Deterministic learning algorithm, noisy data: $ϵ\sim c\cdot D^{-1/2}$; noises in data make learning harder.确定性算法，数据有噪：ϵ∼c⋅D−1/2。数据乱，学起来就费劲。
4. Stochastic learning algorithm, noisy data: $ϵ\sim c\cdot D^{-1}+E$; here the irreducible loss $E$ is the residual error that a stochastic learner cannot reduce further, for example when the model runs out of capacity on large data. All four types of learning curves follow a power law:随机算法，数据有噪：ϵ∼c⋅D−1+E。E 是消不掉的残差，比如模型容量到头了。四种曲线，都走幂律。

where $E$ can be 0 and $\alpha =-2,-1,-1/2$. Although their theoretical setup is based on a simplified binary classification task, it points in a useful direction for building empirical ML loss prediction models.E 可为 0，α 取 -2、-1 或 -1/2。理论虽是简单的二分类，但给机器学习损失预测指了路。

One of the earliest empirical studies by Hestness et al. (2017) explained the relationship between generalization error, model size and data. For a given training data size, they identified the best-fit model size via grid search and then plotted loss against training dataset size. Across four different domains in deep learning (neural machine translation, image classification, language modeling, and speech recognition), a recurring pattern was observed where:2017 年，Hestness 等人做了早期的实证研究，理清了误差、模型大小、数据量三者的关系。他们用网格搜索找最优模型，再画出损失随数据量的变化。在神经翻译、图像分类、语言模型、语音识别四个领域，规律反复出现：

• Generalization error scales as a power law across a set of factors (e.g. data size).泛化误差随数据量等因素，呈幂律变化。
• Model improvements shift the error curve but do not seem to affect the power-law exponent.模型改进能平移误差曲线，但改不了幂律指数。
• Interestingly, architecture changes the offset ($E$) of the power-law fit but does not change the exponent ($\alpha$). The slope of the power law appears to be a property of the problem domain rather than the model architecture.架构变了，幂律的截距 E 会变，但指数 α 不变。斜率是问题本身的属性，跟模型架构没关系。
• The number of model parameters $N$ needed to fit a dataset of size $D$ also scales as a power law.要拟合 D 大小的数据集，需要的参数量 N，也符合幂律。

A conceptual illustration breaks the learning curve into three stages. In the small-data region, when there are not enough learning signals, the model performs only slightly better than random guessing. In the middle (“power-law region”), we observe a power-law relationship between loss, data, and model size. The final irreducible-error region can be attributed to factors such as noise in the data.学习曲线分三段。数据少时，信号不足，模型跟瞎猜差不多。中间是“幂律区”，损失、数据、模型大小呈幂律关系。最后是“不可约误差区”，多半是数据噪声闹的。

Rosenfeld et al. (2020) pushed this further by trying to model error as a joint function of both model size $N$ and data size $D$, across a diverse set of architectures (ResNet, WRN, LSTM, Transformer) and optimizers (Adam, SGD variants). Empirically they observed that, holding one axis fixed, the error decays as a power law in the other:2020 年，Rosenfeld 等人更进一步，把误差看作模型大小 N 和数据大小 D 的联合函数。他们试了 ResNet、Transformer 等架构，用了 Adam 和 SGD 等优化器。实验发现，固定一轴，另一轴的误差呈幂律衰减：

which can be combined into a joint form:合在一起就是：

where $A>0,B>0,\alpha \ge 0,\beta \ge 0$ are scalar constants and $E$ is not dependent on either $N$ or $D$.A、B、α、β 是常数，E 跟 N、D 无关。

Thus, they can build a prediction model in the form of a simple parametric function with $\mathit{\theta }=⟨A,B,E,\alpha ,\beta ⟩$ to predict the expected loss for $(D,N)$ > certain thresholds by only training on a set of smaller training configs, $(D,N)$ < certain thresholds.只要在小规模配置下训练，就能定下参数 θ=⟨A,B,E,α,β⟩。往后预测大规模下的损失，准。

Side note: These early works lean on classical learning-theory intuition like the VC dimension (the cardinality of the largest set of points a model can shatter) as a proxy for capacity, but in modern deep learning work the VC dimension is often too coarse to explain the behavior and the empirical power laws turned out to be much cleaner and more practical than the worst-case bounds that theory provides.注：早期工作爱用 VC 维（衡量模型容量）做理论支撑。但在现代深度学习里，VC 维太粗，解释不了现象。实证出来的幂律，比理论上的最坏情况界限要干净、实用得多。

Scaling Laws in Data-Infinite Region数据无限区域的缩放定律#

Kaplan et al.’s Scaling LawsKaplan 等人的缩放定律#

Kaplan et al. (2020) popularized the concept of scaling laws in the language modeling community. They found that the cross-entropy test loss $L$ scales as a power law with each of model size $N$ (excluding embedding layers), dataset size $D$, and training compute $C$ across many orders of magnitude. The findings are aligned with early work in the last section, but Kaplan et al. formalized the concept with a focus on Transformer language models and empirical experimentation at a larger scale, with model size ranging from 768M to 1.5B non-embedding parameters and dataset size from 22M to 23B tokens. All training runs in the paper used a learning rate schedule with a 3000 step linear warmup, followed by a cosine decay to zero.2020 年，Kaplan 等人把缩放定律带火了。他们发现，Transformer 语言模型的交叉熵测试损失 L，随模型大小 N（不含嵌入层）、数据量 D、训练算力 C，跨越多个数量级，呈幂律变化。他们专注于大规模实验，模型参数从 7.68 亿到 15 亿，数据量从 2200 万到 230 亿 token。训练用线性预热加余弦衰减。

List of key findings:核心发现：

• The loss $L$ scales as a power law with $N$, $D$, and $C$ individually; for optimal performance all three must scale in tandem.L 随 N、D、C 单独呈幂律。想效果好，三者得一块儿涨。
• Training curves follow predictable power laws whose parameters are roughly independent of model size.训练曲线遵循幂律，参数基本不随模型大小变。
• Larger models are more sample-efficient, meaning that they reach a given loss with fewer optimization steps and fewer data points than small models.大模型更省样本。同样的损失，大模型用的步数和数据更少。
• Architectural details (width, aspect ratio, etc.) matter less than sheer scale.架构细节（宽度、长宽比等）没规模重要。
• Train loss and test loss are positively correlated. (Sounds trivial but this is the foundation for pretraining work. On the other hand, whether pretraining loss improvement transfers to posttraining evaluation needs separate studies.)训练损失和测试损失正相关。这听着废话，却是预训练的基石。至于预训练损失降了，下游任务能不能好，得另说。
• Given a fixed compute budget, it is more efficient to train a very large model and stop before convergence than to train a smaller model all the way to convergence. This finding is where the Chinchilla scaling laws (the next section) disagree: Kaplan et al. overestimated the optimal model size as their fitted exponent was larger.算力固定时，练个超大模型，半途而废，比练个小模型练到收敛要划算。Chinchilla 论文不同意这点，认为 Kaplan 高估了最优模型大小。

They summarize the joint dependence on $N$ and $D$ in a single equation:他们总结了 N 和 D 的联合依赖方程：

A nice consequence of this form is that the extent of overfitting (i.e. model is complex or data is small) depends predominantly on the ratio $N^{\alpha /\beta }/D$, which indicates that the data needs to grow in a specific proportion to the growth of the model size to avoid training being data-limited.这方程有个好处：过拟合程度主要看 Nα/β/D 这个比值。想不缺数据，模型变大时，数据量得按比例涨。

The most influential and, in hindsight, most contested conclusion was the compute-optimal allocation. Kaplan et al. found $N_{\text{opt}}\propto C^{0.73}$ and concluded that model size should grow faster than dataset size. Concretely, for a 10x increase in compute they suggested scaling the model size by ~5.5x but the training tokens by only ~1.8x. The Chinchilla paper would later overturn this recommendation, arguing that it leaves large models badly undertrained.最受争议的结论是算力最优分配。Kaplan 认为 Nopt∝C0.73，模型得比数据长得快。算力翻 10 倍，模型加 5.5 倍，数据只加 1.8 倍。后来 Chinchilla 论文推翻了这点，说这么练，大模型根本没练透。

Another useful analysis in Kaplan et al. approximates the number of training FLOPs needed based on $D$ and $N$. Each multiply-add is counted as ~2 FLOPs.Kaplan 还给出了 FLOPs 估算。一次乘加算 2 FLOPs。

Given a standard config where $d_{\text{attn}}=d_{\text{model}}=d_{\text{ff}}/4$, and excluding embedding layers from $N$ and the per-token forward compute:假设 dattn=dmodel=dff/4，不算嵌入层参数和前向算力：

Then we count backward-pass FLOPs as twice the forward-pass FLOPs, because backpropagation runs two matrix multiplications, for gradients with respect to the input activations and the weights, respectively. Thus, in total, the training FLOPs per token are approximately $6N$, and the total FLOPs for training over $D$ tokens are $C\approx 6ND$.反向传播算两次矩阵乘法，算力是前向的两倍。每个 token 训练约需 6N FLOPs，训练 D 个 token 总算力 C≈6ND。

Chinchilla Scaling LawsChinchilla 缩放定律#

The Chinchilla paper (Hoffmann et al. 2022) studied the relationship between the optimal model size $N$ (total parameters, including embeddings) and the number of tokens $D$ under a fixed compute budget $C$ with a more careful experimental design and arrived at a somewhat different answer from Kaplan et al..2022 年，Hoffmann 等人（Chinchilla 团队）重新研究了最优模型大小 N 和数据量 D 的关系。设计更严谨，结论跟 Kaplan 不一样。

The central question is on the best strategy to allocate resources given a constraint $\text{FLOPs}(N,D)=C\approx 6ND$. In other words, when we have only limited FLOPs (a given number of GPUs running for a given period of time), how should we choose between more data tokens and more model parameters?核心问题是：算力 C≈6ND 固定时，怎么分配才最划算？是多加 token，还是多加参数？

The Chinchilla paper presented three neatly designed methods for scaling laws fitting.Chinchilla 论文给了三种拟合方法。

The empirical experiments scanned over 400 models, with sizes from 70M to over 16B parameters and training tokens from 5B to 500B. The experiments were under the assumption that every training token is unique (the infinite-data regime). All runs used a cosine learning-rate schedule decaying by 10x over the training horizon. Sweeping over model sizes traces out the compute-optimal frontier.实验跑了 400 多个模型，参数从 7000 万到 160 亿，token 从 50 亿到 5000 亿。假设所有 token 都不重复。余弦衰减调学习率。扫一遍模型大小，就能画出算力最优边界。

Method 1: Fix model sizes, vary the token budget方法 1：固定模型大小，变数据量#

For each parameter count $N$, train several runs with different token budgets, and record the minimal loss achieved per FLOP budget $C$.对每个 N，跑不同数据量，记录每个算力预算下的最低损失。

Method 2: IsoFLOP profiles方法 2：等算力曲线（IsoFLOP）#

Fix a compute budget $C$ and plot the final loss against parameter count $N$. Each iso-FLOP curve is roughly a parabola in log-space, and its minimum flags the optimal model size for that compute budget. Then repeating across budgets traces a power-law line in the plot.固定算力 C，画出损失随参数 N 的变化。曲线呈抛物线，最低点就是该算力下的最优模型大小。重复多次，连成幂律线。

Method 3: Parametric fit方法 3：参数拟合#

Fit the same parametric function as in Rosenfeld et al. (2020) directly,直接拟合 Rosenfeld 那套函数。

We can actually get a closed form approximation of optimal $N_{\text{opt}}(C),D_{\text{opt}}(C)$ by minimizing $\hat{L}(N,D)$ under the constraint $\text{FLOPs}(N,D)=C\approx 6ND$.在 C≈6ND 下最小化 L(N,D)，能得到 Nopt(C) 和 Dopt(C) 的近似闭式解。

First let’s reduce the expression to contain only $N$:先把表达式简化，只留 N：

When $\alpha \approx \beta$, model size and training tokens should scale at equal rates.当 α≈β 时，模型大小和数据量得按同样比例涨。

To find the optimal $\mathit{\theta }=⟨A,B,E,\alpha ,\beta ⟩$, the Chinchilla paper adopts a Huber loss (robust to outliers; $\delta ={10}^{-3}$) and the L-BFGS algorithm (good for curve fitting with a small number of parameters).拟合 θ 用了 Huber 损失（抗噪）和 L-BFGS 算法（适合参数少的情况）。

Chinchilla arrives at its answer through three complementary methods whose final results agree with each other, and this is part of why the result was quite convincing.三种方法结论一致，这让结果很有说服力。

The claim in the Chinchilla paper that most large models (at the time, ~2022) were undertrained is supported by a famous demonstration: under the same compute budget as Gopher (Rae et al. 2021; 280B parameter count, 300B token budget), they trained Chinchilla (70B parameter count, 1.4T token budget), a model 4x smaller but trained on roughly 4x more tokens and it outperformed Gopher across the board.Chinchilla 证明大模型练不够：他们用 Gopher（2800 亿参数，3000 亿 token）同样的算力，练了 Chinchilla（700 亿参数，1.4 万亿 token）。模型小了 4 倍，数据多了 4 倍，效果全面吊打 Gopher。

Reconciling Kaplan and Chinchilla调和 Kaplan 与 Chinchilla#

The Chinchilla scaling laws disagree with Kaplan et al. as follows:Chinchilla 这么反驳 Kaplan：

• Instead of “grow the model faster than the data” ($N_{\text{opt}}\propto C^{0.73}$), for every doubling of model size, you should also double the number of training tokens ($N_{\text{opt}}\propto C^{0.5}$).别“模型长得比数据快”（Nopt∝C0.73），模型翻倍，数据也得翻倍（Nopt∝C0.5）。
• Instead of “train a big model and stop before convergence,” you should train a smaller model on more data.别“大模型练半截就停”，练小一点，多喂数据。

Both papers still agree on the same underlying principle, but they disagree on where the optimal size-vs-token tradeoff lies. Why do they disagree so much?原理一致，结论打架。为什么？

Difference 1: Kaplan et al. experimented mostly on small models. Kaplan et al. experimented mostly on smaller models, while the Chinchilla paper’s experiments reached more than 10x larger scales. When we extrapolate in log-log space, a small difference in the fit can result in large differences (See toy simulation).差异 1：Kaplan 实验模型太小。Kaplan 跑的是小模型，Chinchilla 规模大 10 倍。对数空间里，拟合的一点点偏差，外推后差之千里。

Difference 2: Embedding parameter count matters for small models. In the small-parameter regime, embedding parameters are a non-negligible fraction of the total and thus counting them or not matters. Pearce & Song (2024) did a thorough analysis along this line. Let’s use $N_{\setminus E},C_{\setminus E}$ to denote model size and compute when embedding is excluded and use $N,C$ to count total parameters.差异 2：嵌入层参数影响。小模型里，嵌入层占比大。Pearce & Song (2024) 做了分析。用 N∖E 表示不含嵌入层的参数。

• Kaplan et al.: $N_{\setminus E}^{\ast }\propto C_{\setminus E}^{0.73}$ (non-embedding)Kaplan：N∖E∗∝C∖E0.73
• Chinchilla: $N^{\ast }\propto C^{0.50}$ (total)Chinchilla：N∗∝C0.50（总参数）

To bridge them, they fit a relationship between total parameters $N_T$ and non-embedding parameters $N_{\setminus E}$, for some constant $\omega$:把总参数 NT 和不含嵌入层的 N∖E 关联起来：

This form has nice properties of being strictly increasing and $\underset{N\to \mathrm{\infty }}{lim}N=N_{\setminus E}$ (because $\frac{N}{N_{\setminus E}}=1+\omega {N_{\setminus E}}^{-\frac{2}{3}},\underset{N_{\setminus E}\to \mathrm{\infty }}{lim}\frac{N}{N_{\setminus E}}=1$.这式子当 N 趋于无穷时，比例趋于 1。

Plugging this into the Chinchilla laws equation,代入 Chinchilla 方程：

The relationship between $C_{\setminus E}$ and $N_{\setminus E}$ in the above equation is no longer a clean power law. We can only approximate it locally as $N_{\setminus E}^{\ast }\stackrel{\propto }{\sim }{C}_{\setminus E}^g$, where $g$ is a local exponent based on a first-order derivative ($\stackrel{\propto }{\sim }$) rather than a global power-law exponent, resulting in $g=\frac{\mathrm{d}\log C_{\setminus E}}{\mathrm{d}\log N_{\setminus E}}$. See the full details of how the exponent $g$ is approximated in Appendix A.1 in Pearce & Song (2024).现在 C∖E 和 N∖E 之间不再是干净的幂律。只能局部近似为 N∖E∗∼∝C∖Eg。g 是局部指数。详见 Pearce & Song (2024) 附录。

As shown in the visualization above, as $C_{\setminus E}$ gets larger, $g$ converges to the Chinchilla estimate. By generating synthetic training curves using above equation, in the range of model size from 768M to 1.5B (as in Kaplan et al.), they estimated that $g$ is close to the Kaplan coefficient of 0.73 in that region.算力越大，g 越接近 Chinchilla 的 0.5。在 Kaplan 的小规模区间，g 确实接近 0.73。

Why power law?为什么是幂律？#

Power laws are widely observed across many domains outside AI, such as in Zipf’s law, scale-free networks, urban scaling laws, and many other complex systems. The recurring pattern is that large events are rare, small events are common and the relationship between size and frequency often follows a straight line at log-log scale.幂律在 Zipf 定律、网络科学、城市规模里到处都是。大事件少，小事件多，对数坐标下就是直线。

Why do LLM scaling laws also have the shape of a power law?LLM 为什么也这样？

Inspired partly by different domains displaying different exponents (Hestness et al. 2017), one early explanation by Sharma & Kaplan (2020) hypothesizes that language modeling can be viewed as doing regression on a low-dimensional manifold of data. More model parameters can induce a finer partition of the data manifold and therefore smaller generalization error. In the simplest terms, if a model of effective size $N$ partitions a $d$-dimensional manifold into $O(N)$ regions, the typical linear resolution scales like $\sim N^{-1/d}$. This has a similar power-law form to the scaling laws above. This theory applies most cleanly in the infinite-data, underfitting regime, but in reality estimating the intrinsic dimension of a data manifold is quite hard.Sharma & Kaplan (2020) 猜：语言模型是在低维数据流形上做回归。参数越多，流形切分越细，误差越小。若模型把 d 维流形切成 N 块，分辨率就是 N−1/d。这跟幂律形式像。理论在无限数据下成立，但估算数据流形的维度太难。

A later hypothesis (Michaud et al. 2023, Brill 2024) assumes that knowledge or skills are learned in discrete chunks (“quantized”) and that the frequency distribution of these skills follows a power law. The model learns common skills first and rare skills later, resulting in a smooth power-law decay in loss.后来有人猜（Michaud 等人，2023）：知识是离散的“块”，频率分布符合幂律。模型先学常见的，再学罕见的，损失就平滑下降。

I only listed two hypotheses here, but there are more studies on explaining the shape of power-law scaling through spectral tails of data, kernel eigenvalues, natural-language statistics, or phase transitions in training dynamics.还有人从数据谱特征、核特征值、自然语言统计等角度解释。学问多着呢。

Scaling Laws in Data-Limited Region数据受限区域的缩放定律#

Classic scaling laws assume effectively unlimited unique data, no repetition, and no multi-epoch training. As the model size grows significantly, we are running out of enough high-quality unique tokens. In fact, some arguments about how long scaling in AI can continue are centered on whether we are hitting a “data wall”.经典定律假设数据无限、无重复、不跑多轮。现在数据快用完了，这就是“数据墙”。

It is also worth emphasizing that the dataset behind $D$ is expected to be already cleaned. The pretraining data pipeline is often a large part of an effective pretraining pipeline, with common steps like deduplication (exact and fuzzy), quality filtering, boilerplate removal, safety filtering, PII/copyright masking, benchmark decontamination and careful reweighting of data mix components based on language, quality, content type, etc. Even when two datasets contain the same token count $D$, a high-quality dataset and a dataset of Internet slop can yield drastically different compute efficiency.数据得洗干净。去重、去噪、去版权、去隐私、去基准污染。同样 token 数，垃圾数据和高质量数据，算力效率差远了。

The study by Hernandez et al. (2022) focused on a controlled version: a mostly-unique dataset with a small fraction of repeated data. Starting from a large dataset, the data mix keeps 90% non-repeated but replaces the remaining 10% with repeats of a tiny portion of the original. By training a Transformer model for 100B tokens, they observed a double-descent phenomenon, that is, the test loss can actually get worse and then better again as a function of how much the repeated data is emphasized, an effect that becomes more pronounced as the repeated fraction grows.Hernandez 等人 (2022) 做了受控实验：90% 不重复，10% 重复。跑 1000 亿 token，发现“双下降”现象：重复数据一多，测试损失先坏再好。

The flat or increasing trend in the middle of training is possibly due to memorization of repeated data. Learning curves with such shapes make scaling law fitting less accurate. They also concluded repeated data hurts some OOD evaluation and downstream fine-tuning. However, their data mix is constructed in a more lab-like setup, and repetition in real-world data is often more nuanced (e.g. different data has different levels of repetition, semantic repetition, etc.).中间平坦或上升，可能是记住了重复数据。这让缩放定律拟合不准。重复数据还伤下游微调。但现实里的重复比实验室里复杂得多。

Rather than saying data repetition hurts training, we are more interested in how to fit scaling laws, given that the unique high-quality data is not infinite and we likely have to repeat data during training.数据有限，不得不重复，怎么拟合定律？

Muennighoff et al. (2023) took on the research question of how compute should be allocated optimally when model training is data-constrained. Specifically, they empirically studied the impact of data repetition across roughly 400 experiments, 10M–9B parameters, data sizes up to 900B tokens, and up to 1500 epochs. The exact same dataset is repeated each epoch, shuffled between epochs, and evaluated on a held-out test set.Muennighoff 等人 (2023) 研究了重复数据下的算力分配。跑了 400 多个实验，数据量到 9000 亿，最长跑 1500 轮。

The key modeling adjustment is to decompose the total token count $D$ into two parts: (i) the number of unique tokens $U_D$ and (ii) the number of repeats $R_D$ (i.e. num. epochs - 1). Thus we have $D=U_D(1+R_D)$. With a unique-data budget $D_{\text{uniq}}$, by definition $U_D=min\{D_{\text{uniq}},D\}$ and $R_D=(D/U_D)-1$. They use the Chinchilla scaling laws to find the optimal model size $U_N$ for fitting $U_D$, and define excess model size via repeats $R_N=(N/U_N)-1$.把总 token D 分为：唯一 token UD 和重复次数 RD。D=UD(1+RD)。用 Chinchilla 定律找最优 UN，定义过剩参数 RN=(N/UN)−1。

They then update the Chinchilla parametric fit (method 3) to use effective (discounted) data $D^{\prime }$ and model size $N^{\prime }$ in place of the raw quantities:更新 Chinchilla 参数拟合，用有效数据 D′ 和有效模型 N′ 代替：

The intuition is that a token’s value decays exponentially as it is repeated. In their modeling, each repetition costs the token a $(1-1/r_D)$ fraction of its remaining value, where $r_D$ is a learnable “half-life” parameter. When $R_D=0$ or $R_D\ll r_D$, we recover $D^{\prime }\approx D$.直觉是：token 越重复，价值越低。rD 是“半衰期”。RD 很大时，价值衰减。

A symmetric formulation handles excess model size, $N^{\prime }=U_N+U_N{r}_N(1-\exp (-R_N/r_N))$, capturing the idea that “larger models overfit more quickly on repeated data” and that “a model can be too large for its dataset.” This component is less intuitive, and I could not find a satisfactory explanation for why model size needs to appear in such a symmetric form as repeated data. Later work by Lovelace et al. (2026) changed this assumption.对模型大小也做了对称处理。这部分不太直观，后来的 Lovelace 等人 (2026) 改了假设。

Their empirical fit finds that excess parameters decay faster in value than repeated data, $r_N<r_D$, so we should allocate more resources on more epochs rather than more model parameters. One weakness of this modeling, as the authors also pointed out, is that it significantly underestimates the final test loss of failing models (i.e. models whose loss increases midway through training), such as models trained for 44 epochs.实验发现参数价值衰减比数据快，所以多跑几轮比加参数划算。但这模型对那些练着练着损失就涨上去的“失败模型”，估算不准。

Most recently, Lovelace et al. (2026) revisited the same problem with a different approach. Rather than modeling overparameterization as a diminishing return on effective model size, Lovelace et al. model the interaction between model size $\times$ data repetition explicitly. Empirically, they trained about 300 models, spanning 15M to 1B parameters and 50M to 6B unique tokens.Lovelace 等人 (2026) 换了思路。直接建模“模型大小 × 数据重复”的交互。跑了 300 多个模型。

When they plot the fit residual for a fixed model size across a range of data-repetition levels, the observation is intuitive: more epochs cause more damage, and interestingly larger models are more sensitive to repetition. This hints that the loss penalty is likely a function of both model size and data size.数据重复越多，模型越大，过拟合越严重。损失惩罚是两者的函数。

An explicit overfitting penalty term was introduced and built around the capacity ratio $N/U_D$ (parameter count relative to unique tokens):加了个过拟合惩罚项，基于容量比 N/UD：

where:其中：

• $R_D$ is the repetition count;RD 是重复次数；
• the scalar $P$ is a learnable parameter;P 是可学习参数；
• the exponent $\kappa$ (the 2nd learnable parameter) lets the penalty scale nonlinearly with the capacity ratio $N/U_D$;κ 让惩罚与容量比非线性相关；
• the separate exponent $\delta$ (the 3rd learnable parameter) on the repetition count decouples repetition nonlinearity from $\kappa$.δ 解耦了重复非线性。

The added term (in red) is a direct overfitting penalty that grows with both how many times you repeat the data and how over-parameterized the model is relative to the unique data available.红色的加项，直接反映了重复数据和过拟合带来的惩罚。

They also did a case study on how weight decay impacts training with the limited-data constraint and found that strong weight decay reduces the overfitting penalty caused by data repetition.他们还发现，强权重衰减能缓解重复数据带来的过拟合。

Both modeling approaches by Muennighoff et al. and Lovelace et al. are constructed from empirical curve fitting, so it is still unclear why data-constrained scaling laws should have exactly these forms and why each free parameter is needed. Curious about more theoretical work along this line.这俩模型都是经验拟合，理论依据还不够。期待更多理论研究。

Trickiness of Fitting Scaling Laws in Reality现实中拟合缩放定律的坑#

Despite its clean form, in practice, scaling law fitting can be surprisingly sensitive to seemingly trivial procedural choices, like how you count parameters, how you round the precision, how you sum or average the loss, etc.定律虽美，但参数怎么算、精度怎么取、损失怎么平均，都会影响结果。

Because a scaling law is only fit on the (relatively small, relatively cheap) models that we can afford to train, and the prediction is extrapolated for a model orders of magnitude larger. In such a setup, choices that look like rounding error may lead to wild differences in prediction.因为定律是在小模型上拟合的，外推到大模型，一点点舍入误差，结果就差出十万八千里。

Meanwhile, scaling-law fitting assumes the only changing factor is scale, which means that the model architecture, optimizer, learning rate schedule, batch ramp, data mix, tokenizer, and other design choices should remain the same. Another underlying assumption is that all these settings should have been carefully tuned, as cases like undertrained models can lead to a different conclusion.拟合假设只有规模在变，架构、优化器、学习率、数据混合都得死死固定。要是模型没练透，结论全错。

The disagreement between results by Kaplan et al. and Chinchilla is one example to showcase the trickiness of scaling laws fitting.Kaplan 和 Chinchilla 的打架就是个例子。

A second example is a follow-up analysis investigating why Chinchilla method 3 is slightly off from the other two methods. Besiroglu et al. (2024) extracted the raw $(N,D,L)$ data points from Figure 4 of Hoffmann et al. (2022) and re-ran the method 3 parametric fitting. They found a couple of concrete issues:Besiroglu 等人 (2024) 扒了 Chinchilla 的原始数据重算，发现几个坑：

• A high loss scale in the L-BFGS-B minimizer, caused by averaging Huber-loss values over examples instead of summing them, which led to premature termination of the optimization. The early stopping of loss minimization during both the original fit and bootstrapping produced inconsistent estimates and implausibly narrow confidence intervals.Huber 损失平均方式不对，导致优化器提前停止。拟合出的置信区间窄得离谱。
• The reported $\alpha$ and $\beta$ were rounded to 2 digits of precision, which made the derived $A,B$ look more off than they really were.α 和 β 舍入精度太低，导致 A、B 偏差。

Toy simulation模拟演示#

Here is a toy simulation widget, created by ChatGPT, designed to demonstrate three specific failure modes.ChatGPT 做的模拟器，演示三个翻车点。

We assume the ground truth function is:假设真理函数是：

and thus $N_{\text{opt}}\propto C^{0.5126},D_{\text{opt}}\propto C^{0.4874}$. This is the estimate from Besiroglu et al. (2024).Nopt∝C0.5126，Dopt∝C0.4874。这是 Besiroglu 的结论。

The simulation plots the loss prediction $\hat{L}$ vs dataset size $D$, while providing a set of sliders to show case:模拟器画出损失预测，滑块控制变量：

• Loss precision: rounding losses from high to low decimal points can change the fitted parameter values.损失精度：舍入位数变了，拟合参数就变。
• Loss noise: perturbing loss values by only a multiplier of milli-loss (0.001) units leads to different fit.损失噪声：扰动一点点，拟合结果就乱。
• Fit-region sensitivity: fitting only small models, only medium models, or all models gives different apparent scaling laws.拟合区域：只用小模型、只用中模型或全用，结论都不一样。

Citation引用#

Please cite this work as:引用本文，请写：

Weng, Lilian. "Scaling Laws, Carefully". Lil'Log (Jun 2026). https://lilianweng.github.io/posts/2026-06-24-scaling-laws/
copy

Or use the BibTex citation:或用 BibTex 格式：

@article{weng2026scaling,
 title = {Scaling Laws, Carefully},
 author = {Weng, Lilian},
 journal = {lilianweng.github.io},
 year = {2026},
 month = {June},
 url = "https://lilianweng.github.io/posts/2026-06-24-scaling-laws/"
}
copy

References参考文献

[1] S. Amari, N. Fujita, and S. Shinomoto. “Four Types of Learning Curves. Neural Computation.” 4(4):605–618, 1992.[1] S. Amari, N. Fujita, S. Shinomoto. “Four Types of Learning Curves. Neural Computation.” 4(4):605–618, 1992.

[2] Hestness et al. “Deep Learning Scaling is Predictable, Empirically.” arXiv preprint arXiv:1712.00409, 2017.[2] Hestness 等. “Deep Learning Scaling is Predictable, Empirically.” arXiv preprint arXiv:1712.00409, 2017.

[3] Rosenfeld et al. “A Constructive Prediction of the Generalization Error Across Scales.” ICLR 2020.[3] Rosenfeld 等. “A Constructive Prediction of the Generalization Error Across Scales.” ICLR 2020.

[4] Kaplan et al. “Scaling Laws for Neural Language Models.” arXiv preprint arXiv:2001.08361, 2020.[4] Kaplan 等. “Scaling Laws for Neural Language Models.” arXiv preprint arXiv:2001.08361, 2020.

[5] Hoffmann et al. “Training Compute-Optimal Large Language Models.” NeurIPS 2022.[5] Hoffmann 等. “Training Compute-Optimal Large Language Models.” NeurIPS 2022.

[6] Pearce and Song. “Reconciling Kaplan and Chinchilla Scaling Laws.” TMLR 2024.[6] Pearce, Song. “Reconciling Kaplan and Chinchilla Scaling Laws.” TMLR 2024.

[7] Bahri et al. “Explaining Neural Scaling Laws.” arXiv preprint arXiv:2102.06701, 2021.[7] Bahri 等. “Explaining Neural Scaling Laws.” arXiv preprint arXiv:2102.06701, 2021.

[8] Sharma and Kaplan. “A Neural Scaling Law from the Dimension of the Data Manifold.” arXiv preprint arXiv:2004.10802, 2020.[8] Sharma, Kaplan. “A Neural Scaling Law from the Dimension of the Data Manifold.” arXiv preprint arXiv:2004.10802, 2020.

[9] Hernandez et al. “Scaling Laws and Interpretability of Learning from Repeated Data.” arXiv preprint arXiv:2205.10487, 2022.[9] Hernandez 等. “Scaling Laws and Interpretability of Learning from Repeated Data.” arXiv preprint arXiv:2205.10487, 2022.

[10] Muennighoff et al. “Scaling Data-Constrained Language Models.” NeurIPS 2023.[10] Muennighoff 等. “Scaling Data-Constrained Language Models.” NeurIPS 2023.

[11] Lovelace et al. “Prescriptive Scaling Laws for Data Constrained Training.” arXiv preprint arXiv:2605.01640, 2026.[11] Lovelace 等. “Prescriptive Scaling Laws for Data Constrained Training.” arXiv preprint arXiv:2605.01640, 2026.

[12] Besiroglu et al. “Chinchilla Scaling: A Replication Attempt.” arXiv preprint arXiv:2404.10102, 2024.[12] Besiroglu 等. “Chinchilla Scaling: A Replication Attempt.” arXiv preprint arXiv:2404.10102, 2024.

[13] Michaud et al. “The Quantization Model of Neural Scaling” NeurIPS 2023.[13] Michaud 等. “The Quantization Model of Neural Scaling” NeurIPS 2023.

[14] Brill. “Neural Scaling Laws Rooted in the Data Distribution.” arXiv preprint arXiv:2412.07942, 2024.[14] Brill. “Neural Scaling Laws Rooted in the Data Distribution.” arXiv preprint arXiv:2412.07942, 2024.

[15] Rae et al. “Scaling Language Models: Methods, Analysis & Insights from Training Gopher.” arXiv preprint arXiv:2112.11446, 2021.[15] Rae 等. “Scaling Language Models: Methods, Analysis & Insights from Training Gopher.” arXiv preprint arXiv:2112.11446, 2021.