Exploring Dark Knowledge under Various Teacher Capacities and Addressing Capacity Mismatch探索不同教师容量下的“暗知识”及解决容量不匹配问题

Quite a few works focus on understanding and revealing the essence and effects of “dark knowledge” in KD, including the empirical studies and theoretical analysis [39, 40, 13, 12, 41, 42, 43, 32, 11, 44]. [11] unifies the dark knowledge with privileged information, and [45] attributes the dark knowledge in vision tasks to the task-relevant and task-irrelevant visual concepts. The dark knowledge in KD is also related to label smoothing (LS) [41, 46, 47, 48]. Some works decompose the dark knowledge of teachers and aim to understand their effectiveness correspondingly. [49] decomposes the dark knowledge into two parts, explaining the teacher’s ground-truth/non-ground-truth outputs as importance weighting and class similarities. [39] decomposes the dark knowledge into three parts including universal knowledge, domain knowledge, and gradient rescaling. The bias-variance decomposition is also utilized to analyze the property of KD [13, 43]. [50] utilizes the standard deviation of the secondary soft probabilities to reflect the quality of the teachers’ dark knowledge. Recently, [51] proposed an innovative perspective, suggesting that the knowledge of the teacher stems from containing less noise. The most related work [31] decomposes the KD into correct guidance, smooth regularization, and class discriminability, which provides novel insights about the key information in the dark knowledge. Few works have examined the relationship of dark knowledge with teacher capacities, and this paper provides detailed observations and analysis about this.已有不少工作致力于理解和揭示 KD 中“暗知识”的本质和作用，包括经验性研究和理论分析 [39, 40, 13, 12, 41, 42, 43, 32, 11, 44]。[11] 将暗知识与特权信息统一起来，[45] 将视觉任务中的暗知识归因于任务相关和任务无关的视觉概念。KD 中的暗知识也与标签平滑（LS）[41, 46, 47, 48] 相关。一些工作对教师的暗知识进行分解，旨在理解其相应的有效性。[49] 将暗知识分解为两部分，将教师的真实/非真实类别输出解释为重要性加权和类别相似性。[39] 将暗知识分解为通用知识、领域知识和梯度重缩放三部分。偏差-方差分解也被用于分析 KD 的性质 [13, 43]。[50] 利用次级软概率的标准差来反映教师暗知识的质量。最近，[51] 提出了一种创新的视角，认为教师的知识源于包含较少的噪声。最相关的工作 [31] 将 KD 分解为正确引导、平滑正则化和类别可判别性，这为理解暗知识中的关键信息提供了新见解。很少有工作研究暗知识与教师容量之间的关系，本文对此提供了详细的观察和分析。

An intuitive sense after the proposal of KD [9] is that larger teachers could teach students better because their accuracy is higher. However, there exists a peculiar phenomenon named capacity mismatch that excellent teachers can’t completely teach the smaller students well. ESKD [32] first points out the phenomenon and points out that teacher accuracy may poorly predict the student’s performance, i.e., more accurate neural networks don’t necessarily teach better. However, no research so far has provided an accurate explanation or even a quantitative definition for this phenomenon. Previous works have only been made to propose solutions from various perspectives. TAKD [33] solves this problem by introducing an intermediate-sized network (i.e., the teacher assistant) to bridge the knowledge transfer between networks with a large capacity gap. SCKD [34] formulates KD as a multi-task learning problem with several knowledge transfer losses. ResKD [35, 36] utilize the “residual” of knowledge to teach the residual student, and then take the ensemble of the student and residual student for inference. DIST [52] finds that the discrepancy between the student and a stronger teacher may be fairly severe, which disturbs the training process, and they propose a ranking-based loss as the solution. [53] advocates that an intermediate checkpoint will be more appropriate for distillation. [54] explores the strong teachers in few-shot learning. [55] studies the effect of capacity gap on the generated samples in data-free KD. Recently, [56] demonstrated that sharing the same temperature between the teacher and student leads to a logit shift, which is a side-effect in KD. One of our methods is closely related to [31] that proposes Asymmetric Temperature Scaling (ATS) to make the larger networks teach well again. An improved version named Instance-Specific ATS (ISATS) is proposed. Aside from this method, we also propose other simple yet effective solutions from different aspects.KD [9] 提出后的一个直观感受是，由于大型教师模型的准确率更高，它们理应能更好地教导学生。然而，存在一种被称为容量不匹配的特殊现象，即优秀的教师模型无法完全很好地教导较小的学生模型。ESKD [32] 首次指出了这一现象，并指出教师准确率可能无法很好地预测学生性能，即更准确的神经网络未必能带来更好的教学效果。然而，目前还没有研究为这一现象提供准确的解释或定量的定义。以往的工作仅从不同角度提出了解决方案。TAKD [33] 通过引入中间大小的网络（即教师助手）来桥接大容量差距网络之间的知识迁移，从而解决了这一问题。SCKD [34] 将 KD 形式化为一个包含多种知识迁移损失的多任务学习问题。ResKD [35, 36] 利用知识的“残差”来教导残差学生，然后将学生和残差学生的集成用于推理。DIST [52] 发现学生与更强教师之间的差异可能相当严重，这会干扰训练过程，因此他们提出了一种基于排序的损失作为解决方案。[53] 主张中间检查点更适合蒸馏。[54] 探讨了少样本学习中的强教师模型。[55] 研究了容量差距对无数据 KD 中生成样本的影响。最近，[56] 证明了教师和学生共享相同的温度会导致逻辑值偏移，这是 KD 中的一种副作用。我们的方法之一与 [31] 密切相关，该研究提出了非对称温度缩放（ATS）以使大型网络能够再次有效地进行教学。文中还提出了一个名为实例特定 ATS (ISATS) 的改进版本。除了这种方法外，我们还从不同角度提出了其他简单且有效的方法。

An intuitive method to reflect the relationship of dark knowledge produced by teachers with various capacities is visualization. The first step is training teacher networks with different capacities on several benchmarks. Specifically, we train ResNet14 and ResNet110 [57] on the CIFAR-10/CIFAR-100 dataset [37], ResNeXt50-32-4d and ResNeXt101-32-8d [58] on the Stanford Dogs [59] and CUB [60] dataset, and WideResNet26-2 and WideResNet50-2 [61] on the TinyImageNet [62] dataset. We maximize the size difference between the two models on each dataset to make the experimental effects more pronounced. Then, for each dataset, a random sample is sampled from the first class in the training set and the softened probability vectors produced by different teachers are visualized. Fig. 1 shows the results, where “RES”, “RNX”, and “WRN” are abbreviations for corresponding networks. The utilized temperature is common across the subfigures. Comparing the first and second rows, it is obvious that larger teachers provide less varied probabilities for non-ground-truth classes. Although the variance of non-ground-truth class probabilities differs a lot between teachers, the relative probability values seem to be consistent. For example, the several top-$K$ bars between the two rows lie nearly in the same classes. We conclude the two fundamental observations as follows: (1) larger teachers tend to produce probability vectors that are less distinct among non-ground-truth classes; (2) teachers with different capacities are basically consistent in their cognition of relative class affinity. To further support these two observations, we provide explanations and statistical measures in the following.可视化是一种反映不同容量教师产生的暗知识之间关系的直观方法。第一步是在多个基准数据集上训练不同容量的教师网络。具体而言，我们在 CIFAR-10/CIFAR-100 数据集 [37] 上训练 ResNet14 和 ResNet110 [57]，在 Stanford Dogs [59] 和 CUB [60] 数据集上训练 ResNeXt50-32-4d 和 ResNeXt101-32-8d [58]，并在 TinyImageNet [62] 数据集上训练 WideResNet26-2 和 WideResNet50-2 [61]。我们最大化每个数据集上两个模型之间的尺寸差异，以使实验效果更加显著。然后，对于每个数据集，从训练集的第一类中随机抽取一个样本，并将不同教师产生的软化概率向量进行可视化。图 1 展示了结果，其中“RES”、“RNX”和“WRN”是相应网络的缩写。子图间使用了相同的温度。比较第一行和第二行，显而易见的是，较大的教师模型为非真实类别提供了变化较小的概率。尽管非真实类别概率的方差在不同教师之间差异很大，但相对概率值似乎是一致的。例如，两行之间前 K 个条形图中的几个几乎属于相同的类别。我们将这两个基本观察结果总结如下：（1）较大的教师倾向于生成在非真实类别间区分度较低的概率向量；（2）不同容量的教师在对类别相对亲和力的认知上基本保持一致。为了进一步支持这两个观察结果，我们在下文中提供了解释和统计度量。

The explanations for the first observation are provided step by step.我们逐步提供对第一个观察结果的解释。

Low-dimensional features. First, we train MLP networks on MNIST [63] and train VGG [64] networks on SVHN [65]. These two datasets are relatively simpler classification tasks, and we set the final feature dimensions as 2 and 3 correspondingly for better visualization. We do not use the bias parameters in the last layer. We progressively enlarge the depth and width of MLP networks, and use the total number of parameters to denote the network capacity. The capacity of MLP networks ranges from 0.1M to 0.9M. The VGG capacities are determined by the number of layers, which ranges from 8 to 19. We do not use ReLU [66] activation for the final extracted features. These two datasets have 10 types of digits to identify, and the extracted features are scattered in Fig. 2. With the capacity increases, the features of the same class are more compact, while the class centers between different classes are more dispersed. Notably, the larger networks do not overfit the training dataset, because the test accuracy shown in the figure still becomes higher. This implies that larger DNNs obtain a higher performance but may extract more compact intra-class features and more dispersed inter-class features.低维特征。首先，我们在 MNIST [63] 上训练 MLP 网络，并在 SVHN [65] 上训练 VGG [64] 网络。这两个数据集是相对简单的分类任务，为了更好地可视化，我们将最终特征维度分别设置为 2 和 3。我们在最后一层不使用偏置参数。我们逐步扩大 MLP 网络的深度和宽度，并使用参数总数来表示网络容量。MLP 网络的容量范围从 0.1M 到 0.9M。VGG 的容量由层数决定，范围从 8 到 19。我们在最终提取的特征上不使用 ReLU [66] 激活。这两个数据集有 10 种数字需要识别，提取的特征分布在图 2 中。随着容量的增加，同一类别的特征更加紧凑，而不同类别之间的类中心更加分散。值得注意的是，较大的网络并没有对训练数据集过拟合，因为图中显示的测试准确率仍在提高。这意味着更大的 DNN 获得了更高的性能，但可能会提取出更紧凑的类内特征和更分散的类间特征。

High-dimensional features. The above demos show low-dimensional features that may not reflect the change in high-dimensional space. Hence, we extend the findings to CIFAR-10 and CIFAR-100, on which we train a series of ResNet, WideResNet, and ResNeXt networks. Because the high-dimensional features are hard to visualize and the dimensional-reduced plots by T-SNE [67] are sensitive to hyper-parameters, we instead calculate several metrics to indirectly verify the change of feature compactness. Specifically, we calculate the feature angle as follows:高维特征。上述演示显示的是可能无法反映高维空间变化的低维特征。因此，我们将研究结果扩展到 CIFAR-10 和 CIFAR-100，并在其上训练了一系列 ResNet、WideResNet 和 ResNeXt 网络。由于高维特征难以可视化，且 T-SNE [67] 降维图对超参数敏感，我们转而计算几个指标来间接验证特征紧凑性的变化。具体而言，我们按如下方式计算特征角度：

where $\mathbf{h}_{i}$ and $\mathbf{h}_{j}$ are feature vectors of the $i$-th and $j$-th training sample. Then, we define the inter-class and intra-class set as $\{(i,j)\}_{y_{i}=y_{j},\forall i,j}$ and $\{(i,j)\}_{y_{i}\neq y_{j},\forall i,j}$, respectively. The average and standard deviation of angles in the two sets are plotted in Fig. 3. If the teacher becomes more complex, the inter-class feature angle becomes smaller, while the intra-class angle becomes larger. Notably, we use ReLU [66] activation before the classification layer, implying that all of the elements in the extracted features are non-negative. This could already provide some hints for the capacity mismatch phenomenon. As an extreme case, if the features among the same class collapse into a single point, and features among different classes are orthogonal to each other, the inter-class feature angle will be zero and the intra-class one will be 90^∘. Although the training and test accuracy will be $100\%$, the softened probabilities may be one-hot labels, which bring no additional information to the student. An illustration and more detailed discussion can be found in Sect. 7. The inter-class and intra-class distance metrics utilized in linear discriminant analysis [68] or unsupervised discriminant projection [69] show the same trend. We omit the display of the results based on these distance metrics because the trends of feature angles are vivid enough to show how the feature compactness changes.其中 𝐡i 和 𝐡j 分别是第 i 个和第 j 个训练样本的特征向量。然后，我们分别定义类间集 {(i,j)}yi≠yj,∀i,j 和类内集 {(i,j)}yi=yj,∀i,j。这两个集合中角度的平均值和标准差绘制在图 3 中。如果教师模型变得更复杂，类间特征角度会变小，而类内角度会变大。值得注意的是，我们在分类层之前使用了 ReLU [66] 激活，这意味着提取特征中的所有元素均为非负值。这已经可以为容量不匹配现象提供一些线索。作为一种极端情况，如果同一类别的特征坍缩为一个点，且不同类别之间的特征相互正交，则类间特征角度将为 0，类内角度将为 90°。尽管训练和测试准确率将达到 100%，但软化概率可能是独热标签，这不会为学生提供额外信息。插图和更详细的讨论可在第 7 节找到。线性判别分析 [68] 或无监督判别投影 [69] 中使用的类间和类内距离指标显示出相同的趋势。我们省略了基于这些距离指标的结果展示，因为特征角度的趋势已足够直观地展示特征紧凑性是如何变化的。

Visualizaion of classes. According to the softmax properties in classification problems, the classification weight of each class is pulled towards the features from the same class while pushed away from the features of other classes [70]. That is, the change tendency of features could partially reflect the tendency of classification weights in the final layer. An ideal case is that the learned classification weights converge to the feature centers. If the intra-class feature angle becomes larger, the feature of a specific sample will be far away from the classification weights of the other classes. Similarly, the feature of a specific sample will be closer to the corresponding class’s classification weight. This will lead to a larger logit for the ground-truth class. If the intra-class features are nearly orthogonal, the logits for non-ground-truth classes will be less varied. The illustration can be found in Fig. 4, where the larger teacher extracts more compact inter-class features and more dispersed intra-class features. The triangle represents a sample feature in the $1$-st class, and the larger teacher tends to give a larger target logit (i.e., $\mathbf{f}_{1}=\mathbf{h}^{T}\mathbf{w}_{1}$) and less varied non-ground-truth class logits (i.e., $\mathbf{g}=[\mathbf{h}^{T}\mathbf{w}_{c}]_{c\in\{2,3,4,5\}}$). A larger target logit or less varied non-ground-truth class logits will both lead to less varied non-ground-truth class probabilities after applying softmax function [31].类别的可视化。根据分类问题中 softmax 的性质，每个类别的分类权重会被拉向同一类别的特征，同时被推离其他类别的特征 [70]。也就是说，特征的变化趋势可以在一定程度上反映最后一层中分类权重的趋势。理想的情况是学习到的分类权重收敛到特征中心。如果类内特征角度变大，特定样本的特征将远离其他类别的分类权重。同样，特定样本的特征将更接近相应类别的分类权重。这将导致真实类别具有更大的逻辑值。如果类内特征几乎正交，非真实类别的逻辑值将变化较小。插图见图 4，其中较大的教师提取出更紧凑的类间特征和更分散的类内特征。三角形代表第 1 类中的样本特征，较大的教师倾向于给出更大的目标逻辑值（即 𝐟1=𝐡T𝐰1）和变化较小的非真实类别逻辑值（即 𝐠=[𝐡T𝐰c]c∈{2,3,4,5}）。较大的目标逻辑值或变化较小的非真实类别逻辑值在应用 softmax 函数后都会导致非真实类别概率的变化较小 [31]。

The step-by-step analysis and visualization provide explanations for the first observation that larger teachers tend to produce probability vectors that are less varied between non-ground-truth classes.逐步的分析和可视化为第一个观察结果提供了解释，即较大的教师倾向于生成在非真实类别间变化较小的概率向量。

The second observation in Sect. 4.1 implies that the relative probability values given by different teachers seem to be consistent. Hence, we explore this observation further through several quantitative metrics. The intuitive idea is to explore whether different teachers give the same top-$K$ predictions or not. Specifically, we train a smaller teacher $t_{\text{small}}$ and a larger teacher $t_{\text{large}}$ on the same dataset. Then, we calculate the predicted top-$K$ classes for the $i$-th training sample, and denote the predicted class sets as $\mathcal{C}_{i,K}^{t_{\text{large}}}$ and $\mathcal{C}_{i,K}^{t_{\text{small}}}$, respectively. We denote $|\mathcal{C}_{i,K}^{t_{\text{large}}}\cap\mathcal{C}_{i,K}^{t_{\text{small}}}|$ as the number of overlapped classes. Then we show the distribution of the number of overlapped classes when considering different $K$. Fig. 5 displays the results on CUB and TinyImageNet. The percentages in the figure show how many data samples have the number of overlapping classes corresponding to the values of the y-axis. Considering the top-$3$ predicted classes of ResNeXt50-32-4d and ResNeXt101-32-8d on CUB, there are $21.6\%$ training samples that have 3 common predicted classes, and $56.4\%$ training samples that have 2 common predicted classes. If we consider top-$8$ predicted classes, there are nearly $69.4\%$ training samples that have at least 5 common predicted classes. This implies that different teachers indeed have a great deal of consistency in top-$K$ class recognition.4.1 节中的第二个观察结果表明，不同教师给出的相对概率值似乎是一致的。因此，我们通过几个定量指标进一步探索这一观察。直观的想法是探究不同的教师是否给出相同的 Top-K 预测。具体而言，我们在同一数据集上训练一个较小的教师 tsmall 和一个较大的教师 tlarget。然后，我们计算第 i 个训练样本的预测 Top-K 类别，并分别将其预测类别集记为 𝒞i,Ktlarge 和 𝒞i,Ktsmall。我们将 |𝒞i,Ktlarge∩𝒞i,Ktsmall| 记为重叠类别的数量。然后，我们展示在考虑不同 K 值时重叠类别数量的分布。图 5 展示了在 CUB 和 TinyImageNet 上的结果。图中的百分比显示了有多少数据样本具有对应于 y 轴值的重叠类别数量。考虑 ResNeXt50-32-4d 和 ResNeXt101-32-8d 在 CUB 上的 Top-3 预测类别，有 21.6% 的训练样本具有 3 个共同预测类别，有 56.4% 的训练样本具有 2 个共同预测类别。如果考虑 Top-8 预测类别，则有近 69.4% 的训练样本至少具有 5 个共同预测类别。这意味着不同的教师在 Top-K 类别识别方面确实具有极高的一致性。

Then, we define several specific metrics to further explore the consistency. The first metric is named the rank set overlap ratio, which is calculated as follows:然后，我们定义了几个特定指标来进一步探索这种一致性。第一个指标称为秩集重叠比，计算如下：

where $t_{1}$, $t_{2}$ denote teachers. The second metric is the Spearman correlation. It first obtains the rank of all classes in the output logits of two teachers, i.e., $\mathbf{R}_{i}^{t}=\text{argsort}(\mathbf{f}_{i}^{t})$, $t\in\{t_{1},t_{2}\}$. Then, it calculates the Pearson correlation as follows:其中 t1, t2 表示教师。第二个指标是 Spearman 相关系数。它首先获得两个教师输出逻辑值中所有类别的秩，即 𝐑it=argsort(𝐟it), t∈{t1,t2}。然后，计算 Pearson 相关系数如下：

where $j$ is the index of class and $\overline{\mathbf{R}_{i}^{t}}$ means the average of the elements in $\mathbf{R}_{i}^{t}$. Finally, we calculate Kendall’s $\tau$ between $\mathbf{f}_{i}^{t_{1}}$ and $\mathbf{f}_{i}^{t_{2}}$, which directly shows the rank correlation of two teachers. The formulation is:其中 j 是类别索引，𝐑it¯ 表示 𝐑it 中元素的平均值。最后，我们计算 𝐟it1 和 𝐟it2 之间的 Kendall’s τ，这直接显示了两个教师的秩相关性。公式为：

where $j_{1}$ and $j_{2}$ show the index of classes. $C$ is the number of classes, and $\text{sg}(\cdot)$ returns 1 or -1 depending on whether the input is positive or not. These three metrics only depend on classes’ relative magnitudes and are irrelevant to the absolute values. The metrics are calculated on CIFAR-10 under a series of ResNeXt networks. We set $K=5$ for the rank set overlap metric. The above equations only show the metrics on a single training sample, and we report the average result on 50K training samples. The results are in Fig. 6. Excitingly, these metrics among teachers with different capacities do not vary a lot, and the Spearman correlations are almost all larger than $0.85$. According to the interpretation of Kendall’s $\tau$ [71, 72], if the smaller teacher predicts that class $j_{1}$ is more related to the target class than that of class $j_{2}$, then the larger teacher has a probability of $0.875$ to give the same relative affinity. Certainly, the output logits of different networks are not completely identical in order, even for the same networks under different training randomness [73, 74]. These metrics demonstrate that teachers know approximately the same about relative class affinities.其中 j1 和 j2 表示类别索引。C 是类别数量，sg(⋅) 根据输入是否为正返回 1 或 -1。这三个指标仅取决于类别的相对量级，与绝对值无关。这些指标是在 CIFAR-10 上对一系列 ResNeXt 网络进行计算的。对于秩集重叠指标，我们设置 K=5。上述方程仅显示了单个训练样本上的指标，我们报告了 50K 个训练样本上的平均结果。结果见图 6。令人兴奋的是，不同容量教师之间的这些指标差异不大，Spearman 相关系数几乎都大于 0.85。根据 Kendall’s τ [71, 72] 的解释，如果较小的教师预测类别 j1 与目标类别的相关性高于类别 j2，那么较大的教师有 0.875 的概率给出相同的相对亲和力。当然，即使对于在不同训练随机性下 [73, 74] 的相同网络，不同网络的输出逻辑值在顺序上也不完全相同。这些指标证明了教师们对于类别相对亲和力的认知大致相同。

During training, the cross-entropy loss minimizes the divergence between the predicted posterior distribution and the empirical one-hot label distribution. Although one-hot labels do not capture class relationships, the posterior distribution does. The teacher network’s probability vector, $\mathbf{p}^{t}$, inherently represents this distribution, serving as an alternative that may help the student learn class affinities. However, since the true posterior distribution for each sample is unknown, it is difficult to ascertain which teacher’s label is superior.在训练期间，交叉熵损失最小化了预测后验分布与经验独热标签分布之间的差异。尽管独热标签不能捕捉类别关系，但后验分布可以。教师网络的概率向量 𝐩t 本质上代表了这种分布，作为一种替代方案，可能有助于学生学习类别亲和力。然而，由于每个样本的真实后验分布未知，很难确定哪个教师的标签更优。

While precise posterior distribution assignment per training sample remains challenging, group classification scenarios offer valuable inter-category priors. CIFAR-100 contains 100 categories, which contain 20 superclasses and 5 classes for each superclass ¹¹ 1 https://www.cs.toronto.edu/~kriz/cifar.html. For example, the superclass “flowers” contains five fine-grained classes including “orchids”, “poppies”, “roses”, “sunflowers” and “tulips”. One hypothesis is that categories in the same superclass are more similar to each other. We train different teachers (ranges from ResNet14 to ResNeXt29-64-4d) on CIFAR-100 and find that the hypothesis holds true for most superclasses. Fig. 7 presents averaged posterior distribution on three superclasses: “flowers”, “small mammals”, and “trees”. For each superclass, we select its first fine-grained class as the target label, i.e., “orchids”, “hamster”, and “maple”. For each target label, we obtain its corresponding training samples, i.e., 500 samples for each. We calculate the averaged probabilities via $\mathbb{E}_{\mathbf{x},y=c}[\mathbf{p}^{t}(\mathbf{x};\tau)]$ ($\tau$=4.0) for $c$ being one of the three target classes. The circle size and color map represent the probability value. Key observations: (1) Target classes exhibit stronger affinity to intra-superclass members across all networks; (2) Relative intra-group probabilities maintain cross-architectural consistency. For example, different teacher networks agree that fine-grained classes “oak” and “willow” are more relevant to the target category “maple”, while “palm” is less relevant. (3) Probability variance decreases with increasing model capacity, evident from diminishing circle size differences in deeper networks. For example, the difference in circle sizes in the last row is much smaller than that in the first row, which behaves similarly in each subfigure.虽然每个训练样本的精确后验分布分配仍然具有挑战性，但分组分类场景提供了有价值的类间先验。CIFAR-100 包含 100 个类别，其中包含 20 个超类，每个超类有 5 个类别。例如，超类“花卉”包含五个细粒度类别，包括“兰花”、“罂粟”、“玫瑰”、“向日葵”和“郁金香”。一个假设是，同一超类中的类别彼此更为相似。我们在 CIFAR-100 上训练了不同的教师（范围从 ResNet14 到 ResNeXt29-64-4d），发现该假设对大多数超类成立。图 7 展示了三个超类“花卉”、“小型哺乳动物”和“树木”上的平均后验分布。对于每个超类，我们选择其第一个细粒度类别作为目标标签，即“兰花”、“仓鼠”和“枫树”。对于每个目标标签，我们获取其对应的训练样本，即每个标签 500 个样本。我们通过 𝔼𝐱,y=c[𝐩t(𝐱,τ)] (τ=4.0) 计算平均概率，其中 c 为三个目标类别之一。圆圈大小和颜色映射代表概率值。关键观察结果：（1）目标类别在所有网络中对超类内成员表现出更强的亲和力；（2）组内相对概率保持跨架构的一致性。例如，不同的教师网络都认为细粒度类别“橡树”和“柳树”与目标类别“枫树”的相关性更高，而“棕榈”的相关性较低。（3）概率方差随模型容量的增加而减小，这从更深层网络中圆圈大小差异的减小可以明显看出。例如，最后一行中圆圈大小的差异远小于第一行，每个子图中的表现类似。

Then we provide some rules that define the similarity between classes. For example, the rule “$c_{1}$: $c_{2}$, $c_{3}$, $c_{4}$, $c_{5}$” represents that a sample in class $c_{1}$ is predicted with the following rank: $\mathbf{p}_{c_{1}}>\mathbf{p}_{c_{2}}>\mathbf{p}_{c_{3}}>\mathbf{p}_{c_{4}}>\mathbf{p}_{c_{5}}$. If a network gives the rank as “$c_{1}$, $c_{3}$, $c_{2}$, $c_{4}$, $c_{5}$”, then the Kendall’s $\tau$ between the prediction and the rule is 0.8. To measure the absolute difference in the predictions of non-ground-truth classes, we also calculate the standard deviation of the probabilities, i.e., $\sigma\left(\left[\mathbf{p}_{c_{2}},\mathbf{p}_{c_{3}},\mathbf{p}_{c_{4}},\mathbf{p}_{c_{5}}\right]\right)$. The following lists the rules defined in CIFAR-100:然后，我们提供了一些定义类别间相似性的规则。例如，规则“c1: c2, c3, c4, c5”表示类 c1 中的样本预测排序为：𝐩c1>𝐩c2>𝐩c3>𝐩c4>𝐩c5。如果一个网络给出的排序为“c1, c3, c2, c4, c5”，则该预测与规则之间的 Kendall’s τ 为 0.8。为了衡量非真实类别预测中的绝对差异，我们还计算了概率的标准差，即 σ([𝐩c2,𝐩c3,𝐩c4,𝐩c5])。以下列出了 CIFAR-100 中定义的规则：

• •

  C.1: hamster: mouse, rabbit, squirrel, shrewC.1: 仓鼠: 老鼠, 兔子, 松鼠, 鼩鼱

• •

  C.2: orchids: tulips, roses, poppies, sunflowersC.2: 兰花: 郁金香, 玫瑰, 罂粟, 向日葵

• •

  C.3: maple: oak, willow, pine, palmC.3: 枫树: 橡树, 柳树, 松树, 棕榈

• •

  C.4: dolphin: whale, seal, otter, mammals beaverC.4: 海豚: 鲸鱼, 海豹, 水獭, 哺乳动物海狸

• •

  C.5: apples: sweet peppers, oranges, pears, mushroomsC.5: 苹果: 甜椒, 橙子, 梨, 蘑菇

These rules are derived from the averaged posterior probability distributions in Fig. 7. For simplicity, we only consider intra-superclass relationships. We further construct a group classification task using five digit recognition datasets: MNIST, MNIST-M, USPS, SVHN, and SYN. We denote this as Digits-Five, which is often used in domain adaptation tasks [38] or task relationship estimation [75]. The construction process is simple where we only centralize the data together and obtain a task with $5\times 10=50$ classes. Based on transfer performance analyses and task relation coefficients from existing research [38, 75], we abstract several rules as follows:这些规则源自图 7 中的平均后验概率分布。为简单起见，我们仅考虑超类内的关系。我们进一步使用五个数字识别数据集构建了一个分组分类任务：MNIST、MNIST-M、USPS、SVHN 和 SYN。我们将其称为 Digits-Five，这常用于领域自适应任务 [38] 或任务关系估计 [75]。构建过程很简单，我们只是将数据集中在一起，获得一个包含 5×10=50 个类别的任务。基于现有研究 [38, 75] 的迁移性能分析和任务关系系数，我们抽象出以下规则：

• •

  D.1: MNIST@c: USPS@c, SVHN@cD.1: MNIST@c: USPS@c, SVHN@c

• •

  D.2: MNIST@c: MNISTM@c, SYN@cD.2: MNIST@c: MNISTM@c, SYN@c

• •

  D.3: USPS@c: MNIST@c, MNISTM@c, SVHN@cD.3: USPS@c: MNIST@c, MNISTM@c, SVHN@c

MNIST@c represents the $c$-th class in MNIST, and others are defined similarly. Each of the above rules contains a group of fine-grained rules if we set $c\in\{0,1,\cdots,9\}$. Hence, the averaged results across the fine-grained ones are reported for each rule.MNIST@c 表示 MNIST 中的第 c 类，其他类推。如果我们设置 c∈{0,1,⋯,9}，上述每条规则都包含一组细粒度规则。因此，我们报告了每条规则在细粒度类别上的平均结果。

While these rules may not apply to every sample, they may provide an approximate estimation of a sample’s true posterior distribution. Table 1 presents the results for Kendall’s $\tau$ and the standard deviation of probabilities for non-ground-truth classes. The consistency of teacher outputs with the defined rules shows little variation; in the simpler Digits Five task, the predicted ranks are nearly identical (with Kendall’s $\tau$ approaching 1.0). However, larger teachers yield much smaller standard deviations than smaller ones, indicating less variability among non-ground-truth classes. These experiments on group classification further validate the initial observations in Sect. 4.1.虽然这些规则可能不适用于每个样本，但它们可以提供样本真实后验分布的近似估计。表 1 展示了 Kendall’s τ 和非真实类别概率标准差的结果。教师输出与定义规则的一致性几乎没有变化；在更简单的 Digits-Five 任务中，预测排序几乎完全相同（Kendall’s τ 接近 1.0）。然而，较大的教师产生的标准差远小于较小的教师，这表明非真实类别之间的变异性较小。这些关于分组分类的实验进一步验证了 4.1 节中的初步观察。

For each class $c$, we calculate the averaged probabilities via $\mathbb{E}_{\mathbf{x},y=c}[\mathbf{p}(\mathbf{x};\tau_{0})]$, and advocate that this captures the global relationships among classes. According to Fig. 7, the global relationships given by different teachers are similar in relative values, which may be useful for stable distillation. Hence, we fuse this to each training sample of the $c$-th class as follows:对于每个类别 c，我们通过 𝔼𝐱,y=c[𝐩(𝐱,τ0)] 计算平均概率，并主张这捕捉了类别之间的全局关系。根据图 7，不同教师给出的全局关系在相对值上是相似的，这对于稳定的蒸馏可能是有用的。因此，我们将此融合到第 c 类的每个训练样本中，如下所示：

where $\alpha$ is the hyper-parameter. The relative class affinity suggests that small teacher can be regarded as a form of label smoothing (LS) for the large teacher, as both ensure consistency in the rank order of classes’ probabilities. Currently, the large teacher exhibits an overly one-hot output, necessitating LS to increase the variance of non-ground-truth class probabilities. Although LS can hurt knowledge KD [46],  [76] demonstrates that label smoothing is compatible with KD at lower temperatures. Therefore, we set $\tau_{0}<\tau$. As shown in Fig. 7, the averaged probability distribution provided by larger networks still shows less varied values among non-ground-truth classes, which may limit the effectiveness of this method. Hence, we only provide this method as a trial, and it is not the focus of our paper. In fact, the performances of FGCR could surpass the distillation performance of a smaller teacher, while it is still worse than other advanced techniques as shown in Tab. 2.其中 α 是超参数。类别相对亲和力表明，小型教师可以被视为大型教师的一种标签平滑（LS）形式，因为两者都确保了类别概率排序的一致性。目前，大型教师表现出过于独热的输出，需要 LS 来增加非真实类别概率的方差。尽管 LS 可能损害知识蒸馏 [46]，但 [76] 表明标签平滑在较低温度下与 KD 是兼容的。因此，我们设置 τ0<τ。如图 7 所示，大型网络提供的平均概率分布在非真实类别之间仍然显示出变化较小的值，这可能会限制该方法的有效性。因此，我们仅将此方法作为一种尝试提供，它并非我们论文的重点。事实上，FGCR 的性能可以超过小型教师的蒸馏性能，但如表 2 所示，它仍然逊色于其他先进技术。

During the phase of training teachers, we add the following regularization to enhance the variance of non-ground-truth class probabilities:在训练教师的阶段，我们添加以下正则化以增强非真实类别概率的方差：

where $\mathbf{q}(\mathbf{x};1.0)=[\mathbf{p}_{c}(\mathbf{x};1.0)]_{c\neq y}$ and $\beta$ is the coefficient of the regularization term. Equation (8) explicitly encourages teacher network to decrease the influence of the ground-truth class and enhance the distinctness of non-ground-truth classes. Different from other post-processing methods, this way could enhance the variance of non-ground-truth class probabilities in the stage of training teachers, which could provide an excellent initialization for distillation.其中 𝐪(𝐱,1.0)=[𝐩c(𝐱,1.0)]c≠y，β 是正则化项的系数。方程 (8) 明确鼓励教师网络降低真实类别的影响，并增强非真实类别的区分度。与其他后处理方法不同，这种方法可以在训练教师的阶段增强非真实类别概率的方差，从而为蒸馏提供极好的初始化。

The previous work [31] proposes the approach named asymmetric temperature scaling (ATS) to tackle the capacity mismatch in KD. ATS applies different temperatures to the logits of ground-truth and non-ground-truth classes, i.e.,已有研究 [31] 提出了一种名为非对称温度缩放（Asymmetric Temperature Scaling, ATS）的方法，旨在解决知识蒸馏（KD）中的容量失配问题。ATS 对真实类别和非真实类别的 Logits 采用不同的温度，即：

where $\tau_{j}=\tau_{1}$ for $j=y$ and $\tau_{j}=\tau_{2}$ for $j\neq y$. The recommended setting is $\tau_{1}-\tau_{2}\in[1,2]$. This could effectively make the larger networks teach well again, but the effort in searching for proper hyper-parameters of $\tau_{1}$ and $\tau_{2}$ is huge.其中，当 j=y 时 τj=τ1，当 j≠y 时 τj=τ2。推荐的参数设置为 τ1−τ2∈[1,2]。这可以有效地使大型网络重新发挥良好的教学作用，但寻找合适的 τ1 和 τ2 超参数需要付出巨大的努力。

As a step further solution, we extend ATS to Instance-Specific ATS (ISATS) that searches for proper temperatures for each training sample. Given a training sample $\mathbf{x}$, the predicted logit vector is $\mathbf{f}(\mathbf{x})$, and the optimal temperature that could enlarge the variance of softened non-ground-truth classes’ probabilities is:作为进一步的解决方案，我们将 ATS 扩展为实例特定 ATS（Instance-Specific ATS, ISATS），即为每个训练样本搜索合适的温度。给定一个训练样本 𝐱，其预测的 Logit 向量为 𝐟⁡(𝐱)，能够增大软化后的非真实类别概率方差的最优温度为：

where $\mathbf{q}(\mathbf{x};\tau)$ denotes the probability vector of non-ground-truth classes in teacher network, i.e., $\mathbf{q}(\mathbf{x};\tau)=[\mathbf{p}_{c}(\mathbf{x};\tau)]_{c\neq y}$ and $\mathbf{p}(\mathbf{x};\tau)=\text{SF}(\mathbf{f}(\mathbf{x});\tau)$. It is noted that, according to relative class affinity, enlarging the teacher model does not alter the rank order among non-ground-truth classes. Consequently, all non-target classes share the same $\tau$, eliminating the need to apply different $\tau$ values across classes. $v(\cdot)$ calculates the variance of the elements in a vector. That is, the optimal temperature for each instance is searched to make the probabilities of non-ground-truth classes more distinct. Then, we set $\tau_{1}(\mathbf{x})=\tau^{\star}+1$ and $\tau_{2}(\mathbf{x})=\tau^{\star}$ as recommended by ATS [31]. Equation (10) has no closed-form solution. So like ATS, ISATS still requires manual search to ensure an appropriate temperature. Nevertheless, ISATS has two advantages over ATS. First, ISATS saves the effort of finding the proper value of $\tau_{2}$ and $\tau_{1}$. ATS requires performing multiple distillation processes and comparing the experimental results at different temperatures to determine proper $\tau$. In contrast, ISATS can directly compute $\tau^{\star}$ based on (10) in advance, eliminating the need for comparative distillation experiments. Second, ISATS considers a more fine-grained way to release the discriminative information contained in wrong classes for each training sample. In summary, ISATS is an all-around improvement over ATS.其中，𝐪⁡(𝐱,τ) 表示教师网络中非真实类别的概率向量，即 𝐪⁡(𝐱,τ)=[𝐩c​(𝐱,τ)]c≠y，且 𝐩⁡(𝐱,τ)=SF​(𝐟⁡(𝐱),τ)。值得注意的是，根据相对类别亲和力，增大教师模型并不会改变非真实类别之间的排序。因此，所有非目标类别共享同一个 τ，从而消除了跨类别应用不同 τ 值的必要。v⁡(⋅) 计算向量中元素的方差。也就是说，为每个实例搜索最优温度，旨在使非真实类别的概率更具区分度。然后，按照 ATS [31] 的建议，我们设置 τ1​(𝐱)=τ⋆+1 和 τ2​(𝐱)=τ⋆。方程 (10) 没有闭式解，因此与 ATS 一样，ISATS 仍需人工搜索以确保合适的温度。尽管如此，ISATS 相比 ATS 有两个优势。首先，ISATS 省去了寻找 τ2 和 τ1 合适值的繁琐工作。ATS 需要进行多次蒸馏过程，并比较不同温度下的实验结果以确定合适的 τ。相比之下，ISATS 可以预先根据 (10) 直接计算出 τ⋆，无需进行对比蒸馏实验。其次，ISATS 考虑了一种更细粒度的方式，为每个训练样本释放错误类别中所包含的判别性信息。总之，ISATS 是对 ATS 的全面改进。

We compare with SOTA methods and list the results on CIFAR-100, TinyImageNet, CUB, and Dogs in Tab. 2. The compared methods include ESKD [32], TAKD [33], SCKD [34], and ATS [31]. NoKD trains students without the teacher’s supervision. ST-KD trains students under the guidance of a smaller teacher. KD trains students under the guidance of the larger teacher. The larger teachers are ResNet110, WRN50-2, RNX101-32-8d, and RNX101-32-8d for the four datasets, while the smaller teachers are ResNet20, WRN26-2, RNX50-32-4d, and RNX50-32-4d, respectively. The values in “()” display the test accuracy of the large teachers. The last column of the table shows the average performance of corresponding rows. The last three rows present the performances of our proposed algorithms. These methods could improve the performances of students when taught by a larger network and surpass that of KD and ST-KD, which verifies that enhancing the variance of non-ground-truth classes could indeed make larger networks teach well again. Although dataset bias exists in a few circumstances, the proposed ISATS could achieve the SOTA results when compared with ATS. It is easy to understand that ISATS searches for optimal temperatures in an instance-specific manner, which could enhance the variance of non-ground-truth class probabilities more effectively. The two methods presented in Sections 5.1 and 5.2 are also meaningful. RegT belongs to the pre-processing method that enhances the variance of non-ground-truth classes during the phase of training teachers, which also presents good performances. RegT does not directly affect the teacher model’s error class variance like ATS, so its performance is not as good as ATS. However, together with ESKD [32], it indicates that a more suitable teacher is benefical to KD performance improvement and provides a simple and effective method for teacher regularization. Similarly, the performance improvement of FGCR is less pronounced than ATS, but it indirectly supports the rationale for moderate LS. The experimental data for these two methods, from the perspective of wrong-class variance that we introduced, demonstrate the validity of previous works that tried to address capacity mismatch.我们将本文方法与 SOTA 方法进行比较，并在表 2 中列出了在 CIFAR-100、TinyImageNet、CUB 和 Dogs 上的结果。对比方法包括 ESKD [32]、TAKD [33]、SCKD [34] 和 ATS [31]。NoKD 在没有教师监督的情况下训练学生模型。ST-KD 在较小教师的指导下训练学生。KD 在较大教师的指导下训练学生。对于四个数据集，较大的教师分别是 ResNet110、WRN50-2、RNX101-32-8d 和 RNX101-32-8d，而较小的教师分别是 ResNet20、WRN26-2、RNX50-32-4d 和 RNX50-32-4d。“()”中的数值显示了大教师的测试准确率。表格的最后一列显示了相应行的平均性能。最后三行展示了我们所提算法的性能。这些方法能够在由较大网络指导时提高学生的性能，并超过 KD 和 ST-KD，这验证了增强非真实类别的方差确实能使大型网络重新发挥良好的教学作用。尽管在少数情况下存在数据集偏差，但与 ATS 相比，所提的 ISATS 能够达到 SOTA 结果。不难理解，ISATS 以实例特定的方式搜索最优温度，这能更有效地增强非真实类别概率的方差。第 5.1 节和 5.2 节提出的两种方法也具有意义。RegT 属于预处理方法，在教师训练阶段增强非真实类别的方差，其表现也很好。RegT 不像 ATS 那样直接影响教师模型的错误类别方差，因此其性能不如 ATS。然而，它与 ESKD [32] 一起表明，更合适的教师有利于 KD 性能的提升，并为教师正则化提供了一种简单有效的方法。同样，FGCR 的性能提升不如 ATS 明显，但它间接支持了适度标签平滑（LS）的合理性。从我们引入的错误类别方差角度来看，这两种方法的实验数据证明了以往试图解决容量失配问题的研究的有效性。

We also verify the ensemble performances after distillation because ResKD [35, 36] improves the students’ performances by introducing the residual student and taking the two residual students’ ensemble. Although ResKD surpasses the performance of ATS, this comparison is not fair. Hence, we also provide ensemble performances of two separately trained students under different initialization and training seeds. For a fair comparison, we also test the performances of our methods by repeating the corresponding algorithms two times and making predictions via the ensemble of the obtained two student networks. The results are listed in Tab. 3. Clearly, our proposed ISATS could also achieve better performances when compared with ATS and other methods.我们还验证了蒸馏后的集成性能，因为 ResKD [35, 36] 通过引入残差学生并对两个残差学生进行集成来提高学生性能。尽管 ResKD 的性能超过了 ATS，但这种比较并不公平。因此，我们也提供了在不同初始化和训练种子下分别训练的两个学生的集成性能。为了公平比较，我们还通过重复相应算法两次，并对获得的两个学生网络进行集成预测，来测试我们方法的性能。结果列于表 3 中。显然，与 ATS 和其他方法相比，我们提出的 ISATS 也能实现更好的性能。

Fig. 8 further demonstrates that ISATS can achieve superior KD performance with a given teacher model size, avoiding the substantial effort required to find an appropriate teacher model in the situation of TS. Notably, ISATS outperforms ATS in distillation performance, particularly when the teacher model is large.图 8 进一步证明，ISATS 可以在给定教师模型大小时实现更优的 KD 性能，避免了在 TS 情况中寻找合适教师模型所需的大量工作。值得注意的是，ISATS 在蒸馏性能上优于 ATS，特别是在教师模型较大时。