Knowledge Distillation of Black-Box Large Language Models黑盒大型语言模型的知识蒸馏

Traditional knowledge distillation (KD) research predominantly employs white-box teachers and is typically classified into three main branches: feature-based, response-based, and relation-based methods. Feature-based methods seek to replicate the teacher’s intermediate representations, such as attention scores (Jiao et al., 2020), attribution maps (Wu et al., 2023), and hidden representations of tokens (Sun et al., 2019). Response-based methods train the student model by minimizing divergences like Kullback–Leibler (KL) divergence (Hinton et al., 2015; Sanh et al., 2019), reverse KL (Gu et al., 2023; Wen et al., 2023), Jensen–Shannon Divergence (JSD) (Fang et al., 2021; Yin et al., 2020), and Total Variation Distance (TVD) (Wen et al., 2023) based on the teacher’s output distribution. Relation-based methods train the student model by learning pairwise distances and triple-wise angles among token representations from the teacher (Park et al., 2021), or extracting structural relations from multi-granularity representations (Liu et al., 2022).传统的知识蒸馏（KD）研究主要采用白盒教师，通常分为三个主要分支：基于特征、基于响应和基于关系的方法。基于特征的方法试图复制教师的中间表示，如注意力分数（Jiao et al., 2020）、归因图（Wu et al., 2023）和标记的隐藏表示（Sun et al., 2019）。基于响应的方法通过基于教师输出分布最小化诸如Kullback-Leibler（KL）散度（Hinton et al., 2015; Sanh et al., 2019）、反向KL（Gu et al., 2023; Wen et al., 2023）、Jensen-Shannon散度（JSD）（Fang et al., 2021; Yin et al., 2020）和全变差距离（TVD）（Wen et al., 2023）等散度来训练学生模型。基于关系的方法通过学习教师标记表示之间的成对距离和三元角度（Park et al., 2021），或从多粒度表示中提取结构关系（Liu et al., 2022）来训练学生模型。

Given the remarkable performance achieved by proprietary LLMs like GPT-4 (OpenAI, 2023), Claude 3 Anthropic (2024), and Gemini (Team et al., 2023), recent studies like Alpaca (Taori et al., 2023), Vicuna (Chiang et al., 2023), and Orca (Mukherjee et al., 2023) have focused on transferring diverse capabilities from these black-box teachers into smaller open-source models. For instance, Li et al. (2024) and Liu et al. (2023) improved the mathematical capability of small models by training on tailored rationale samples generated by GPT-3.5-Turbo and GPT-4. To transfer the code generation capability, Azerbayev et al. (2023) prompted Codex (Chen et al., 2021) to create natural language-code pairs and fine-tuned a smaller model on these samples. To transfer the tool usage capability, Gou et al. (2023) utilized GPT-4 to generate interactive tool-use trajectories as training samples for the target model. Other approaches, such as Hsieh et al. (2023); Ho et al. (2022); Chen et al. (2023), utilize rationales generated by black-box teachers as training data to transfer their general reasoning capabilities.鉴于GPT-4（OpenAI, 2023）、Claude 3 Anthropic（2024）和Gemini（Team et al., 2023）等专有LLM所取得的显著性能，近期研究如Alpaca（Taori et al., 2023）、Vicuna（Chiang et al., 2023）和Orca（Mukherjee et al., 2023）专注于将这些黑盒教师的各种能力迁移到较小的开源模型中。例如，Li et al.（2024）和Liu et al.（2023）通过使用GPT-3.5-Turbo和GPT-4生成的定制推理样本训练小模型，提高了小模型的数学能力。为了迁移代码生成能力，Azerbayev et al.（2023）提示Codex（Chen et al., 2021）创建自然语言-代码对，并在这些样本上微调较小模型。为了迁移工具使用能力，Gou et al.（2023）利用GPT-4生成交互式工具使用轨迹作为目标模型的训练样本。其他方法，如Hsieh et al.（2023）; Ho et al.（2022）; Chen et al.（2023），利用黑盒教师生成的推理作为训练数据来迁移其一般推理能力。

White-box knowledge distillation (KD) efficiently distills knowledge by leveraging the internal states of the teacher model. However, white-box teachers typically possess a more limited capacity compared to their black-box counterparts. In contrast, black-box KD capitalizes on the superior performance of the teacher models but is restricted to fine-tuning on teacher-generated samples. This approach captures input-output patterns without accessing the deeper, intrinsic knowledge of the teacher model. To bridge these gaps, we propose Proxy-KD, a straightforward method that combines the strengths of both white-box and black-box KD while mitigating their respective limitations.白盒知识蒸馏（KD）通过利用教师模型的内部状态有效地蒸馏知识。然而，白盒教师通常比黑盒教师具有更有限的容量。相比之下，黑盒KD利用了教师模型的优越性能，但仅限于在教师生成的样本上进行微调。这种方法捕获输入-输出模式，而无法访问教师模型更深层的内在知识。为了弥补这些差距，我们提出了Proxy-KD，一种简单的方法，结合了白盒和黑盒KD的优点，同时减轻了它们各自的局限性。

The proposed Proxy-KD method draws inspiration from TAKD (Mirzadeh et al., 2020), as both methods use an intermediate network to aid knowledge distillation, but they differ in three significant ways. Firstly, the motivation behind each approach is distinct: TAKD focuses on mitigating the capacity gap between the teacher and student in white-box settings, whereas Proxy-KD addresses the challenges posed by black-box teacher models and seeks to incorporate the benefits found in white-box scenarios. Secondly, the methodologies diverge, with Proxy-KD introducing a crucial proxy alignment phase that includes preference optimization to better align the proxy model with the black-box LLM. This step is essential for reducing discrepancies between the proxy and teacher models, thereby improving the effectiveness of the distillation process. Lastly, they operate in different domains: TAKD is applied in the field of computer vision, while Proxy-KD is specifically designed for natural language processing, targeting the distillation of proprietary large language models (LLMs).所提出的Proxy-KD方法借鉴了TAKD（Mirzadeh et al., 2020），因为两种方法都使用中间网络来辅助知识蒸馏，但它们在三个重要方面有所不同。 首先，每种方法的动机不同：TAKD专注于缓解白盒设置下教师与学生之间的容量差距，而Proxy-KD解决黑盒教师模型带来的挑战，并寻求融合白盒场景中的优势。 其次，方法论不同，Proxy-KD引入了一个关键的代理对齐阶段，包括偏好优化，以更好地将代理模型与黑盒LLM对齐。这一步骤对于减少代理和教师模型之间的差异，从而提高蒸馏过程的有效性至关重要。 最后，它们应用领域不同：TAKD应用于计算机视觉领域，而Proxy-KD专门为自然语言处理设计，针对专有大型语言模型（LLM）的蒸馏。

To facilitate the transfer of knowledge from a black-box teacher LLM $\pi_{t}$ to a smaller, open-source student LLM $\pi_{s}$, we introduce a proxy model $\pi_{p}$. The training dataset $\mathcal{D}$ consists of input-output pairs $(x,y)$, where $x$ represents the input prompt and $y$ is the output sequence generated by the teacher model $\pi_{t}$. This dataset is strategically divided into three parts: 10% ($\mathcal{D}_{w}$) for the warm-up phase, 45% ($\mathcal{D}_{p}$) for aligning the proxy model with the teacher, and the remaining 45% ($\mathcal{D}_{s}$) for the knowledge distillation training of the student model.为了促进知识从黑盒教师LLM πtsubscript𝜋𝑡\pi_{t}... 向更小的开源学生LLM πssubscript𝜋𝑠\pi_{s}... 的迁移，我们引入代理模型 πpsubscript𝜋𝑝\pi_{p}...。训练数据集 𝒟𝒟\mathcal{D}... 由输入-输出对 (x,y)... 组成，其中 x... 表示输入提示，y... 是教师模型 πt... 生成的输出序列。该数据集被策略性地分为三部分：10%（𝒟w...）用于预热阶段，45%（𝒟p...）用于代理模型与教师对齐，其余45%（𝒟s...）用于学生模型的知识蒸馏训练。

The process begins with a warm-up phase where the proxy model $\pi_{p}$ is trained on $\mathcal{D}_{w}$. This phase helps $\pi_{p}$ develop a basic capability to generate responses to input prompts. Following this, the proxy model undergoes alignment with the teacher model $\pi_{t}$ using the next dataset, $\mathcal{D}_{p}$. This alignment is achieved through two methods: hard-label knowledge distillation (KD) and preference learning. These methods enable $\pi_{p}$ to approximate the behavior and outputs of the teacher model. Once aligned, $\pi_{p}$ acts as an intermediary, facilitating the transfer of knowledge to the student $\pi_{s}$ on $\mathcal{D}_{s}$.该过程首先进入预热阶段，代理模型πpsubscript𝜋𝑝\pi_{p}...在𝒟w...上训练。此阶段帮助πpsubscript𝜋𝑝\pi_{p}...发展生成输入提示响应的基本能力。之后，代理模型使用下一个数据集𝒟p...与教师模型πt...进行对齐。这种对齐通过两种方法实现：硬标签知识蒸馏（KD）和偏好学习。这些方法使πpsubscript𝜋𝑝\pi_{p}...能够近似教师模型的行为和输出。一旦对齐，πpsubscript𝜋𝑝\pi_{p}...作为中介，促进在𝒟s...上向学生πs...的知识迁移。

The proxy model $\pi_{p}$ is typically a larger white-box LLM than the student model $\pi_{s}$. For effective knowledge transfer, it’s crucial to first align the output distribution of the proxy model with that of the black-box teacher model $\pi_{t}$. This alignment ensures that the proxy accurately captures the teacher’s behavior.代理模型πpsubscript𝜋𝑝\pi_{p}...通常是一个比学生模型πssubscript𝜋𝑠\pi_{s}...更大的白盒LLM。为了实现有效的知识迁移，首先将代理模型的输出分布与黑盒教师模型πtsubscript𝜋𝑡\pi_{t}...的输出分布对齐至关重要。这种对齐确保代理准确捕获教师的行为。

The proxy model $\pi_{p}$ first undergoes supervised fine-tuning on a warm-up dataset $\mathcal{D}_{w}$. Following this, the proxy is further trained on the $\mathcal{D}_{p}$ dataset by minimizing the NLL loss:代理模型πpsubscript𝜋𝑝\pi_{p}...首先在预热数据集𝒟w...上进行监督微调。之后，代理进一步在𝒟p...数据集上通过最小化NLL损失进行训练：

To enhance the alignment of the proxy model with the teacher, we further introduce a preference learning-based alignment objective, with the hypothesis that the teacher model’s responses are of higher quality compared to those from the unaligned proxy model. The objective is to iteratively adjust the proxy model so that it increasingly favors responses similar to those of the teacher while reducing its preference for its own initial outputs. To implement this, we employ the Direct Preference Optimization (DPO) algorithm (Rafailov et al., 2024), which refines the proxy model by systematically preferring the teacher’s responses.为了增强代理模型与教师的对齐，我们进一步引入基于偏好学习的对齐目标，其假设是教师模型的响应比未对齐代理模型的响应质量更高。目标是迭代调整代理模型，使其越来越偏好与教师相似的响应，同时减少对其自身初始输出的偏好。为此，我们采用直接偏好优化（DPO）算法（Rafailov et al., 2024），通过系统性地偏好教师的响应来改进代理模型。

Specifically, for a given input $x$, we iteratively sample a response $y$ from the teacher and $\hat{y}$ from the proxy. These responses form a preference pair $(x,y,\hat{y})$. To train the proxy model to prefer $y$ over $\hat{y}$, we define the following preference loss function:具体来说，对于给定输入x，我们迭代地从教师采样响应y，从代理采样响应y^。这些响应形成偏好对(x,y,y^)。为了训练代理模型偏好y而非y^，我们定义以下偏好损失函数：

where $\pi_{p}^{(i-1)}$ is the proxy model from the previous training iteration. The overall preference loss over all the preference samples is defined as:其中πp(i−1)是来自前一次训练迭代的代理模型。所有偏好样本上的总体偏好损失定义为：

At each iteration $i$, the proxy model is updated based on the combined objective that includes both the NLL loss and the preference loss:在每次迭代i时，代理模型基于包含NLL损失和偏好损失的组合目标进行更新：

This iterative process continues for a fixed number of iterations $k$ or until the proxy model converges. Through this method, the proxy model $\pi_{p}$ is aligned to emulate the distribution of the black-box teacher $\pi_{t}$, becoming an effective intermediary for transferring knowledge to the student model.该迭代过程持续固定迭代次数k，直到代理模型收敛。 通过这种方法，代理模型πpsubscript𝜋𝑝\pi_{p}...被对齐以模拟黑盒教师πtsubscript𝜋𝑡\pi_{t}...的分布，成为向学生模型迁移知识的有效中介。

To transfer knowledge from the black-box teacher to the student model $\pi_{s}$, we define the first training objective using teacher-generated sequences and the hard-label knowledge distillation objective:为了将知识从黑盒教师迁移到学生模型πssubscript𝜋𝑠\pi_{s}...，我们使用教师生成的序列和硬标签知识蒸馏目标定义第一个训练目标：

Based on the proxy model aligned with the black-box teacher, which delivers accessible output distributions, we define another training objective for the student through soft-label knowledge distillation:基于与黑盒教师对齐且可访问输出分布的代理模型，我们通过软标签知识蒸馏为学生定义另一个训练目标：

In this process, the proxy model functions as an intermediary for the black-box teacher, facilitating the transfer of knowledge to the student model. However, as illustrated in Figure 5 in Appendix, discrepancies between the teacher’s and the proxy’s output distributions persist even after aligning the proxy model, potentially degrading the effectiveness of knowledge distillation. To address these discrepancies, we propose a weighted approach to the soft-label knowledge distillation objective. By introducing weights, we dynamically adjust the influence of each sample based on the alignment quality between the proxy and the black-box teacher. This approach ensures that the student model prioritizes samples where the proxy’s distribution closely matches the teacher’s distribution and reduces focus on samples where it does not. The weights are calculated based on the log-likelihood of the teacher’s output generated by the proxy, normalized by the mean and variance of these log-likelihoods:在此过程中，代理模型充当黑盒教师的中间媒介，促进知识向学生模型的迁移。然而，如图5（附录）所示，即使在代理模型对齐后，教师和代理的输出分布之间仍存在差异，可能降低知识蒸馏的有效性。为了解决这些差异，我们提出一种加权的软标签知识蒸馏目标。通过引入权重，我们根据代理与黑盒教师之间的对齐质量动态调整每个样本的影响。这种方法确保学生模型优先处理代理分布与教师分布紧密匹配的样本，并减少对不匹配样本的关注。权重基于代理生成的教师输出的对数似然计算，并通过这些对数似然的均值和方差进行归一化：

where $w(x,y)$ is a weight reflecting the quality of the proxy’s prediction for the sample $(x,y)$, $\mathbb{V}\text{ar}(\cdot)$ is the variance operation, $\gamma$ is the standard deviation, $\sigma$ is the sigmoid function. Based on Equation (8), we derive the sample-level weighted version of $\mathcal{L}_{\text{Student-KL}}$ as follow:其中w(x,y)是反映代理对样本(x,y)预测质量的权重，𝕍ar(·)是方差运算，γ是标准差，σ是sigmoid函数。基于公式(8)，我们推导出ℒStudent-KL的样本级加权版本如下：

Therefore, the overall objective for student knowledge distillation can be derived as:因此，学生知识蒸馏的总体目标可推导为：

where $\alpha$ is a hyperparameter utilized to adjust the strength of the weighted KL loss.其中α是一个超参数，用于调整加权KL损失的强度。

This knowledge distillation strategy effectively blends the advantages of both black-box and white-box knowledge distillation methods, employing the proxy model to bridge the gap between black-box LLMs and open-source student LLMs.这种知识蒸馏策略有效地融合了黑盒和白盒知识蒸馏方法的优点，利用代理模型弥合黑盒LLM与开源学生LLM之间的差距。

Teacher/Proxy/Student Models. In Proxy-KD, we choose GPT-4 (OpenAI, 2023) as the teacher, which is a powerful proprietary large language model. We select Llama-2-70b (Touvron et al., 2023b) and Llama-2-13b (MetaAI, 2024) as the proxy, respectively. Our student models come from two model types: Llama-1-7B (Touvron et al., 2023a) and Llama-2-7B (Touvron et al., 2023b).教师/代理/学生模型。在Proxy-KD中，我们选择GPT-4（OpenAI, 2023）作为教师，这是一个强大的专有大型语言模型。我们分别选择Llama-2-70b（Touvron et al., 2023b）和Llama-2-13b（MetaAI, 2024）作为代理。我们的学生模型来自两种模型类型：Llama-1-7B（Touvron et al., 2023a）和Llama-2-7B（Touvron et al., 2023b）。

All experiments are conducted on 8×A100 Nvidia GPUs with 80GB memory. All proxy and student models are trained for only one epoch. We use a constant learning rate of 1e-5 and the Adam optimizer, with a max sequence length of 1024. We set hyperparamter $\alpha=100$ in Equation (11), and $k=16$ for the number of proxy alignment iterations. All models are trained using LoRA (Hu et al., 2021) with mixed-precision: frozen parameters in bfloat16 and LoRA-trained parameters in float32.所有实验均在8×A100 Nvidia GPU上运行，显存为80GB。所有代理和学生模型仅训练一个epoch。我们使用恒定的学习率1e-5和Adam优化器，最大序列长度为1024。 我们在公式(11)中设置超参数α=100，并设置k=16作为代理对齐迭代次数。 所有模型均使用LoRA（Hu et al., 2021）和混合精度训练：冻结参数为bfloat16，LoRA训练参数为float32。

We compare Proxy-KD with different white-box KD and black-box KD methods.我们将Proxy-KD与不同的白盒KD和黑盒KD方法进行比较。

White-Box KD. For knowledge distillation with white-box teachers, we compare forward KL methods (Hinton et al., 2015; Agarwal et al., 2024) and reverse KL methods including MiniLLM (Gu et al., 2023) and GKD (Agarwal et al., 2023) (with the same hyperparameters set in the paper.). The chat version of Llama-2-70b is utilized as the white-box teacher. We also compare with using the aligned proxy as white-box teacher to perform distillation.白盒知识蒸馏。对于白盒教师的知识蒸馏，我们比较前向KL方法（Hinton et al., 2015; Agarwal et al., 2024）和反向KL方法，包括MiniLLM（Gu et al., 2023）和GKD（Agarwal et al., 2023）（使用论文中设置的相同超参数）。 我们使用Llama-2-70b的对话版本作为白盒教师。我们还比较了使用对齐代理作为白盒教师进行蒸馏的方法。

Black-Box KD. For knowledge distillation with black-box teachers, we compare the vanilla black-box KD methods (Mukherjee et al., 2023; Mitra et al., 2023; Xu et al., 2023), which directly fine-tunes the student on the data generated by the black-box teacher. We also compare Proxy-KD with the TAKD (Mirzadeh et al., 2020) method.黑盒知识蒸馏。对于黑盒教师的知识蒸馏，我们比较原始黑盒KD方法（Mukherjee et al., 2023; Mitra et al., 2023; Xu et al., 2023），该方法直接微调学生模型于黑盒教师生成的数据。我们还将Proxy-KD与TAKD（Mirzadeh et al., 2020）方法进行比较。

For baselines implemented by us, we start from the same student checkpoint as Proxy-KD and use the same input prompts. In white-box KD, output sequences are generated by the white-box teacher, while in black-box KD, output sequences are generated by the black-box teacher.对于我们自己实现的基线，我们从与Proxy-KD相同的学生检查点开始，并使用相同的输入提示。在白盒KD中，输出序列由白盒教师生成；在黑盒KD中，输出序列由黑盒教师生成。

We show the comparison of Proxy-KD against baselines in Table 1, the proxy models in Proxy-KD are based on Llama-2-70B backbone. Overall, the performance of black-box KD methods outperforms that of white-box KD methods, demonstrating the efficacy of distilling knowledge from powerful black-box models.我们在表1中展示了Proxy-KD与基线的比较，Proxy-KD中的代理模型基于Llama-2-70B骨干。总体而言，黑盒KD方法的性能优于白盒KD方法，证明了从强大的黑盒模型中蒸馏知识的有效性。

Proxy-KD outperforms white-box KD and black-box KD methods. Notably, Proxy-KD further enhances the performance, consistently achieving higher scores across most evaluated benchmarks compared to the white-box KD methods (e.g. MiniLLM and GKD) and the black-box KD methods. Improvement is particularly pronounced in the challenging datasets like BBH and GSM8K, where Proxy-KD obtains scores of 53.40 and 53.07, respectively, outperforming even larger models trained using traditional black-box KD methods.Proxy-KD优于白盒KD和黑盒KD方法。值得注意的是，Proxy-KD进一步提升了性能，在大多数评估基准上相较于白盒KD方法（如MiniLLM和GKD）和黑盒KD方法始终取得更高分数。在BBH和GSM8K等具有挑战性的数据集上提升尤为显著，Proxy-KD分别获得53.40和53.07的分数，甚至优于使用传统黑盒KD方法训练的更大模型。

Proxy-KD outperforms TAKD consistently. TAKD performs even worse than vanilla Black-Box KD. When using Llama-1-7B as the student, vanilla Black-Box KD achieves an average of 49.11%, while TAKD only reaches 46.63%. Similarly, with Llama-2-7B as the student, vanilla Black-Box KD attains 53.66% compared to TAKD’s average of 52.82%. This decline in performance is likely due to TAKD’s failure to account for the proxy alignment process, which is essential for effective closed-source KD. Introducing an unaligned proxy not only fails to enhance performance but actually degrades the performance of the student model.Proxy-KD始终优于TAKD。TAKD的性能甚至低于原始黑盒KD。当使用Llama-1-7B作为学生时，原始黑盒KD的平均得分为49.11%，而TAKD仅达到46.63%。同样，以Llama-2-7B作为学生时，原始黑盒KD达到53.66%，而TAKD的平均为52.82%。这种性能下降可能是由于TAKD未考虑代理对齐过程，而这对于有效的闭源KD至关重要。引入未对齐的代理不仅无法提升性能，反而会降低学生模型的性能。

Proxy-KD outperforms white-box KD with an aligned proxy as the teacher. Relying solely on an aligned proxy for white-box KD offers limited knowledge to the student. This suggests that the capabilities of closed-source teachers are more beneficial than those of open-source teachers, even after alignment, underscoring the superiority of distilling from closed-source LLMs.Proxy-KD优于使用对齐代理作为教师的白盒KD。仅依赖对齐代理进行白盒KD为学生提供的知识有限。这表明闭源教师的能力即使在对齐后也比开源教师更有益，凸显了从闭源LLM蒸馏的优越性。

We also present the performance changes of student models during the distillation process in Figure 6 in Appendix. We show the accuracy curves of students on the benchmark test sets for every 40K training steps. We compare three methods: vanilla black-box KD, Proxy-KD, and white-box KD (forward KL). The results show that Proxy-KD stands out with the most significant enhancements, indicating its superior capability to efficiently transfer the comprehensive knowledge of black-box teachers to student models. The steeper and more consistent improvement curves of Proxy-KD across benchmarks such as AGIEval, ARC, and particularly in complex tasks like BBH and GSM8K, underscore its robust and effective approach in leveraging proxy models for knowledge distillation.我们还在附录图6中展示了蒸馏过程中学生模型的性能变化。我们展示了学生在每40K训练步后在基准测试集上的准确率曲线。我们比较了三种方法：原始黑盒KD、Proxy-KD和白盒KD（前向KL）。 结果表明，Proxy-KD的提升最为显著，表明其高效地将黑盒教师的综合知识迁移到学生模型的优越能力。在AGIEval、ARC以及BBH和GSM8K等复杂任务上，Proxy-KD的曲线更陡峭且更一致，凸显了其利用代理模型进行知识蒸馏的稳健有效方法。

In this section, we examine the impact of different components within Proxy-KD. Llama-2-7B and Llama-2-70B are utilized as the backbones of the student and the proxy models, respectively.本节检验Proxy-KD中不同组件的影响。学生模型和代理模型分别使用Llama-2-7B和Llama-2-70B作为骨干。

Effect of the Proxy Model. The proxy model $\pi_{p}$ is crucial for the effectiveness of Proxy-KD. Removing the proxy model forces the distillation process to revert to hard-label knowledge distillation, leading to significant performance drops across multiple benchmarks: a decrease of 4.24 on ARC, 6.72 on BBH, and 3.56 on GSM8K, as shown in Table 2. These declines underscore the proxy model’s essential role in capturing and transferring the distributional knowledge from the black-box teacher, which is particularly important for tasks involving complex reasoning and mathematical challenges. Without the proxy, the student model fails to benefit from the detailed distributional guidance, resulting in markedly lower performance.代理模型的效果。 代理模型πp对Proxy-KD的有效性至关重要。移除代理模型会使蒸馏过程退化为硬标签知识蒸馏，导致多个基准上的性能显著下降：ARC下降4.24，BBH下降6.72，GSM8K下降3.56，如表2所示。这些下降凸显了代理模型在捕获和迁移黑盒教师分布知识方面的关键作用，尤其对于涉及复杂推理和数学挑战的任务。没有代理，学生模型无法从详细的分布指导中受益，导致性能明显降低。

Effect of Proxy Model Alignment. The proxy model alignment, facilitated by the loss $\mathcal{L}_{\text{Proxy}}$, is vital for effective knowledge transfer. Table 2 shows that when the proxy is initialized directly from the Llama-2-70B checkpoint without alignment, the performance drops notably on BBH (-10.40), GSM8K (-5.53), and MMLU (-3.26). This decline illustrates the adverse effect of an unaligned proxy, which fails to approximate the teacher’s distribution and consequently underperforms compared to models directly fine-tuned on teacher data. The slight increase on CSQA (+0.86) when skipping alignment might be attributed to the simplicity of the task, indicating potential overfitting to teacher outputs without proxy guidance. This reinforces the necessity of the alignment process to ensure the proxy effectively bridges the knowledge transfer from the black-box teacher to the student model across diverse and complex tasks.代理模型对齐的效果。 由损失ℒProxy促进的代理模型对齐对于有效的知识迁移至关重要。表2显示，当代理直接以Llama-2-70B检查点初始化而不进行对齐时，性能在BBH（-10.40）、GSM8K（-5.53）和MMLU（-3.26）上显著下降。这种下降说明未对齐代理的负面影响，它无法近似教师的分布，因此与直接在教师数据上微调的模型相比性能不佳。在CSQA（+0.86）上跳过对齐时性能略有提升，可能归因于该任务的简单性，表明没有代理指导时可能对教师输出过拟合。这强化了对齐过程的重要性，以确保代理能够有效地在多样化和复杂任务中桥接从黑盒教师到学生模型的知识迁移。

Effect of Preference Optimization. Table 2 illustrates the significant role of preference optimization in enhancing the performance of both the proxy and student models. When the proxy preference loss $\mathcal{L}_{\text{Pref}}$ is removed, reducing the proxy alignment loss to $\mathcal{L}_{\text{Proxy-NLL}}$, we observe notable performance drops across various benchmarks. Specifically, the alignment of the proxy model with the black-box teacher deteriorates, as evidenced by decreases in scores on benchmarks like BBH and MMLU, which subsequently impacts the student model. The overall trend confirms that preference optimization is crucial for refining the proxy model’s ability to emulate the teacher effectively.偏好优化的效果。 表2展示了偏好优化在提升代理和学生模型性能方面的重要作用。当移除代理偏好损失ℒPref，将代理对齐损失简化为ℒProxy-NLL时，我们观察到各基准上的显著性能下降。具体来说，代理模型与黑盒教师之间的对齐会恶化，BBH和MMLU等基准上的分数下降即为证据，这进而影响学生模型。总体趋势证实，偏好优化对于改进代理模型有效模拟教师的能力至关重要。

Effect of Weighted KL. When $\mathcal{L}_{\text{Weight-KL}}$ is replaced with the standard KL loss $\mathcal{L}_{\text{Student-KL}}$, we also observe declines in performance across most benchmarks, indicating that the effectiveness of the distillation process diminishes. The results shown in Table 2 highlight that focusing on high log-likelihood distributions from the proxy, as facilitated by the weighted KL loss, significantly enhances the quality of knowledge transfer. The overall declines underscore that this weighting mechanism significantly improves the quality of knowledge distillation, enhancing the student’s ability to learn from a well-aligned proxy.加权KL的效果。 当ℒWeight-KL被标准KL损失ℒStudent-KL替换时，我们同样观察到大多数基准上的性能下降，表明蒸馏过程的有效性降低。表2中的结果强调，通过加权KL损失关注代理的高对数似然分布，显著提高了知识迁移的质量。总体下降表明这种加权机制显著改善了知识蒸馏的质量，增强了学生从良好对齐代理中学习的能力。

How well the proxy aligned with the teacher can directly affect the performance of the student. The final alignment effectiveness of the proxy model depends on two factors: the design of the alignment algorithm and the inherent alignment capability of the proxy backbone model itself. In this section, we investigate the impact of the latter. We hypothesize that the size of the proxy model’s parameters is crucial for its capacity to align with the black-box teacher’s capability, especially when the teacher’s parameter size is significantly larger than the proxy’s. Experiments are conducted with Llama-2-70B and Llama-2-13B as the proxy backbone models. We show the performance of these aligned proxy models. As depicted in Figure 3, the proxy model based on Llama-2-70B performs better than the one based on Llama-2-13B, the latter has fewer parameters. We also examine the impact of proxy models with different capacities on student performance. We observe that the stronger proxy based on Llama-2-70B yields better student performance than the weaker proxy based on Llama-2-13B. Furthermore, when using a proxy based on a backbone model with a larger capacity, the student demonstrates a greater potential for achieving higher performance.代理模型与教师的对齐程度会直接影响学生的表现。代理模型的最终对齐效果取决于两个因素：对齐算法的设计和代理骨干模型本身固有的对齐能力。在本节中，我们研究后者的影响。我们假设代理模型的参数量对其与黑盒教师能力对齐的能力至关重要，尤其是当教师的参数量远大于代理时。我们使用 Llama-2-70B 和 Llama-2-13B 作为代理骨干模型进行实验。我们展示了这些对齐后的代理模型的性能。如图 3 所示，基于 Llama-2-70B 的代理模型表现优于基于 Llama-2-13B 的代理模型，后者的参数更少。我们还考察了不同能力的代理模型对学生表现的影响。我们观察到，基于 Llama-2-70B 的更强代理比基于 Llama-2-13B 的较弱代理带来了更好的学生表现。此外，当使用基于更大容量骨干模型的代理时，学生展现出实现更高性能的更大潜力。