Distilling the Knowledge in a Neural Network

神经网络中的知识蒸馏

Geoffrey Hinton *↑
Google Inc.
Mountain View
geoffhinton@google.com
Oriol Vinyals †
Google Inc.
Mountain View
vinyals@google.com
Jeff Dean
Google Inc.
Mountain View
jeff@google.com

Abstract

摘要

A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions [3]. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators [1] have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.

提高几乎任何机器学习算法性能的一种非常简单的方法是，在相同数据上训练许多不同的模型，然后对它们的预测结果取平均值[3]。遗憾的是，使用整个模型集成进行预测较为繁琐，并目计算成本可能过高，难以部署给大量用户，尤其是在各个模型都是大型神经网络时。Caruana及其合作者[1]已表明，可以将一个集成中的知识压缩到单个模型中，从而使其更易于部署；我们使用一种不同的压缩技术进一步发展了这种方法。我们在MNIST上取得了一些令人惊讶的结果，并表明，通过将模型集成中的知识蒸馏到单个模型中，我们可以显著改进一个被广泛使用的商业系统的声学模型。我们还引入了一种新型集成，它由一个或多个完整模型以及许多专家模型组成，这些专家模型学习区分完整模型容易混淆的细粒度类别。与专家混合模型不同，这些专家模型可以快速并行训练。

1 Introduction

1引言

Many insects have a larval form that is optimized for extracting energy and nutrients from the environment and a completely different adult form that is optimized for the very different requirements of traveling and reproduction. In largescale machine learning, we typically use very similar models for the training stage and the deployment stage despite their very different requirements: For tasks like speech and object recognition, training must extract structure from very large, highly redundant datasets but it does not need to operate in real time and it can use a huge amount of computation. Deployment to a large number of users, however, has much more stringent requirements on latency and computational resources. The analogy with insects suggests that we should be willing to train very cumbersome models if that makes it easier to extract structure from the data. The cumbersome model could be an ensemble of separately trained models or a single very large model trained with a very strong regularizer such as dropout [9]. Once the cumbersome model has been trained, we can then use a different kind of training, which we call “distillation” to transfer the knowledge from the cumbersome model to a small model that is more suitable for deployment. A version of this strategy has already been

pioneered by Rich Caruana and his collaborators [1]. In their important paper they demonstrate convincingly that the knowledge acquired by a large ensemble of models can be transferred to a single small model.

许多昆虫具有一种幼虫形态，该形态经过优化以从环境中获取能量和营养；而其完全不同的成虫形态则经过优化，以满足迁徙和繁殖的截然不同的需求。在大规模机器学习中，尽管训练阶段和部署阶段的需求非常不同，我们通常为两者使用非常相似的模型：对于语音和目标识别等任务，训练必须从规模极大、高度冗余的数据集中提取结构，但它不需要实时运行，并且可以使用大量计算资源。然而，面向大量用户的部署对Latency和计算资源有更严格的要求。与昆虫的类比表明，如果这能使从数据中提取结构变得更容易，我们应当愿意训练非常笨重的模型。这个笨重的模型可以是由独立训练的模型组成的集成，也可以是使用诸如 dropout[9]之类的强正则化器训练得到的单个超大型模型。一旦这个复杂模型训练完成，我们就可以采用一种不同的训练方式，我们称之为“蒸馏”，将复杂模型中的知识迁移到一个更适合部署的小模型中。Rich Caruana 及其合作者已经率先提出了这一策略的一个版本[1]。他们在一篇重要论文中令人信服地证明，由大型模型集成所获得的知识可以迁移到单个小模型中。

A conceptual block that may have prevented more investigation of this very promising approach is that we tend to identify the knowledge in a trained model with the learned parameter values and this makes it hard to see how we can change the form of the model but keep the same knowledge. A more abstract view of the knowledge, that frees it from any particular instantiation, is that it is a learned

一个可能阻碍人们进一步研究这种极具前景方法的概念性障碍在于，我们倾向于将训练后模型中的知识等同于学习到的Parameter值，这使得我们难以理解如何在改变模型形式的同时保留相同的知识。对知识的一种更抽象的看法，即将其从任何特定实例化中解放出来，是它是一种学习到的

mapping from input vectors to output vectors. For cumbersome models that learn to discriminate between a large number of classes, the normal training objective is to maximize the average log probability of the correct answer, but a side-effect of the learning is that the trained model assigns probabilities to all of the incorrect answers and even when these probabilities are very small, some of them are much larger than others. The relative probabilities of incorrect answers tell us a lot about how the cumbersome model tends to generalize. An image of a BMW, for example, may only have a very small chance of being mistaken for a garbage truck, but that mistake is still many times more probable than mistaking it for a carrot.

从输入向量到输出向量的映射。对于学习区分大量类别的笨重模型，常规训练目标是最大化正确答案的平均对数概率但学习的一个副作用是，训练后的模型会为所有错误答案分配概率；即使这些概率非常小，其中一些也会比其他概率大得多。错误答案的相对概率告诉我们很多关于笨重模型倾向于如何泛化的信息。例如，一张BMW的图像被误判为垃圾车的概率可能非常小，但这种错误仍然比将其误判为胡萝卜的概率高很多倍。

It is generally accepted that the objective function used for training should reflect the true objective of the user as closely as possible. Despite this, models are usually trained to optimize performance on the training data when the real objective is to generalize well to new data. It would clearly be better to train models to generalize well, but this requires information about the correct way to generalize and this information is not normally available. When we are distilling the knowledge from a large model into a small one, however, we can train the small model to generalize in the same way as the large model. If the cumbersome model generalizes well because, for example, it is the average of a large ensemble of different models, a small model trained to generalize in the same way will typically do much better on test data than a small model that is trained in the normal way on the same training set as was used to train the ensemble.

人们普遍认为，用于训练的目标函数应尽可能准确地反映用户的真实目标。尽管如此，当真正的目标是良好地泛化到新数据时，模型通常仍被训练为优化其在训练数据上的性能。显然，若能训练模型使其具备良好的泛化能力会更好，但这需要有关正确泛化方式的信息，而这类信息通常不可获得。然而，当我们将知识从一个大型模型蒸馏到一个小型模型时，可以训练这个小型模型以与大型模型相同的方式进行泛化。如果这个庞大的模型能够良好泛化，例如因为它是大量不同模型组成的大型集成的平均结果，那么，经过训练以相同方式泛化的小型模型，通常会比以常规方式在用于训练该集成的同一训练集上训练的小型模型，在测试数据上表现得好得多。

An obvious way to transfer the generalization ability of the cumbersome model to a small model is to use the class probabilities produced by the cumbersome model as “soft targets" for training the small model. For this transfer stage, we could use the same training set or a separate “transfer" set. When the cumbersome model is a large ensemble of simpler models, we can use an arithmetic or geometric mean of their individual predictive distributions as the soft targets. When the soft targets have high entropy, they provide much more information per training case than hard targets and much less variance in the gradient between training cases, so the small model can often be trained on much less data than the original 将庞大模型的泛化能力迁移到小型模型的一种显而易见的方法，是使用庞大模型生成的类别概率作为训练小型模型的 “软目标”。在这一迁移阶段，我们可以使用同一训练集，或使用单独的“迁移”数据集。当庞大模型是由多个较简单 模型组成的大型集成时，我们可以将其各自预测分布的算术平均或几何平均作为软目标。当软目标具有较高熵时，相比 硬目标，它们为每个训练样本提供的信息更多，并且不同训练样本之间的梯度方差更小，因此小型模型通常可以使用远 少于原始庞大模型的数据进行训练，并采用高得多的Learning Rate。

For tasks like MNIST in which the cumbersome model almost always produces the correct answer with very high confidence, much of the information about the learned function resides in the ratios of very small probabilities in the soft targets. For example, one version of a 2 may be given a probability of $10^{-6}$ of being a 3 and $10^{-9}$ of being a 7 whereas for another version it may be the other way around. This is valuable information that defines a rich similarity structure over the data (i. e. it says which 2’s look like 3’s and which look like $7$ but it has very little influence on the cross-entropy cost function during the transfer stage because the probabilities are so close to zero. Caruana and his collaborators circumvent this problem by using the logits (the inputs to the final softmax) rather than the probabilities produced by the softmax as the targets for learning the small model and they minimize the squared difference between the logits produced by the cumbersome model and the logits produced by the small model. Our more general solution, called “distillation”, is to raise the temperature of the final softmax until the cumbersome model produces a suitably soft set of targets. We then use the same high temperature when training the small model to match these soft targets. We show later that matching the logits of the cumbersome model is actually a special case of distillation.

对于像MNIST这样的任务，繁琐模型几乎总是以极高置信度给出正确答案，所学习函数的大部分信息都存在于软目标中极小概率之间的比率里。例如，一个版本的2可能被赋予为3的概率 $10^{-6}$ 以及为7的概率 $10^{-9}$ ，而另一个版本则可能恰好相反。这是定义数据上丰富相似性结构的宝贵信息(即它说明哪些2看起来像3，哪些看起来像7)，但在迁移阶段，它对交叉熵代价函数的影响很小，因为这些概率太接近于零。Caruana及其合作者通过使用Logits(最终Softmax的输入）而不是Softmax产生的概率，作为学习小模型的目标来规避这一问题；他们最小化繁琐模型产生的Logits 与小模型产生的Logits 之间的平方差。我们更通用的解决方案称为“蒸馏”，即提高最终 Softmax的Temperature，直到庞大模型产生一组足够软的目标。然后，在训练小模型以匹配这些软目标时，我们使用相同的高Temperature。我们随后将表明，匹配庞大模型的Logits 实际上是蒸馏的一种特殊情况。

The transfer set that is used to train the small model could consist entirely of unlabeled data [1] or we could use the original training set. We have found that using the original training set works well, especially if we add a small term to the objective function that encourages the small model to predict the true targets as well as matching the soft targets provided by the cumbersome model. Typically, the small model cannot exactly match the soft targets and erring in the direction of the correct answer turns out to be helpful

用于训练小模型的迁移集可以完全由无标签数据组成[1]，或者我们也可以使用原始训练集。我们发现，使用原始训练集效果很好，尤其是在目标函数中加入一个小项，以鼓励小模型在匹配庞大模型提供的软目标的同时，也预测真实目标。通常，小模型无法完全匹配这些软目标，而朝着正确答案的方向产生误差被证明是有帮助的。

2 Distillation

2蒸馏

Neural networks typically produce class probabilities by using a “softmax" output layer that converts the logit, zi, computed for each class into a probability, $q_i,$ by comparing Zi with the other logits.

神经网络通常通过使用“Softmax”输出层来生成类别概率，该输出层通过将为每个类别计算的Logits $z_i$ 与其他Logits 进行比较，将其转换为概率 $q_i$

$$q_i=\frac{\exp (z_i/T)}{{\sum }_j\exp (z_j/T)}$$

where ${}_T$ is a temperature that is normally set to 1 . Using a higher value for T produces a softer probability distribution over classes.

其中，T 是通常设置为1 的 Temperature。为 T 使用更高的值会产生在各类别上的更平滑概率分布。

In the simplest form of distillation, knowledge is transferred to the distilled model by training it on a transfer set and using a soft target distribution for each case in the transfer set that is produced by using the cumbersome model with a high temperature in its softmax. The same high temperature is used when training the distilled model, but after it has been trained it uses a temperature of 1 .

在最简单形式的蒸馏中，通过在迁移集上训练蒸馏模型，并为迁移集中的每个样本使用软目标分布来将知识传递给蒸馏模型；该软目标分布通过在其 Softmax 中使用高 Temperature 的复杂模型生成。训练蒸馏模型时使用相同的高Temperature，但训练完成后，它使用 Temperature 1。

When the correct labels are known for all or some of the transfer set, this method can be significantly improved by also training the distilled model to produce the correct labels. One way to do this is to use the correct labels to modify the soft targets, but we found that a better way is to simply use a weighted average of two different objective functions. The first objective function is the cross entropy with the soft targets and this cross entropy is computed using the same high temperature in the softmax of the distilled model as was used for generating the soft targets from the cumbersome model. The second objective function is the cross entropy with the correct labels. This is computed using exactly the same logits in softmax of the distilled model but at a temperature of 1 . We found that the best results were generally obtained by using a condiderably lower weight on the second objective function. Since the magnitudes of the gradients produced by the soft targets scale as $1/T^2$ it is important to multiply them by $T^2$ when using both hard and soft targets. This ensures that the relative contributions of the hard and soft targets remain roughly unchanged if the temperature used for distillation is changed while experimenting with meta-parameters.

当迁移集中的全部或部分样本具有已知的正确标签时，还可以通过训练蒸馏模型产生正确标签来显著改进该方法。一种做法是使用正确标签来修改软目标，但我们发现，更好的方法是简单地使用两个不同目标函数的加权平均。第一个目标函数是与软目标的交叉熵，并且该交叉熵在蒸馏模型的Softmax中使用与从复杂模型生成软目标时相同的高Temperature 进行计算。第二个目标函数是与正确标签的交叉熵。它使用蒸馏模型 Softmax 中完全相同的 Logits 进行计算，但Temperature为1。我们发现，通常在第二个目标函数上使用显著更低的权重能够获得最佳结果。由于由软目标产生的梯度幅度按 $1/T^2$ 缩放，因此在同时使用硬目标和软目标时，必须将它们乘以 $T^2$ 。这确保了在试验元参数时，如果改变用于蒸馏的Temperature，硬目标和软目标的相对贡献仍大致保持不变。

2.1 Matching logits is a special case of distillation

2.1 匹配 Logits 是蒸馏的一种特殊情况

Each case in the transfer set contributes a cross-entropy gradient, $dC/d{z}_i$ , with respect to each logit, zi of the distilled model. If the cumbersome model has logits $v_i$ which produce soft target probabilities Pi and the transfer training is done at a temperature of T, this gradient is given by

迁移集中的每个样本都会对蒸馏模型的每个 Logits zi 产生一个交叉熵梯度 $dC/d{z}_i$ 。如果复杂模型具有Logits $v_i$ 其生成软目标概率 $p_i$ ，并且迁移训练在 Temperature ${}_T$ 下进行，则该梯度为：

$$\frac{\partial C}{\partial z_i}=\frac{1}{T}(q_i-p_i)=\frac{1}{T}\left(\frac{e^{z_i/T}}{{\sum }_j{e}^{z_j/T}}-\frac{e^{v_i/T}}{{\sum }_j{e}^{v_j/T}}\right)$$

If the temperature is high compared with the magnitude of the logits, we can approximate:

如果 Temperature 相对于 Logits 的幅值较高，我们可以近似为：

$$\frac{\partial C}{\partial z_i}\approx \frac{1}{T}\left(\frac{1+z_i/T}{N+{\sum }_j{z}_j/T}-\frac{1+v_i/T}{N+{\sum }_j{v}_j/T}\right)$$

If we now assume that the logits have been zero-meaned separately for each transfer case so that $\begin{array}{r}{\sum }_j{z}_j={\sum }_j{v}_j=0_{\mathrm{Eq}}.\end{array}$ 3 simplifies to:

如果我们现在假设对于每个迁移样例，Logits 都已分别进行零均值化，那么 $\begin{array}{r}{\sum }_j{z}_j={\sum }_j{v}_j=0_{}\end{array}$ 3 可简化为：

$$\frac{\partial C}{\partial z_i}\approx \frac{1}{N{T}^2}(z_i-v_i)$$

So in the high temperature limit, distillation is equivalent to minimizing $1/2(z_i-v_i{)}^2$ , provided the logits are zero-meaned separately for each transfer case. At lower temperatures, distillation pays much less attention to matching logits that are much more negative than the average. This is potentially advantageous because these logits are almost completely unconstrained by the cost function used for training the cumbersome model so they could be very noisy. On the other hand, the very negative logits may convey useful information about the knowledge acquired by the cumbersome model. Which of these effects dominates is an empirical question. We show that when the distilled model is much too small to capture all of the knowledege in the cumbersome model, intermediate temperatures work best which strongly suggests that ignoring the large negative logits can be helpful.

因此，在高 Temperature 极限下，蒸馏等价于最小化 $1/2(z_i-v_i{)}^2$ 前提是对于每个迁移样例，Loqits 都已分别进行零均值化。在较低Temperature下，蒸馏对匹配那些远低于平均值的Logits的关注要少得多。这可能是有利的，因为这些Logits几乎完全不受用于训练笨重模型的代价函数约束，因此它们可能具有很大的噪声。另一方面，这些非常负的Logits可能传达了有关笨重模型所获得知识的有用信息。哪种效应占主导地位是一个经验问题。我们表明，当蒸馏模型太小而无法捕获笨重模型中的全部知识时，中等 Temperature 的效果最佳，这强烈表明忽略较大的负 Logits 可能是有帮助的。

3 Preliminary experiments on MNIST

3 MNIST 上的初步实验

To see how well distillation works, we trained a single large neural net with two hidden layers of 12oo rectified linear hidden units on all 6o,ooo training cases. The net was strongly regularized using dropout and weight-constraints as described in [5]. Dropout can be viewed as a way of training an exponentially large ensemble of models that share weights. In addition, the input images were

为考察蒸馏的效果，我们使用全部60,000个训练样本，训练了一个具有两个隐藏层、每层包含1200个修正线性隐藏单元的大型单一神经网络。按照[5] 中所述，该网络通过 dropout 和权重约束进行了强正则化。Dropout 可以被视为一种训练权重共享的、规模呈指数增长的模型集成的方法。此外，输入图像还被

jittered by up to two pixels in any direction. This net achieved 67 test errors whereas a smaller net with two hidden layers of 8oo rectified linear hidden units and no regularization achieved 146 errors. But if the smaller net was regularized solely by adding the additional task of matching the soft targets produced by the large net at a temperature of 2o, it achieved 74 test errors. This shows that soft targets can transfer a great deal of knowledge to the distilled model, including the knowledge about how to generalize that is learned from translated training data even though the transfer set does not contain any translations.

沿任意方向最多抖动了两个像素。该网络在测试集上产生了67个错误，而一个具有两个隐藏层、每层包含800个修正线性隐藏单元且未进行正则化的较小网络产生了146个错误。但是，如果仅通过增加一项额外任务来正则化这个较小网络，即在 Temperature 为 20 时匹配由大型网络生成的软目标，它便取得了74 个测试错误。这表明，软目标能够向蒸馏后的模型传递大量知识，其中包括从翻译后的训练数据中学到的有关如何泛化的知识，尽管迁移集不包含任何翻译。

When the distilled net had 3oo or more units in each of its two hidden layers, all temperatures above 8 gave fairly similar results. But when this was radically reduced to 3o units per layer, temperatures in the range 2.5 to 4 worked significantly better than higher or lower temperatures.

当蒸馏网络的两个隐藏层各自具有 300个或更多单元时，所有高于 8的 Temperature 都给出了相当相似的结果。但是，当每层的单元数被大幅减少到 30 个时，范围在 2.5 到 4 之间的 Temperature的效果显著优于更高或更低的Temperature.

We then tried omitting all examples of the digit 3 from the transfer set. So from the perspective of the distilled model, 3 is a mythical digit that it has never seen. Despite this, the distilled model only makes 2o6 test errors of which 133 are on the

101o threes in the test set. Most of the errors are caused by the fact that the learned bias for the 3 class is much too low. If this bias is increased by 3.5 (which optimizes overall performance on the test set), the distilled model makes 109 errors of which 14 are on 3s. So with the right bias, the distilled model gets 98.6% of the test 3s correct despite never having seen a 3 during training. If the transfer set contains only the 7s and 8s from the training set, the distilled model makes 47.3% test errors, but when the biases for 7 and 8 are reduced by 7.6 to optimize test performance, this falls to 13.2% test errors.

随后，我们尝试从迁移集中过滤掉所有数字3的样本。因此，从蒸馏模型的角度来看，3是一个它从未见过的神秘数字。尽管如此，蒸馏模型仅产生了206 个测试错误，其中133个发生在测试集中的1010个3上。大多数错误是由于为类别3学得的偏置过低所致。如果将该偏置增加3.5（这会使测试集上的总体性能达到最优)，蒸馏模型会产生109个错误，其中14个发生在3上。因此，在使用正确的偏置时，尽管训练期间从未见过3，蒸馏模型仍能正确识别98.6%的测试集3。如果迁移集只包含训练集中的7和8，蒸馏模型的测试错误率为 47.3%；但当将7和8的偏置各降低7.6 以优化测试性能时，测试错误率降至13.2%。

4 Experiments on speech recognition

4 语音识别实验

In this section, we investigate the effects of ensembling Deep Neural Network (DNN) acoustic models that are used in Automatic Speech Recognition (ASR). We show that the distillation strategy that we propose in this paper achieves the desired effect of distilling an ensemble of models into a single model that works significantly better than a model of the same size that is learned directly from the same training data

在本节中，我们研究对用于自动语音识别（ASR）的深度神经网络（DNN）声学模型进行集成的效果。我们表明，本文提出的蒸馏策略实现了预期效果：将一个模型集成蒸馏为单个模型，而该单个模型的性能显著优于使用相同训练数据直接学习得到的同等规模模型。

State-of-the-art ASR systems currently use DNNs to map a (short) temporal context of features derived from the waveform to a probability distribution over the discrete states of a Hidden Markov Model (HMM) [4]. More specifically, the DNN produces a probability distribution over clusters of tri-phone states at each time and a decoder then finds a path through the HMM states that is the best compromise between using high probability states and producing a transcription that is probable under the language model.

当前最先进的 ASR 系统使用 DNN，将从波形中提取的特征的（短)时间上下文映射为隐马尔可夫模型（HMM)离散状态上的概率分布[4]。更具体地说，DNN 在每个时间点生成 tri-phone 状态簇上的概率分布，随后解码器在 HMM 状态中寻找一条路径，使其在使用高概率状态与生成在语言模型下具有较高概率的转录结果之间达到最佳折中。

Although it is possible (and desirable) to train the DNN in such a way that the decoder (and, thus, the language model) is taken into account by marginalizing over all possible paths, it is common to train the DNN to perform frame-by-frame classification by (locally) minimizing the cross entropy between the predictions made by the net and the labels given by a forced alignment with the ground truth sequence of states for each observation:

尽管可以（且最好）以这样一种方式训练DNN：通过对所有可能路径进行边缘化来考虑解码器（以及语言模型），但通常会训练DNN通过(局部地)最小化网络预测结果与由每个观测的真实状态序列强制对齐所给出的标签之间的交叉熵，来执行逐帧分类：

$$\theta =\arg \underset{{\theta }^{\prime }}{\max }P\left(h_t\mid {\mathbf{s}}_t;{\theta }^{\prime }\right)$$

where $\theta$ are the parameters of our acoustic model P which maps acoustic observations at time $t,{\mathbf{s}}_t,$ to a probability, $P\left(h_t\mid {\mathbf{s}}_t;{\theta }^{\prime }\right)$ , of the “correct” HMM state $h_t,$ which is determined by a forced alignment with the correct sequence of words. The model is trained with a distributed stochastic gradient descent approach.

其中，θ 是我们的声学模型 $P$ 的参数，该模型将时刻t的声学观测 ${\mathbf{s}}_t$ 映射为“正确”HMM 状态 $h_t$ 的概率$P\left(h_t\mid {\mathbf{s}}_t;{\theta }^{\prime }\right)$ ；该状态由与正确词序列的强制对齐确定。该模型采用分布式随机 Gradient Descent 方法进行训练。

We use an architecture with 8 hidden layers each containing 256o rectified linear units and a final softmax layer with 14,000 labels (HMM targets $h_t)$ . The input is 26 frames of 40 Mel-scaled filterbank coefficients with a 1oms advance per frame and we predict the HMM state of 21st frame. The total number of parameters is about 85M. This is a slightly outdated version of the acoustic model used by Android voice search, and should be considered as a very strong baseline. To train the DNN acoustic model we use about 2ooo hours of spoken English data, which yields about 7ooM training examples. This system achieves a frame accuracy of 58.9%, and a Word Error Rate (WER) of 10.9% on our development set.

我们使用一种架构，其中包含8个隐藏层，每层含有2560个修正线性单元，以及一个具有14,000 个标签的最终Softmax 层（HMM 目标 h）。输入为 26 帧、每帧包含 40 个 Mel 标度滤波器组系数，帧间推进为 10ms，我们预测$21^{\mathrm{st}}$ 帧的 HMM 状态。Parameter 总数约为 85M。这是 Android 语音搜索所使用声学模型的一个略显过时的版本，应被视为一个非常强的基线。为训练 DNN 声学模型，我们使用了约 2000 小时的英语语音数据，由此产生约 700M个训练样本。该系统在我们的开发集上实现了58.9%的帧准确率和10.9%的词错误率(WER)。

System系统	Test Frame Accuracy测试帧准确率	WER
Baseline基线	58.9%	10.9%
10xEnsemble10x 集成模型	61.1%	10.7%
Distilled Single model蒸馏后的单一模型	60.8%	10.7%

Table 1: Table 1: Frame classification accuracy and WER showing that the distilled single model performs about as well as the averaged predictions of 1o models that were used to create the soft targets

表1：帧分类准确率和 WER，表明蒸馏后的单一模型的性能与用于创建软目标的10个模型平均预测的性能大致相当。

4.1 Results

4.1 结果

We trained 1o separate models to predict $P ( h _ { t } \mid \mathbf { s } _ { t } ; \pmb { \theta } ) $ , using exactly the same architecture and training procedure as the baseline. The models are randomly initialized with different initial parameter values and we find that this creates sufficient diversity in the trained models to allow the averaged predictions of the ensemble to significantly outperform the individual models. We have explored adding diversity to the models by varying the sets of data that each model sees, but we found this to not significantly change our results, so we opted for the simpler approach. For the distillation we tried temperatures of [1, 2, 5, 10] and used a relative weight of o.5 on the cross-entropy for the hard targets, where bold font indicates the best value that was used for table 1.

我们训练了10个独立模型来预测 $P(h_t\mid {\mathbf{s}}_t;\mathbf{\theta })$ ，其架构和训练流程与基线完全相同。这些模型以不同的初始Parameter值随机初始化，我们发现这会在训练后的模型中产生足够的多样性，使集成模型的平均预测结果显著优于各个单独模型。我们曾探索通过改变每个模型所见的数据集来为模型增加多样性，但发现这并未显著改变结果，因此选择了更简单的方法。在蒸馏过程中，我们尝试了[1,2,5,10] 的 Temperature，并对硬目标的交叉熵使用了 0.5 的相对权重，其中粗体表示用于表1的最佳值。

Table 1 shows that, indeed, our distillation approach is able to extract more useful information from the training set than simply using the hard labels to train a single model. More than 8o% of the improvement in frame classification accuracy achieved by using an ensemble of 1o models is transferred to the distilled model which is similar to the improvement we observed in our preliminary experiments on MNIST. The ensemble gives a smaller improvement on the ultimate objective of WER (on a 23K-word test set) due to the mismatch in the objective function, but again, the improvement in WER

achieved by the ensemble is transferred to the distilled model.

表1表明，事实上，我们的蒸馏方法能够从训练集中提取比仅使用硬标签训练单个模型更多的有用信息。通过使用由10个模型组成的集成所获得的帧分类准确率提升中，超过80%被迁移到了蒸馏模型中，这与我们在MNIST上的初步实验中观察到的提升类似。由于目标函数不匹配，该集成在最终目标WER（在一个包含23K个词的测试集上）上的提升较小，但同样，集成所实现的 WER 改进被迁移到了蒸馏模型中。

We have recently become aware of related work on learning a small acoustic model by matching the class probabilities of an already trained larger model [8]. However, they do the distillation at a temperature of 1 using a large unlabeled dataset and their best distilled model only reduces the error rate of the small model by 28% of the gap between the error rates of the large and small models when they are both trained with hard labels.

我们最近了解到一项相关工作，该工作通过匹配一个已训练的大型模型的类别概率来学习一个小型声学模型[8]。然而，他们使用一个大型无标签数据集，以Temperature 为1 进行蒸馏；其最佳蒸馏模型仅将小模型的错误率降低了如下差距的28%：即当大模型和小模型都使用硬标签训练时，两者错误率之间的差距。

5 Training ensembles of specialists on very big datasets

5 在超大型数据集上训练专家集成

Training an ensemble of models is a very simple way to take advantage of parallel computation and the usual objection that an ensemble requires too much computation at test time can be dealt with by using distillation. There is, however, another important objection to ensembles: If the individual models are large neural networks and the dataset is very large, the amount of computation required at training time is excessive, even though it is easy to parallelize.

训练模型集成是利用并行计算的一种非常简单的方法，而通常对集成模型的反对意见——即其在测试时需要过多计算可以通过蒸馏来解决。然而，集成模型还存在另一个重要问题：如果各个模型都是大型神经网络，并且数据集非常大，那么训练时所需的计算量会非常庞大，尽管这些计算很容易并行化。

In this section we give an example of such a dataset and we show how learning specialist models that each focus on a different confusable subset of the classes can reduce the total amount of computation required to learn an ensemble. The main problem with specialists that focus on making fine-grained distinctions is that they overfit very easily and we describe how this overfitting may be prevented by using soft targets.

本节将给出这样一个数据集的示例，并展示学习专门模型如何降低训练集成模型所需的总计算量，其中每个专门模型都聚焦于类别中一个不同的易混淆子集。专门模型在进行细粒度区分时面临的主要问题是它们极易过拟合；我们将介绍如何通过使用软目标来防止这种过拟合。

5.1 The JFT dataset

5.1 JFT 数据集

JFT is an internal Google dataset that has 100 million labeled images with 15,0oo labels. When we did this work, Google’s baseline model for JFT was a deep convolutional neural network [7] that had been trained for about six months using asynchronous stochastic gradient descent on a large number of cores. This training used two types of parallelism [2]. First. there were many replicas of the neural net running on different sets of cores and processing different mini-batches from the training set. Each replica computes the average gradient on its current mini-batch and sends this gradient to a sharded parameter server which sends back new values for the parameters. These new values reflect all of the gradients received by the parameter server since the last time it sent parameters to the replica. Second, each replica is spread over multiple cores by putting different subsets of the neurons on each core. Ensemble training is yet a third type of parallelism that can be wrapped

JFT 是 Google 的一个内部数据集，包含1 亿张带标签的图像和 15,000 个标签。我们开展这项工作时，Google 用于JFT的基线模型是一个深度卷积神经网络[7]，它使用大量核心上的异步随机 Gradient Descent 训练了大约六个月。该训练使用了两种类型的并行化[2]。首先，许多神经网络副本运行在不同的核心集合上，并处理训练集中的不同小批量数据。每个副本计算其当前小批量数据上的平均梯度，并将该梯度发送给分片Parameter服务器，后者返回Parameter的新值。这些新值反映了自上次向该副本发送Parameter 以来 Parameter服务器接收到的所有梯度。其次，每个副本通过将不同的神经元子集放置在各个核心上而分布在多个核心中。集成训练则是第三种类型的并行化，可以包裹在

around the other two types, but only if a lot more cores are available. Waiting for several years to train an ensemble of models was not an option, so we needed a much faster way to improve the baseline model.

其他两种类型之外，但前提是有更多的核心可用。等待数年以训练一个模型集成并不可行，因此我们需要一种快得多的方法来改进基线模型。

5.2 Specialist Models

5.2 专家模型

When the number of classes is very large, it makes sense for the cumbersome model to be an ensemble that contains one generalist model trained on all the data and many “specialist” models, each of which is trained on data that is highly enriched in examples from a very confusable subset of the classes (like different types of mushroom). The softmax of this type of specialist can be made much smaller by combining all of the classes it does not care about into a single dustbin class.

当类别数量非常大时，让笨重模型成为一个集成模型是合理的，该集成模型包含一个在所有数据上训练的通才模型，以及许多“专家”模型，每个专家模型都在某个极易混淆类别子集的样本高度富集的数据上进行训练（例如不同种类的蘑菇）。通过将该专家模型不关心的所有类别合并为一个垃圾箱类别，可以使这类专家模型的 Softmax大幅缩小。

To reduce overfitting and share the work of learning lower level feature detectors, each specialist model is initialized with the weights of the generalist model. These weights are then slightly modified by training the specialist with half its examples coming from its special subset and half sampled at random from the remainder of the training set. After training, we can correct for the biased training set by incrementing the logit of the dustbin class by the log of the proportion by which the specialist class is oversampled.

为了减少过拟合并共享学习低层特征检测器的工作，每个专家模型都使用通才模型的权重进行初始化。随后，通过训练专家模型来稍微修改这些权重，其中一半样本来自其专属子集，另一半从训练集的其余部分中随机采样。训练后，我们可以通过将垃圾箱类别的logit增加专家类别被过采样比例的对数，来校正有偏的训练集。

5.3 Assigning classes to specialists

5.3 将类别分配给专家模型

In order to derive groupings of object categories for the specialists, we decided to focus on categories that our full network often confuses. Even though we could have computed the confusion matrix and used it as a way to find such clusters, we opted for a simpler approach that does not require the true labels to construct the clusters.

为了为各个专家模型推导出对象类别分组，我们决定重点关注完整网络经常混淆的类别。尽管我们可以计算混淆矩阵，并将其用作寻找此类簇的方法，但我们选择了一种更简单的方法，该方法无需真实标签即可构建这些簇。

In particular, we apply a clustering algorithm to the covariance matrix of the predictions of our generalist model, so that a set of classes Sm that are often predicted together will be used as targets for one of our specialist models, m. We applied an on-line version of the K-means algorithm to the columns of the covariance matrix, and obtained reasonable clusters (shown in Table 2). We tried several clustering algorithms which produced similar results.

具体而言，我们将聚类算法应用于通用模型预测结果的协方差矩阵，因此，一组经常被共同预测的类别 $S^m$ 将被用作某个专家模型m的目标。我们将 K-means 算法的在线版本应用于协方差矩阵的各列，并获得了合理的簇（如表 2所示）。我们尝试了几种聚类算法，它们产生了相似的结果。

5.4 Performing inference with ensembles of specialists

5.4 使用专家模型集成进行 Inference

Before investigating what happens when specialist models are distilled, we wanted to see how well ensembles containing specialists performed. In addition to the specialist models, we always have a generalist model so that we can deal with classes for which we have no specialists and so that we can decide which specialists to use. Given an input image x, we do top-one classification in two steps:

在研究蒸馏 specialist 模型时会发生什么之前，我们想先了解包含 specialist 的集成模型表现如何。除了 specialist 模型之外，我们始终保留一个 generalist 模型，以便处理没有对应 specialist 的类别，并决定使用哪些 specialist。给定张输入图像 x，我们通过以下两步进行 top-one 分类：

Step 1: For each test case, we find the n most probable classes according to the generalist model. Call this set of classes k. In our experiments, we used n = 1.

步骤1：对于每个测试样本，我们根据 generalist 模型找出概率最高的 n 个类别。将该类别集合称为 。在我们的实验中，我们使用了 n = 1。

Step 2: We then take all the specialist models, m, whose special subset of confusable classes, Sm, has a non-empty intersection with k and call this the active set of specialists Ak (note that this set may be empty). We then find the full probability distribution q over all the classes that minimizes:

步骤 2：然后，我们选取所有specialist 模型m，其易混淆类别的特殊子集 $S^m$ 与存在非空交集，并将其称为active specialist 集合 $A_k$ (请注意，该集合可能为空)。接着，我们在所有类别上寻找使下式最小化的完整概率分布q :

$$KL\left({\mathbf{p}}^g,\mathbf{q}\right)+\sum _{m\in A_k}KL\left({\mathbf{p}}^m,\mathbf{q}\right)$$

Figure 1: Table 2: Example classes from clusters computed by our covariance matrix clustering algorithm

图1：表2：由我们的协方差矩阵聚类算法计算得到的聚类中的示例类别

where KL denotes the KL divergence, and ${\mathbf{p}}^m{\mathbf{p}}^g$ denote the probability distribution of a specialist model or the generalist full model. The distribution $ { \mathbf { p } } ^ { m }$ is a distribution over all the specialist classes of m plus a single dustbin class, so when computing its KL divergence from the full q distribution we sum all of the probabilities that the full q distribution assigns to all the classes in m 's dustbin.

其中 KL 表示 KL 散度， $ { \mathbf { p } } ^ { m } { \mathbf { p } } ^ { g }$ 表示专家模型或通才完整模型的概率分布。分布 $ { \mathbf { p } } ^ { m }$ 是对 m的所有专家类别加上一个单独的垃圾箱类别的分布，因此，在计算其相对于完整9分布的 KL 散度时，我们将完整9分布分配给m的垃圾箱中所有类别的概率相加。

System 系统	Conditional Test Accuracy 条件测试准确率	Test Accuracy 测试准确率
Baseline 基线	43.1%	25.0%
+ 61 Specialist models
+ 61 个专家模型	45.9%	26.1%

Table 2: Table 3: Classification accuracy (top 1) on the JFT development set.

表 2：表 3：JFT 开发集上的分类准确率（top1）。

# of specialists covering覆盖的专家数量	# of test examples测试样例数量	delta in top1 correcttop1 正确数变化	relative accuracy change相对准确率变化
0	350037	0	0.0%
1	141993	+1421	+3.4%
2	67161	+1572	+7.4%
3	38801	+1124	+8.8%
4	26298	+835	+10.5%
5	16474	+561	+11.1%
6	10682	+362	+11.3%
7	7376	+232	+12.8%
8	4703	+182	+13.6%
9	4706	+208	+16.6%
10 or more10或更多	9082	+324	+14.1%

Table 3: Table 4: Top 1 accuracy improvement by # of specialist models covering correct class on the JFT test set.
表 3：表 4：JFT 测试集上，覆盖正确类别的 specialist 模型数量带来的 Top 1 准确率提升。
Eq. 5 does not have a general closed form solution, though when all the models produce a single probability for each class the solution is either the arithmetic or geometric mean, depending on whether we use KL(p, q) or KL(q, P) ). We parameterize q = softmax(z) $(\mathrm{with}T=1\dot{})$ ) and we use gradient descent to optimize the logits z w.r.t. eq. 5. Note that this optimization must be carried out for each image.

公式5 没有一般的闭式解，不过当所有模型都为每个类别产生单一概率时，解为算术平均值或几何平均值，具体取决于我们使用 KL(p, q)还是 KL(q, p)）。我们对 q = softmax(z)（使用 T = 1）进行参数化，并使用 GradientDescent 针对公式 5 优化 Logits z。请注意，必须针对每张图像执行此优化。

5.5 结果

Starting from the trained baseline full network, the specialists train extremely fast (a few days instead of many weeks for JFT). Also, all the specialists are trained completely independently. Table 3 shows the absolute test accuracy for the baseline system and the baseline system combined with the specialist models. With 61 specialist models, there is a 4.4% relative improvement in test accuracy overall. We also report conditional test accuracy, which is the accuracy by only considering examples belonging to the specialist classes, and restricting our predictions to that subset of classes

从训练好的基线完整网络开始，specialist 的训练速度极快(对于JFT，只需几天而非数周)。此外，所有 specialist都完全独立地进行训练。表3 展示了基线系统以及与specialist 模型结合后的基线系统的绝对测试准确率。使用 61 个specialist 模型后，整体测试准确率相对提升了4.4%。我们还报告了条件测试准确率，即仅考虑属于 specialist类别的样本，并将预测限制在该类别子集时的准确率。

For our JFT specialist experiments, we trained 61 specialist models, each with 3oo classes (plus the dustbin class). Because the sets of classes for the specialists are not disjoint, we often had multiple specialists covering a particular image class. Table 4 shows the number of test set examples, the change in the number of examples correct at position 1 when using the specialist(s), and the relative percentage improvement in top1 accuracy for the JFT dataset broken down by the number of specialists covering the class. We are encouraged by the general trend that accuracy improvements are larger when we have more specialists covering a particular class, since training independent specialist models is very easy to parallelize.

在我们的JFT专家模型实验中，我们训练了61 个专家模型，每个模型包含300个类别（外加垃圾箱类别）。由于专家模型所涵盖的类别集合并不互斥，我们经常会有多个专家模型覆盖某个特定图像类别。表4按覆盖该类别的专家模型数量，对JFT数据集展示了测试集样本数量、使用专家模型时位置1 上预测正确的样本数量变化，以及top1 准确率的相对百分比提升。我们受到总体趋势的鼓舞：当有更多专家模型覆盖某个特定类别时，准确率提升更大，因为训练彼此独立的专家模型非常容易进行并行化。

6 Soft Targets as Regularizers

6 作为正则化器的软目标

One of our main claims about using soft targets instead of hard targets is that a lot of helpful information can be carried in soft targets that could not possibly be encoded with a single hard target. In this section we demonstrate that this is a very large effect by using far less data to fit the 85M parameters of the baseline speech model described earlier. Table5shows that with only 3% of the data (about 2oM examples), training the baseline model with hard targets leads to severe overfitting (we did early stopping, as the accuracy drops sharply after reaching 44.5%), whereas the same model trained with soft targets is able to recover almost all the information in the full training set (about 2% shy). It is even more remarkable to note that we did not have to do early stopping: the system with soft targets simply “converged” to 57%. This shows that soft targets are a very effective way of communicating the regularities discovered by a model trained on all of the data to another model.

我们关于使用软目标而非硬目标的主要主张之一是，软目标能够携带大量有用信息，而这些信息不可能被编码到单一硬目标中。在本节中，我们通过使用远少于前文所述基线语音模型拟合其 85M 个 Parameter的数据，来证明这一效应非常显著。表5表明，仅使用3%的数据(约20M个样本)时，使用硬目标训练基线模型会导致严重过拟合(我们进行了早停，因为准确率在达到44.5%后急剧下降)；而使用软目标训练相同模型，则能够恢复完整训练集中的几乎所有信息(仅相差约2%)。更值得注意的是，我们甚至不需要进行早停：使用软目标的系统仅仅“收敛”到57%。这表明，软目标是将一个在全部数据上训练的模型所发现的规律传递给另一个模型的极其有效方式。

System & training set系统与训练集	Train Frame Accuracy训练帧准确率	Test Frame Accuracy测试帧准确率
Baseline (100% of training set)基线（100%的训练集）	63.4%	58.9%
Baseline (3% of training set)基线（3%的训练集）	67.3%	44.5%
Soft Targets (3% of training set)软目标（3%的训|练集）	65.4%	57.0%

Table 4: Table 5: Soft targets allow a new model to generalize well from only 3% of the training set. The soft targets are obtained by training on the full training set.
表4：表 5：软目标使新模型仅使用 3%的训练集也能很好地泛化。软目标通过在完整训练集上进行训练获得。

6.1 Using soft targets to prevent specialists from overfitting

6.1 使用软目标防止专家模型过拟合

The specialists that we used in our experiments on the JFT dataset collapsed all of their non-specialist classes into a single dustbin class. If we allow specialists to have a full softmax over all classes, there may be a much better way to prevent them overfitting than using early stopping. A specialist is trained on data that is highly enriched in its special classes. This means that the effective size of its training set is much smaller and it has a strong tendency to overfit on its special classes. This problem cannot be solved by making the specialist a lot smaller because then we lose the very helpful transfer effects we get from modeling all of the non-specialist classes.

我们在JFT数据集上的实验中使用的专家模型，将所有非专家类别合并为一个单一的垃圾桶类别。如果允许专家模型在所有类别上拥有完整的Softmax，那么可能存在一种比使用早停更好的方法来防止其过拟合。专家模型是在其专长类别高度富集的数据上训练的。这意味着其训练集的有效规模要小得多，并且它很容易在其专长类别上过拟合。通过大幅缩小专家模型无法解决这个问题，因为那样我们会失去通过建模所有非专家类别所获得的非常有帮助的迁移效应。

Our experiment using 3% of the speech data strongly suggests that if a specialist is initialized with the weights of the generalist, we can make it retain nearly all of its knowledge about the non-special classes by training it with soft targets for the non-special classes in addition to training it with hard targets. The soft targets can be provided by the generalist. We are currently exploring this approach.

我们使用3%语音数据进行的实验强烈表明，如果专家模型使用通用模型的权重进行初始化，那么除了使用硬目标进行训练外，还可以通过使用非专家类别的软目标进行训练，使其保留几乎全部关于非专家类别的知识。软目标可以由通用模型提供。我们目前正在探索这种方法。

7 与专家混合模型的关系

The use of specialists that are trained on subsets of the data has some resemblance to mixtures of experts [6] which use a gating network to compute the probability of assigning each example to each expert. At the same time as the experts are learning to deal with the examples assigned to them, the gating network is learning to choose which experts to assign each example to based on the relative discriminative performance of the experts for that example. Using the discriminative performance of the experts to determine the learned assignments is much better than simply clustering the input vectors and assigning an expert to each cluster, but it makes the training hard to parallelize: First, the weighted training set for each expert keeps changing in a way that depends on all the other experts and second, the gating network needs to compare the performance of different experts on the same example to know how to revise its assignment probabilities. These difficulties have meant that mixtures of experts are rarely used in the regime where they might be most beneficial: tasks with huge datasets that contain distinctly different subsets.

使用在数据子集上训练的专家与专家混合模型[6]有一定相似之处；后者使用一个门控网络来计算将每个样本分配给各个专家的概率。在专家学习处理分配给它们的样本的同时，门控网络也在学习根据各专家对该样本的相对判别性能，选择将每个样本分配给哪些专家。使用专家的判别性能来确定学习得到的分配方式，远优于简单地对输入向量进行聚类并为每个聚类分配一个专家，但这使得训练难以并行化：首先，每个专家的加权训练集会不断变化，并且其变化依赖于所有其他专家；其次，门控网络需要比较不同专家在同一样本上的表现，才能知道如何修正其分配概率。这些困难意味着，专家混合模型很少被用于它们可能最具优势的场景：包含明显不同子集的超大规模数据集任务。

It is much easier to parallelize the training of multiple specialists. We first train a generalist model and then use the confusion matrix to define the subsets that the specialists are trained on. Once these subsets have been defined the specialists can be trained entirely independently. At test time we can use the predictions from the generalist model to decide which specialists are relevant and only these specialists need to be run.

对多个专家模型的训练进行并行化要容易得多。我们首先训练一个通用模型，然后利用混淆矩阵来定义专家模型所训练的数据子集。一旦定义了这些子集，各个专家模型便可以完全独立地进行训练。在测试时，我们可以利用通用模型的预测结果来决定哪些专家模型是相关的，因此只需运行这些专家模型。

8 Discussion

8讨论

We have shown that distilling works very well for transferring knowledge from an ensemble or from a large highly regularized model into a smaller, distilled model. On MNIST distillation works remarkably well even when the transfer set that is used to train the distilled model lacks any examples of one or more of the classes. For a deep acoustic model that is version of the one used by Android voice search, we have shown that nearly all of the improvement that is achieved by training an ensemble of deep neural nets can be distilled into a single neural net of the same size which is far easier to deploy.

我们已经表明，蒸馏在将知识从一个集成模型或一个大型、高度正则化的模型迁移到一个较小的蒸馏模型时效果非常好。在MNIST上，即使训练蒸馏模型所使用的迁移集缺少一个或多个类别的任何样本，蒸馏依然表现得非常出色。对于一个与Android 语音搜索所使用模型版本相同的深度声学模型，我们已经证明，通过训练深度神经网络集成所获得的几乎全部改进，都可以蒸馏到一个大小相同、但更易于部署的单一神经网络中。

For really big neural networks, it can be infeasible even to train a full ensemble, but we have shown that the performance of a single really big net that has been trained for a very long time can be significantly improved by learning a large number of specialist nets, each of which learns to discriminate between the classes in a highly confusable cluster. We have not yet shown that we can distill the knowledge in the specialists back into the single large net.

对于真正庞大的神经网络，即使训练一个完整的集成模型也可能不可行，但我们已经表明，通过学习大量专家网络，可以显著提升一个经过极长时间训练的单个超大网络的性能，其中每个专家网络都学习区分一个高度易混淆簇中的类别。我们尚未证明能够将专家网络中的知识蒸馏回单个大型网络中。

We thank Yangqing Jia for assistance with training models on ImageNet and Ilya Sutskever and Yoram Singer for helpful discussions.

我们感谢 Yangqing Jia 在 ImageNet 上训练模型时提供的帮助，也感谢 Ilya Sutskever 和 Yoram Singer 提供的有益讨论。