Efficient Diversity-Driven Ensemble for Deep Neural Networks

Ensemble learning is an art of combining the predictions of different machine learning models, and it has achieved state-of-the-art performance in many challenges such as the Netflix Prize and various Kaggle competitions [24]. From the algorithmic level, Bagging, Boosting and Stacking are very representative algorithms.

Bagging is an ensemble strategy which uses the bootstrap aggregation to train multiple models while reducing variance. As a popular bagging method, Majority Voting [25] counts the votes from all base learners and makes the prediction using the most voted label. The other method is Averaging [26], which averages the softmax outputs of base learners. By averaging six residual networks, He et al [21] won the first place in ILSVRC 2015. Similarly, Snapshot Ensemble and BANs adopts an average strategy in its prediction process.

Another popular ensemble method is Boosting. It trains multiple base models sequentially and re-weights the training samples which are harder to train so that the latter models focus more on the difficult examples [27]. The idea of boosting neural networks has been around for many years [28]. BoostCNN incorporates boosting weights into the deep learning architecture [29], and it can select the best network structure in each iteration. Besides, deep incremental boosting [11] uses transfer learning to accelerate the initial warm-up phase of training each Boosting round. To enhance diversity, AdaBoost.NC propose an ambiguity decomposition strategy to negatively correlate the error made by other models.

Unlike Bagging and Boosting, Stacking [14] combines the outputs of each base model via a meta-learner. Based on the same idea, deep super learner [30] circularly appends the outputs of base models to the training data and uses the super learner to combine their outputs. Cheng Ju [31] adopts super Learner from the convolution neural network perspective, in which a CNN network is used to combine each output of the base model.

Besides, according to the usage of the training data, current methods of training ensemble of the deep neural networks can be classified into two categories. The first one is to train each network architecture with the entire dataset [12, 32], and the other one is to train it with a different subset of replacements from the original dataset [29]. Usually, a deep neural network has a large parameter space and it can get low bias if trained with the entire dataset [32]. However, this may bring higher training cost. As a sub-sampling method, Bagging and Boosting can enhance diversity but result in higher bias of base models since we train them with less unique samples in each iteration.

As we know, base models in an ensemble contribute more if they are diverse, otherwise there may be no gain in the combining process. However, all these methods individually train each base model without correlation and thus they cannot introduce diversity in an explicit way. Since there is no feedback from the combination stage to the individual design stage, it is possible that some of the independently designed networks may not make any contribution to the whole ensemble model. Furthermore, some of them train each network from scratch, requiring a lot of time and training resource.

Current methods for designing neural network ensemble usually train individual neural networks independently. However, the base models are likely to be trapped in the same nearby local minimums and result in low diversity, especially if we transfer the knowledge of the pre-trained base model [12]. Both theoretical and experimental studies show that the ensemble generalization ability can be largely enhanced if the base models in the ensemble are negatively correlated [33].

Negative correlation learning(NCL) is proposed to achieve this goal, all the individual networks in the NCL are trained simultaneously through the correlation penalty terms in their loss functions, thus the diversity can be introduced in an explicit way.

Based on the NCL, cooperative ensemble learning system (CELS) [34] encourages the individual networks to learn different aspects of a given dataset cooperatively and simultaneously. Besides, learning via Correlation-Corrected Data(NCCD) [35] embeds penalty values to every training sample instead of the error function.

However, all of their penalty terms are unfit to implement in classification ensembles. The AdaBoost.NC is proposed to solve this problem, it defines the penalty term with the correct/incorrect decision and introduces the error correlation information into the weights of the training data.

Although this definition can be used for classification ensembles with NCL, such coarse-grained definition may lose lots of useful information the model has learned before. Besides, AdaBoost.NC only provides a diversity-aware heuristic to adjust the weight of training data but fails to formalize an objective function that takes diversity into consideration. Therefore, the diversity of base models in AdaBoost.NC can be inferior, which limits the ensemble performance.

Our method EDDE differs AdaBoost.NC in two aspects. First, we propose a new diversity-driven loss function to explicitly increase diversity, while AdaBoost.NC introduce diversity by adjusting the sample weights. Compared with our diversity-driven loss function, using the sample weights to enhance diversity is not direct and robust. Besides, EDDE captures more information as we negatively correlate the soft target of former ensemble model.

Seen from Figure 2, during the training process of base model $h_{t}$, we transfer the generic knowledge of the earlier fitted model $h_{t-1}$ to accelerate the convergence. Besides, we adopt a diversity-driven optimization to negatively correlate the soft-target of the former ensemble model $H_{t-1}$. After each iteration, we add the $h_{t}$ to the ensemble $H_{t-1}$ and get a new ensemble model $H_{t}$. Note that the weights of training samples are updated in each iteration and we use a Boosting-based strategy to construct the full pipeline.

The learning algorithm of EDDE is summarized in Algorithm 1, and all notations we used are defined in Table I. We will explain Algorithm 1 in detail in the following parts.

As the size of the ensemble grows, the training time grows linearly with the size of the ensemble. Therefore, it’s desirable to find a method to accelerate the convergence of each base model.

A simple way to achieve this goal is to use Transfer Learning. Transferred with the previous knowledge, an additional network can be hatched from the former pre-trained neural network and trained significantly faster than trained from scratch. Practically, this allows us to train very large ensembles of deep neural networks in the time taken to train just a couple from scratch [38].

Transfer learning [39] is a strategy where a machine learning model developed for a task is reused as the starting point for another model on a second task. In transfer Learning, we first train a base network on a base dataset, and then we repurpose the learned features, or transfer them, to a second target network to be trained on a target dataset. We can naturally transfer the knowledge of previously trained neural network $h_{t-1}$ to train another neural network $h_{t}$, and $h_{t}$ may converge more quickly as it lies in a region near a good local minimum [40].

Take the Snapshot Ensemble for an example, all the weights of the pre-trained neural network $h_{t-1}$ are transferred to the current base neural network $h_{t}$. Indeed, we can rapidly get a converged model $h_{t}$. However, the model $h_{t}$ may make similar predictions as $h_{t-1}$ because $h_{t}$ starts training at the local minimum proposed by $h_{t-1}$. That’s why the diversity in Snapshot Ensemble is limited. Classically, transfer learning aims to transfer the information from one domain to another domain and it hasn’t taken the diversity between each base models into consideration.

To guarantee the diversity in an ensemble, it is unsuitable to transfer all the pre-trained knowledge without selection. Fortunately, DNN generally learns generic features (e.g., edge detectors or color blob detectors) in its lower layers and task-specific features (e.g., parts of specific objects) in its upper layers [7]. The features in the upper layers are (weighted) combinations of the features in the lower layers and they are directly responsible for making predictions. Besides, the generic features contained in the earlier several layers of a neural network should be suitable for both base and target tasks, instead of specific to the base task. Inspired by this discovery, we naturally come up with the idea of using these generic features to accelerate the training process of the next base model.

The full procedure of our knowledge transfer is shown in Figure 3. We transfer the parameters of the earlier several layers of the teacher model $h_{t-1}$ to student model $h_{t}$. Besides, in order not to decrease the diversity, we randomly initialize $h_{t}$’s remaining high-level layers since they contain the specific knowledge of the given dataset.

Note that we fine-tune both the transferred wights of the low layers and the randomly initialized weights of the high layers in each iteration, which means the transferred knowledge will not be fixed during the retraining processes. That’s because the fixed transferred knowledge adds too strong constraints (i.e., most of the parameters are not allowed to be updated) to the optimization task, and a better local minimum is not likely to be obtained given such constraints. By fine-tuning all the weights, each individual base model has the ability to converge to some good and different local minimums in a short time.

Like Snapshot Ensemble, we accelerate the convergence of each base model by transferring the knowledge of the former pre-trained neural network. The main difference is that we just transfer the generic knowledge. The core problem we face is how to find such generic features, which is also equal to say how many layers of weights we should transfer from one pre-trained neural network to the other untrained one.

We propose an efficient and adaptive method to tackle this problem. In our proposed method EDDE, we denote the parameter $\beta$ to determine the proportion of parameters we should transfer. If it gets too much, the diversity may be reduced; if it is too small, we cannot maximize the training speed. Therefore, we aim to find a $\beta$ that leverages a trade-off between model diversity and training speed.

As shown in Figure 4, to better analyze this problem, we assume the training set $D$ is split into $n$ folds, and we leave the last fold as the test set. We train a neural network $h_{t-1}$ using the first $n$-1 folds, and initialize the next network $h_{t}$ according to $\beta$. Then, we train $h_{t}$ with the first $n$-2 folds. Finally, we predict $h_{t}$’s accuracy on the ($n$-1)-th fold and the $n$-th fold. In the above case, the pre-trained network $h_{t-1}$ has seen the ($n$-1)-th fold while the next base model $h_{t}$ hasn’t. Besides, $h_{t-1}$ generally gets a higher accuracy on the ($n$-1)-th fold than the $n$-th fold due to overfitting. Different choices of $\beta$ can lead to different accuracy results:

• •

  If $\beta$ is large, we transfer a large portion of weights of $h_{t-1}$, yielding a $h_{t}$ similar to $h_{t-1}$. Since $h_{t-1}$ has specific knowledge on the ($n$-1)-th fold, the accuracy of $h_{t}$ on the ($n$-1)-th fold is higher than that on the $n$-th fold.

• •

  If $\beta$ is small, we transfer a small portion of weights of $h_{t-1}$, and therefore $h_{t}$ can be more different. Accordingly, $h_{t}$ has more possibility to forget the specific knowledge learned by $h_{t-1}$ from the ($n$-1)-th fold. Since $h_{t}$ has never seen both the ($n$-1)-th fold and the $n$-th fold, the accuracy on these two folds are similar.

Motivated by the above analysis, we find that we can choose a $\beta$ according to $h_{t}$’s performance on the dataset, which $h_{t-1}$ saw but $h_{t}$ did not. In practice, we can start from $\beta=1$ and gradually reduce it, until $h_{t}$ performs similarly on two datasets — an exclusive dataset owned by $h_{t-1}$ and a shared test dataset. With this strategy, we can achieve a good trade-off between model diversity and training speed.

We also run an experiment to validate the above analysis. With different $\beta$, we split the training set CIFAR-100 into six folds ($n=6$). We firstly pre-train the model $h_{1}$ using the first 5 folds, then transfer the weights of $h_{1}$ to initialize our next base model $h_{2}$. During the training process of $h_{2}$, we calculate $h_{2}$’s mean accuracy of the first 5 epochs on the sixth fold and the fifth fold, the experimental result is shown in Figure 5. Seen from the experiments on ResNet-32 and DenseNet-40, as we gradually reduce $\beta$ to a suitable value, $h_{2}$’s accuracy on the fold $n-1$ may be similar to that on the untrained fold $n$.

Remarkably, the pre-trained model $h_{1}$ can still be used in the ensemble process. Besides, the parameter $\beta$ only needs to be tuned with the first base model $h_{1}$ and we just fix this tuned value for the latter base models. In this way, we only need a few epochs to find how many parameters we need to transfer, which will not incur a large extra cost.

It’s true that such a definition can be applied to a classification task, and can also be used in the negative correlated learning. However, it has severe drawbacks in two aspects. On the one hand, they ignore the original softmax outputs of each base model and use the concrete classification label, such coarse-grained definitions may lose lots of useful information the model has learned before. On the other hand, as the ensemble output is the hard target, we cannot get useful gradient information with such a definition. As a result, it is impossible to directly optimize a diversity-driven loss function using this definition. That’s also the reason why AdaBoost.NC enhance diversity just by changing the sample weights.

To sum up, it is desirable to propose a new diversity measure which can make full use of the softmax outputs of each network and can also be used in a diversity-driven optimization.

Therefore, we define a new diversity measure for two networks $h_{j}$ and $h_{k}$. The diversity between these two base models is denoted as

Compared with the Eq. 1, our proposed diversity measure adopts the soft target, which contains more knowledge of the trained neural network. Besides, for each node in the softmax layer, we can get its gradient information with this definition. Such a feature is indispensable for our diversity-driven loss function.

Besides, we define the similarity between $h_{j}$ and $h_{k}$ as

The larger $Sim_{h_{j},h_{k}}$ means model $h_{j}$ is more similar to model $h_{k}$, thus their diversity $Div_{h_{j},h_{k}}$ becomes smaller accordingly. This definition will be used in our diversity-driven optimization latter.

As both the $\boldsymbol{h_{t}(x_{i})}$ and $\boldsymbol{H(x_{i})}$ are model’s softmax outputs, we have:

For a $k$-dimension vector $\boldsymbol{x}$,

According to Eq. 4 and 5, we get:

According to Eq. 2, 3 and 6, both $Div_{h_{j},h_{k}}$ and $Sim_{h_{j},h_{k}}$ range in $\left[0,1\right]$.

Last, to calculate the diversity of the ensemble network $H$, we define

For numerically comparing the diversity of two ensemble model, we can use Eq. 7 to calculate their $Div_{H}$ respectively.

According to Knowledge Distillation, a simple student model can be trained with the objective of matching the full softmax distribution of the complex teacher model. As the soft targets can provide much more information per training case and get much less variance in the gradient between training cases, a student network can achieve better accuracy by virtue of knowledge transferred from the teacher model than it would if trained directly [7].

Motivated by the idea of Knowledge Distillation, EDDE also adopts the soft target in our loss function. However, the usage of the soft target is totally different in two aspects. First, we optimize the loss function with our self-defined diversity measure. More importantly, the student network in Knowledge Distillation matches the soft target of the teacher model, but EDDE aims to negatively correlate this soft-target since our goal is not to enhance accuracy but to increase the difference between each base model and the ensemble model. Note that the student model in EDDE has the same architecture as the teacher model, so the accuracy of base models can be guaranteed in EDDE.

Specially, we define the ensemble teacher model by a function $y=H(x;w)$ where $x$ is the training sample, and $w$ is the learned weights of $H$. Both the $H(x_{i};w)$ and $h(x_{i};w^{\prime})$ are the vector of probabilities representing the conditional distribution over object categories given sample $x_{i}$. One of our objectives is to find a new set of parameters $w^{\prime}$ for a student neural network $h(x;w^{\prime})$ such that

According to Eq. 8, the base model $h(x;w^{\prime})$ in each iteration is more likely to make different prediction to the former ensemble model $H(x_{i};w)$.

Based on our diversity measure and the above objective, we define the penalty term as

Suppose we use the categorical cross-entropy as the loss function. We have two objectives when designing the objective function. The first objective is to minimize the error (Bias in Figure 6) of the base models, which is the cross entropy with the correct labels. The other is to maximize their difference (Variance in Figure 6). Based on this intuition, we define a weighted diversity-driven loss function on sample $x$ as

The hyperparameter $\gamma$ is used to adjust the strength of the second objective. Larger $\gamma$ may introduce higher diversity, but the bias of the neural network may also be increased accordingly. As a result, this parameter needs to be carefully tuned in our experiments.

It is easy to get the gradient information with our diversity-driven loss function. For $h_{t}$’s output node $c$, the partial derivative of $L$ with respect to the output $h_{t,c}$ is

We use the standard back propagation (BP) algorithm  [37] to optimize each network and its weights are updated with Eq. 11. To explain the optimization process in detail, we take the ensemble of three base models for an example. As shown in Figure 6, if we pre-train two base models $h_{1}$ and $h_{2}$, we can combine them and get the ensemble model $H_{2}$. Given the training data $D$ with sample weights $W_{2}$, the pre-trained knowledge by $h_{2}$, and two parameters $\gamma$ and $\beta$, we train the third model $h_{3}$ to negatively correlate the softmax outputs of $H_{2}$, such procedure is correspond to the $I(D,W_{2},h_{2},H_{2},\gamma,\beta)$ in line 7 of algorithm  1.

To further enhance diversity, we use a Boosting-based framework to combine the diversity-driven optimization and the knowledge transfer strategy. As an art of combining the predictions of different machine learning models, Boosting is a very popular ensemble method and it has been widely used in many deep learning scenes [29]. It increases the weight of the misclassified samples for the next training process, as the weight of each sample can be changed in each cycle and the loss function is closely related to the sample weights, the optimization path is changed correspondingly and each base model may make different predictions as they can converge to different local minimums.

For each sample $x_{i}$, we set $Sim_{t}(x_{i})$ as the similarity between model $h_{t}$ and $H_{t-1}$ on sample $x_{i}$ (line 8 of Algorithm 1). Besides, we set $Bias_{t}(x_{i})$ as the bias of model $h_{t}$ on sample $x_{i}$ (line 9 of Algorithm 1)

For a misclassified sample $x_{i}$, we update its weights by (line 10 of Algorithm 1)

Larger $Sim_{t}(x_{i})$ means the individual base model $h_{t}$ has the same opinion with the ensemble model $H_{t-1}$ on sample $x_{i}$. If sample $x_{i}$ is mis-classified by $h_{t}$, the ensemble model $H_{t-1}$ probably cannot classify this sample either, thus we must give more attention on sample $x_{i}$ and the weight $W_{t}(x_{i})$ will be increased more. On the contrary, if $Sim_{t}(x_{i})$ is small, which means $H_{t-1}$ may has better ability to classify the sample $x_{i}$, as it’s prediction is less similar to $h_{t}$. In this way, it’s better to increase the $W_{t}(x_{i})$ less aggressively.

Note that the base models of traditional Boosting algorithm only needs to be weak and it is true that we can get a strong ensemble model if we have a large number of weak base learners. However, it’s hard to meet such condition in deep learning as each base model needs a high training cost. So for EDDE, we hope the base models have both high diversity and accuracy. As the sample weights we used for these base models are only to enhance diversity, we just update the weights based on $W_{1}$. Due to this operation, the weight of each sample can be greatly changed. Since the loss function is closely related to the sample weights, the optimization path is changed correspondingly and the diversity can be further increased.

At last, we define the weight of each base model as (line 12 of Algorithm 1)

After we get the weight $W_{t}(x_{i})$ and the similarity $Sim_{t}(x_{i})$ of each sample $x_{i}$, we can accordingly calculate the weight of each base model. A large $Sim_{t}(x_{i})$ means $h_{t}(x_{i})$ is very similar to $H_{t-1}(x_{i})$, which indicates the model $h_{t}$ has the same opinion with the ensemble model $H_{t-1}$. Usually, the prediction of the ensemble model $H_{t-1}$ is relatively more accurate, thus $h_{t}$’s prediction becomes more important accordingly. In this way, we can increase or decrease $\alpha_{t}$ more if the $Sim_{t}(x_{i})$ is large.

For the prediction process, we average the softmax outputs of each base model $h_{t}$ with the model weight $\alpha_{t}$, and the ensemble model $H_{T}$ is defined as (line 16 of Algorithm 1)

Previous works have shown that training neural network ensemble through sub-sampled dataset may lead to low generalization accuracy as it reduces the number of unique data items seen by an individual neural network. A neural network has a large number of parameters and it is affected relatively more from this reduction in unique data items than the ensemble of other classifiers such as decision trees or SVMs [32]. Therefore, we use all the training set in each iteration.

To sum up, our method differs from traditional Boosting algorithms in three aspects.

• •

  we use all the training data instead of using the sub-sampling technique in each iteration as the less unique training samples may lead to low bias.

• •

  We train each base models based on the knowledge of the pre-trained one while the traditional Boosting methods train every network from scratch and introduce a high training cost.

• •

  we use a diversity-driven loss function to get each base model, and the weights update we use is totally different from the traditional Boosting methods.

To validate the effectiveness of EDDE, we train different ensemble methods on different tasks, and the main results are summarized in Table II and Table III. For CIFAR dataset, the methods in the same group are trained for 200 epochs. EDDE is trained for 50 epochs for the IMDB and MR dataset, and the other methods in the same group are trained for 100 epochs. Note that the results of our method are colored in blue, and the best result for each network/dataset pair is bolded.

For the CV task, the experiments on CIFAR data show that EDDE can always get the highest ensemble accuracy in all cases. Take the result on CIFAR-100 using ResNet-32 for an example, EDDE achieves the accuracy rate of 74.38%, far outperforming the next-best model’s 72.17% under the same training cost. For the NLP task, EDDE only need half time to achieve 87.69% test accuracy in IMDB dataset and 76.98% test accuracy in the MR dataset, which means EDDE is superior the other ensemble methods both in the speed and accuracy. Through the experiments on the CV and NLP tasks, we observe that EDDE is capable of getting a high ensemble accuracy with a limited training budget.

As the most concern of EDDE is to efficiently get a high ensemble accuracy with a limited training budget, it’s necessary to make an end-to-end comparison to other ensemble methods. For EDDE and other baselines, we train each base model with the same network structures and dataset. Therefore, the training time per epoch for each ensemble method is same. Besides, the extra time cost brought by different ensemble method is trivial compared to the training time of deep neural networks. As a result, the training epochs can be treated as the training expense. We compare the ensemble accuracy of each method trained with different epochs in the CV tasks, and the result is shown in Figure 7.

We firstly see that the AdaBoost.M1, AdaBoost.NC and Bagging all have relatively low test accuracy during each period. The main reason is that they train each base model individually on a sub-sampled dataset, thus it’s hard to get enough base models with high accuracy in a short time. For BANs, it also gets a low ensemble accuracy in many cases. That is because it randomly initialized each base model without the prior knowledge and it cannot get high diversity and accuracy at the same time.

As we can see in Figure 7, EDDE gets a higher test accuracy than other methods in all cases. More concretely, from the left picture, we see EDDE achieves the accuracy of 73.67% within only 130 epochs, while the next-best model Snapshot Ensemble needs 400 epochs to achieves 72.98% ensemble accuracy, which means EDDE is more than 3 times faster than the next-best model in this scenario.

Similarly, as shown in the right picture in Figure 7, EDDE always gets higher ensemble accuracy than other methods using DenseNet-40. Because our knowledge transfer method is capable of accelerating the training process and the diversity-driven method can explicitly enhance diversity, EDDE can get the highest efficiency.

We run an experiment on CIFAR100 using ResNet-32 and analyze the diversity between their first 8 base models. According to Eq. 3, we compute the pairwise similarity among each base model of Snapshot Ensemble, EDDE, and AdaBoost.NC. The result is shown in Figure  8.

For Snapshot Ensemble, the similarity between two nearby base models is high and it becomes higher when the model is trained longer. That’s because the next model is initialized with the former one, it’s more prone to converge to nearby local minimums during the two nearby training cycle. Besides, if trained longer, the base models becomes more accurate and they have more chance to make similar predictions.

On the contrary, it’s clear to see that the pairwise similarity of EDDE and AdaBoost.NC is smaller than Snapshot Ensemble. To further explore the significance of the diversity, we compute the increased accuracy of each ensemble methods. Besides, we also compute their concrete diversity according to Eq. 7. The experimental results are summarized in Table IV.

As shown in Table IV, the AdaBoost.NC can get the highest diversity of 0.1787. That’s because it trains the base models on different sub-sampled datasets. Besides, it randomly initializes the weights of each base model and then negatively correlates the former neural network by adjusting the sample weights. However, even given 400 training epochs, it gets the lowest average accuracy of 66.81%, which means all the base model cannot converge to a good local minimum given limited time.

As for the Snapshot Ensemble, it gets the highest average accuracy of 68.53% among these three methods since it transfers the knowledge of the pre-trained neural network to accelerate the training process of the current model. Unfortunately, as it transfers all the knowledge without selection, it gets the lowest diversity.

Seen from Table IV, the base models in EDDE can get both high diversity and average accuracy. Accordingly, EDDE gets the highest increased accuracy of 7.26% . Besides, EDDE achieves the 75.30% accuracy only using 250 epochs, while Snapshot Ensemble and AdaBoost.NC cannot get such accuracy even if trained with 400 epochs. Therefore, EDDE is efficient in the ensemble of deep neural networks.

In order to investigate the influences of hyperparameters on EDDE, we vary their values and compare the ensemble accuracy accordingly. EDDE has two hyperparameters, $\gamma$ controls the strength of the diversity-driven loss and $\beta$ determines the proportion of knowledge we should transfer from the pre-trained network. As we introduced in Section IV-B, we have proposed an effective method to find the optimal $\beta$, so the only parameter we should tune in EDDE is $\gamma$. Therefore, we vary the parameter $\gamma$ and run the experiment on ResNet-32 using CIFAR 100 dataset. The result is summarized in Table V.

For analyzing the influence of $\gamma$, we set $\gamma$ to 0, 0.1,0.3,0.5 and 1. We can observe from Table V that EDDE with the setting $\gamma$ of 0.1 get the highest accuracy of 74.38% since EDDE can get the most balanced tradeoff between diversity and accuracy in this setting. However, when setting $\gamma$ to 0, the ensemble accuracy is decreased to 73.86%. In this situation, our diversity-driven loss function is equal to the normal loss function, and the diversity of EDDE is reduced accordingly. On the contrary, if we set $\gamma$ to 1, the ensemble accuracy has a sharp decline. A higher $\gamma$ means we negatively correlate the previous soft target more, and it also means we give less attention to the true target. Accordingly, the network cannot converge well with a very high $\gamma$. However, according to Table II, even set $\gamma$ to 1, EDDE trained with 200 epochs still outperforms Snapshot Ensemble trained with 400 epochs. Therefore, our method EDDE is robust to the hyperparameters.

In order to show the effectiveness of our diversity-driven loss function and the knowledge transfer strategy, we add the ablation studies to measure their effect respectively.

We firstly train an ensemble model using EDDE without diversity-driven loss function, this strategy is denoted as EDDE (normal loss). Besides, like Snapshot Ensemble, we transfer all the pre-trained knowledge in EDDE, and we name this method EDDE (transfer all). Next, we individually train each base model without knowledge transfer, and this method is denoted as EDDE (transfer None). At last, we also compare EDDE with AdaBoost.NC using transfer learning (initialize each model using the weights of the pre-trained one), we name this method AdaBoost.NC (transfer). All these methods are tested in the CIFAR100 dataset using ResNet-32 and the training budget is 200 epochs for EDDE and 400 epochs for Adaboost.NC . The experimental results are summarized in Table VI.

As shown in Tabel VI, the ensemble accuracy may decrease if we use a normal loss function. Besides, we can get a higher average accuracy of 68.16% if we transfer all the pre-trained knowledge, but the diversity may decrease accordingly. On the contrary, from the result of EDDE (Transfer none), we can get the highest diversity of 0.1854. However, given limited training budget, each base model cannot converge well without the knowledge transfer. As a result, EDDE (Transfer none) gets the lowest average accuracy and ensemble accuracy.

As for the AdaBoost.NC with transfer learning, it can get higher average accuracy compared with the original AdaBoost.NC, but its ensemble accuracy is not as high as EDDE due to its lower diversity and average accuracy. That’s because it trains base models on the sub-sampled datasets and transfers all the previous knowledge without selection.

Among all these methods, EDDE can get the most balanced tradeoff between average accuracy and diversity and it can get the highest ensemble accuracy of 74.38%, thus both the diversity-driven loss and the knowledge transfer are effective in EDDE.