Midpoint Regularization: from High Uncertainty Training to Conservative Classification

We consider a standard classification setting with a given training data set $(X;Y)$ associated with $K$ candidate classes $\{1,2,\cdots,K\}$. For an example $x_{i}$ from the training dataset $(X;Y)$, we denote the ground truth distribution $q$ over the labels as $q(y|x_{i})$ ( $\sum_{y=1}^{K}q(y|x_{i})=1$). Also, we denote a neural network model to be trained as $f_{\theta}$ (parameterized with $\theta$), and it produces a conditional label distribution over the $K$ classes as $p_{\theta}(y|x_{i})$:

with $\sum_{y=1}^{K}p_{\theta}(y|x_{i})=1$, and $z$ is noted as the logit of the model $f_{\theta}$. The logits are generated with two steps: the model $f_{\theta}$ first constructs the $m$-dimensional input embedding $S_{i}\in R^{m}$ for the given input $x_{i}$, and then passes it through a linear fully connected layer $W_{l}\in R^{K\times m}$:

During learning, the model $f_{\theta}$ is trained to optimize the parameter $\theta$ using the $n$ examples from $(X;Y)$ by minimizing the cross-entropy loss:

Instead of using one-hot encoded vector for each example $x_{i}$ in $(X;Y)$, label smoothing (LS) adds a smoothed label distribution (i.e., the prior distribution) $u(y|x_{i})$ to each example $x_{i}$, forming a new target label, namely soft label:

where hyper-parameter $\alpha$ is a weight factor ($\alpha\in[0,1]$) needed to be tuned to indicate the smoothing strength for the one-hot label. This modification results in a new loss:

Usually, the $u(y|x_{i})$ is a uniform distribution, independent of data $x_{i}$, as $u(y|x_{i})=1/K$, and hyper-parameter $\alpha$ is tuned with cross-validation.

For a sample, denoted as $x_{i}$, from the provided training set $(X;Y)$ for training, PLS first randomly selects ¹¹1For efficiency purpose, we implement this by randomly selecting a sample from the same mini-batch during training. another training sample $x_{j}$. For the pair of samples $(x_{i};y_{i})$ and $(x_{j};y_{j})$, where $x$ is the input and $y$ the one-hot encoding of the corresponding class, PLS then generates a synthetic sample through element-wisely averaging both the input features and the labels, respectively, as follows:

These synthetic samples can be considered as the midpoint samples of the original sample pairs. It is also worth noting that, the midpoint samples here are equivalent to fixing the linear interpolation mixing ratios as 0.5 in the Mixup [28] method. Doing so, for the midpoint sample $x_{ij}$ we have the ground truth distribution $q$ over the labels as $q(y|x_{ij})$ ($\sum_{y=1}^{K}q(y|x_{ij})=1$). The newly resulting midpoint $x_{ij}$ will then be used for label smoothing (will be discussed in detail in Section 3.3) before feeding into the networks for training. In other words, the predicted logits as defined in Equation 2 is computed by first generating the $m$-dimensional input embedding $S_{ij}\in R^{m}$ for $x_{ij}$ and then passing through the fully connected linear layer to construct the logit $z$:

Hence, the predicted conditional label distribution over the $K$ classes produced by the networks is as follows:

PLS leverages a learned distribution, which depends on the input $x$, to dynamically generate the smoothing distribution mass for midpoint samples to distribute their ground truth target distribution to the non-target classes. To this end, the PLS implements this by adding a fully connected layer to the network $f_{\theta}$. That is, the $f_{\theta}$ produces two projections from the penultimate layer representations of the network: one for the logits as the original network (Equation 9), and another for generating the smoothing distribution as follows.

In specific, an additional fully connected layer $W_{t}\in R^{K\times m}$ is added to the original networks $f_{\theta}$ to produce the smoothing distribution over the $K$ classification classes. That is, for the given input image $x_{ij}$, its smoothing distributions over the $K$ classification targets, denoted as $u^{\prime}_{\theta}(y|x_{ij})$, are computed as follows:

where $\sigma$ denotes the Sigmoid function, and $S_{ij}$ is the same input embedding as that in Equation 9. In other words, the two predictions (i.e., Equations 9 and 12) share the same networks except the last fully connected layer. That is, the only difference between PLS and the original networks is the added fully connected layer $W_{t}$. The added Sigmoid function here aims to squash the smoothing distributions learned for different targets to the same range of [0, 1].

After having the smoothing distributions $u^{\prime}_{\theta}(y|x_{ij})$, PLS then uses them to smooth the ground truth labels $q(y|x_{ij})$ as described in Equation 7, with average:

The loss function of PLS thus becomes the follows:

Coefficient 0.5 here helps prevent the network from over-distributing its label distribution to the non-ground truth targets. Over-smoothing degrades the model performance, as will be shown in the experiments. Also, 0.5 here causes the resulting training signals to have high uncertainty regarding the ground truth targets. Such high uncertainty helps train the models to make conservative predictions.

For training, PLS minimizes, with gradient descent on mini-batch, the loss $\ell^{\prime}$. One more issue needs to be addressed for the training. That is, the midpoint samples used for training, namely $(x_{ij};y)$, may lack information on the original training samples $(x_{i};y)$ due to the average operation in midpoint generation. To compensate this, we alternatively feed inputs to the networks with either a mini-batch from the original inputs, i.e., ${x}_{i}$ or ${x}_{ii}$, or a mini-batch from the midpoint samples, i.e., ${x}_{ij}$. Note that, when training with the former, the networks still need to learn to assign the smoothing distribution $u^{\prime}_{\theta}(y|x_{ii})$ to form the soft targets $q^{\prime}(y|x_{ii})$ for the sample ${x}_{ii}$. As will be shown in the experiment section, this training strategy is important to PLS’ regularization effect.

We use six benchmark image classification tasks. MNIST is a digit (1-10) recognition dataset with 60,000 training and 10,000 test 28x28-dimensional gray-level images. Fashion is an image recognition dataset with the same scale as MNIST, containing 10 classes of fashion product pictures. SVHN is the Google street view house numbers recognition data set. It has 73,257 digits, 32x32 color images for training, 26,032 for testing, and 531,131 additional, easier samples. Following literature, we did not use the additional images. Cifar10 is an image classification task with 10 classes, 50,000 training and 10,000 test samples. Cifar100 is similar to Cifar10 but with 100 classes and 600 images each. Tiny-ImageNet [5] has 200 classes, each with 500 training and 50 test 64x64x3 images.

We conduct experiments using the popular benchmarking networks PreAct ResNet-18 and ResNet-50 [11]. We compare with the label smoothing methods [18, 16] (denoted as ULS) with various smoothing coefficients (i.e., $\alpha$ in Equation 5), where ULS-0.1, ULS-0.2, and ULS-0.3 denote the smoothing coefficient of 0.1, 0.2, and 0.3, respectively. We also compare with the input-pair based data augmentation method Mixup [28]. We further compare with methods that stacking the ULS on top of Mixup, denoted Mixup-ULS0.1, Mixup-ULS0.2, and Mixup-ULS0.3.

For Mixup, we use the authors’ code at ²²2https://github.com/facebookresearch/mixup-cifar10 and the uniformly selected mixing coefficients between [0,1]. For PreAct ResNet-18 and ResNet-50, we use the implementation from Facebook ³³3https://github.com/facebookresearch/mixup-cifar10/models/. For PLS, the added fully connected layer is the same as the last fully connected layer of the baseline network with a Sigmoid function on the top. All models are trained using mini-batched (128 examples) backprop, with the exact settings as in the Facebook codes, for 400 epochs. Each reported value (accuracy or error rate) is the mean of five runs on a NVIDIA GTX TitanX GPU with 12GB memory.

The error rates obtained by ULS, Mixup, Mixup-ULS, and PLS using ResNet-18 as baseline on the six test datasets are presented in Table 1. The results with ResNet-50 as baselines are provided in Table 2.

Table 1 shows that PLS outperforms, in terms of predictive error, the ResNet-18, the label smoothing models (ULS-0.1, ULS-0.2, ULS-0.3), Mixup, stacking ULS on top of Mixup (Mixup-ULS0.1, Mixup-ULS0.2, Mixup-ULS0.3) on all the six datasets. For example, the relative improvement of PLS over ResNet-18 on Cifar10 and MNIST are over 30% and 24%, respectively. When considering PLS and the average error obtained by the three ULS models, on both Cifar10 and MNIST, the relative improvement is over 23%. It is also promising to see that on Tiny-ImageNet, PLS reduced the absolute error rates over Mixup, Mixup-ULS, and ULS from about 38% to 35%.

For the cases with ResNet-50 as baselines, Table 2 indicates that similar error reductions are obtained by PLS. Again, on all the testing datasets, PLS outperforms all the comparison baselines. For example, for Cifar10, the relative improvement achieved by PLS over ResNet-50 and the average of the three label smoothing strategies (i.e. ULS) are 25.47% and 28.67%, respectively. For Cifar100 and Tiny-ImageNet, PLS reduced the absolute error rates of ULS from 23% and 36% to 18.65% and 30.73% respectively.

An important observation here is that, stacking label smoothing on top of Mixup (i.e., Mixup-ULS) did not improve, or even degraded, Mixup’s accuracy. For example, for Cifar10, Cifar100, and Tiny-ImageNet (the last three columns in Tables 1 and 2), Mixup-ULS obtained similar or slightly higher error rate than Mixup. The reason here is that Mixup creates samples with soft labels through linearly interpolating between [0, 1], which is a form of label smoothing regularization [3]. Consequently, stacking another label smoothing regularizer on top of Mixup can easily mess up the soft training targets, resulting in underfitting. Promisingly, PLS was able to improve over Mixup and Mixup-ULS. For example, on Tiny-ImageNet PLS outperformed Mixup and Mixup-ULS by reducing the error from 38.06% to 35.26% and from 32.36% to 30.73%, respectively, when using ResNet-18 and ResNet-50. Similar error reduction can be observed on Cifar10 and Cifar100.

Impact of Learned Distribution and Original Samples We evaluate the impact of the key components in PLS using ResNet-18 and ResNet-50 on Cifar100. Results are in Table 3. The key components include 1) removing the learned smoothing distribution mass in Equation 13, 2) excluding the original training inputs as discussed in the method section, and 3) replacing the learned smoothing distribution mass with uniform distribution with weight coefficients of 0.1, 0.2, and 0.3 (denoted UD-0.1, UD-0.2 and UD-0.3).

The error rates obtained in Table 3 show that, both the learned smoothing distribution and the original training samples are critical for PLS. In particular, when excluding the original samples from training, the predictive error of PLS dramatically increased from about 19% to nearly 24% for both ResNet-18 and ResNet-50. The reason, as discussed in the method section, is that, without the original training samples, the networks are trained with midpoint samples only, thus may lack information on the validation samples with one-hot labels.

Also, Table 3 indicates that, replacing the learned smoothing distribution in PLS with manually tuned Uniform distribution (i.e., UD) obtained slightly larger errors. This indicates that PLS is able to learn the distribution to smooth the two target labels in the midpoint samples, resulting in superior accuracy and excluding the need for the coefficient search for different applications.

Learning Smoothing Coefficient We also conducted experiments of learning the smoothing coefficient, instead of learning the smoothing distribution, for each midpoint example. That is, we replace Equation 12 with $\sigma(W_{t}S_{ij})$ where $W_{t}$ is $R^{1\times m}$ instead of $R^{K\times m}$. Table 4 presents the error rates (%) obtained using PreAct ResNet-18 and ResNet-50 on Cifar100 and Cifar10. These results suggest that learning to predict smoothing distribution significantly better than predicting the smoothing coefficient in PLS.

Re-weight Smoothing Strength PLS distributes half of a midpoint sample’s ground truth distribution mass over the non-ground truth targets (Equation 13). We here evaluate the impact of different weight factors between the ground truth and non-ground truth targets, by varying it from 0.1 to 0.9 (0.5 equals to the average used by PLS). The results obtained by PLS using ResNet-18 on Cifar100 are in Figure 3. The error rates in Figure 3 suggest that average used by midpoints in PLS provides better accuracy than other weighting ratios. This is expected as discussed in Section 3.3.

This section will show that PLS utilizes high uncertainty labels from midpoint samples for training and produces very conservative classification decisions in testing.

Section 4.4 shows that PLS produces very conservative classification decisions for in-distribution samples. We here explore the effect of PLS on out-of-distribution data.

In this experiment, we first train a ResNet-18 or ResNet-50 network (denoted as vanilla models) with in-distribution data (using either Cifar10 or Cifar100) and then let the trained networks to predict on the testing samples from the SVHN dataset (i.e., out-of-distribution samples). We compare our method with the vanilla baseline (ResNet-18 or ResNet-50), Mixup, and ULS-0.1. The winning predicted softmax scores on the SVHN testing samples are presented in Figure 7, where the top and bottom rows depict the results for ResNet-18 and ResNet-50, respectively.

Figure 7 shows that, PLS again produces low winning predicted softmax scores (namely less confident) against the samples from the SVHN dataset when training with either Cifar10 or Cifar100 data, when compared to Mixup, ULS, and the vanilla models. For example, when being trained with Cifar10 and tested on the SVHN data with ResNet-18 (the top-left subfigure), PLS produced (orange curve) a spike of score distribution near the midpoint 0.5, with extremely spare distribution in the regions of high confidence. In contrast, while Mixup and ULS (yellow/green curves) are more conservative than the vanilla model on out-of-distribution data, it is noticeably more overconfident than the PLS strategy by producing a spike of prediction with 90% confidence at the right end of the figure.