Disturbing Target Values for Neural Network Regularization

DisturbLabel (DL) [1] regularizes the loss layer by replacing some ground truth labels with incorrect labels in each iteration. DL selects $k$ substitutes randomly to generate disturb labels based on a noise rate $\alpha\%$ that determines the amount of $k=\frac{\alpha}{100}\times(batch\,size)$. When $\alpha=0\%$, a batch-training works without any change. When $\alpha>0\%$, each of selected true labels $y$ is converted into a disturb label by a Multinoulli distribution with the following probabilities:

where $p_{c}$ and $p_{i}$ replaces 1 and 0 respectively in the one-hot vector and C is the number of classes. The DL does not consider the confidence of labels to generate disturb labels.

Our hypothesis is that high confident labels cause the overfitting problem, hence our method Directional DisturbLabel (DDL) considers only confident labels as candidates to be disturbed. To be specific, we define the confident label as $\hat{y}$ satisfying $\frac{y\cdot\hat{y}}{\parallel y\parallel\parallel\hat{y}\parallel}\geq cos\theta$ where $\theta$ is a permissible angle. In $\mathbb{R}^{C}$, the angle $\rho$ between $y$ and $\hat{y}$ is calculated by the cosine similarity and we say that $\hat{y}$ is a confident label if $\rho<\theta$. To simplify the formula $\frac{y\cdot\hat{y}}{\parallel y\parallel\parallel\hat{y}\parallel}\geq cos\theta$, we use the unit vectors, $u=\frac{\hat{y}}{\parallel\hat{y}\parallel}$ and $y$, with $\theta=\frac{\pi}{3}$ so that $y\cdot u\geq 0.5$ is applied to select substitutes for disturb labels in each batch data. The range of $\rho$ is $[0,\frac{\pi}{2}]$. $\rho=0$ when $\hat{y}$ is the same as $y$(i.e. the same direction). Plus, $y$ and $\hat{y}$ always have positive elements so that the maximum angle between them is $\frac{\pi}{2}$. In our setting, non-confident labels from the candidates are excluded by the non-confident interval of $(\frac{\pi}{3},\frac{\pi}{2}]$. For instance, Figure 1 shows that all the outputs of classification models with three classes are on the three dimensional space and the natural basis of the vector space are the one-hot vectors of the three true labels, $(1,0,0),(0,1,0)$, and $(0,0,1)$. Therefore, there exist $y$ and $\hat{y}$ are on the same space such that $y\cdot\hat{y}\geq 0.5$ is calculated.

To prevent overfitting in the training session, DisturbValue (DV) adds Gaussian noise to some of the target values at random. To be specific, a target value $y$ is replaced by $\tilde{y}=y+\epsilon$ following $\epsilon\sim N(0,\sigma^{2})$ based on a noise rate $\alpha$. When $\alpha=p\%$, $\epsilon$ is added randomly to $k$ target values in each batch data where $k\leq\frac{p}{100}\times(batch\,size)$. As hyperparameters of DV, $\alpha$ and $\sigma$ have a huge impact on the regularization performance, but it is too hassle for controlling them depending on datasets. To reduce the number of the hyperparameters, we transform the domain of target values into $[0,1]$ by the MinMax scaler $f(x)=\frac{x-min(x)}{max(x)-min(x)}$ and $\sigma=0.01$ is fixed as a default. Thus, all of the domain of target values is $[0,1]$ regardless of datasets so that the same standard variance $\sigma$ can be used for any datasets. Finally, we consider the only one hyperparameter $\alpha$ during the training.

DV adds noise to target values at random. However, DisturbError (DE) adds Gaussian noise $\epsilon\sim N(0,\sigma^{2})$ to target values satisfy $\mid y-\hat{y}\mid<\rho$ where $\rho$ is a residual boundary. We say that $\hat{y}$ is a high confident value if there exists small constant $\rho$ such that $\mid y-\hat{y}\mid<\rho$. When there are many high confident values, it is highly possible that a model is overfitted to train data. To prevent overfitting, DE disturbs the error of high confident values by redefining the error as $\mid y+\epsilon-\hat{y}\mid$. In DE, we should control $\rho$ and $\sigma$ originally, however we can consider only $\rho$ based on the same scaling and $\sigma$ as DV.

Therefore, we find an optimal hyperparameter of DV and DE depending on datasets and compare to other regularization techniques in Section 4. In this paper, mean square loss function $L=\frac{1}{N}\sum_{i=1}^{N}(\hat{y_{i}}-y_{i})^{2}$ is used such that $\tilde{L}=\frac{1}{N}\sum_{i=1}^{N}(\hat{y_{i}}-(y_{i}+\epsilon_{i}))^{2}$ is defined as an objective function with our disturb methods. Considering the gradient of $\tilde{L}$,

is derived. It shows that the noise $\epsilon$ controls the gradient of prediction values to prevent overfitting.

Evaluation metric. We report test misclassification rate to compare the performance of the methods. Experiments for each dataset method combination were run for 5 times, average value alongside with standard deviation is reported.

Baselines. In our experiments we compare the performance of suggested DDL method against same baselines as the reference paper (including exploring cooperation of the methods) and DL method itself; no regularization, dropout, DistrubLabel (DL), DistrubLabel (DL) + dropout, Directional DisturbLabel (DDL), Directional DisturbLabel (DDL) + dropout.

Datasets. Having a hypothesis that regularization effects should appear more vividly on ’easier’ datasets, we conduct experiments on six collections of images of various complexity: MNIST [24], FMNIST [35], CIFAR-10 [25], CIFAR-100 [16], INTEL [34] , and ART [33]. The characteristics of the datasets are summarized in Table 1.

Architecture. LeNet [20] is modified in accordance to baseline paper (two convolution units for MNIST and FMNIST dataset and three convolution units for CIFAR10, CIFAR100, ART and INTEL followed by ReLU and max pooling). ResNet18 [19] is modified with additional dropout after average pooling step for our experiment on models combination before feeding the input to the last layer with the softmax loss function.

Optimizer. The training procedure was performed using SGD optimizer with momentum and decaying learning rate. We start with learning rate of 0.001 and reduce it by factor 0.1 after 40, 60 and 80 epochs. Each experiment was run for 100 epochs in total.

Hyperparameters. This hyperparameter $\alpha$ (probability of the label to be disturbed) was tuned for each dataset separately. The optimal values of hyperparameters used are summarized in Table 2. The dropout probability was set to 0.5 in all corresponding experiments.

Experimental Results. The results of the experimented on LeNet are summarized in Table 3. DDL method (in combination with Dropout) demonstrated the best results for half of the datasets and is in top-2 results for all of them. Intel and Art dataset considered as simpler datasets which have higher degree of overfitting the other more complex datasets. MNIST is also known for being easily classified by modern networks, therefore we can say that the results prove the hypothesis of DDL method having better regularization capacity and is useful on the datasets which needed it at most. The four newly tested datasets demonstrated similar behaviour except for CIFAR100, which is the most challenging of them, having far more classes. We assume that memorizing the ground truth was not really in place for CIFAR100, that is why perturbing the labels even more has not facilitated the training procedure. A more lightweight method - dropout - nevertheless has still improved the result compared to full absence of regularization.

The results of the experimented on ResNet18 is shown in Table 4 confirming the results obtained for LeNet. For 5 out of 6 tested datasets, DDL (either alone or in combination with Dropout) has gained the best result compared to baselines and is second best for the remaining. FMNIST and CIFAR10 presumably do not require such strong regularization as the most simple MNIST, Intel and Art datasets, therefore using combination of DDL and dropout can be excessive in this case and DDL method obtains the best result alone. CIFAR100 still does not suffer overfitting even with the deeper network to the extend as the other datasets do and does not profit from strong regularization.

Evaluation metric. We use root-mean-square error (RMSE) [7]. For every experiment and every dataset we average RMSE for 20 runs and measure standard deviation.

Baselines. For all eight datasets we compare the results of our methods with the baselines which represent state-of-the-art on neural network regularization to the best of our knowledge and also combination of our methods with baselines: $L_{2}$ regularization, dropout, DV - Gaussian noise, DV - Laplacian noise, DV - cosine annealing [17], DE, DV + dropout, DV + $L_{2}$, DV + DE.

Datasets. We use eight datasets with different sizes and complexity for our evaluations: Boston House Prices (BHP) [8], Bike Sharing (BS) [36], Air Quality (AQ) [10], Make-sklearn (MS) [8], Housing Price (HP) [11], Superconductivty (SC) [12], Communities and Crime (CC) [13], Appliances Energy Prediction (AEP) [14]. The characteristics of the datasets are summarized in Table 5. Also, Minmax scaling is used to standardize the features to present in the data in a fixed range.

Architecture. We implement our methods using neural network with two hidden layers and ReLU activation function.

Optimizer. We use ADAM optimizer [6] with learning rate of 0.001 in all experiments.

Hyperparameters. For each dataset a grid-search is used to find the best value for hyper-parameters:

1. 1.

   $L_{2}$ penalty for $L_{2}$ regularization,

2. 2.

   Drop rate (%) for Dropout,

3. 3.

   Disturb rate (%) for Disturb Value,

4. 4.

   Residual (e) for Disturb Error.

Experimental Results. To tune hyper-parameters all datasets were split on training set (50%) and testing set (50%). For each hyper-parameter we fit our models on training set and then evaluate accuracy metric on testing set. For every run every dataset values were shuffled before split. Table 6 shows the results of experiments, which proved that DV approach or combination of DV with DE, cosine-annealing, Dropout or $L_{2}$ outperforms baselines for all eight datasets. We were interested to know if the most suitable approach depends on data size and data complexity. To check this we form Table 6 starting from a dataset with smaller complexity (smaller number of features) and ending with a dataset with bigger complexity. No dependencies were revealed during this analysis.

We also analyze standard deviations for every method to be sure that offered approaches (DV and DE) are robust. There is no significant deviation between standard deviation for the model without regularization and models with regularization.