Adaptive Neuron-wise Discriminant Criterion and Adaptive Center Loss at Hidden Layer
for Deep Convolutional Neural Network

Deep convolutional neural networks (CNN) have achieved great success in image classification problems. CNN consists of several convolution layers and fully-connected layers. The computation in the convolution layer is the filtering with the trainable weights. The fully-connected layers integrate the features extracted by the convolution layers for classification. Usually nonlinear activation function is introduced at each neuron in the hidden layers.

In the deep neural networks, it is known that the gradients of the loss function exponentially decrease with the number of layers. This phenomenon is known as a gradient vanishing problem and is troublesome for the gradient-based learning algorithm, such as the error backpropagation. To prevent the gradient vanishing problem in the deep neural networks, ReLU function has been frequently used as the standard activation function of the neurons in the hidden layers. The ReLU function $\phi(\eta)$ is defined by

where $\eta$ denotes the input of the ReLU function.

The softmax function is often used at the output layer for classification and the softmax cross-entropy loss has been widely used as the objective function for training the parameters of the network.

Let $\{(I_{i},\bm{t}_{i})|i=1\ldots N\}$ be a set of training samples, where $I_{i}$ is the $i$-th image and $\bm{t}_{i}$ is the class label vector of the $i$-th image $I_{i}$. We assume that the class label vector $\bm{t}_{i}=\begin{bmatrix}t_{i1}&t_{i2}&\cdots&t_{iK}\end{bmatrix}^{T}$ is represented as one hot vector. Then the output of the $k$-th neurons in the output layer is defined by using softmax function as

where $\bm{z}_{i}=\begin{bmatrix}z_{i1}&z_{i2}&\cdots&z_{iK}\end{bmatrix}^{T}\in\mathbb{R}^{K}$ and the $k$-th element of the vector $\bm{z}_{i}$, namely $z_{ik}=z_{k}(I_{i})\in\mathbb{R}$, denotes the input of $k$-th neuron at the output layer for the input image $I_{i}$.

The softmax cross-entropy loss for the training samples $\{(I_{i},\bm{t}_{i})|i=1\ldots N\}$ is defined as

where $K$, and $N$ are the number of classes, and the number of training samples.

We can observe that probability distributions for each class of the input of the neuron at the output layer become close to the Gaussian distribution. Since we can consider that the neurons of the output layer is doing the binary classification between the target class and the other classes, we can accelerate the discrimination by introducing the neuron-wise discriminant criterion for this binary classification [11].

The neuron-wise discriminant criterion is a measure of discrimination between the distributions of each class and is defined as

where

The means of the target class and the non-target class of the inputs of $k$-th neuron are denoted by $\mu_{k}$ and $\hat{\mu}_{k}$ respectively. The total mean of the inputs of the $k$-th neuron is denoted as $\mu_{Tk}$. Similarly the number of samples of the $k$-th class is denoted as $N_{k}=\sum_{i}^{N}t_{ik}$. Then the number of samples of the non-traget class is given by $\hat{N}_{k}=\sum_{i}^{N}(1-t_{ik})=1-N_{k}$. Then the means are given as

The objective function for training the parameters of the network is defined by combining the softmax cross-entropy loss and the neuron-wise discriminant criterion as

where $\lambda$ is a hyper parameter to balance the softmax cross-entropy loss and the neuron-wise discriminant criterion.

The neuron-wise discriminant criterion can accelerate discrimination between two classes, but it is necessary to keep all the input values of each neuron to calculate the discriminant criterion. In this paper, we apply online computation with the exponential forgetting to compute the statistics such as $\sigma_{W_{k}}$ and $\sigma_{T_{k}}$. We call this method the adaptive neuron-wise discriminant criterion, and the details are explained in the next section.

It is known that trained CNN with the softmax cross-entropy loss and the ReLU function leads to better accuracy for image classification and the other classification tasks. The center loss [12] was introduced to improve the recognition accuracy for the face recognition tasks further at the output layer similar to the neuron-wise discriminant criterion.

The center loss minimizes the distance between the extracted feature vectors and the mean vector of each class and is defined as

where $\bm{z}_{i}$ is the feature vector for the $i$-th training image $I_{i}$ at the output layer, and $\bm{c}_{k}$ denotes the mean vector of the feature vectors for $k$-th class.

The authors proposed the method to update the mean vector using the samples in the mini-batch. Let $B$ be a set of indexes of the samples in the mini-batch. Then the update rule of the mean vector $\bm{c}_{k}$ of the $k$-the class is defined as

where $\beta\in[0,1]$ is a hyper parameter and

The objective function for training is defined by the combination of the standard softmax cross-entropy loss and the center loss as

where $\lambda$ is a hyper parameter to balance the softmax cross-entropy loss and the center loss.

The center loss can make the feature vectors at the output layer more discriminative. In this paper, we apply the center loss at a hidden layer instead of the output layer. Also, we introduce online computation with the exponential forgetting to compute the mean vectors of each class. We call this method the adaptive center loss, and the details are explained in the next section.

Fig. 3 (a) shows the 2-dimensional extracted features before the ReLU function of the hidden layer. We can observe the elongated shapes of the clusters of each class. It is expected that this phenomenon can be reduced by introducing the center loss at the hidden layer. Thus, we proposed to use the center loss at the hidden layer instead of the output layer and to combine it with the neuron-wise discriminant criterion at the output layer. The locations of the center loss and the neuron-wise discriminant criteria on CNN are shown in Fig. 1.

Also, we introduce the online computation of the means and the variances with the exponential forgetting to calculate the neuron-wise discriminant criterion and the center loss. We named them the adaptive neuron-wise discriminant criterion and the adaptive center loss, respectively. The adaptive neuron-wise discriminant criterion can make the input features of the neurons at the output layer discriminative. The adaptive center loss can also make the extracted feature vectors before the ReLU function at the hidden layer compact and discriminative.

Thus the objective function for the training of the proposed method is defined as

where $\lambda_{1}$, and $\lambda_{2}$ are the hyper parameters to balance the softmax cross-entropy loss $\mathcal{L}_{S}$, the adaptive neuron-wise discriminant criterion $\mathcal{L}_{AD}$, and the adaptive center loss $\mathcal{L}_{AC}$, respectively.

In the neuron-wise discriminant criterion, we have to keep all the features to calculate the within-class variance shown in Eq. (5) and the between-class variance shown in Eq. (6). In the proposed method, they are calculated by using online computation with the exponential forgetting weights. The exponential weight is defined as

The weight $w(s)$ becomes smaller when $s$ becomes the bigger. Then the total mean of the features $\{z_{ik}|i=1,\ldots,n\}$ of the $k$-th neuron is defined by using the forgetting weights as

where $n$ and $\alpha\in(0,1)$ are the number of samples, and the hyper parameter to define the forgetting rate, respectively. The last equation gives the update rule of the online computation. By this online computation, we can compute the total mean for $n$ samples $\mu_{Tk}^{(n)}$ from the total mean of $n-1$ samples $\mu_{Tk}^{(n-1)}$ and the $n$-th sample $z_{nk}$.

Similar with the total mean shown in Eq. (III-B), we can derive the update rules of online computation for the mean of the target class and the non-target classes of the inputs of $k$-th neuron as

The details of the derivations are shown in Appendix.

Similarly we can derive the update rules of online computation for the within-class and the total variances with forgetting weights as

The details of the derivations are also shown in Appendix.

Then the adaptive neuron-wise discriminant criterion is defined by using Eq.(III-B) and Eq.(III-B) as

The original center loss was introduced at the output layer to make the feature more discriminative [12], but it is possible to apply it at a hidden layer. We can observe that the feature vectors extracted before the ReLU function at a hidden layer give the elongated shapes for each class, as shown in Fig. 3 (a). In this paper, we introduce the center loss at the hidden layer instead of the output layer to make the features of each class more circular. Since the neuron-wise discriminant criterion makes the features at the output layer more discriminative, it expected that the recognition accuracy could be improved by combining the center loss at the hidden layer with the neuron-wise discriminant criterion at the output layer.

Also, we propose to use the online computation of the mean vectors of each class with exponential forgetting weights instead of the update computation expressed as Eq.(12) within mini-batch which is proposed in the original center loss [12].

Let $\{\bm{x}_{i}\in\mathbb{R}^{D}|i=1,\ldots,n\}$ be a set of feature vectors extracted before ReLU function at a hidden layer shown in Fig.1, where $D$ is the dimension of an extracted feature vectors. Then the total mean vector of these vectors is defined by using the exponential forgetting weights as

The details of the derivation are shown in Appendix.

The adaptive center loss is defined by using Eq. (III-C) as

To confirm the effectiveness of the proposed approach, we have performed experiments using MNIST dataset. The activation function in this network is ReLU function in the hidden layers, and the softmax function is used in the output layer. Two fully-connected layers are used for classification. The details of the network architecture are shown in Table I. In this table, Conv3x3+1 denotes a convolution layer with $3\times 3$ convolution filters, and the stride of the convolution computation is set to 1, and the padding size is 1. Maxpool2x2 denotes a max-pooling with the size $2\times 2$, and FC denotes a fully-connected layer.

The parameters of this network are trained by minimizing the proposed objective function shown in Eq. (14). Stochastic Gradient Descent (SGD) is used with a momentum of $0.9$ as the optimizer. Both the number of epochs and the number of samples in mini-batch are set to $100$. The initial learning rate is set to $0.01$ and is divided by $10$ at every $50$ epochs. The weight decay parameter is set to $0.01$ to prevent overfitting.

We have performed preliminary experiments to investigate the effectiveness of the proposed approach. We compared the adaptive neuron-wise discriminant criterion in Eq. (21) with the original neuron-wise discriminant criterion [11] in Eq. (4) and the standard CNN without acceleration for feature discrimination (baseline CNN). Also, the adaptive center loss in Eq. (23) at the hidden layer is compared with the original center loss [12] in Eq.(10) at the hidden layer and the standard CNN without acceleration for feature discrimination.

For the neuron-wise discriminant criterion, 10-dimensional features of the output layer are used, and 100-dimensional features before the ReLU activation function at the last fully-connected layer are used to compute the center loss. For the neuron-wise discriminant criterion, we set $\lambda$ to $0.01$. For the center loss, we set $\beta$ and $\lambda$ to $1.0$ and $1.0$, respectively. For the adaptive neuron-wise discriminant criterion, we set $\alpha$ and $\lambda$ to $0.99$ and $0.01$, respectively. For the adaptive center loss, we set $\alpha$ and $\lambda$ to $0.99$ and $1.0$, respectively.

Table II shows the results of these experiments. It is noticed that the adaptive neuron-wise discriminant criterion gives better accuracy than the original neuron-wise discriminant criterion and the baseline CNN, and the adaptive center loss also at a hidden layer gives better test accuracy than the center loss and the baseline CNN. These results show that the adaptive neuron-wise discriminant criterion and the adaptive center loss at the hidden layer can improve the recognition accuracy.

The recognition accuracy obtained by the integration of the adaptive neuron-wise discriminant criterion and the adaptive center loss at the hidden layer is also included in Table II. The parameters $\lambda_{1}$, $\lambda_{2}$, and $\alpha$ are set to $0.001$, $1.0$, and $0.99$, respectively. From this table, we can confirm that the proposed integration of the adaptive neuron-wise discriminant criterion and the adaptive center loss at the hidden layer can give the best test accuracy.

The effectiveness of the proposed approach is confirmed by drawing histograms of the input features of a neuron in the output layer of the CNN which are trained using a training set (60000) of MNIST. In this case, the dimension of the feature vector before the ReLU activation function at the last fully-connected layer is changed from $100$ to $2$ in the network architecture shown in Table I. Fig. 2 (a), (b), and (c) show the histograms obtained by the standard softmax cross-entropy loss, the adaptive neuron-wise discriminant criterion, and the integration of the adaptive neuron-wise discriminant criterion and the adaptive center loss at the hidden layer, respectively.

From this figure, it is noticed that the separations in the histograms (b) and (c) are better than the baseline (a). This result shows the effectiveness of the adaptive neuron-wise discriminant criterion. Also, it is noticed that there are multiple distributions in the non-target class (orange). We think that this phenomenon is caused by the adaptive center loss at the hidden layer.

Fig. 3 shows the scatter plots of the 2-dimensional feature vectors extracted from the hidden layer learned by CNN with ReLU function using training data (60000) on MNIST. In this figure, colors indicate the classes, and black points are the means of each class. Fig. 3 (a), (b), and (c) are the scatter plots of the feature vectors obtained by using the standard softmax cross-entropy loss, the adaptive center loss, and the integration of the adaptive neuron-wise discriminant criterion and adaptive center loss, respectively.

It is noticed that the distributions of each class are more compact in Fig. 3 (b) and (c) than the plot shown in Fig. 3 (a). This means that the adaptive center loss at the hidden layer is useful to make the feature vectors discriminative. It is also noticed that the integration of the adaptive neuron-wise discriminant criterion and adaptive center loss can make the feature vectors more discriminative.

The proposed integration of the adaptive neuron-wise discriminant criterion and adaptive center loss is compared with the original neuron-wise discriminant criterion and the center loss at the hidden layer with several datasets such as FashionMNSIT, CIFAR10, CIFAR100, and STL10. For the FashionMNIST data set, the same network with the preliminary experiment shown in Table I. The more complex network architecture shown in Table III is used for the data sets CIFAR10, CIFAR100, and STL10.

In Table III, $h$ and $w$ denote the height and the width of input images. Conv3x3+1 denotes the convolution layer with the size $3\times 3$, and the stride and the padding are 1. Batchnorm denots batch normalize [17]. Maxpool2x2 denotes max-pooling with the size $2\times 2$. Conv3x3 denotes the convolution layer with the size $3\times 3$, where the stride is 1, but the padding is 0. FC denotes the fully-connected layer, and ReLU is the activation function. Dropout denotes drop out [18].

These networks are trained by using the training samples of each data set. For the training, the number of epochs and the number of samples in the mini-batch are set to $500$ and $100$, respectively. As the optimizer, we use Stochastic Gradient Decent (SGD) with a momentum of $0.9$. The learning rate is set to $0.01$ and is divided by $10$ at every $100$ epochs. The weight decay parameter is set to $0.01$ for FashionMNIST, $0.01$ for CIFAR10, $0.001$ for CIFAR100, and $0.01$ for STL10. As the preprocessing, we apply the affine transformation to the inputs for CIFAR10, CIFAR100, and STL10 to prevent the overfitting.

Similar to the preliminary experiments, each feature of the output layer is used to compute the neuron-wise discriminant criterion, and the 100-dimensional feature vectors before the ReLU function at the hidden layer are used to compute the center loss.

For the neuron-wise discriminant criterion, the hyper parameter $\lambda$ is set to $0.001$ for FashionMNIST, $0.001$ for CIfAR10, $0.01$ for CIFAR100, and $0.01$ for STL10. For the center loss, the parameter $\beta$ is set to $1.0$, and the parameter $\lambda$ is set to $1.0$ for FashionMNIST, $0.08$ for CIFAR10, $0.01$ for CIFAR100, and $0.01$ for STL10, respectively. For the adaptive neuron-wise discriminant criterion and the adaptive center loss, the parameter $\alpha$ is set to $0.99$, and $\lambda_{1}$ and $\lambda_{2}$ to $0.0001$ and $1.0$ for FashionMNIST, $0.0001$ and $0.08$ for CIFAR10, $0.01$ and $0.001$ for CIFAR100, and $0.01$ and $0.001$ for STL10.

The results are shown in Table IV. It is obvious from this Table that the proposed integration of the adaptive neuron-wise discriminant criterion and adaptive center loss at the hidden layer give the best test accuracy for all data sets. Thus we can say that the proposed approach can improve the classification accuracy of the trained CNN.