Moment Centralization based Gradient Descent Optimizers for Convolutional Neural Networks

Deep learning has become very prominent to solve the problems in computer vision, natural language processing, and speech processing [17]. Convolutional neural networks (CNNs) have been exploited to utilize deep learning in the computer vision area with great success to deal with the image and video data [15], [22], [9], [24], [1], [19]. The CNN models are generally trained using the stochastic gradient descent (SGD) [3] based optimization techniques where the parameters of the model are updated in the opposite direction of the gradient. The performance of any CNN model is also sensitive to the chosen SGD optimizer used for the training. Hence, several SGD optimizers have been investigated in the literature.

The vanilla SGD algorithm suffers from the problem of zero gradients at local minimum and saddle regions. The SGD with momentum (SGDM) uses the accumulated gradient, i.e., momentum, to update the parameters instead of the current gradient [25]. Hence, it is able to update the parameters in the local minimum and saddle regions. However, the same step size is used by SGDM for all the parameters. In order to adapt the step size based on the gradient consistency, AdaGrad [8] divides the learning rate by the square root of the accumulated squared gradients from the initial iteration. However, it leads to a vanishing learning rate after some iteration as the squared gradient is always positive. In order to tackle the learning rate diminishing problem of AdaGrad, RMSProp [10] uses a decay factor on the accumulated squared gradient. The idea of both SGDM and RMSProp is combined in a very popular Adam gradient descent optimizer [13]. Basically, Adam uses a first order moment as the accumulated gradient for parameter update and a second order moment as the accumulated squared gradient to control the learning rate. A rectified adam (Radam) [18] performs the rectification in Adam to switch to SGDM to improve the precision of the convergence of training. Recently, Adabelief optimizer [27] utilizes the residual of gradient and first order moment to compute the second order moment which improves the training at saddle regions and local minimum. Other notable recent optimizers include diffGrad [6], AngularGrad [20], signSGD [2], Nostalgic Adam [11], and AdaInject [7].

The above mentioned optimization techniques do not utilize the normalization of the moment for smoother training. However, it is evident that normalization plays a very important role in the training of deep learning models, such as data normalization [23], batch normalization [12], instance normalization [5], weight normalization [21], and gradient normalization [4]. The gradient normalization is also utilized with SGD optimizer in [26]. However, as the first order moment is used for the parameter updates in adaptive momentum based optimizers, we proposed to perform the normalization on the accumulated first order moment. The contributions of this paper can be summarized as follows:

• •

  We propose the concept of moment centralization to normalize the first order moment for the smoother training of CNNs.

• •

  The proposed moment centralization is integrated with Adam, Radam, and Adabelief optimizers.

• •

  The performance of the existing optimizers is tested with and without the moment centralization.

The rest of the paper is structured as follows: Section 2 presents the proposed moment centralization based SGD optimizers; Section 3 summarizes the experimental setup, datasets used and CNN models used; Section 4 illustrates the experimental results, comparison and analysis; and finally Section 5 concludes the paper.

In this paper, a moment centralization strategy is proposed for the adaptive momentum based SGD optimizers for CNNs. The proposed approach imposes the explicit normalization over the first order moment in each iteration. The imposed zero mean distribution helps the CNN models for a smoother training. The proposed moment centralization is generic and can be integrated with any adaptive momentum based SGD optimizer. In this paper, we use it with state-of-the-art optimizers, including Adam [13], Radam [18] and Adabelief [27]. The Adam optimizer with moment centralization is termed AdamMC. Similarly, the Radam and Adabelief optimizers with moment centralization are referred to as RadamMC and AdabeliefMC, respectively. In this section, we explain the proposed concept with Adam, i.e., AdamMC.

Let a function $f$ represents a CNN model having parameters $\theta$ for image classification. Initially, the parameters $\theta_{0}$ are initialized randomly and updated using backpropagation of gradients in the subsequent iterations of training. During the forward pass of training, the CNN model takes a batch of images ($I_{B}$) containing $B$ images as the input and returns the cross-entropy loss as follows:

where $p_{c_{i}}$ is the softmax probability of given input image $I_{i}$ for the correct class level $c_{i}$. Note that the class level is represented by the one-hot encoding in the implementation. During the backward pass of the training, the parameters are updated based on the gradient information. Different SGD optimizers utilize the gradient information differently for the parameter update. The adaptive momentum based optimizers such as Adam are very common in practice which utilizes the first order and second order momentum which is the exponential moving average of gradient and squared gradient, respectively. Hence, we explain the proposed concept below with the help of Adam optimizer.

Consider $g_{t}$ to be the gradient of an objective function w.r.t. the parameter $\theta$, i.e., $g_{t}=\nabla_{\theta}f_{t}(\theta_{t-1})$ in the $t^{th}$ iteration of training, where $\theta_{t-1}$ represents the parameter values obtained after $(t-1)^{th}$ iteration. The first order moment ($m_{t}$) in the $t^{th}$ iteration is computed as follows,

where $m_{t-1}$ is the first order moment in $(t-1)^{th}$ iteration and $\beta_{1}$ is a decay hyperparameter. Note that the distribution of first order moments is generally not zero centric which leads to the update in a similar direction for the majority of the parameters. In order to tackle this problem, we propose to perform the moment centralization by normalizing the first order moment with zero mean as follows,

The second order moment ($v_{t}$) in the $t^{th}$ iteration is computed as follows,

where $v_{t-1}$ is the second order moment in $(t-1)^{th}$ iteration and $\beta_{2}$ is a decay hyperparameter. Note that we do not perform the centralization of second order moments as it may lead to a very inconsistent effective learning rate.

The first and second order moments suffer in the initial iteration due to very small values leading to very high effective step size. In order to cope up with this bias, Adam [13] has introduced a bias correction as follows,

where $\widehat{m_{t}}$ and $\widehat{v_{t}}$ are the bias corrected first and second order moment, respectively. Finally, the parameter update in $t^{th}$ iteration is performed as follows,

where $\theta_{t}$ is the updated parameter, $\theta_{t-1}$ is the parameter after $(t-1)^{th}$ iteration, $\alpha$ is the learning rate and $\epsilon$ is a very small constant number used for numerical stability to avoid the division by zero. The steps of Adam optimizer without and with the proposed moment centralization concept are summarized in Algorithm 1 (Adam) and Algorithm 2 (AdamMC), respectively. The changes in the AdamMC are highlighted in blue color. Similarly, we also incorporate the moment centralization concept with Radam [18] and Adabelief [27] optimizers. The steps for Radam, RadamMC, Adabelief, and AdabeliefMC are illustrated in Algorithm 3, 4, 5, and 6, respectively.

All experiments are conducted on Google Colab GPUs using the Pytorch 1.9 framework. We want to emphasise that our Moment Centralization (MC) approach does not include any additional hyper-parameters. To incorporate MC into the existing optimizers, only one line of code is required to be included in the code of existing adaptive momentum based SGD optimizers, with all other settings remaining untouched.

In order to demonstrate the improved performance of the optimizers with the proposed moment centralization, we conduct the image classification experiments using VGG16 [22] and ResNet18 [9] CNN models on CIFAR10, CIFAR100 and TinyImageNet datasets [14] and report the classification accuracy. The results are compared with the corresponding optimization method without using moment centralization. Moreover, the results are also compared with the corresponding optimization method with the gradient centralization [26]. We repeat all the experiments three times with independent initializations and consider the mean and standard deviation for the comparison purpose. All results are evaluated with the same settings as described above.

The classification accuracies on the CIFAR10, CIFAR100 and TinyImagenet datasets are reported in Table 1, Table 2, and Table 3, respectively. It is noticed from these results that the optimizers with moment centralization (i.e., AdamMC, RadamMC, and AdabeliefMC) outperform the corresponding optimizers without moment centralization (i.e., Adam, Radam, and Adabelief, respectively) and with gradient centralization (i.e., AdamGC, RadamGC, and AdabeliefGC, respectively) in most of the scenario. The performance of the proposed AdamMC optimizer is also comparable with Adam on the CIFAR10 dataset, where AdamGC (i.e., Adam with gradient centralization [26]) fails drastically. It’s also worth mentioning that the classification accuracy using the proposed optimizers is very consistent in different trials with a reasonable standard deviation in the results, except on TinyImageNet dataset using VGG16 model.

The accuracy plot w.r.t. epochs is depicted in Fig. 1 on CIFAR100 dataset for different optimizers. Note that the learning rate is set to 0.001 for first 80 epochs and 0.0001 for last 20 epochs. It is noticed in all the plots that the performance of the proposed optimizers boosts significantly and outperforms other optimizers when the learning rate is dropped. It shows that the proposed moment centralization leads to better regularization and reaches closer to minimum.

In order to justify the improved performance of the proposed Adam_MC method, we show the convergence plot in terms of the optimization trajectory in Fig. 2 with random intializations for following toy example:

It is a quadratic ‘bowl’ with two gaussian looking minima at (1, 0) and (-1, 0), respectively. It can be observed in Fig. 2(a) that the Adam_MC optimizer leads to a shorter path and faster convergence to reach minimum as compared to other optimizers. Figure 1 illustrates how the moment centralization for Adam, RAdam, and Adabelief achieved higher accuracy after the 80th epoch than without. This demonstrates that optimizers with MC variation have faster convergence speed than without. The SGD shows much oscillations near the local minimum. In Fig. 2(b), Adam and Adam_GC depict fewer turns as compared to SGD with momentum, while Adam_MC exhibits smoother updates. In Fig. 2(c-d), Adam_MC leads to less oscillations throughout its course as compared to other optimizers. Hence, it is found that the proposed optimizer leads to shorter and smoother path as compared to other optimizers.

In this paper, a moment centralization is proposed for the adaptive momentum based SGD optimizers. The proposed approach explicitly imposes the zero mean constraints on the first order moment. The moment centralization leads to better training of the CNN models. The efficacy of the proposed idea is tested with state-of-the-art optimizers, including Adam, Radam, and Adabelief on CIFAR10, CIFAR100 and TinyImageNet datasets using VGG16 and ResNet18 CNN models. It is found that the performance of the existing optimizers is improved when integrated with the proposed moment centralization in most of the cases. Moreover, the moment centralization outperforms the gradient centralization in most of the cases. Based on the findings from the results, it is concluded the moment centralization can be used very effectively to train the deep CNN models with improved performance. It is also observed that the proposed method leads to shorter and smoother optimization trajectory. The future work includes the exploration of the proposed idea on different types of computer vision applications.