AdaInject: Injection Based Adaptive Gradient Descent Optimizers for Convolutional Neural Networks

Deep learning has shown a great impact over the performance of the neural networks for a wide range of problems [1]. In recent past, convolutional neural networks (CNNs) have shown very promising results for different computer vision applications, such as object recognition [2], [3], [4], [5]; object detection [6], [7]; face recognition [8], [9]; image quality assessment [10]; gesture recognition [11]; Covid-19 grading [12]; and many more. CNNs have also been used as basic building blocks in other networks like Autoencoder [13], [14], [15], Siamese Network [16], [17], Generative Adversarial Networks [18], [19], etc.

The training of different types of deep neural networks (DNNs) is mainly performed with the help of stochastic gradient descent (SGD) based optimization [20]. SGD optimizer updates the parameters of the network based on the gradient of objective function w.r.t. the corresponding parameters [21]. The vanilla SGD optimization suffers from three problems, including 1) zero gradient in local minimum and saddle regions leading to no update in the parameters, 2) a jittering effect along steep dimensions due to the inconsistent changes in the loss caused by the different parameters, and 3) noisy updates due to the gradient computed from the batch of data. SGD with moment (SGDM) [22] considers the first order moment (i.e., velocity) as an exponential moving average (EMA) of gradient for each parameter while training progresses [23]. The parameter is updated in SGDM based on the EMA of gradient which resolves the problem of zero gradient.

Several SGD based optimization techniques have been proposed in the recent past [24], [25], [26], [27], [28], [29], [30], and etc. AdaGrad [24] controls the learning rate by dividing it with the root of the sum of the squares of the past gradients. However, it makes the learning rate very small after certain iterations and kills the parameter update. AdaDelta [25] resolves the diminishing learning rate issue of AdaGrad by considering only a few immediate past gradients. However, it is not able to exploit the global information. In another attempt to resolve the problem of AdaGrad, RMSProp [26] divides the learning rate by the root of the exponentially decaying average of squared gradients. In 2015, Kingma and Ba [27] proposed the adaptive moment based Adam optimizer. Adam combines the ideas of SGDM and RMSprop and uses first order and second order moments. Adam computes the first order moment as the EMA of gradients and uses it to update the parameter. Adam also computes the second order moment as the EMA of the square of gradients and uses it to control the learning rate. Adam performs well in practice to train the convolutional neural networks (CNNs) [27]. However, it suffers from overshooting and oscillations near minimum and varying gradient variance due to batch wise computation. diffGrad [28] resolved the issues as posed by Adam by introducing a friction term in parameter update using the rate of change in gradients. Radam [29] resolved the variance issue as posed by Adam by rectifying the variance of gradients during parameter update. AdaBelief [30] uses the belief in gradients to compute the second order moment. The belief in gradients is computed as the difference between the gradient and the first order moment of the gradient. Other recently proposed and notable gradient descent optimizers are Proportional Integral Derivative (PID) [31], Nesterov’s Moment Adam (NADAM) [32], Nostalgic Adam (NosAdam) [33], YOGI [34], Adaptive Bound (AdaBound) [35], Adaptive and Momental Bound (AdaMod) [36], Aggregated Moment (AggMo) [37], Lamb [38], Adam Projection (AdamP) [39], Gradient Centralization (GC) [40], AdaHessian [41], and AngularGrad [42].

The adaptive SGD optimization techniques have led to a promising performance on deep CNN models. The majority of the above mentioned adaptive gradient descent optimizers suffer due to the overshooting of the minimum and oscillation near minimum. However, it is evident that a robust online stepsize adaptation in optimization plays an important role in gradient descent optimization [43]. We resolve the above issues by injecting the second order moment in first order for the parameter update, which is weighted by the short-term parameter update history to incorporate the robust adaptation of step size. The major contribution of this work is summarized as follows:

• •

  We propose AdaInject for the adaptive optimizers by injecting the short-term parameter change weighted second order moment in EMA of gradient used for parameter update.

• •

  We provide an intuitive explanation in support of the effectiveness of the proposed AdaInject in different optimization scenarios.

• •

  We show the effect of the proposed approach using toy examples. The convergence analysis is also conducted using regret bound which shows the convergence property of the proposed approach.

• •

  We validate the superiority of the proposed injection concept with the recent state-of-the-art optimizers, including Adam [27], diffGrad [28], Radam [29] and AdaBelief [30] using a wide range of CNN models for image classification over four benchmark datasets.

• •

  The proposed concept is generic and can be easily integrated with any existing adaptive moment based SGD optimizer.

The remaining paper is structured by presenting the proposed Injection based optimizers in Section 2; Intuitive explanation and empirical analysis in Section 3; Convergence analysis in Section 4; Experimental analysis in Section 5; and Concluding remarks in Section 6.

As per the conventions used in Adam [27], the aim of gradient descent optimization is to minimize the loss function $f(\theta)\in\mathbb{R}$ where $\theta\in\mathbb{R}^{d}$ is the parameter. The gradient ($g_{t}$) at $t^{th}$ step is computed as $g_{t}\leftarrow\nabla_{\theta}f_{t}(\theta_{t-1})$. Adam computes the first order moment ($m_{t}$) and second order moment ($v_{t}$) as the exponential moving average (EMA) of $g_{t}$ and $g_{t}^{2}$, respectively, which can be written as,

where $\beta_{1}$ and $\beta_{2}$ are the smoothing hyperparameters, typically set as $\beta_{1}=0.9$ and $\beta_{2}=0.999$. The $g_{t}^{2}$ is computed as $g_{t}\cdot g_{t}$ as in the Adam. A bias correction is performed as $\widehat{m_{t}}\leftarrow m_{t}/(1-\beta_{1}^{t})$, $\widehat{v_{t}}\leftarrow v_{t}/(1-\beta_{2}^{t})$ to avoid very large step size in the initial iterations. The parameter update rule in Adam [27] is given as,

where $\alpha$ is the learning rate and $\epsilon=1e^{-8}$ is a small number for numerical stability to avoid division by zero. A detailed algorithm of Adam optimizer is summarized in Algorithm 1. The first order moment $m_{t}$ is used to update the parameters in Adam wherein, the second order moment $v_{t}$ is used to control the learning rate. It can be noticed that Adam mainly relies on the gradients.

However, the SGDM considers only the momentum to update the parameters as follows:

In order to utilize the parameter update history information during optimization, we propose a novel concept named AdaInject. Basically, we inject the short-term parameter change weighted second order moment into first order moment to compute the injected moment using the EMA of $(g_{t}+\Delta\theta\cdot g_{t}^{2})/k$ as,

where $\Delta\theta=\theta_{t-2}-\theta_{t-1}$ is the short-term change in parameter $\theta$ to utilize the parameter history information and $k$ is an injection controlling hyperparameter, typically set to $2$ in the experiment. The injection of parameter history guided second order moment helps the optimizers to perform the smaller updates near minimum (i.e., “steep and narrow” valley) to avoid the overshooting and oscillation, while reasonably large updates are used in the small curvature regions. This phenomenon is depicted in Fig. 1 with a detailed analysis in the next section. We perform the bias correction of injected moment and second order moment as $\widehat{s_{t}}\leftarrow s_{t}/(1-\beta_{1}^{t})$ and $\widehat{v_{t}}\leftarrow v_{t}/(1-\beta_{2}^{t})$, respectively.

The parameter ($\theta$) update of AdamInject optimizer is given as,

where $\alpha$ is the learning rate and $\epsilon=1e^{-8}$ is a small number for numerical stability to avoid the division by zero. We refer to Adam optimizer with the proposed second order moment injection as AdamInject optimizer. A detailed algorithm of AdamInject optimizer is presented in Algorithm 2 with highlighted changes in blue color as compared to vanilla Adam optimizer which is shown in Algorithm 1.

Basically, we use the proposed AdaInject concept with four existing state-of-the-art optimizers, including Adam [27], diffGrad [28], Radam [29] and AdaBelief [30], and propose the corresponding AdamInject (i.e., Adam + AdaInject), diffGradInject (i.e., diffGrad + AdaInject), RadamInject (i.e., Radam and AdaInject) and AdaBeliefInject (i.e., AdaBelief + AdaInject) optimizers, respectively. The algorithms for different optimizers (i.e., without and with AdaInject), such as diffGrad, diffGradInject, Radam, RadamInject, AdaBelief, and AdaBeliefInject, are provided in Supplementary. Though we test the proposed injection concept with four optimizers, it can be extended to any EMA based gradient descent optimization technique. In the next section, we analyze the property of the proposed approach.

In this section, we present an intuitive explanation using a one dimensional optimization landscape having three scenarios and an empirical analysis using three toy examples.

We use the online learning framework to show the convergence property of the proposed injection based AdamInject optimizer, similar to Adam [27]. Let’s represent the unknown sequence of convex cost functions as $f_{1}(\theta)$, $f_{2}(\theta)$,$...$, $f_{T}(\theta)$. We want to estimate parameter $\theta_{t}$ at each iteration $t$ and assess over $f_{t}(\theta)$. The regret bound is commonly used in such scenarios to assess the algorithm where the information of the sequence is not known in advance. The sum of the difference between all the previous online guesses $f_{t}(\theta_{t})$ and the best fixed point parameter $f_{t}(\theta^{*})$ from a feasible set $\chi$ of all the previous iterations are used to compute the regret bound. The regret bound is given as,

where $\theta^{*}=\mbox{arg }\mbox{min}_{\theta\in\chi}\sum_{t=1}^{T}{f_{t}(\theta)}$. The regret bound for the proposed injection based AdamInject is noticed as $O(\sqrt{T})$ which is the same as Adam and is comparable to general convex online learning approaches. We provide the proof in the Supplementary. We consider $g_{t,i}$ as the gradient for the $i^{th}$ element in the $t^{th}$ iteration, $g_{1\mathrel{\mathop{\mathchar 58\relax}}t,i}=[g_{1,i},g_{2,i},...,g_{t,i}]\in\mathbb{R}^{t}$ is the gradient vector in the $i^{th}$ dimension up to $t^{th}$ iterations, and $\gamma\triangleq\frac{\beta_{1}^{2}}{\sqrt{\beta_{2}}}$.

The aggregation terms over the dimension ($d$) can be very small as compared to the corresponding upper bounds, such as $\sum_{i=1}^{d}{||g_{1\mathrel{\mathop{\mathchar 58\relax}}T,i}||_{2}}<<dG_{\infty}\sqrt{T}$, $\sum_{i=1}^{d}{||g_{1\mathrel{\mathop{\mathchar 58\relax}}T,i}||_{2}^{2}}<<dG_{\infty}\sqrt{T}$ and $\sum_{i=1}^{d}{\sqrt{T\hat{v}_{T,i}}}<<dG_{\infty}\sqrt{T}$. The adaptive methods such as the proposed optimizers and Adam show the upper bound as $O(\log d\sqrt{T})$, which is better than $O(\sqrt{dT})$ of non-adaptive optimizers. By following the convergence analysis of Adam [27], we also use the decay of $\beta_{1,t}$.

We show the convergence of average regret of AdamInject in below corollary with the help of the above theorem and $\sum_{i=1}^{d}{||g_{1\mathrel{\mathop{\mathchar 58\relax}}T,i}||_{2}}<<dG_{\infty}\sqrt{T}$, $\sum_{i=1}^{d}{||g_{1\mathrel{\mathop{\mathchar 58\relax}}T,i}||_{2}^{2}}<<dG_{\infty}\sqrt{T}$ and $\sum_{i=1}^{d}{\sqrt{T\hat{v}_{T,i}}}<<dG_{\infty}\sqrt{T}$.

In this section, first we describe the experimental setup. Then, we present the detailed results using different optimizers. Finally, we analyze effects of the hyperparameters.

In this paper, we present a novel and generic injection based EMA of gradients for parameter update by utilizing the parameter change information along with the second order moment. The proposed injection approach leads to an accurate and precise update by performing smaller updates near minimum to avoid the overshooting as well as oscillation and reasonably higher updates in the small curvature regions. The effect of the proposed injection based optimizers is observed using toy examples. The convergence property of the proposed optimizer is also analyzed. The object recognition results for different CNN models over benchmark datasets using four optimizers show the superiority of the proposed injection concept. It is noticed that the injection hyperparameter as 2 yeilds to better results in majority of the cases using the AdamInject optimizer. It is also noted that the batch size as 64 and learning rate as 0.001 are better suitable with the proposed AdamInject optimizer. The intuitive explanation, empirical, convergence, and experimental analyses are evident that the proposed injection based optimizers lead to better optimization of CNNs by avoiding the overshooting of the minimum and reducing the oscillation near minimum to a greater extent.