Benchmarking Inference Performance of Deep Learning Models on Analog Devices

Deep Learning (DL) has enjoyed and continues to enjoy renewed interest from researchers due to their ability to achieve or even surpass human-level performance for many cognitive applications. The resurgence can be attributed to large scale dataset, high-performance hardware, new activation functions and more sophisticated optimization methods [1]. These factors have led to an advancement in the field of deep learning favoring the design of deep, large, and very complex models. Despite their unprecedented level of performance and improvement in their design in recent years, DL models require high computational and energy resources during training and inference [2, 3]. The high computational resource requirement is because of the intense fundamental operations by these models, such as dot product of vector and matrix, and multiplications of matrices, during training and inference [4, 5]. This is further complicated by the large increase in the quantity of these operations with the increase in the size of the models. The high energy requirement can be attributed to the data intensive nature of the models and the huge memory and memory bandwidth requirement of the deep and large models. As a result, typical CPU and GPU on edge devices do not have enough memory resource to process these models efficiently as the CMOS technology is approaching its limit and making it a performance bottleneck [2, 4]. Hence, models are stored in the off-chip memory leading to the presence of a memory wall, a physical separation between the processing unit and the memory [4]. This separation means there is a need for constant shuttling for data access between the processing unit and the off-chip memory that leads to high energy consumption and high latency [6].

At the same time, there has been increasing demand for high performance and energy efficient computing in recent years. This demand has been further fueled by the desire to deploy DL models on Internet-of-Things (IoT) and edge devices which operate within a tight computational resource and power envelope [7]. This tight requirement and the desire to fix compute and memory transfer bottlenecks in current set of hardware has led to a significant interest in analog specialized hardware for DL, as they have the potential to deliver at least 2X better performance than the conventional digital hardware in both speed and energy efficiency [8, 9]. In fact, they can deliver at projected throughput of multiple tera-operations (TOPs) per seconds and also achieve femto-joule energy budgets per multiply-and-accumulate (MAC) operation [5, 10, 11, 12]. The improvement can be attributed to the use of non-volatile memory cross bar arrays to encode DL model weights and biases, a form of computing known as in-memory computing. The arrays have a multi-level storage capability and also allow a single time step matrix-vector multiplication based on Kirchhoff’s circuit laws [7, 4, 13, 6]. Despite these advantages, analog accelerators do not have the bit-exact precision enjoyed by digital hardware and suffer from noise [7], as shown in Figure 1. This can be attributed to many factors such as thermal noise, quantization noise, circuit non-linearity, and device failure [2]. These disadvantages can affect the reliability of DL models in the form of performance degradation depending on the prevailing factors. Although there are some studies on the robustness of DL models to label noise [14], very little is known on the inference behavior of trained DL models when the weights are subject to analog noises.

The objective of this paper is to systematically study the effect of additive noise on the performance of trained DL models implemented in analog accelerators. Specifically, we try to establish the benchmark of performance degradation and provide observations on the detailed behavior of some popular computer vision DL models for image classification task in the presence of analog noise induced weight changes, as illustrated in Figure 1. The noise is modeled as additive white noise to the weights of trained DL models implemented in analog accelerators. We vary the strength of the noise and the impact of the noise on each layer of the model is investigated. The remainder of this paper is organized as follows: The methodology used for this work is discussed in Section II. Experimental results and analysis are given in Section III. Further discussions and related works are reviewed in Section IV. Section V concludes the paper.

The effect of additive white noise on the performance of pre-trained DL models for computer vision task during inferencing is investigated. The DL models used for this work are selected models used for image classification task. The performance metrics under observation is the percentage of classification accuracy (100% - error) of the model on the test data.

The noise was modeled as a white Gaussian noise of zero mean and a standard deviation of $\sigma_{noise}$. The standard deviation of a white noise can be interpreted as the energy of the noise. The value of $\sigma_{noise}$ was decided using a term known as the signal to noise ratio (SNR). The SNR in this context is defined as the ratio of the standard deviation of the weights in a layer $\sigma_{w}$, to the standard deviation of the noise added to that layer $\sigma_{noise}$. This is defined mathematically as:

$$SNR=\frac{\sigma_{w}}{\sigma_{noise}}$$

(1)

A Gaussian noise of zero mean and standard deviations equivalent to 1%, 10%, 20%, 40%, 60% and 100% of the standard deviation of the weights of a particular layer $\sigma_{w}$, is added to the same layer. This is equivalent to the SNR of 100, 10, 5, 2.5, 1.67 and 1, respectively.

The pre-trained model of interest was put in inference mode and performance measurement on the test data is taken to establish the baseline. Thereafter, the effect of the additive noise was investigated by adding Gaussian noise of the desired standard deviation or energy to the first layer of the model. The model was then set to inference mode and the performance measurement was taken. This was done multiple times and the average of the testing accuracy was obtained. This process was then repeated for the rest of the layers and the testing accuracy was recorded. The classification accuracy due to the present of noise $a_{i}$ in the model is then normalized with the baseline classification accuracy $a_{o}$, that is given by:

$$A_{i}=\frac{a_{i}}{a_{o}}$$

(2)

where $A_{i}$ is the normalized classification accuracy due to the present of noise in layer $i$.

There are several works in the literature on the effect of analog noise on neural network models. The use of noise for regularization purpose of deep learning model was proposed in [24]. It also considered 3 approaches to controlling trade-off between the bias and the variance of a neural network. The works of [25, 26] focused on the injection of noise to the synaptic weight of neural networks during training to improve their noise resistant ability.The improvement in noise resistant ability is very important as analog devices suffers from limited precision and also presence of analog noise in the hardware. The use of knowledge distillation,a method to transfer knowledge from a teacher model to a student model, was used to improve further the noise resistant ability of a DL model in [7].In this case, the student and the teacher model are the same model. The student model obtained from this method showed better performance than the DL model obtained from the method used in [25] during inferencing in the presence of analog noise.

Instead of training the DL models on digital devices, the training of neural networks on analog hardware in order to improve their noise resistant ability was done in [27, 28]. The DL model is trained using stochastic gradient descent and back-propagation in the presence of noise it will later experience during testing. The resulting model adapt to the noise by developing some resistance to it in the process of learning its weight.Despite its advantages, this method has some demerits. The need to design a back-propagation circuitry on a device for only inferencing complicates the chip design and also increases the area of the chip though the back-propagation is only needed once. This method is laborious as DL training might have to be done on every chip that will be used during testing. This is because the analog noise power might vary from one chip to another [2]. Chip-in-the-loop method, a method that involves measuring the error that the model is going to experience in the hardware and then using the error to update the weights in software, was used in [29, 30].This method is used to tune pre-trained model for the chip in order to make them adaptable to the chip. The process can be slow and very inefficient. The use of linear and non-linear analog error correcting codes to protect the analog weights was done in [31]. A study on the design of code rate of error correction code for different layers of the neural network was also conducted. A comparison was done between this method and the binarized neural network which is an alternative to this method. The use of deep reinforcement learning and selective protection scheme to choose the important bit for error correcting codes protection was done in [32]. The reinforcement learning algorithm was used to determine the complex relationship between the bits to protect and the model performance in order to determine the optimal trade off between redundancy and performance. The paper also showed that the most important bit in the weights of a DL model that is worthy of protection is not always the most significant bit.

The existing works are different from this study as they explore ways to use analog noise to improve on the noise resistant properties of DL models. On the other hand, the objective of this study is to investigate and provide intuitions and insights about the noise resistant ability of popular deep neural networks, especially those DL models for image classification. Although some bench marking studies were done in [31], digital noise (bit flipping) was used unlike this work where we used analog noise (additive white Gaussian noise). Furthermore, a comprehensive study has been performed in this work, including examining the performance of the DL models when the noise is added to all the weights, and the detailed effects of the noise on each individual layer of the DL models. The effect of noise on the layers of each neural network is important as layers are the building blocks of DL models. In addition, this study compared the performance differences between models of difference design and provide insights on the effect of model compression on the noise resistance ability of DL models.

Analog hardware implemented deep learning models are considered a promising approach for edge learning because they have faster execution speed and at the same time very energy efficient comparing to digital devices. However, one of the major concerns of such an approach is the analog noise incurred weights change in the deep learning models. In this paper, the noise resistance ability of various deep learning models in the presence of additive white Gaussian noise was investigated. Specifically, systematic experiments are carried out by adding white Gaussian noise of various power level to the weights in some of the popular deep learning models for image classification such as VGG, ResNet, and Shufflenet models. In addition to adding noise to all the weights of the models, experiments of adding noise to weights layer-by-layer are also conducted to examine the role of each layer in the deep learning models in terms of noise resistance. This has been done for deep learning models with different design principles, models with similar design principles but different depths or different number of parameters, as well as models with and without compression. A performance metric to assess the resistance of a model to the presence of noise in its layer was introduced and used to compare the performance of various models.

In this work, it was observed that deep learning models do have some noise resistant ability measured by their performance degradation due to various noise levels and it varies from model to model. It was shown that the amount of degradation experience due to the presence of noise could be affected by factors such as number of parameters (model compression), depth of the model and design philosophy of the model. It was also observed that deeper layers in the models are more resistance to noise than the earlier layers. This is an important step towards understanding how deep learning models would perform when implemented in analog hardware with noise affecting the performance of such models. The hope is that through comprehensive study, practitioners may be able to choose the appropriate deep learning models for given analog hardware to meet performance requirements. Furthermore, designers of deep learning models may find new ways to design the structure of the models and new methods to train the models that make them more robust to noise when implemented in analog hardware.

This research work is supported in part by the U.S. Dept. of Navy under agreement number N00014-17-1-3062 and the U.S. Office of the Under Secretary of Defense for Research and Engineering (OUSD(R&E)) under agreement number FA8750-15-2-0119. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Dept. of Navy or the Office of the Under Secretary of Defense for Research and Engineering (OUSD(R&E)) or the U.S. Government.