Evaluating complexity and resilience trade-offs in emerging memory inference machines

A variety of neuromorphic systems have been implemented using emerging memory devices, but few perform at industrial levels due to the difficulties of implementing software-quality synapses and gradients. These challenges are somewhat simplified in the inference-only application, where imported weights from a separately pre-trained network are used to accelerate predictions on a known task. Nevertheless, electrical issues such as parasitics, cycle-to-cycle read noise, and device-to-device variability may limit the ultimate size and accuracy of inference-only neuromorphic accelerators [1, 2]. These issues can quickly degrade the performance of deep networks with hundreds of thousands of parameters. Accelerators have been proposed to leverage memristive crossbars such as PUMA [3], ISAAC, [4], but they typically discuss the implications of these issues at minimal length. We explore the idea that these systems may be unexpectedly fragile to combined perturbations, and seek mitigations.

Neural networks with an intrinsic temporal behavior may yield new sources of power and resilience [5]. In order to efficiently map standard networks to temporal ones, the efficiency of fundamental operations in a given graph must be considered. As illustrated in Fig. 1, while the overall size of an artificial neural network (ANN) graph may be very large, parts of the necessary computation are done repeatedly. Mosaics in this context refers to a temporal form of neuromorphic multiplexing, whereby certain computations (in this example, the neurons, which are often a limiting resource) can be reused for different stages of an algorithm; allowing larger scale algorithms to be implemented on a resource restricted neuromorphic platform. By exploring the partitioning of a neural graph into sub-graphs that enable the computation to be performed on a smaller subset of available computing nodes - trading space for time - we open a new avenue for failure and resilience analysis. Concretely, in the following sections we explore this contrast through three relatively well-known ANNs: a) a simple, low parameter multi-layer perceptron, b) a complex, medium parameter convolutional neural network (CNN), and c) a medium-parameter, medium complexity recurrent neural network (RNN) inspired by Mosaics re-use concepts. In general, RNNs are an attractive emerging option for the demonstration of on-chip learning or inference, as they are powerful general computational models [6] . Recently, long-short-term memory (LSTM) networks have been the most heavily considered for implementation with dense non-volatile memory arrays [7]; however, such schemes involve complex crossbar partioning and hardware implementation of many transcendental functions (e.g. $tanh$) on the edge of the periphery. Feasibly, LSTM implementations can be done on-chip during inference mode, such as with phase-change memory (PCM) NVM synapses [8], yet the full overhead required to implement such systems has not been documented. Drawing on previous work which shows that an RNN network using appropriate, simple (ReLU) activation functions can still perform competitively to an LSTM on certain tasks [9], in the following sections we demonstrate that a vanilla RNN system can perform competitively to a compact CNN on two state of the art machine learning (ML) tasks, at a lower energy budget.

We trained CNN models using the Keras framework [10] on the iconic MNIST data task [11] as well as the newer fashion-MNIST data task [12]. We then imported these models into CrossSim, a crossbar simulation tool that helps model resistive memory crossbars as highly parameterizable neural cores [13], and which has recently been extended to perform physics-rich analysis of inference operations. Our imported models for CNNs contain 119,322 trainable parameters and contain both convolutional and fully connected (FC) layers, each followed by a bounded rectified linear function (ReLU, given as $f(x)=min(max(0,x),1)$) as visible in Fig. 2(a); exact model is given in Table 1. [14] and as visible in Fig. 2(a). Our imported MLP models are standard consisting of one hidden layer (128 ReLU functions) and a logit of 10; this model has 101,770 trainable parameters. Our recurrent neural network design, newly developed for this work and visible in Fig. 2(b), consists of a recurrent matrix which is re-used during multiple time steps. Notably it is partioned into two cores, a core which always receives a component of the input and a part which receives the hidden layer activation $h_{t}$ from the previous time step $t-1$ (or null if $t=0$) . The number of output nodes $D_{o}$ is set at 400, and depending on the chosen time steps $t$, the number of hidden nodes $D_{hl}$ receiving activation from the previous time step is given as $301-M/t$ , where $M=784$ is the dimensionality of our task. The number of parameters of our RNN networks vary from 118,37 trainable parameters at $t=7$ up to 158,656 parameters at $t=56$ ( time step integers are divisors of $M$). While the the number of physical weights does not increase with the number of time-steps in CrossSim, the number of parameters tracked in Keras tracked for backprogagation of error does.

The RNN core crossbar is connected to a read-out/logit crossbar of dimensions $D_{o}$ x $L$, where $L$ is the number of classes (here $L=10$); critically, the second core is only activated at the final time step (every $t$ steps). In our resilience testing strategy, we have considered two classes of pre-trained networks: noise-prepared or regularized networks, which train with jittered gaussian filters on every hidden neuron’s ReLu function during training following the scheme given in [14] and standard/un-prepared networks which have been trained without noise. As first suggested in [15], the use of noise regularization provides a definitive improvement in inference (Test ) performance of models to internal and external noise or perturbation effects. During training phase, noise centered around 0 with a distribution width $\sigma_{\text{neu}}$ is injected into every ReLu neuron; during testing phase $\sigma_{\text{syn}}$ is then injected on a synapse-per-synase basis . This device-by-device synaptic dispersion most closely relates to intrinsic noise effects that would occur during the operation of large arrays (Johnson-Nyquist noise), but can approximate shot noise or device-specific perturbations or weight variance as suggested in [1]. In addition, given the test set $\mathcal{Y}$, we have also considered the addition of additive gaussian noise at dispersion $\sigma_{\text{te}}$ where $\mathcal{Y}_{\text{noisy}}=\mathcal{Y}+y(\sigma_{\text{te}})$. We have perturbed all weight matrices in every system consistently based on the generation of random numbers with seeds different in every run.

One phenomenon suggested but not proven in our analysis is the idea that noise regularization is still helpful , but less critical than in more complex CNN models, due to the presence of attractor basins in recurrent architectures which help to provide some intrinsic level of noise resilience [20]. We have also discovered that binary stochastic neurons may not be sufficient for full RNN regularization; a more complex function or method may be required. One interesting method suggested in [21] would be the use of temporal skip connections. Another fruitful extension of this work could be the exploration of how recurrent , convolutional , or mixed architectures using effective regularization mtehods (tuned based on the time-step dynamics of networks) could provide defensive properties against adversarial noise or perturbation effects [22, 23]. Finally, while we have showed the Mosaics concept implemented in the temporal domain in this work, the efficient parallel or horizontal stacking of convolution operations may yield more efficient and/or resilient convolutional operations (Figure 1c).

Making emerging memory inference systems more reslient is an important goal for the neuromorphic engineering field, but so far analysis has focused almost exclusively on the limitations of CNN systems. In this work, we have put CNN resilience in conversation with other approaches and in particular our novel RNN approach. We have shown that an ostensibly simple recurrent neural network design efficiently implements the idea of temporal stacking or weight re-use and reduces energy costs by at least 13.5x while achieving results that are competitive with CNNs - at least on tasks of intermediate complexity. In the future, we plan to extend our physics-aware methods to analyse the cross-over points at which various deeper ANNs scale to state-of-the-art tasks given both temporally and physically sequential sub-systems.

Sandia National Laboratories is a multimission laboratory managed and operated by NTESS, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525. This paper describes objective technical results and analysis. Any subjective views or opinions that might be expressed in the paper do not necessarily represent the views of the U.S. Department of Energy or the United States Government.