Enabling Fast Deep Learning on Tiny Energy-Harvesting IoT Devices

Modeling challenges: There are several challenges during DNN modeling. 1) SRAM/FRAM size 2) CPU frequency and inference time 3) Overflow error 4) Model accuracy. Aware of the above challenges, $RAD$ trained the DNN model offline. The model must fit into the FRAM with acceptable inference time and accuracy. $RAD$’s architecture search technology finds a suitable model and further compresses it. As we deal with fixed-point calculation in a resource constraint device, there is possible data overflow during vector operations like MAC and FFT. As a result, $RAD$ performs several normalizations so that data stays between ranges during calculation.

Fixed-point quantization: Traditionally, DNN models are trained using high precision floating-point calculation. During on-device deployment, this is costly in terms of latency and memory. In this work, we use low-precision fixed-point representation. To implement the low-precision fixed-point representation, $RAD$ maps high precision floating-point to 16-bits fixed-point so that $ACE$ can deploy the DNN model with fixed-point calculation.

Normalization: For normalization, $RAD$ first sets the data range $G_{i}$ with a minimum value, $G_{min}$, as -1 and a maximum value, $G_{max}$, as 1. $RAD$ then normalizes the data within this range of $[-1,1]$. Finally, to avoid the value of the computed intermediates exceeding this range during inference, $RAD$ uses the cosine normalization [12] to constrain the values of the computed intermediates into $[-1,1]$.

Compression: The fully connected (FC) layers are partitioned into 2D blocks of square sub-matrices and each of sub-matrices is a circulant matrix. This block circulant matrix (BCM) [6] enables the FFT-based computation for the FC layers. In addition, the structured pruning is also applied on the convoluitonal (CONV) layers to reduce the redundancy of CONV. The combination of BCM on FC and structured pruning on CONV can achieve drastically reduced memory footprint with a very small accuracy drop. Especially for the BCM, shown in Table I, the memory footprint reduction for FC layers is up to 99.60%.

Alternating Direction Method of Multipliers (ADMM)-Regularized Optimization: The structured pruning problem can be effectively solved using the ADMM-regularized optimization. After the BCM-based pretrained DNN model is obtained, we define the constraint for the structured pruning separately on accelerators via the ADMM-regularized optimization [4].

Consider an $N$-layer DNN, where the weights and biases in the $i$-th CONV layer are ${\bf{W}}_{i}$ and ${\bf{b}}_{i}$. The optimization problem is shown below.

where ${\bf{S}}_{i}$ = {remaining non-zeros weights in convolutional layer satisfy the requirement of structured pruning} is the pruning constraint set. We use the method proposed in ADMM-NN [15] to solve the problem.

Acceleration-aware dataflow: Figure 3 demonstrates the on-device data flow of LeNet-5 model as an example. Before executing each layer, the input and kernel are buffered in the SRAM allocated specifically for the accelerators. The buffered vectors are then sent to the accelerator for computation. Next, the outputs from the accelerators, buffered in SRAM, are stored in the FRAM. The subsequent Maxpooling layer and activation function (ReLU) is loaded directly into the CPU without SRAM buffering. $ACE$ also selects the right kind of data movement method based on the energy and latency of moving the data. For example, large vector of data is moved with DMA while a single data is moved with CPU. During the inference, more time is spent on data movement, thus utilizing DMA with bulk data transfer achieves significant improvement over CPU-based data transfer.

Quantization: $ACE$ provides software support for 16-bit fixed-point quantization to represent the floating-point data. $B=>A*2^{b-1}$ is the common quantization rule for embedded devices where “A” is a floating-point number, “B” is a quantized number, and “b” represents quantization bit.

Hardware Acceleration of FC layer: With the FFT/IFFT low energy vector operations, the DNN inference time and energy can be reduced significantly for BCM-based fully connected (FC) layers. We redesign the computation of FC layer as shown in Algorithm 1, which describes the on-device implementation of BCM computation. In lines 3 and 4, input and weight data are scaled-down to overcome FFT operation overflow. Since data is not in complex form during the convolution, the real input needs to be converted into the complex number before performing FFT operation. $ACE$ performs complex/real operations in lines 5-8. Line 7 performs the core function of the BCM based DNN. It performs FFT, complex number multiplication on both inputs, and then IFFT on the result. After that, real number is retrieved at line number 8 and stored to FRAM. In Line 9, output data is scaled up.

Hardware Acceleration of CONV layer: Instead of element-wise computation, $ACE$ implements bulk computation by computing the whole kernel at a time. For example, as Figure 4, a typical convolution of a 3x3 kernel needs 18 operations (9 multiplication and 9 addition) with an input window. Instead, an accelerator can replace the whole with a single MAC [17] operation, resulting in drastic improvement in overall convolution performance.

Overflow-aware Computation: Fixed point calculation in resource constraint devices frequently suffers from data overflow errors. During FFT computation, there is a possible data overflow if the totaling of the input array exceeds the capacity of the quantized bit. For example, in an 8-bit data quantization, the FFT will produce wrong results if the addition of the input array elements exceeds $2^{8}=256$. To address the challenge, $ACE$ performs data scaling based on the array size in Algorithm 1. Besides, there is also possible overflow during accelerator operation. The solution is provided using normalization as discussed in Section III-A.

Circular Buffer Convolution: $ACE$ optimizes the memory usage by reusing the input/output buffer after a layer-level computation takes place. In the existing DNN implementation, an inference with $N$ layers requires N buffers, as shown in figure 5. Instead of allocating memory for individual layers, $ACE$ requires only two buffers (input and output) at most. $ACE$ implemented circular buffer convolution, restructured the DNN inference, which reuses the buffer by interchanging and overwriting the input and output pointer after finishing a layer-level computation. $ACE$ can considerably reduce the memory footprint, irrespective of the layer size in a DNN inference. The size required for the buffer is $max(L_{i}),1\leq i\leq N$.

BCM-based FC layer: The left part of Figure 6 demonstrates the drawback of $TAILS$ when we try to employ FFT-based BCM computation under the intermittent power supply. An FFT-based BCM computation has to go through several steps including DMA, FFT/IFFT, and MPY as shown in algorithm 1. The arrays highlighted in red, $x,w,y,y^{\prime}$ are intermediate values. In loop-index based techniques, these are very much susceptible to data loss whenever a power failure happens. On power restore, the program has to roll back to the initial $DMA$ operation because only the loop index is insufficient to retrieve the current state of the Algorithm. It causes wasted work and hurts the progress. As a solution, $FLEX$ stores block index, intermediate result and state bit of the Algorithm. As shown in the right part of Figure 6, $FLEX$ specifies the 4 additional bits $b_{0}$-$b_{2}$ to indicate the current state, resulting in a successful continuation from the interrupted operation on power failure. As the control bits are small, it requires small memory footprint and read/write energy cost.

Other layer: In BCM-based FC layers, with the help of a voltage monitor system, $FLEX$ predicts a power failure and checkpoint the latest intermediate result. Apart from that, for all other layers, $FLEX$ follows loop index-based checkpointing mechanism that checkpoint and restores the system, based on loop indices.

Hardware Setup: The proposed DNN implementation framework is evaluated with TI’s MSP430FR5994 [18] ultra-low-power evaluation board, consisting of a 16 MHz MCU, a 8KB volatile SRAM, a 256KB nonvolatile FRAM memory, and a low-energy Accelerator (LEA) that runs independently of CPU. The LEA accelerator supports FFT, IFFT, MAC, and MPY operation. The board is connected to the function generator SIGLENT SDG1032X [1] to simulate the energy harvesting scenario. Energy is buffered with a capacitor of 100µF. For the energy measurement, we used CCS energy trace technology [16].