In-Network Accumulation: Extending the Role of NoC for DNN Acceleration

The increasing popularity of Deep Neural Networks (DNN) can be attributed to the availability of training data, advancements in semiconductor technology, and rapid evolution in DNN algorithms. However, there is also a disparity between the rate at which DNN architecture is evolving and the underlying hardware that executes DNN to satisfy the need of real-world applications. DNN execution demands high computing and power budget, however, many AI applications demand the execution on hardware with limited computation and power budget. Hence, traditional general-purpose processors are no longer able to cater to this need, which drives the research on domain-specific processors i.e., accelerators [1].

Spatial architecture [2] is a popular class of domain-specific architecture where a number of Processing Elements (PEs) are connected via a communication fabric to exploit the computing and communication parallelism. DNN accelerators typically consist of an array of PEs each configured with its own local memory, and control unit, as well as a global buffer that hides the DRAM access latency. These PEs are typically connected via the Network-on-Chip (NoC) [3] which serves a critical role in the overall performance of the DNN execution [4]. NoC enables parallelism in computation and communication by allowing multiple PE support and parallel data movement.

The dataflow model determines the processing order and where data is stored and reused, i.e., the way data (including inputs, weights, and partial sums) communication happens between the PEs and the global memory. DNN algorithms can be mapped onto the PEs using various dataflow models [5] like Output Stationary (OS), Weight Stationary (WS), Row Stationary (RS), etc. each of them has its own memory usage and energy advantage.

Authors in [6] demonstrated that more than $90\%$ of a certain class of  DNN workload execution time on a single-threaded CPU was spent on a convolution layer. Similarly, in most commonly used DNNs the multiply and accumulate (MAC) operations in CONV and FC layers consume more than $90\%$ of the total operations involved. Due to the volume of MAC operations and data (weights, inputs) involved, the DNN execution bottleneck in hardware can be broadly categorized into communication and computation.

The computation bottleneck is fairly straightforward to resolve. This involves adding or increasing the number of PEs or computation nodes. However, it is also to note that increasing computation resources will increase a fair amount of complexity in the communication infrastructure. Additionally, DNN comes in a variety of shape and sizes which include the diversity in layers, kernels, etc. In order to handle these diverse workloads, the communication needs should be reconfigurable. In this paper, we propose the In-Network Accumulation (INA) method which focuses on reducing the network load and potentially memory transactions that will help in alleviating the communication bottleneck.

The rest of the paper is organized as follows. Section II explains the related background and motivation behind this study. Section III presents the proposed architectural support for INA. Section IV presents the result of the performance evaluation and finally, Section V summarizes this paper.

Fig. 1 shows an example of the Weight Stationary (WS) dataflow model where each PE keeps the filter weight stationary in a PE while the input activations and partial sums move across the PE array. When these weights are no longer needed, they will be replaced with other weights. This setting reuses the weights and minimizes the energy consumption in reading the weights. Google’s TPU [2] is one example of a design that uses the WS dataflow model.

The dominant computing architecture today relies on memory to provide data to the PE when required. This approach of computing has hit the memory wall i.e., there is a big gap to fill in between the memory latency and CPU execution latency. To overcome this gap, near memory computing or in-memory computing [7] is considered a promising candidate. The main philosophy behind these architectures is to process the data close to where it is stored. Inspired by this philosophy, in this paper we present in-network accumulation to accelerate the DNN workload in accelerators. The fundamental principle of this architecture is to process the data in the network while routing the data.

There exist enough operations in the DNN workload that can be offloaded to the memory as near-memory computing. Authors in [8] proposed an in-memory computing architecture for neural network applications, however, this method is not suitable for CNN workloads. This architecture does not offer flexibility in data reuse which leads to frequent reloading of input feature maps and weights during the DNN execution. Similarly, authors in [9] propose a Computing-On-the-Move (COM) architecture to address the shortcomings of [8]. COM [9] is implemented on a 2D Mesh NoC to enable the inter-memory computation like psum addition.

However, the aforementioned architectures cannot be easily integrated with existing accelerator designs adopting different dataflow models. Let us assume a WS dataflow model, where filter weights are stored at local PEs and kept there until all the MAC operations involving the weights are exhausted. The major bottleneck in this approach is to keep the filter weights in the local PE where we have limited scratch memory or a register file. To address this problem, only a portion of the weights are kept in a PE, and overall multiple PEs share and store the weights of one filter. This distribution of filter weights results in the distribution of the partial sum across multiple PEs which need accumulation. A similar distribution also occurs in the RS dataflow model.

Fig. 2 shows the partial sum (psum) generation method which is explained by showing the MAC operation in different dataflow models. A weight vector consisting of $m$ elements ${W_{1},...,W_{m}}$ and an input vector consisting of $m$ elements ${I_{1},...,I_{m}}$ are streamed from the streaming bus to the PE in the OS dataflow model where the psum accumulation is performed at a single PE as shown in Fig. 2 (a). However, due to the distributed nature of weights/inputs in the WS/RS dataflow model particularly due to the memory limitation, psum accumulation needs to happen across different PEs as shown in Fig. 2 (b). Assuming that each PE can hold only $x$ elements from the weights and input vector, only a part of the psum is generated at each PE. This provides an opportunity to optimize the way psum accumulation happens in the WS/RS dataflow model.

In this paper, INA is provided as a solution to perform the psum accumulation on memory constraint models like WS/RS where only a portion of calculation happens in a PE. INA eliminates the additional network traffic due to the limited memory of a PE. The proposed method helps in reducing unnecessary movement of the psum by allowing the psum accumulation perform at the router.

The INA design modifies the router to support psum accumulation which makes it easy to integrate with the existing accelerator design. The INA is controlled by a central controller, hence the accelerator can be optimized for individual DNN layers unlike in COM [9] where the dataflow control is distributed. Fig. 3 shows an example of the WS dataflow model on a $4\times 4$ mesh. Assume that a CONV operation with a filter in $\{F_{1},F_{2},F_{3},F_{4}\}$, input activation in $\{I_{1},I_{2},...,I_{P}\}$ each with a dimension of $C\times R\times R$ is implemented on a $4\times 4$ mesh where each PE can hold half the elements of a filter. In order to generate one output activation, two PEs are required as shown in Fig. 3 where the filter weights are divided into two parts $P1,\ P2$ and distributed among two adjacent PEs in the same column. After both PEs generate a psum, they can be accumulated across the PEs to get the final output activations.

Fig. 4 shows the flow for psum accumulation with and without in-network support. Fig. 4 (a) shows the flow where - are the sequential steps that need to be performed in order to process the accumulation of the partial sum generated at each node on a WS dataflow model as shown in Fig. 1. In the absence of in-network accumulation, the incoming packet   with partial sum  is first ejected to the PE where accumulation happens locally and then the packet with the new accumulation result needs to be injected back to the network for the next hop .

Fig. 4 (b) shows the alternative approach of accumulation of the partial sum generated at each node on a WS dataflow model in the presence of an in-network accumulation unit. Steps - show the flow of partial sum accumulation in the presence of in-network accumulation support. This method of accumulation brings the computation close to the data source and helps in removing the redundant and obvious network transactions. The incoming packet can be directly accumulated in the router with the partial sum as generated by the local node before forwarding it to the next hop . One thing to notice here is ejection and injection of the packets are saved with the support of in-network accumulation which not only helps in improving the latency but also saves the network power consumption.

We assume that there is a higher level entity or a mapping framework similar to  [2], [20] that does the task of mapping neurons to the PEs, streaming the inputs and weights in parts, controlling timing for better synchronization without stalls, so that our focus is on evaluating the performance of the on-chip network.

To evaluate the performance of INA for the WS dataflow model, we ran simulations for different CNN workloads on 8x8 mesh-based NoCs modified with a two-way streaming architecture [12]. For the WS dataflow model, all the weights are streamed in full or in parts as needed depending on the memory availability, these weights are stored in the local memory of the PE and used for multiple rounds of MAC operation for all the input activations. Hence, input activation will not be streamed, and the PE should not start the MAC operation until all the weights are streamed to the local memory of the PE. In this section, we describe the experiment settings, followed by presenting the results.

DNN workloads can be executed with a variety of dataflow models. Supporting different dataflow models is important to ensure effective data reuse during the DNN execution. In this paper, we present the In-Network Accumulation (INA) architecture to support partial sum accumulation without ejecting the packet from the network on a WS dataflow model. To support the psum accumulation, an INA block enclosing a digital adder will be added to each router. We performed the simulation on different DNN workloads and showed $1.22\times$ improvement in the runtime latency and $2.16\times$ improvement in the power consumption. Further comparison with the OS dataflow model with gather support, the WS dataflow model with INA achieves up to $1.19\times$ latency improvement and $2.16\times$ improvement in the power consumption across different DNN workloads. In our future work, this method will be explored further with other dataflow models and NoC topologies.

This work is supported in part by the National Science Foundation under grant no. 1949585.