A Real Time Super Resolution Accelerator with Tilted Layer Fusion

The adopted model [7] has seven layers. The first six layers are 3x3 convolutions with ReLU activation, and the final layer is a 3x3 convolution followed by a residual-like structure. Based on this model, we propose its hardware architecture as shown in Fig. 3. The proposed system architecture consists of 28 PE blocks, 2-stage pipelined accumulator, activation block, ping-pong buffer, overlap buffer, weight SRAM, bias SRAM, and residual SRAM. This design accesses 8-bit input images, weights, and biases from off-chip DRAM and then stores them to the corresponding SRAM buffers.

The whole processing is as follows. First, to process a layer, one of the ping-pong buffers will serve as an input provider to supply input to 28 PE blocks for simultaneously processing 28 channels of input. The output of 28 PE blocks will be sent to the 2-stage pipelined accumulator to complete the convolutions. The results will then go through the activation block for ReLU activation function. Then, the results will be stored in the other ping-pong buffer. Second, to process the next layer, the previous output buffer will serve as input, and the previous input buffer will serve as output. The whole processing is repeated. With this, all intermediate data will be within the chip and save external DRAM access.

This design has 28 PE blocks, where each PE block consists of three PE arrays, and a PE array has 5x3 MACs. To ease the arrangement of these MACs, we adopt the input broadcasting as shown in Fig. 4(a), where seven input images are broadcasted horizontally, and three weights are broadcasted vertically. These two are multiplied and then added up along the diagonal direction to generate five partial sums for the accumulators.

In the adopted model [7], all the intermediate layers have 28 channels except for the first layer(3 channels) and the final layer(27 channels). By splitting MACs into 28 PE blocks, and letting one PE block process a channel of input at a time, our design can reach an average of $87\%$ hardware utilization with little control overhead. The three PE arrays within one PE block can finish a 3x3 convolution in a cycle, which will be further discussed later.

Fig. 4(b) shows the accumulator, which is two-stage pipelined to shorten the critical path. Three partial sums from three PE arrays of the same PE(channel) are added up first, and then the 28 output channels from 28 PE blocks are summed up with a tree adder divided into two partial tree adders. In the second stage of the 2-stage pipelined accumulator, a multiplexer selects whether biases or residuals should be added depending on the current working layer.

Fig. 6(a) shows the data flow of a PE block process example from Fig. 5. In our data flow, a column of an input image is broadcasted horizontally while a column of filter weights is broadcasted vertically. With this data flow, the products to be summed up together to complete a convolution are along the diagonal direction. Hence, we use parallelogramical arranged PEs. This design can reduce the control overhead with the expense of allowing two more input images to the PE array.

This vectorwise scheduling is illustrated by the mathematical equations in Fig. 6(b). By sending three consecutive columns of input images and filter weights to three PE arrays, the convolution of one column of output is completed within a cycle. In this case, the example in Fig. 5 only takes three cycles to be completed. To sum up, this data flow achieves high parallelism, simple control, regular structure, and high hardware utilization.

The ping pong buffer stores a tile of data for each layer. When it starts computing a new tile, the system loads the input image data from the external DRAM to the left ping pong buffer. Then, the data are consecutively loaded from the left ping pong buffer to PEs for computation and then stored in the right ping pong buffer after finishing all operations of the current layer. After that, the role of two ping pong buffers is switched to the next layer of computing. This process is repeated layer by layer until finishing the output layer.

The overlap buffer is a queue-style addressing memory for left and right boundary data. For easy addressing, the current computing layer is regarded as the back layer of the queue, and the last layer is the front layer of the queue. When computing, the data required by the first two columns of the tile are loaded from the front layer, and the results of the last two columns are stored in the back layer. After finishing the computation of a layer, it pops the front layer and the next layer becomes the new back of the queue.

We implement the queue data structure on the overlap buffer by saving the address of the front layer. This addressing helps us calculate the corresponding address of each pixel in both front and back layers in the queue.

The buffers in this design serve two purposes. One is for storing the values of feature maps and residuals; the other is for model weights. The following analysis will focus on improving the former since the latter depends on the adopted model. This analysis assumes that the classical layer fusion [15] uses a 60x60 tile, which is similar to the one used in  [15] and has the same tile height as our 8x60 tile size.

In the following, $M_{p}$, $M_{o}$, and $M_{r}$ represent the memory size of ping-pong buffer, overlap buffer, and residual buffer, respectively. $R$ is the number of rows of a tile (length of a tile), $C$ is the number of columns of a tile (width of a tile), $L$ represents the number of layers, and $Ch_{i}$ represents the number of channels of layer $i$; for example, $Ch_{0}$ represents the number of channels of the input layer.

Table I shows the implementation result of the proposed design and the comparison with other designs. This design is synthesized with Synopsys Design Compiler under the TSMC 40nm CMOS process. It achieves 60 fps for x3 scale FHD image generation while running at the 600 MHz clock frequency. The total chip area only occupies 3.11 $mm^{2}$ with 102 KB on-chip SRAM.

Compared to designs without layer fusion, [11] and [12], the required memory bandwidth is significantly reduced. The amount of off-chip DRAM access of this design is reduced from 5.03 GB/sec to 0.41 GB/sec, a total reduction of $92\%$. Thus, even DDR2 DRAM can work well with this design. Compared to designs with layer fusion [13], this design has a smaller on-chip SRAM due to the adoption of the tilted layer fusion and has a lower area cost.