ONNX-to-Hardware Design Flow for the Generation of Adaptive Neural-Network Accelerators on FPGAs

Cyber-Physical System (CPS) integrate “computation with physical processes whose behavior is defined by both the computational (digital and other forms) and the physical parts of the system”¹¹1https://csrc.nist.gov/glossary/term/cyber physical systems. They are characterized by a considerable information exchange with the environment and by dynamic and reactive behaviors with respect to environmental changes. In modern systems, CPS or not, decisions making can be brought directly at the edge on small embedded platforms exploiting the capabilities of the NNs. This calls for real-time, low-energy execution, which can be achieved by leveraging different resources on the same platforms, making heterogeneous computing fundamental.

Field Programmable Gate Arrays devices can guarantee hardware acceleration, execution flexibility, and energy efficiency, as well as heterogeneity, and that is why this type of device is a valuable option for NNs inference at the edge [17]. Indeed, their integration on heterogenous Multi-Processor System on Chip opened up a wide range of possibilities exploiting the FPGA potentials along with the CPU capabilities of managing control flow and communication.

While many solutions exist to deploy AI models on these platforms, they often lack full support for advanced features, such as flexibility. This paper focuses on putting the basis to achieve future runtime adaptivity, which is key in our opinion for addressing the dynamic and reactive nature of CPS. Designing and deploying reconfigurable accelerators with these functionalities still needs to be investigated, requiring an in-depth knowledge of the underlying hardware and hand-tailored solutions. This belief is at the basis of this study that, starting from a state-of-the-art framework [1], intends to ultimately seek automated strategies to deploy lightweight, flexible, and adaptive NN accelerators for CPS, achieving different Pareto optimal working points that could be merged into an adaptive accelerator and exploited at runtime to serve variable and evolvable applications.

Several software libraries and frameworks have been developed to facilitate the development and high-performance execution of CNNs. Tools such as Caffe²²2https://caffe.berkeleyvision.org/, CoreML³³3https://developer.apple.com/documentation/coreml, PyTorch⁴⁴4https://pytorch.org/, Theano⁵⁵5https://github.com/Theano and TensorFlow⁶⁶6https://www.tensorflow.org/ aim to increase the productivity of CNN developers by providing high-level APIs to simplifying data pipeline development. Such environments are currently flanked High Level Synthesis (HLS) that are used to generate FPGA-based hardware designs from a high level of abstraction, to fasten porting of complex algorithms at the edge. Examples of HLS environments are AMD’s Vitis HLS, Intel FPGA OpenCL SDK, Maxeler’s MaxCompiler [9], and LegUp [7]. They employ commonly used programming languages such as C, C++, OpenCL, and Java, to fill the gap between software-defined applications and their hardware implementation.

To execute NN at the edge, three main types of architectures can be found in literature [2]: the Single Computational Engine architecture, based on a single computation engine, typically in the form of a systolic array of processing elements or a matrix multiplication unit, that executes the CNN layers sequentially [11]; Vector Processor architecture, with instructions specific for accelerating operations related to convolutions [13]; the Streaming architecture consists of one distinct hardware block for each layer of the target CNN, where each block is optimized separately [3, 4]. In our studies, we focus mainly on the latter.

The main innovation of the design flow proposed in [1] is the possibility of supporting runtime adaptivity in the hardware accelerator through reconfiguration.

To assess the re-engineered flow described in Section III-B, a wide exploration targeting the MNIST classifier has been carried out, as described in Table II. The intent was also to show the impact of quantization on both model accuracy and hardware performance, which is generally in line with the expectations that AC can offer, as discussed in Section II-B. It can be noticed that accuracy is not as affected by reducing parameter precision as it is by reducing activations precision. Moreover, reduced parameter precision leads to a reduced memory footprint (BRAM column) and a high percentage of zero weights. This latter can be exploited to combine quantization with pruning, which skips multiplications by zero. To have a fair comparison with state-of-the-art solutions presented in Table I, onboard-running experiments that consider also memory accesses are needed. However, we can see that the preliminary results show competitive performance in terms of utilized resources and latency/throughput. A broader comparison against state-of-the-art, based on significant onboard measurements and targeting more complex datasets, will be carried out in the future. Nonetheless, it is worth recalling that state-of-the-art approaches are not conceived to support runtime adaptivity, which is motivating our research instead.

Indeed, our future work intends to explore mixed precision in adaptive NN accelerators. To have available a fully automated flow, with reconfiguration support capabilities, was a key preliminary step to save the manual effort in the accelerator definition and exploration. The analysis carried out so far on non-reconfigurable accelerators shows that promising trade-offs are present, e.g. trading off accuracy for reduced energy consumption. The ultimate goal will be the efficient runtime management of the system that implies, as a first step, the combination of the different working points over a reconfigurable substrate. This latter can certainly be achieved by leveraging on the whole set of functionality offered by the MDC tool to design and operate reconfigurable and evolvable NN accelerators for CPS, including the one presented in this study. Resulting accelerators will be able to switch configuration at runtime to adapt to the desired goal, e.g. when a limited energy budget is left a reduction in energy consumption is worth the cost of some accuracy loss.

One of the challenges we expect to face in this research’s future steps is the limited onboard memory, which could constrain us to the execution of relatively small models (e.g. TinyML), especially when runtime switching among algorithms/configurations is required. The adoption of a reconfigurable approach, capable of sharing weights among configurations, should help us tackle that issue limiting the impact on the memory footprint of having more than one network available. This may limit the advantages in terms of accuracy achievable with Quantization Aware Training [18]. However, the preliminary results with Post Training Quantization show a limited drop in accuracy even with 4-bit weights.

The authors would like to thank Stefano Esposito for his contribution to this work during his Master’s thesis.