HALO-CAT: A Hidden Network Processor with Activation-LOcalized CIM Architecture and Layer-Penetrative Tiling

Hardware acceleration of Neural Networks (NNs) typically follows a layer-by-layer processing approach, where each layer’s computation starts only after completing the previous one. This method, while straightforward, leads to large storage needs for intermediate results and increased latency. An alternative approach under exploration is cross-layer (CL) computing, also referred to as depth-first (DF) execution (Goetschalckx and Verhelst, 2019; Goetschalckx et al., 2023) or layer-fusion (Alwani et al., 2016; Houshmand et al., 2022). This approach initiates the computation of the next layer’s output pixel as soon as the necessary input pixels are available, rather than waiting for the entire feature map’s completion. A similar commonly-used technique is tiling (Alwani et al., 2016; Mei et al., 2022), which, in the context of CL computing, involves blockwise traversal of the feature map to reduce the maximum data size required at any given time, thereby diminishing storage requirements(Goetschalckx and Verhelst, 2019; Goetschalckx et al., 2023).

However, the limitations on achievable layer depth in existing CL computing are evident, as presented in Fig. 1. Despite being labeled as ’depth-first,’ the depth it can explore is somewhat shallow, particularly in area-constrained environments. Normally, fusing more than three CONV3x3 layers is generally inefficient, as seen in (Hirose et al., 2022) and (Houshmand et al., 2022), where the former involves only one CONV3x3 layer and the latter fuses four, but with significant overhead in activation memory access. This is primarily attributed to the following two reasons:
Data Dependency Issue: In Cross-Layer (CL) computing, the reliance on data from adjacent tiles for convolution operations increases as the depth of fused layers grows. This results in an escalation of on-chip memory demands and reduced efficiency (Li et al., 2021).
Weight Reloading Issue: Multiple fused layers in CL computing lead to an increase in weight parameters. With limited weight buffer sizes in standard ASICs, this necessitates frequent reloading of weights for different input portions, increasing energy consumption and limiting the feasible depth of fused layers.

Inspired by the spirit of CL computing, we propose Layer-Penetrative Tiling (LPT), an approach uniquely enabled by the attributes of HNNs. LPT represents a profound implementation of CL tiling, demonstrating the capability to efficiently process through more than 10 layers while significantly reducing the need for on-chip storage. By leveraging the inherent structure of HNNs (Ramanujan et al., 2020) and incorporating block convolution (Li et al., 2021), LPT effectively overcomes the challenges traditionally associated with layer depth and storage requirements.

While the Lottery Ticket Hypothesis (Frankle and Carbin, 2018) inspired researchers to bravely seek valuable parameters from the start to reduce training costs, Hidden Network (HNN) (Ramanujan et al., 2020) takes an even bolder approach by directly exploring a subnetwork on an untrained, or random-weighted network. The development of an HNN involves initializing a neural network with random weights and employing a supermask to selectively retain critical weights. This results in a subnetwork, used during inference, which combines the initial network with the supermask, as shown in Fig. 2. Despite using untrained weights, HNNs achieve accuracies on par with conventionally trained networks on complex datasets like ImageNet.

The random generation of weights in HNNs, achieved using simple hardware circuits, eliminates the need for energy-intensive data transfers from DRAM, allowing for ’almost-free’ weight usage. Moreover, the supermask used in HNNs, being sparse and binary, can be conveniently stored on-chip. These attributes significantly alleviate the bottleneck typically associated with off-chip weight access and facilitate a more effective implementation of LPT, especially considering the reduced costs associated with weight reloading.

As illustrated in Fig. 1, we propose Layer Penetrative Tiling (LPT), an extreme case of CL computation for fortified depth exploration, targeting significantly reduced storage requirement and energy consumption particularly for HNN. The LPT process starts with a single tile of input and penetrates through multiple layers (more than 10) in depth. LPT ensures that computation for the next tile commences only after the entire computation of the current tile is completed. Recall that traditional CL computation was constrained to just two to three layers due to the Weight Reloading Issue, since all weights spanning across layers needed to be stored on-chip for cost-effective inference. LPT, however, effectively overcomes this challenge. With our HNN-based accelerator design, weight data is generated directly on-chip, enabling almost ’free’ transmission of these weights. This advancement paves the way for a more aggressive approach in deepening the integration of layers, which leads to a marked improvement in energy efficiency.

Another challenge in conventional CL computation is the Data Dependency Issue, which arises because convolution operations require activation data from neighboring tiles, leading to a substantial amount of activation data being written to and retrieved from buffer memory, potentially undermining the efficiency gains achieved by LPT. To address this, we introduce Blocked HNN with Tile Concatenation. Blocked HNN incorporates block convolution(Li et al., 2021) into HNN’s CONV layers. As illustrated in Fig. 2, which divides the input activations into multiple small tiles, each separated by padding, to ensure that convolution within each tile is an isolated operation. This independence eliminates the need for data from neighboring tiles and confines the data dependency within the boundaries of each tile.

One limitation of the conventional blocked convolution, however, is the restriction on the maximum fused depth imposed by the tile size. When encountering a pooling layer or a CONV layer with a stride greater than 1, the tile size reduces, which could lead to distorted computation results. For instance, applying a CONV3x3 on a 2x2 tile result in substantial accuracy degradation. In the strategy outlined in (Li et al., 2021), small tile sizes are addressed by switching to standard convolution; however, this approach falls short in achieving the desired memory reduction in hardware.

To address this issue, we introduce the Tile Concatenation (TC) technique. This method is applied after processing up to a certain layer, denoted as layer K, and the output from layer K is temporarily stored in additional memory. Then, the computation of an adjacent tile is restarted from layer 1 and continues up to layer K. Once this is complete, the previously stored output tile is retrieved and concatenated with the current output, effectively doubling the tile size. This larger, concatenated tile is then used for further computations beginning from layer K+1. TC thus provides a practical solution to the tile size limitation in LPT, allowing for accurate and efficient processing even in scenarios where layer characteristics would otherwise lead to reduced tile sizes and potential computation inaccuracies.

As a result, our implementation of LPT with blocked HNN enables the end-to-end inference of ResNet50@ImageNet with a mere 72KB of activation memory. This signifies a 14.2x decrease on activation storage, which is elaborated in the following section.

In response to this challenge, leveraging (1) the unique advantages of HNNs where the data movement of weights is almost ’free’ and (2) the reduced storage requirement enabled by LPT, we shift our focus towards minimizing activation movement. To this end, we introduce an Activation-Localized (AL) CIM dataflow, as shown in Fig. 3. This approach marks a distinct departure from the existing activation-stationary (AS) CIM structure (Kim et al., 2021; Ju et al., 2023), which only maintain activation stationary during computation. Our AL CIM dataflow extends this concept by preserving the locality of activations between the output of one layer and the input of the next. This is achieved by employing the CIM core as both the computation core and output memory interchangeably, allowing direct initiation of the next layer’s computation with the output results stored in the CIM core. Such dual functionality of CIM core eliminates the need for additional transportation from the output memory back to the computation core, a requirement in the conventional AS dataflow, thereby effectively circumventing the dominant issue of communication power cost.

The architecture of HALO-CAT, an activation-Localized (AL) CIM HNN processor, is depicted in Fig. 4, which features a meticulously designed for efficient HNN computation. The HALO-CAT key components and functionalities are as follows:
CIM Cores: Central to the processor are three 16KB CIM cores, each dedicated to AL computation. The cores are dynamically assigned roles; one serves as the input memory (iCIM) and another as the output memory (oCIM), facilitating a streamlined flow of activation data.
Near-Memory Pipeline (NMP): Located proximately to the memory units, the NMP handles post-processing tasks. This strategic placement significantly reduces the distance data must travel, enhancing processing speed and energy efficiency.
HNN Weight Management: The random weight generator (WGEN) is responsible for creating weights, which are subsequently masked in the weight buffer (WBUF). This masking is guided by the supermask data stored in the superMask memory (MMEM), ensuring precise weight application.
Operational Flow: In operation, weights generated by WGEN are routed to iCIM for MAC operations. Post-MAC, the data undergoes further processing in the NMP before being transferred to oCIM for storage.
Layer Execution and Activation Localization: To maintain activation locality, the roles of iCIM and oCIM are interchangeably assigned for each layer’s execution, allowing seamless data transition between layers.
Support for Residual Connections: The trio of CIM cores is specifically chosen to effectively support residual connections, a common feature in modern neural network architectures.
Tile Concatenation (TC) Mechanism: To facilitate TC (as discussed in Section 3), TMEM, a 24KB Tile memory, temporarily stores intermediate tiles. These tiles are then concatenated within the CIM core, optimizing the computation process.
Memory Usage Efficiency: Remarkably, the combined memory footprint, including the three CIM cores and TMEM, totals just 72KB, which is 14.2$\times$ less than Hiddenite. This demonstrates the processor’s ability to manage complex computations within a compact and energy-efficient memory allocation.

The structure and activation mapping of HALO-CAT CIM core is illustrated in Fig. 5. The core is organized into eight columns of CIM groups, corresponding to the tile width $W$. Each group consists of a vertically aligned compressor and eight rows of macros, mapping to the activation precision $P$. These macros consist of 128 clusters and local computing units (LCUs), corresponding to the channel depth $C$, with each cluster containing 16 rows of 6T SRAM cells, unfolding to the tile height $H$. In addition, the core accommodates shallow channel depth by storing multiple pixels in one macro row and adjusts to deep channel depth by storing one pixel across several macro rows.

Every operation cycle activates one out of 16 SRAM rows, sending its data to the LCU for CIM computations across 128 SRAM columns. The resulting CIM outputs are subsequently distributed to the compressor and aggregated among 8 macros to generate MAC outputs for all 8 CIM groups in parallel. Lastly, these 8 channels of MAC outputs are transmitted to the NMP for convolution summation and post-processing. The execution of $H$ is bit-serial, while $C$, $P$, and $W$ are processed in a bit-parallel manner, optimizing computational efficiency.

Our AL CIM operational principle is ideally suited for CONV operations, eliminating unnecessary data accesses. For $K\times K$ CONV, the weight is partitioned into $K^{2}$ pixels, each sequentially processed in iCIM. This is illustrated in Fig. 6, where a 1x3 weight interacts with a 1x8 input tile, producing an array of partial products (PPs). These PPs are conditionally shifted and accumulated in the NMP (Fig.6), with MAC shifters facilitating efficient data manipulation. A standard CONV3x3 operation, for instance, involves repeating this process thrice with varied operand rows. Post-processing tasks like scaling, ReLU, and quantization are then executed in the NMP, before storing the results back to oCIM.

Contrary to most WS CIM architectures (Zhang et al., 2023; Houshmand et al., 2022), which require additional input buffers for activation reuse and often struggle to maintain high utilization rates for different kernel sizes, our dataflow ensures optimal utilization for any kernel size and negates the need for extra buffering.

Fig. 7 illustrates HALO-CAT’s CIM circuit, comprising an SRAM Cluster with 16 bitcells, alongside a local computing unit (LCU). In pursuit of a more compact CIM core, we implement intra-macro A/D conversion method for our analog-domain CIM (ACIM). This approach negates the necessity for a large capacitor array in conventional SAR-ADCs. We repurpose the CIM compute capacitor for two functions: charge sharing and DAC reference voltage generation. During the CIM operation phase, activation is fetched using LBL, while weight and DAC signals are dispatched through GBL and GBLB. By selectively switching between $IN*W$ and DAC signal, we can efficiently derive both the MAC output and reference voltage using the same capacitors. While similar A/D conversion scheme has been employed for CIM with bitcell-wise computation in previous works (Yoshioka, 2023; Yao et al., 2023), we are the first to integrate it into CIM with cluster-wise computation, resulting in an even more compact area density and efficient design.