Shared-PIM: Enabling Concurrent Computation
and Data Flow for Faster Processing-in-DRAM

A modern DRAM organization typically involves multiple DRAM arrays organized in a hierarchical structure to optimize performance and density. A DRAM cell consists of a capacitor that stores the logical state in the cell, and an access transistor. The DRAM cell is connected to a sense amplifier, which detects and amplifies the logical state of the cell, resulting in a steady voltage level. Bitlines link the DRAM cells in one column to their respective sense amplifiers, whereas the wordlines control the cell’s access transistor.

A standard read operation in a DRAM cell involves three stages: ⓪ Activate, ① Charge Sharing, and ② Sensing (Fig.1). Initially, the bitlines ($BLs$) are pre-charged to a $\frac{1}{2}V_{dd}$ state. After a read operation is initiated, an ACTIVATE signal is sent to the DRAM bank along with the source row address. The row decoders then enable the wordline ($WL$) corresponding to the address, thereby initiating the Activate phase ⓪. Once the $WL$ is enabled, the capacitor ($C_{1}$) in the DRAM cell is connected to the $BL$, and the Charge Sharing phase ① begins, which transfers charge between $C_{1}$ and $BL$. The charge on the $BL$ results in $\frac{1}{2}V_{dd}+\Delta$ , where $\Delta$ can be positive or negative depending on whether the bit stored in $C_{1}$ is a ‘Logic 1’ or a ‘Logic 0’, respectively. When charge sharing is completed, the Sensing phase ② begins. Here, the sense amplifier is activated, which drives the $BL$ to a stable state based on the value of $\Delta$: When $\Delta$ is positive, the $BL$ is driven to $V_{dd}$, while the charge on $\overline{BL}$ goes to 0V. The charge on the $BL$ is reversed when $\Delta$ is negative [14]. After the read operation is concluded and the $WL$ is deactivated, a PRECHARGE signal is sent to the bank to bring the bitlines back to the equilibrium state $(\frac{1}{2}V_{dd})$.

Fig. 2 depicts the standard DRAM architecture along with modifications required for our proposed Shared-PIM highlighted in red (these modifications will be discussed in Sec. III). In DRAM, memory is structured hierarchically: (1) Ranks are logical or physical groupings of DRAM modules on a memory channel; (2) each rank contains multiple DRAM chips, each of which is further divided into banks; (3) banks are then divided into smaller subarrays per Fig. 2(a); (4) within each subarray, we reach the finest level of granularity – i.e., the tiles, which contain individual memory cells per Fig. 2(b). Subarrays within a bank operate largely independently of each other and can be activated per [15] in parallel. Banks within a DRAM chip share a global row buffer, which supports I/O operations. Previous work [11] has demonstrated how the global row buffers can be used for inter-bank data transfer, as well as inter-subarray transfer by first copying the data to a temporary bank. However, as the global row buffer has a lower bit width compared to the bank row size, data must be copied serially/in smaller segments, thereby creating a bottleneck concerning inter-subarray data movement (as indicated by Rowclone’s [11] PSM mode). Hence, no structure exists to facilitate data movement between subarrays in contemporary DRAMs, which negatively impacts PIM designs (i.e., AMBIT, pLUTo) that leverage standard DRAM organization.

pLUTo [3] is a SOTA in-DRAM PIM architecture that leverages the high density afforded by DRAM to store large LUTs, together with newly added logic to perform in-DRAM LUT-based computation. In addition to the match logic, the pLUTo-BSA architecture extends the structure of standard DRAM by adding (a) a matchline and a transistor to every bitline in the subarray, along with (b) a buffer to store match results. When evaluating cryptographic and neural network compute workloads, pLUTo could offer speedups of 713$\times$ and energy savings of 1855$\times$ when compared to a CPU. pLUTo also outperforms other DRAM PIM designs by an average of 18.3$\times$, owing to its ability to handle parallel lookup queries directly in DRAM (i.e., pLUTo can efficiently manage complex operations while reducing data transfer demands).

pLUTo requires extensive data transfers between subarrays during computations, which are managed by the LISA framework [10]. pLUTo encounters two distinct data transfer overhead scenarios during computations: one arises from the need to transfer data between LUTs across subarrays due to size limitations, and the other stems from the data transfer required between operations within applications, such as in matrix multiplication. The first type of overhead arises when performing complex operations, such as a 32-bit addition that cannot be contained within a subarray and instead requires the coordination of multiple 4-bit LUTs to support said addition that are distributed across subarrays. This leads to processing delays as data must be moved to the appropriate subarray. The second type of overhead occurs when computational steps are interdependent, necessitating a complete data transfer from previous operations before subsequent ones can proceed. This requirement for sequential data handling, especially in application like matrix multiplication where multiplications and additions are depend on each other’s outputs. For example, in matrix multiplication, each product of matrix element pairs needs to be aggregated to compute the final matrix elements, resulting in data transfers accounting for approximately 60% of the total operations involved in collecting and summing these results. The pLUTo architecture, utilizing the LISA framework, lacks the capability for concurrent computing and data transfer, resulting in significant system latency as operations must wait for previous data transfers to complete before proceeding.

We briefly review existing strategies to mitigate the bottlenecks associated with data transfer in PIM architectures, specifically Rowclone [11] and LISA [10].

We propose Shared-PIM to reduce the latency of inter-subarray data movement in DRAM-based PIM solutions, aiming to efficiently manage unavoidable data dependencies. Our approach is rooted in the observation that when a subarray is active, its bitlines and sensing amplifiers are engaged. As a result, concurrent computation and data transfer between active subarrays is not possible without the presence of a separate set of bitlines/sense amplifiers to support such data movement. In the following, we discuss the physical structure of Shared-PIM and its implications to the in-DRAM PIM paradigm.

Here, we explain the setup used to assess Shared-PIM’s functionality at the circuit-level, as well as to evaluate its performance at the application-level.

Fig. 5 depicts the waveforms from our SPICE simulations of Shared-PIM to simultaneously copy one source row to four destination rows in different subarrays over the BK-bus (i.e., a “broadcasting” operation). Once the source row is activated, the charge sharing stage between the cells and the BK-bus begins. At the beginning of the operation we observe the voltage of the source cell (blue line) declining, while the voltage on the BK-bus bitline (red line) rises as they share charge. Once the cell and the bitline reach equilibrium, the BK-SAs start sensing and amplifying the voltage on the bitline. We then activate the destination row(s) $4ns$ later, which causes the voltage drop at $854ns$, while the voltage in destination cell (green line) rises. Similarly, after the charge sharing stage is finished, the BK-SAs begin amplifying the voltage on the bitline until the data is restored into the source and destination rows, which concludes the data transfer operation. Although four destination rows is not an absolute limit for the broadcast operation, and our simulations show that broadcasting to five or even six destination rows is possible, we keep the maximum number of destination rows to four to ensure it stays within the standard DDR timing limits.

Table II reports the inter-subarray latency and energy for copying one row (8KB of data) from one subarray to another within the same bank using four different approaches: (1) memcpy, which uses the memory channel, (2) Rowclone (RC-InterSA), (3) LISA, and (4) our proposed Shared-PIM (with two shared rows per subarray).

With respect to inter-subarray copy latency, we further optimize Shared-PIM’s copy latency by overlapping the two ACTIVATE commands with only a 4ns delay between them as proposed in [8], resulting in a total copy latency of $52.75ns$. Fig. 6 shows the command timeline and latency improvement of Shared-PIM with respect to RC-InterSA [11] and LISA [10].

The energy consumption reported in Table II was obtained with the method described in IV-A1, i.e., by calculating the power usage of the DRAM commands, and then multiplying this by the latency of a copy operation via Shared-PIM. Notably, the addition of two shared rows per subarray does not significantly affect the subarray’s energy, since it only adds two transistors per bitline (16K transistors per subarray, in total). Meanwhile, the addition of the BK-SAs that enable Shared-PIM’s data transfer raises the power consumption per copy operation. However, because the copy process is performed on the BK-bus with improved latency, the overall energy dissipation per copy operation is minimized compared to Rowclone [11] and the conventional memcpy. When comparing against LISA [10], it is apparent that Shared-PIM’s energy savings (1.2$\times$) are not as good as the latency improvements (5$\times$). This can be attributed to Shared-PIM’s additional BK-SAs, since the copy operation over the bus activates the entire bus which consists of four rows of sense amplifiers as explained in Sec. III, i.e., Shared-PIM’s copy operation activates four times the number of sense amplifiers that LISA does, which results in a trade off of power dissipation for latency improvements.

Finally, Table III shows the area breakdown of the base DRAM, the pLUTO architecture (using the BSA mode, which is pLUTo’s more compact implementation), and Shared-PIM (combined with pLUTo). We estimate a 7.16% overhead to pLUTO’s chip area with Shared-PIM, which is a result of the addition of BK-bus, BK-SAs, the GWL transistors (for shared cells), and GWL drivers. To further reduce the physical implementation complexity and the area overhead of Shared-PIM, the BK-bus lines could be implemented using a different metal layer than that of the regular bitlines.

Here, we evaluate how data transfer in two architectures influences application-level performance. The architectures considered are pLUTo combined with LISA [3], and a scenario where pLUTo is integrated with our proposed Shared-PIM. We carry out our evaluation on two different fronts: (1) with operations of varied number of bits, and (2) by running five application benchmarks.

For varied number of bits, we study pLUTo’s performance for addition and multiplication operations. Given the size constraints of the LUTs in pLUTo, a single subarray cannot accommodate the entire LUT required for computations with higher number of bits. As a result, these computations are distributed across multiple subarrays. This distribution necessitates inter-subarray data transfers when executing a single operation, which can impact performance. In our experiments, we consider that a single pLUTo subarray can efficiently perform a 4-bit addition and 4-bit multiplication as reported in [3]. We then test pLUTo’s performance across varying bit-widths: 16, 32, 64 and 128 bits for both addition and multiplication. The inclusion of 128-bit operations demonstrates the advantages of using Shared-PIM in pLUTo for applications requiring high-precision computing and cryptography. Throughout these experiments, we assume that pLUTo executes these computations with maximum parallelism.

Fig. 7 compares the latency of pLUTo paired with LISA and pLUTo combined with Shared-PIM, for addition and multiplication, with varied number of bits. The benefits of Shared-PIM become increasingly apparent in computations involving operations with higher number of bits, as they require more data movement.

The speedup of Shared-PIM for addition operations can be attributed to its capability to execute all the 4-bit additions simultaneously. After these parallel operations, the results are forwarded to a subarray for final aggregation via the BK-bus. Shared-PIM enables efficient data transfer, allowing it to achieve a 40% performance improvement in 128-bit addition compared to LISA.

On the other hand, the scenario for multiplication is more challenging. Data dependencies in multiplication, especially for operations with higher number of bits, make it more challenging to achieve the same level of parallelism as in addition. This complexity stems from the combination of 4-bit multiplication, addition, and shifting. However, Shared-PIM offers significant advantages by enabling simultaneous computation and data transfer. For instance, while intermediate multiplication results are being transferred for final aggregation, Shared-PIM allows the next layer of multiplication and shifting operations to proceed immediately. Shared-PIM paired with pLUTo leads to a 40% improvement in performance for 128-bit multiplication compared to pLUTo combined with LISA.

For the study with application benchmarks, we explored the performance impact of our Shared-PIM design on pLUTo across five programs, which were selected for their varying levels of data dependencies. This selection allows us to compare and contrast the performance benefits of using the Shared-PIM architecture across different dependency scenarios. The benchmark programs are:

1. 1.

   Number Theoretic Transform (NTT): The NTT transforms polynomials into their modular equivalents in a finite field, functioning similarly to the Fast Fourier Transform (FFT). We evaluate the NTT using polynomial degree of 300.

2. 2.

   Breadth-First Search (BFS): The BFS application analyzes a 1000-node densely connected graph, where each node links to every other node, marking the highest amount of data dependencies among all of the selected programs.

3. 3.

   Depth-First Search (DFS): The DFS is similar to BFS, being a worst-case DFS that needs to visit each node and explore its neighbors.

4. 4.

   Polynomial Multiplication (PMM): Polynomial multiplication is the process of multiplying two polynomials to obtain their product. We use polynomials with a degree of 300. We employ a naive approach to polynomial multiplication (not using NTT to optimize).

5. 5.

   Matrix Multiplication (MM): Two matrices with a size of 200 $\times$ 200 multiply each other. MM has a high amount of data dependencies and requires significant data transfer between the elements of the matrices.

All the computations in these benchmark programs use 32-bit operations, and the latency and energy consumption are derived from previous tests for pLUTo with LISA and Shared-PIM architectures. We assume that both pLUTo with LISA and pLUTo with Shared-PIM can run these applications under full parallelism, which in this context means that an ideal number of computing arrays are available for these benchmark programs.

Fig. 8 presents the latency and energy consumption metrics for the benchmarks executed under pLUTo with LISA and the Shared-PIM architectures. The latency comparison reveals that Shared-PIM confers a substantial speed advantage in all tasks, with a 40% latency reduction in MM, 44% in PMM, and 31% in NTT. Shared-PIM enhances performance by enabling concurrent computation and data transfer within pLUTo subarrays, which facilitates the pipelining of data transfer with computation processes. Additionally, as shown in Fig.7, pairing Shared-PIM with pLUTo achieves an 18% speedup in 32-bit addition and a 31% speedup in 32-bit multiplication compared to the pLUTo with LISA setup, significantly boosting performance for individual operations.

The speedup for MM and PMM are particularly significant, as they are attributed to similar degrees of parallelism and computational steps. Compared to NTT, MM and PMM demonstrate greater speedup. This is primarily due to relative lower data dependencies in MM and PMM compared to NTT, allowing Shared-PIM to more effectively parallelize data transfer and computation in MM and PMM (as discussed in Sec. III-B). In our analysis, BFS and DFS show equal performance in worst-case scenarios on the same graph, as they follow identical processes.

With respect to energy consumption, a clear difference emerges despite the parallel capabilities of both designs. The Shared-PIM architecture proves to be more energy-efficient in data transfers than LISA, as indicated by the consistent reduction in energy usage. As a result, Shared-PIM achieves an average of 18% energy savings in data transfers across all benchmarks.