Big Memory Servers and Modern Approaches to Disk-Based Computation

In recent years, datasets of interest are growing larger in size and at a faster rate. Although we expect the capability of our processors to increase every few years, traditional memory models cannot keep up with that growth. While volatile memory such as DRAM is fast, it is limited by its capacity and cost. Large data must be placed in stable storage and retrieved when needed. Bottlenecks arise in performance-critical workloads, and therefore a new paradigm is required to handle emergence of this trend. This paper is targeted at improving High Performance Computing (HPC) and data-center applications in terms of scaling systems in affordable ways.

A Big Memory server is one with a RAM subsystem ranging from 6-24 TB, larger than that available on contemporary systems. This is a somewhat arbitrary definition, but consider that it is difficult to get even 4TB of DRAM in a modern server motherboard. Additionally, we look at the whole memory stack when discussing main memory. While we may never be able to do much off-processor improvements with L1 and L2 cache, we can likely do a lot with L3 (and L4). In modern HPC centers, we often discuss the server architectures as multicore and multi-processor. Our Big Memory techniques unequally favor multicore scale-up design improvements, since, in the perennial search to remove every source of latency in our compute stack, we remove network cards where we can.

Jun et al. [1] showed experimental evidence that, if servers executing a distributed computation have to retrieve data from disk even five percent of the time, their performance falls to a level that’s comparable with flash. Big Memory systems address this issue and deliver improvements to a range of applications.

Many performance-critical workloads can benefit from such systems. Examples may include real-time analytic, financial data analysis, and other Machine Learning applications. For instance, recommendation engine models can be hundreds of gigabytes in size and their embedding table can be even terabytes. It is important to consider scalability in production, and the model inference time can be greatly reduced if the computation can be done in-memory.

Additionally, a large RAM subsystem can be used to enumerate the possible branches on a computation tree and is useful for implementing large proof-by-exhaustion of a theorem. For example, mathematicians spent over two decades hunting the number of moves an optimal algorithm would take in the worst case to solve any configuration of the Rubik’s Cube. Before cracking the mystery, its upper-bound was reduced to 26 by the use of disk-based computation [2]. In this work, Kunkle and Cooperman enumerated and stored 4.3 x $10^{19}$ possible configurations of the cube which required terabytes of main memory. They argued that because distributed disks offer approximately the same bandwidth as a single RAM subsystem, the local disks of a computer cluster can be regarded as a single large RAM subsystem.

In Section II, we explore the possible existing solutions to handle the situations when the size of data becomes greater than memory. In Section III, we investigate performance of a disk-based computation alternative using NVMe NAND-flash SSDs. In Section IV, we take a look at the future directions of the industry to serve the Big Memory community.

Flash storage is an order of magnitude less expensive compared to DRAM in terms of costs and power consumption. We propose a software-defined memory abstraction layer with relatively cheaper NVMe NAND-flash SSDs to lower the total cost of ownership of HPC centers without sacrificing the performance drastically.

We espouse a Big Memory approach that leverages a processor agnostic approach like CXL that will be able to take advantage of NVMe-centric memory support. We anticipate that the industry will move into a more tiered approach to a multilevel memory model as flash costs decrease and PCI-e bus speeds increase to make them competitive with traditional DRAM solutions.

We thank the MIT Lincoln Laboratory Supercomputing Center team for providing reviews and edit suggestions.