DisaggRec: Architecting Disaggregated Systems for Large-Scale Personalized Recommendation

A scale-up strategy aims to equip a single server node with sufficient resources to serve end-to-end model inference. Figure 3(a) shows the end-to-end model inference configuration on a scale-up dual-socket (SU-2S) server. Two inference tasks are launched in parallel on the two processors. The two front-end network interface cards (NICs) receive incoming queries and return prediction outcomes. Given the dynamic query arrival pattern and the configured batch size, a large query is split into multiple sub-batches and multiple small queries are fused into one large batch. These batches are queued and wait for execution. The inference task exploits three types of parallelism.

Model Parallelism. The model’s computation graph $G_{M}$ is partitioned into three sub-graphs, preprocessing $G_{P}$, SparseNet with embedding tables $G_{S}$, and DenseNet $G_{D}$. Sub-graphs are launched concurrently and pipelined. $G_{P}$ and $G_{S}$ co-locate on the CPU’s 40 cores while $G_{D}$ runs on the GPUs within a SU-2S server.

Operator Parallelism. Physical cores are statically assigned to one single thread where independent operators inside the computation graph can be executed in parallel. On a SU-2S server, we assign 20 CPU cores to the preprocessing thread and 20 to the SparseNet thread such that $G_{P}$’s random hash operators and $G_{S}$’s embedding operators are executed in parallel on their assigned CPU cores.

Data Parallelism. Query batches are executed in parallel across the GPUs. Each GPU launches a replica of DenseNet’s threads. On a SU-2S, a processor socket is connected to four A100 GPUs and four $G_{D}$ threads serve batches of intermediate results from $G_{S}$.

We observe degraded performance due to non-uniform memory accesses (NUMA) when SparseNet exceeds the memory capacity of a single socket. As embeddings occupy both memory nodes, the two SparseNet threads on the two processors route half of its accesses to local memory and the other half to remote memory via processor socket interconnect—Intel’s Ultra Path Interconnect (UPI)—leading to unbalanced memory bandwidth utilization. Inter-socket bandwidth ($\sim$55 GB/s at peak) is much lower than local memory bandwidth ($\sim$145 GB/s at peak).

Following the scale-out strategy, the model’s SparseNet is sharded and distributed across multiple servers when the embedding tables cannot fit into a single server’s memory. The model’s computational graph $G_{M}$ is partitioned into three sub-graphs for preprocessing ($G_{P}$), SparseNet shards ($G_{S}.Shard.0$, $G_{S}.Shard.1$), and DenseNet ($G_{D}$).

Figure 3(b) shows the distributed inference [35] across two scale-out single-socket (SO-1S) servers. Each server launches two processes, the primary inference task and one SparseNet shard. The primary task receives incoming queries, performs preprocessing, routes SparseNet’s embedding lookups to the appropriate shards, receives the final summation (Fsum) back from the remote shards, and then executes DenseNet to get the final prediction. The embedding operations in the primary task are issued by remote procedure calls (RPCs) using pre-stored destination metadata (e.g., model ID, SparseNet shard ID). Embedding operations in one SparseNet shard is packed in one packet which is routed to that shard’s serving process hosting targeted SparseNet partitions. The dedicated back-end network connects servers’ back-end NICs using RDMAs, permitting efficient communication between servers.

To compare the scaling up and scaling out, we perform the end-to-end model inference on a scale-up dual-socket server (SU-2S) with 8 GPUs, and distributed inference on a group of scale-out single-socket servers (SO-1S) with 4 GPUs.

Latency. Figure 4 shows the lifecycle of an incoming query, queuing delay and model inference pipelines. We find the model inference on the SU-2S server suffers from unbalanced local-remote sockets’ memory bandwidth utilization. In Figure 4(b), half of the memory accesses are routed to the local socket’s memory (93 GB/s at peak), and the other half of memory accesses are routed to the remote socket’s memory (only 52 GB/s at peak) that is bounded by UPI links.

To eliminate memory accesses through UPI interconnect, we implement NUMA-aware inference on the SU-2S adopting the SparseNet sharding scheme to perform memory operations inside local sockets. Similar to distributed inference, this optimization shards SparseNet $G_{s}$ into $G_{s}.Shard.0$ and $G_{s}.Shard.1$. Two shards are separately launched on the two processor sockets in parallel. All memory accesses are routed to the processor socket’s local memory. The embedding reduction is performed inside SparseNet shards before sending them to the remote socket. In Figure 4(a), eliminating NUMA’s effect within the SU-2S server reduces SparseNet execution time by more than 60%. Moreover, the communication overheads are minimal ($<$8%) that only required for the embeddings’ input (lookup indices) and output (Fsum).

The major difference between NUMA-aware inference and distributed inference is the communication interface, UPI links vs NICs. Today’s network communication bandwidth is approaching that of the processor interconnect. In Figure 4(a), the model inference latency of distributed inference on two distributed SO-1S servers only has a minor increment over the NUMA-aware inference on one SU-2S server. The network bandwidth between SO-1S servers achieves $\sim$25 GB/s at peak, around half of UPI bandwidth, incurring less than 5% performance degradation for scale-out. One single server will finally meet the difficulty of holding enough resources to serve the model, (e.g., power delivery of one rack). Thus, distributed inference provides higher scalability and is a better strategy to support future model growth.

Throughput. Unlike datacenter batch workloads that only maximize throughput, the recommendation system maximizes throughput subject to a strict latency SLA. During workload initialization, we use a hill-climbing algorithm and a pressure test that sweeps query arrival rates and batch sizes [53]. The algorithm halts when latency-bounded throughput plateaus or decreases. Figure 5(a) shows that the end-to-end latency increases and gets dominated by queuing delay when query arrival rates are high. Figure 5(b) indicates that a batch size of 128 maximizes latency-bounded throughput and further increasing batch size harms throughput. Target latency cannot be met when batch size is 2048 as latency violates the SLA.

System Architecture. Figure 6 describes how hardware resources in traditional monolithic servers are disseminated into network-attached compute nodes (CNs) and memory nodes (MNs). This architecture resembles those in prior work [50, 57, 9]. But whereas prior MN designs offer transparent physical memory with no processing power  [50, 49, 43], DisaggRec’s MN includes an ASIC or a light-weight processor to perform embedding reduction locally.

The decision to process near memory arises from our insights in Sec  III. The embedding reduction performed inside the remote SparseNet shards can reduce communication traffic by transferring only the embeddings’ input and output values (indices, Fsum). Without such processing, the recommendation system would access raw embedding entries at the remote MNs and incur significant network overheads.

An RDMA-supported network topology connects all CNs and MNs together. Every node has a dedicated back-end NIC connected with the back-end ToR switch. The fast network enables low-latency and high-bandwidth communication, leading to only minor latency overhead when transferring embeddings’ indices and Fsum.

Managing Tasks. Adopting the distributed inference scheme, one serving unit consists of {$n$ CNs, $m$ MNs}. There are $n$ primary inference tasks launched on the $n$ CNs, and $m$ remote SparseNet serving shards launched on the $m$ MNs. Every primary task manages a private local memory region and a shared remote memory region. A CN’s local memory and a GPU’s HBM are mapped to the private local memory region. The remote $M$ MNs’ memory is mapped to the shared remote memory region.

Illustrated in Figure 6, a primary task with a unique global task ID (TID) is launched on every CN with dedicated CPUs for preprocessing $G_{P}$ and GPUs for DenseNet $G_{D}$. SparseNet RPC client operators are launched in the primary task. Based on the MemAccess routing table, the input data (i.e., embedding lookup indices) of all embeddings are partitioned into $m$ packets and sent to $m$ designated MNs via RDMA writes. MNs respond to the primary CN with an acknowledgment signal and the remote memory addresses of the results (embeddings’ Fsum). After receiving acknowledgments from all SparseNet shards, the primary CN loads results from remote MNs to the local GPU’s HBM memory via RDMA reads. Then, DenseNet is launched on the GPU to calculate the final prediction.

Handling Failures. CN and MN failures are independent and handled separately in a disaggregated serving unit (unlike failures in monolithic servers). In Figure 7(b), when a CN in a serving unit fails, only the primary task running on that CN is affected and migrated to a backup, over-provisioned node. The other tasks in the serving unit are unaffected.

When an MN fails, there are two possible scenarios. First, when there are multiple replicas of embedding tables allocated in the $m$ MNs, at least one copy of each embedding table is likely still available in the remaining MNs. We only need to update the MemAccess routing table, re-running the greedy routing to evenly distribute the embedding accesses on the remaining MNs. Second, when multiple MN failures lead to a loss of all replicas for an embedding table, memory is re-initialized to re-distribute all embedding tables across MNs after adding backup MNs to the serving unit.

Provisioning Heterogeneous Components. Many emerging hardware technologies, such as compute-centric accelerators [45] and near-memory processing solutions [33], benefit recommendation performance. However, deploying new components in existing datacenter infrastructure is a cumbersome process. As datacenters deploy new servers and platforms to host specific devices, such as ZionEX to host GPUs [10], system architects are exposed to many produces from varied vendors. Resource disaggregation simplifies how new hardware is deployed by introducing a new resource pool and allowing workloads to request desired hardware combinations from multiple pools.

To balance memory capacity, bandwidth utilization, and fault tolerance, we perform greedy embedding allocation and memory access routing during task initialization. Embedding tables are read-only during inference, so remote memory does not cause data consistency and correctness issues.

Embedding Allocation. In Figure 7(a), one SparseNet shard contains a subset of embedding tables and $m$ shards are allocated to $m$ MNs. We take the embedding table as the basic unit for memory allocation and greedily assign embedding tables to the $m$ MNs following the scheme in Figure 7(c). Given the memory capacity provided by the $m$ MNs, the algorithm calculates the number of embedding replicas ($nReplicas$) that can be held by the $m$ MNs. Then, the algorithm picks the top $nReplicas$ MNs, ranked by available capacity, to allocate the $nReplicas$ of each embedding table. The replicas of an embedding on different MNs provide the backup when memory node failure happens.

MemAccess Routing. All embeddings’ memory accesses must be routed from the $n$ CNs to the $m$ MNs. We construct a MemAccess routing table that distributes memory accesses to the $m$ MNs. As depicted in Figure 7(c), for every embedding’s memory access, the destination is picked from the $nReplicas$ MNs where that embedding table is allocated. First, the memory accesses of each individual embedding table is calculated by the average pooling factor multiply the embedding entry’s dimension. The average pooling factor is profiled from embedding pooling operations shown in historical queries. The greedy method selects the destination MN in the $nReplicas$ MNs that has the minimal memory accessed have been routed to. Once an MN has been selected, the tuple (task ID, embedding table ID, the destination MN) is added as an entry to the MemAccess routing table.

Why Not Random? A naive method would randomly pick $nReplicas$ of MNs to allocate an embedding table, and also randomly pick the destination MN from the $nReplicas$ MNs to route the memory accesses. In Figure 7(d), thousands of embedding tables are allocated on 8 MNs. The random embedding management leads to unbalanced memory capacity allocation and memory accesses among the 8 MNs whereas our greedy method balances accesses among the 8 MNs.

DisaggRec uses a task manager to coordinate scheduling within one serving unit. The manager can either perform interleaved or sequential query processing. In distributed inference, we find that sequential query processing offers lower latency and sustains higher throughput while satisfying the SLA.

Figure 8(a) illustrates one serving unit with {2 CNs, 2 MNs}. The embeddings’ request packets are generated from primary tasks on the two CNs and forwarded to SparseNet shards on the two MNs. Interleaved query processing executes the packets on SparseNet shards in first-come-first-serve (FCFS) order, which seems like a natural design to maximize throughput. Under this scheme, packets from different queries are interleaved and both queries finish late. In contrast, sequential query processing starts query execution only after all of its packets are received on all SparseNet shards. This scheme processes packets in lock step to finish one query’s embedding operations together, allowing one of the queries to finish earlier.

As shown in Figure 8(b), interleaved and sequential query processing achieve similar peak throughput if ignoring the latency target. However, when the latency SLA at 250ms needs to be met, the latency-bounded throughput of the sequential scheme is 28% higher than that of the interleaved scheme.

DisaggRec’s global manager performs sequential query processing to maximize latency-bounded throughput. In one serving unit, after all input data (embedding indices) belonging to one query are transferred from one CN to $m$ MNs, the global manager starts the query’s embedding operations on the $m$ MNs simultaneously. After $m$ MNs finish all embedding operations for that query, the global manager schedules the $m$ MNs to proceed to the next query. The naive adoption of previous disaggregated designs [57, 9] usually considers interleaved query processing among remote MNs, allowing an MN to respond to multiple packets (for different queries) at the same time to maximize remote memory utilization. However, we find that such interleaving will harm response latency.

To guarantee quality-of-service for billions of users, cluster resource management must allocate a sufficient number of serving units to ensure availability of the recommendation system. One serving unit consists of {$n$ CNs, $m$ MNs} in the disaggregated cluster, or $n$ servers in the monolithic server-based cluster, as one group to serve a recommendation model. We formulate resource allocation as a constrained optimization problem with two main steps—offline workload characterization and online resource allocation. The optimization’s objective is the efficient allocation of serving units to achieve performance and availability goals.

During offline workload characterization, we measure the throughput and power for each model-system pair, the achieved latency-bounded throughput $QPS_{M,S}$ and the peak power consumption $Power_{M,S}$ of model $M$ launched on one serving unit $S$, are input parameters when optimizing resource allocations for online serving.

During online resource allocation, subject to two constraints, Equation (1) allocates $N$ serving units (system type $S$ to model $M$) to minimize the total cost of ownership (TCO) for a time period t (e.g., 10s of minutes) where Capex is the acquisition cost of physical machines and Opex captures the operational cost due to electricity. Constraint (2) states the number of allocated serving units must be sufficient to guarantee availability given the highly-fluctuating diurnal loads and the expected machine failure rates. Backup machines are over-provisioned by $R$% based on historical data on load variance and also by $F$% given a machine failure rate. Constraint (3) states that provisioned power should be sufficient for allocated serving units.

The failure rate for a machine is dictated by the failure rate of its least reliable component. When any component fails, the whole machine becomes inaccessible and its jobs must migrate to a backup server. Figure 9 details reliability and failure rates in a production datacenter fleet. It reports four types of machine states during a day.

• •

  Server available all day

• •

  Server inaccessible all day (blue region)

• •

  Server becomes available mid-day (green region)

• •

  Server initially available but fails during day (red region)

We guarantee service availability by provisioning backup machines, which assume the responsibilities of failed servers in the fourth category.

Monolithic servers handle failures inefficiently because they bundle resources together despite the distinct reliability characteristics of each system component. We find that CPU servers (with high-capacity memory systems) and GPU servers (with multiple powerful compute GPUs) exhibit very different failure rates. The daily failure rate for GPU servers (7%) is much higher than that for CPU servers (0.4%) in the historical, 90-day datacenter logs. The monolithic SU-2S/SO-1S server deploys CPUs, GPUs, and DRAMs together into single nodes, their failure rates follow the higher rate of 7%. This higher failure rate implies more conservative over-provisioning, specifically on CPU and DRAM.

DisaggRec separates these unnecessarily bundled failures via a disaggregated architecture. It exploits the distinct failure rates of CNs and MNs to reduce the over-provisioning factor, in particular, for the more reliable MNs.

Table I describes GPU servers used to evaluate monolithic servers for recommendation systems. These servers are used to evaluate model inference on the state-of-the-art (SOTA) baseline, (a) end-to-end inference on a single scale-up server, and (b) distributed inference on multiple scale-out servers.

SU-2S is a scale-up dual-socket server with two 40-core Intel IceLake CPU processors, two terabytes (TB) of memory, and eight NVIDIA A100 GPUs, each with 80GB HBM. Front-end NICs are attached to the two CPU processors to receive inference queries from the front-end networks.

SO-1S is a scale-out single-socket server with one 40-core Intel IceLake CPU processor, one terabyte (TB) of memory, and four Nvidia A100 GPUs. One front-end NIC is attached to the CPU processor to receive inference queries from the front-end networks. Two back-end NICs are connected through the PCIe switch to communicate intermediate model inference data, embedding indices and Fsum, between the servers multiple SO-1S servers. We emulate different types of SO-1S servers by limiting GPU usage to one, two, or four GPUs.

Table I describes two types of compute nodes (CNs) and memory nodes (MNs) that we consider for the disaggregated system design. We emulate the recommendation system on disaggregated CNs and MuccjbhdcgblnjgulkklltdkkbbjguevbjlkjnjdbnjreedvfcvcdevigiNs by launching stages of the inference pipeline on the CPU and GPU servers. All performance numbers are measured from real systems except those with NMP-enabled memory, which is estimated with cycle-level simulation.

Compute Nodes. One CN is configured with one light-weight CPU and either one or four GPUs, matching the number of GPUs in the two SO-1S configurations. Each CN has one front-end NIC that receives incoming inference queries. It also has one back-end NIC that sends embedding indices to MNs and loads the embedding Fsum from MNs.

CN emulation accounts for two stages of the inference pipeline. Preprocessing $G_{P}$ is launched on the light-weight CPU server with one Intel CooperLake CPU and 64GB memory. DenseNet $G_{D}$ is launched on an SO-1S server using the selected number of GPUs.

Memory Nodes. First, the DDR-MN is configured with 1TB memory, matching the memory capacity of the SO-1S server, and a light-weight ASIC as the MN-side processing unit. Alternatively, the NMP-MN represents a memory node with near-memory processing enabled (NMP-DIMMs) [31, 33]. The NMP-MN exploits DIMM- and rank-level parallelism to increase effective memory bandwidth by $4\times$ relative to DDR-MN. DIMM-level parallelism doubles bandwidth by provisioning two NMP-DIMMs on a single memory channel. Rank-level parallelism further doubles the bandwidth by provisioning two ranks on one NMP-DIMM. We conservatively estimate the power dissipated by the MN’s ASIC to be 23.9 Watts based on the power profile of an internal ASIC accelerator fabricated in TSMC 7nm.

DDR-MN emulation launches SparseNet $G_{S}$ on SO-1S while disabling all GPUs. NMP-MN emulation simulates emerging NMP-DIMMs, following the earlier methodology for NMP studies [61, 31, 62, 6].

Communication. High-performance communication between CNs and MNs is the key for efficiency. One serving unit is defined by $n$ CNs and $m$ MNs and we explore a range of design points, ranging from the minimum (1 CN, 1 MN) to the maximum (8 CNs, 8 MNs). We emulate communication within the recommendation serving unit by microbenchmarking communication between 16 (8 + 8) GPUs on two SU-2S servers. Communication uses NVIDIA’s Collective Communication Library (NCCL) [51] and RDMAs. For each inference task, a CN sends embedding indices to multiple MNs with a Scatter (one-to-all) operation and the CN loads embedding Fsums from multiple MNs with a Gather (all-to-one) operation.

We estimate the total cost of ownership (TCO) using public market prices for all commodity devices and assuming a three-year machine lifetime. Since there is no NMP-DIMM commercially available on the market, we assume the price of one NMP-DIMM is 2$\times$ that of a regular DDR-DIMM because NMP-DIMM can theoretically double effective memory bandwidth to the same memory capacity. This estimate is conservative because NMP-DIMM will be a value-added product over today’s DIMM with only a small, fractional increase in cost when the technology is standardized.

For monolithic servers, the performance of model inference on the scale-up single server and the scale-out multiple servers are evaluated on real systems, building atop of the DeepRecSys serving framework [53].

For disaggregated systems, latency and power dissipated in the three pipeline stages are recorded for each inference query. Latency in the three stages—preprocessing $G_{P}$ on a CPU, SparseNet $G_{S}$ on DDR-MN, and DenseNet $G_{D}$ on GPUs—are individually measured and recorded on real systems. During real model inference execution, the dummy serving processes of the corresponding pipeline stages are replayed with the recorded latency for each query for each pipeline stage. When NMP-MNs are evaluated, cycle-level simulation estimates SparseNet $G_{S}$ performance. CPU and DDR4 power is read from Intel RAPL [18], and GPU power is measured by Nvidia API nvidia-smi.

Takeaway_A: While system heterogeneity permits server configurations to be tailored for diverse recommendation models, the exponential model growth drives up the datacenter costs substantially.

We consider recommendation services RM1 and RM2 and their respective model evolution for the next three years, as illustrated in Figure 1(c). We explore the server design space and optimize monolithic server configurations to minimize the TCO subject to performance goals. We consider five system configurations, including naive and NUMA-aware model inference on a SU-2S server as well as distributed inference on three types of SO-1S servers with 1, 2, and 4 GPUs. We follow method in Sec. IV-D to estimate the cluster TCO for online serving based on offline measured throughput and power consumption.

In Figure 10, when the model size grows larger than 2TB, one SU-2S server no longer provides sufficient memory capacity, and distributed inference is required. The number of distributed SO-1S servers in one serving unit is determined by the model size. For memory-intensive RM1, distributed inference on SO-1S (1 GPU) servers is the most cost-effective. For compute-intensive RM2, distributed inference on SO-1S (4 GPUs) servers is the best system choice. Thus, a heterogeneous cluster based on monolithic design would host two types of servers. Despite deploying these heterogeneous servers optimize for each recommendation service, as models grow in size and complexity, the datacenter’s TCO increases substantially—by 6.8$\times$ for RM1 and 12.4$\times$ for RM2—over the three-year period.

Takeaway_B: Because monolithic servers bundle compute and memory in a fixed ratio, up to 30% of datacenter cost is wasted on idle resources (23.1%) and over-provisioned capacity (6.8%). These inefficiencies arise due to unbalanced model pipelines and heterogeneous fault rates.

Figure 11(a) shows server allocation and utilization in the span of a day. The blue region shows the activated capacity for serving queries while the red region represents the over-provisioned capacity assuming a 7% machine failure rate observed from a production datacenter fleet. We assume servers are abundant and workloads always receive their preferred machines, optimally SO-1S (1 GPU) for RM1 and SO-1S (4 GPUs) for RM2.

Figure 11(b) shows resource utilization for active servers. RM1 models are constrained by SparseNet (dark blue for busy) while CPUs and GPUs are under-utilized during the Preprocessing and DenseNet stages (light orange and light green for idling), respectively. RM2 models are first constrained by SparseNet (dark blue), and then by DenseNet (dark green), as RM2 models grow primarily in the DenseNet.

Figure 11(c) shows the percentage of TCO wasted on idle resources attributed by two sources. We assume the CPU costs for carrying out Preprocessing and SparseNet are the same. For both models, over-provisioned capacity accounts for 6.8% of TCO. In addition, RM1’s unbalanced pipeline costs 15.6%–23.1% of TCO while RM2’s costs 2.8%–16.2%. RM1 pays higher for idleness because its DenseNet computation poorly utilizes the expensive GPUs; in contrast, RM2 only pays the toll in less expensive CPUs and memories during Preprocessing and SparseNet.

Takeaway_C: Scaling the number of monolithic servers used for distributed inference improves latency-bounded throughput and cost efficiency.

We explore the benefits of distributed inference and scaling out monolithic servers by examining the diagonals in Figure 12. A serving unit may scale out with two to eight monolithic SO-1S servers for RM1.V0. Adding more servers helps reduce query response latency, and then improves latency-bounded throughput for serving units that perform sequential (and not interleaved) query processing. In Figure 12(a), a serving unit with two, four, and eight SO-1S servers achieves 65%, 76%, and 90.6% of that serving unit’s peak throughput, respectively. Scaling out causes performance to increase superlinearly as throughput improves by 2.4$\times$ and 5.6$\times$ with 2$\times$ and 4$\times$ the number of servers.

A serving unit’s throughput and the over-provisioning ratio based on machine failure rates—which we take to be 7% for SO-1S, 7% for CNs, and 0.04% for MNs—determine how many serving units are required to ensure the cluster satisfies peak query load (Figure 12(d)). The recommendation serving cluster’s cost depends on the number of hosted serving units and their power costs (Figure 12(c)). Our analysis indicates that scaling out with monolithic serves causes normalized TCO to decrease from 2.55$\times$ to 1.83$\times$ (Figure 12(e), diagonal).

Takeaway_D: Disaggregation further improves cost efficiency by improving resource utilization and reducing over-provisioned backups for machine failures.

We explore the benefits of distributed inference on disaggregated compute and memory nodes by examining the whole 2D grid in Figure 12. Disaggregation permits a serving unit to scale compute and memory nodes independently, which better matches resources to workload needs. For memory-intensive model RM1.V0, throughput is relatively insensitive to the number of CNs in the serving unit. Disaggregation permits a serving unit with fewer CNs and more MNs, which in turn reduces cost (Figure 12(e), blueish region). A serving unit that deploys 3 CNs and 8 MNs minimizes the TCO with negligible impact on performance. The throughput of this cost-efficient solution is only 2% less than that of eight monolithic SO-1S servers (Figure 12(b)).

Broadening our evaluation, Figure 13 details disaggregation’s benefits for six generations of recommendation models. For memory-intensive RM1, disaggregation reduces the TCO by up to 49.3% compared to distributed inference on monolithic SO-1S servers. Most of the saving (40.9% of 49.3%) comes from reducing the number of CNs, which are equipped with expensive GPUs. The remaining saving comes from exploiting MNs’ low failure rates and over-provisioning resources by a smaller factor.

For compute-intensive RM2, serving units require similar amounts of hardware whether using monolithic servers or using disaggregated CNs and MNs. For example, the optimal configurations for V2, V3, and V4 models use the same ratio of CNs and DDR-MNs. Yet even for these models, disaggregation reduces cost by 4.3% to 9.3%.

The cost savings come from two sources. First, the serving unit with disaggregated nodes is more power-efficient. CNs deploy low-end CPUs and high-end GPUs. MNs deploy ASICs for a modest amount of computation near the data. These CN and MN configurations dissipate less power than the monolithic SO-1S servers deploying a high-end CPU (i.e., 40-core IceLake). Lower operating expenses account for 7.2% of the cost savings. Second, the MN’s lower failure rate permits the serving unit to over-provision resources by a smaller factor.

Takeaway_E: Resource disaggregation provides flexible support for resource heterogeneity in production datacenters. Hardware components are organized into disparate resource pools rather than integrated into monolithic servers. This organization improves utilization and reduces costs.

We explore technology scenarios, comparing and contrasting a cluster with monolithic servers and one with disaggregated resources as recommendation models evolve and grow over three years. The initial clusters are built with commodity CPUs, regular DDR-DIMMs, and GPUs. The cluster with monolithic servers optimally deploys SO-1S servers with 1 GPU and DDR-DIMMs for memory-intensive RM1.V0 and optimally deploys SO-1S servers with 4 GPUs and DDR-DIMMS for compute-intensive RM2.V0. On the other hand, the cluster with disaggregated compute and memory nodes optimally deploys {CN with 1 GPU, DDR-MN} and {CN with 4 GPUs, DDR-MN} for RM1.V0 and RM2.V0, respectively. We assume that deployed servers and nodes will remain deployed for their three-year machine lifetimes.

In future model generations, near-memory processing leads to new system components (NMP-DIMM). The cluster with monolithic servers deploys two new server configurations, namely an SO-1S server with 1 GPU and NMP-DIMMs as well as an SO-1S server with 4 GPUs and NMP-DIMMs. On the other hand, the disaggregated cluster deploys NMP-DIMMs as a new type of memory node, NMP-MN (Table I).

Figure 14(a) shows the TCO savings from NMP-DIMMs. When monolithic servers are used, NMP-DIMMs reduce costs for memory-intensive RM1 models but increase costs for compute-intensive RM2 models. Emerging NMP-DIMMs are more expensive than conventional DDR-DIMMs starting in the V2 generation. Their costs are justified only for memory-intensive workloads that experience throughput gains when adopting the technology.

Specifically, NMP-DIMMs increase effective memory bandwidth by 4$\times$ through DIMM- and rank-level parallelism. Greater memory bandwidth accelerates embedding operations. The SO-1S server with NMP-DIMMs improves RM1 throughput by up to 3.64$\times$. However, for compute-dominated RM2, the memory bandwidth of NMP-DIMMs on the SO-1S server is under-utilized, and the 2$\times$ cost of NMP-DIMMs (Table II) over DDR-DIMMs eventually leads to a higher TCO.

In contrast, when disaggregated nodes are used, NMP-DIMMs reduce costs for both RM1 and RM2 models because the emerging technology (NMP-DIMM) is deployed as a new resource pool and allocated flexibly. The independent scaling of CNs and MNs in the disaggregated cluster allows a smaller ratio of MNs in a serving unit, which prevents the under-utilization of NMP-DIMMs’ memory bandwidth.

Figure 14(b) shows total cluster TCOs over multiple model generations when the cluster is provisioned by continuously deploying optimal system configurations for new generations of RM1 and RM2. When monolithic servers are used, the cluster expands capacity for evolving recommendation models by deploying SO-1S servers with 1 GPU and NMP-DIMMs for RM1.V1–V5, and deploying SO-1S servers with 4 GPUs and DDR-DIMMs for RM2.V1–V5. When disaggregated resource nodes are used, the cluster deploys {CN with 1 GPU, NMP-MN} and {CN with 4 GPUs, NMP-MN} for RM1 and RM2, respectively. Overall, the disaggregated cluster allows 21%–43.6% TCO saving over the monolithic server-based cluster across the three-year model evolution.