ConServe: Harvesting GPUs for Low-Latency and High-Throughput Large Language Model Serving

Today’s LLMs typically generate outputs following the autoregressive process—they take a sequence of tokens as input, and repeatedly predict the next token given the input sequence and all the previously generated tokens. This process involves two distinct phases: prefill and decode. An LLM inference starts with the prefill phase that deals with a new input sequence and generates the first output token. Thanks to the modern model architecture such as Transformer [76], an LLM can process all input tokens in parallel, making the prefill phase compute-bound. After the prefill phase, the LLM enters the decode phase that sequentially generates subsequent tokens. Because each decode iteration generates only one token per request, it is less compute-intensive and typically bounded by the GPU memory bandwidth due to the need to load model weights. To improve its compute intensity and hardware efficiency, a common practice is to batch multiple requests in a single inference run to reuse the model weights for computation and memory load.

More specifically, during LLM inference, each token has its own intermediate state, represented by a series of key vectors and value vectors. As token states remain unchanged throughout the inference process, the inference engine [54, 45, 47] caches them in GPU memory (a.k.a., KV cache) to avoid recomputation in each decode step. However, due to the large size of KV vectors and the excessive number of tokens in a batch, the KV cache can consume a significant amount of GPU memory, which limits the batch size and hence hardware efficiency.

The rise of compound AI systems and LLM agents dictates that LLMs be used for various demands and in different forms. Generally, LLM serving can be categorized into two types: online serving, which generates responses to user inputs in real time, and offline serving, which processes user inputs and generates outputs in batchs. Typical scenarios for offline serving including document summarization [27], data wrangling [42], LLM-enhanced data analytics [37], LLM benchmarking and evaluation  [34], among other emerging applications. However, online and offline serving exhibit drastically different characteristics, as elaborated below.

Online serving is mostly suitable for latency-critical requests. Unlike traditional cloud services, where processing times are often predictable, a request to an LLM generates a sequence of tokens with varying lengths in multiple steps. As a result, LLM serving latency is measured on a per-token basis. Specifically, because LLM inference consists of two distinct phases, an LLM’s serving latency is defined by two key metrics: time to first token (TTFT), which is the duration of the prefill phase, and time per output token (TPOT), which is the execution time of each step in the decode phase.²²2In this paper, we measure TPOT as the generation time of each decode token (also referred to as inter-token latency in some literature) rather than the average token generation latency of each request. To provide a low end-to-end response latency and smooth user experience, it is critical for an online LLM serving system to optimize both TTFT and TPOT under stringent SLOs. For example, an online chatbot may set a 99^th percentile TTFT SLO of 1.5 seconds and a 99^th percentile TPOT SLO of 100 milliseconds to respond faster than a typical human can read [60, 41].

In contrast, offline serving is better suited for best-effort requests that are insensitive to response latency. Users typically submit offline requests in batch, while the serving system processes them in a best-effort manner to maximize hardware efficiency. Unlike online serving that emphasizes low latency, offline serving often involves processing large corpora or datasets, where the first-order performance metric is throughput that measures how many tokens are generated per second.

The different service-level performance objectives between online and offline serving poses challenges to LLM serving systems. To overcome the tension between low response latency (TTFT and TPOT) and high generation throughput, today’s LLM serving systems usually adopt different configurations and designs for different serving scenarios, typically requiring the service provider to set up separate clusters for online and offline serving. Next, we will discuss how existing systems optimize LLM serving and why they fall short in achieving high GPU utilization.

A large body of system optimizations have been proposed to serve LLMs with low latency and high throughput since the launch of ChatGPT.

Another line of work moves away from continuous batching scheduling, as exemplified by DistServe [91], Splitwise [57]. Instead, they duplicate the model and disaggregate the prefill and the decode computation onto different GPUs. Specifically, new requests are sent to a dedicated cluster for the prefill computation, and the generated token and KV cache are populated to another dedicated cluster that performs the decode computation. Llumnix [71] is another recent work that migrates KV caches between servers to reduce load imbalance. These systems achieve low TTFT and TPOT at the cost of more GPUs. Due to the additional data transfer of KV caches between servers, they also lead to reduced throughput.

Unfortunately, these problems are more pronounced when serving real-world inference workloads, which can contain both online and offline requests with high load variation. Such a challenge makes it hard to provision the right amount of GPU resources a priori.

ConServe is an LLM serving system designed to co-serve online and offline requests. Figure 3 shows its overall architecture. At its frontend, ConServe provides similar APIs to other LLM serving systems. For online requests, it uses a real-time streaming API that returns outputs once each token is generated. For offline requests, it adopts an interface similar to OpenAI’s Batch API [51], which takes a batch of requests from a pool and returns responses asynchronously.

ConServe relies on its backend to schedule and execute inference jobs, which consists of three major components. The scheduler runs as a daemon thread and continuously fetches and schedules offline requests. Upon the arrival of an online request, the scheduler reactively preempts offline requests and immediately schedules the incoming online request (§4.2). The scheduler leverages a set of workers to host the LLM and serve the scheduled batch. A model can span multiple GPUs, where each GPU is managed by one worker. ConServe supports both tensor and pipeline parallelism for high efficiency and flexibility (§4.3). To quickly recover preempted requests with minimized recomputation costs, ConServe leverages host memory and incrementally checkpoints KV caches during its execution (§4.4). Finally, because ConServe can schedule both online and offline requests in the same batch, it adopts an SLO-aware policy to adaptively adjust the batch size to preserve online TTFT and TPOT latency objectives while maximizing the hardware efficiency (§4.5).

ConServe’s scheduler is the key component that serves both online and offline requests in a unified fashion. As with previous LLM serving schedulers [86, 30], ConServe adopts continuous batching in its scheduler. To prevent long prefills from interfering decode iterations, ConServe also adopts chunked prefill [2] that partitions and computes long prefills into small chunks over multiple iterations and limits the batch size per iteration.

The overall scheduling logic is shown in Algorithm 1. It maintains two separate queues for online and offline requests, and continuously monitors the online queue for incoming requests. In each scheduling step, it first calculates a budget batch size given the latency objectives for TTFT and TPOT (Line 1). The budget limits both the number of tokens and the number of requests in a batch. The detailed policies are elaborated in §4.5. The scheduler then checks and schedules incoming online requests within the budget allowance (Lines 1-1). To prevent previously scheduled offline requests from blocking incoming online requests (i.e., head-of-line blocking), the scheduler excludes offline tokens from the budget first (Line 1-1), and then reactively preempts offline requests if the budget is over-saturated (Line 1). This preemption continues until all scheduled requests can fit into the budget (Lines 1-1). After accommodating all online requests, the scheduler opportunistically schedules offline requests using the remaining budget (Line 1-1). The detailed scheduling policies (i.e., SloAwareSchedule) to maintain online latency SLOs and manage GPU memory usage will be discussed shortly in §4.5.

However, in the worst case where incoming online requests are about to exceed their TTFT objectives, ConServe must deal with the urgent case and preempt offline requests even if they are in a running batch to schedule new requests in a timely fashion. To this end, ConServe scheduler invokes an asynchronous handler upon the arrival of new online requests, as shown in Algorithm 2. The handler first pushes the incoming online request into the online request queue, and then leverages the profiler (discussed shortly in §4.5) to estimate the queuing delay and the execution time (Lines 2-2). If the estimated serving time exceeds the TTFT objective, the scheduler signals the worker to preempt offline requests in the current running batch until it can meet the TTFT objective (Lines 2-2).

To host the model on GPUs and execute inference requests, ConServe employs a set of workers to manage GPU resources. Each worker is responsible for handling the physical KV cache, launching model computation, and communicating with other workers. ConServe supports both tensor and pipeline parallelism in the Megatron-LM fashion [70]. Additionally, ConServe’s workers support preempting a running batch (i.e., PreemptRunning in Algorithm 2 Line 2) as the last resort to avoid head-of-line blocking. This is particularly crucial in offline batching mode, where large batch sizes are used for maximized throughput at the cost of high latency. However, preempting model execution while it is running involves several challenges in balancing responsiveness and runtime overhead.

The first challenge is how the scheduler should notify all workers. When the model is deployed with tensor parallelism, workers perform collective communications. Preempting a subset of workers indiscriminately may leave others waiting during communication operations, potentially causing the program to hang. To prevent this, ConServe periodically synchronizes all workers using a distributed barrier [62] and signals preemption only after synchronization. The ConServe scheduler shares a preemption flag with its workers, which is set when it decides to preempt the current batch, and checked by workers during their execution. Upon detecting the flag is set. Workers check this flag during execution, and upon detecting it, they immediately abort the remaining execution and return an empty result. The scheduler then cleans up the partial computation states, restores its own state, and quickly schedules any incoming online requests.

The second challenge is determining the appropriate temporal granularity for preemption. If the granularity is too coarse, the system may not respond timely to load bursts; conversely, if it is too fine-grained (e.g., per GPU kernel as in previous work [21]), synchronization overhead may become significant and reduce overall efficiency. To strike a balance, ConServe chooses to preempt at the granularity of model layers for two reasons. First, while architectures and sizes of model may vary drastically, LLMs all consist of multiple layers. The execution time for a single layer accounts for only a small fraction of the end-to-end latency, which allows for responsive preemption. Second, LLM layers involve dozens of GPU kernels and communication operations, so the overhead of preemption and synchronization is small relative to the layer execution time.

To avoid impacting online serving latency, ConServe restrict layer-wise preemption to the offline batching mode. In co-serving mode, where the batch contains both online and offline requests, ConServe leverages chunked prefill and SLO-aware scheduling (discussed shortly) to limit chunk sizes and bound execution time per iteration. While in offline batching mode, ConServe aggressively increases batch sizes and chunk sizes and relies on layer-wise preemption to maintain responsiveness.

To further mitigates costs for small models whose layer executes fast, ConServe allows workers to adjust the preemption granularity by batching multiple layers before reaching a synchronization barrier. In our evaluation, preempting per eight layers achieves a favorable balance of low runtime cost and high responsiveness (details in §6.4.2).

While preempting offline requests can free up GPU compute resources immediately, ConServe must also address the GPU memory pressure caused by the excessive KV cache usage of offline serving. To quickly reclaim memory in response during online load bursts and minimize the scheduling delay, one strawman approach is to directly discard KV caches for victim offline requests that are preempted, and recompute them later once online load subsides (see Figure 4(a)). However, this approach sacrifices GPU efficiency by wasting compute resources on redundant recomputation, thereby reducing overall serving throughput.

To avoid expensive recomputation, a common optimization applied by existing serving systems such as vLLM [30] is to swap out KV caches of victim requests to host memory, as shown in Figure 4(b). However, swapping out by itself still takes time, which delays the scheduling of incoming online requests. To make matters worse, the amount of data to be swapped out increases proportionally to the number of tokens in offline requests, while the interconnection bandwidth between GPUs and host memory is limited. For example, an Nvidia A100 GPU connecting to host DRAM via PCIe 4.0x16 offers only 32 GBps of bandwidth. Swapping out all KV caches for large offline requests can easily take dozens or even thousands of milliseconds and significantly block online requests. Moreover, when resuming preempted offline requests, the GPU must swap in the evicted KV cache blocks before continuing the computation, leaving the GPU idle and again, hurting overall serving throughput.

ConServe manages GPU memory similar to vLLM [30] by reserving GPU memory ahead of time and virtually mapping KV cache blocks to physical GPU memory. But unlike swapping which frees KV caches right after they are swapped out, checkpointing keeps KV caches in GPU memory until incoming online requests are scheduled. Discarding KV caches in ConServe is as fast and lightweight as freeing victim KV cache blocks and remapping new ones virtually, which finishes in dozens of microseconds.

Inspired by the asynchronous swap system design in the OS kernel [63] and the conventional wisdom of random early detection [15], ConServe adaptively controls the checkpointing rate based on GPU memory pressure to limit resources used by checkpointing, and it gradually increases the checkpointing rate as GPU memory usage goes up and always tries to match checkpointing speed with memory consumption speed. Specifically, ConServe starts checkpointing when available GPU memory is running low (by default the threshold is set to $50\%$). but it will only checkpoint a small number of offline requests first and gradually increase the number of offline requests to checkpoint after observing constantly increasing memory usage. This is sufficient for most cases when GPU memory pressure is moderate, and ConServe can selectively preempt offline requests that have been checkpointed. Under the extreme case when memory pressure still persists after preempting all checkpointed requests, ConServe will discard the KV cache for the remaining offline requests to ensure incoming online requests can get scheduled without any delay.

Finally, incremental checkpointing can also help online serving to accommodate swapping/preemption caused by continuous batching. With the continuous batching policy, the scheduler will eagerly add new requests to the batch to enlarge the batch size when GPU memory permits, but this may consume all GPU memory when requests in the batch generate long output sequences and keep allocating memory for KV caches. In such scenarios, the scheduler will have to swap or preempt online requests to make room. ConServe will also incrementally checkpoint online requests under GPU memory pressure, thereby eliminating the cost of swapping out or recomputation for online requests as well.

The last challenge that ConServe must address is to determine the right batch size (i.e., the number of offline requests and tokens) to compute and the right set of KV cache blocks to checkpoint/prefetch for each inference iteration. (1) As for the batch size, if the scheduler batches too few offline requests, a substantial portion of GPU compute resources and memory bandwidth would be underutilized and lead to suboptimal overall serving throughput; on the flip side, if the scheduler batches too many offline requests, they will saturate the GPU and increase the inference computation time and hence online serving latencies, resulting in SLO violations. To make matters more complicated, the model execution time depends on not only the batch size but also the context lengths of each request due to the non-linear compute complexity of the attention kernel ($O(n^{2})$ for prefill phase and $O(n)$ for decode phase where $n$ is the context length)³³3Chunked prefill alleviates this effect by splitting long sequences into smaller chunks, but its performance is still prone to prefix lengths due to repeated KV cache access [2].. Therefore, the scheduler must also decide the number of offline tokens that can be batched and processed together with online tokens while meeting the SLO. (2) As for the KV cache blocks to checkpoint/prefetch, swapping too many blocks, even if in the background, will exhaust the host-device bandwidth and GPU streaming multiprocessors (SMs) resources and block the computation kernel, while checkpointing too few blocks will make it unable to keep up with the GPU memory consumption rate, leading to memory exhaustion and triggering swap that blocks incoming online requests.

To tackle this challenge, ConServe adopts an SLO-aware scheduler to collect the model execution profiles with an offline profiler and dynamically adjust the batch size and the degree of background swapping leveraging the collected information and SLOs specified by the users. To flexibly batch varying numbers of offline tokens into each batch, ConServe leverages chunked prefill [2] to break offline sequences if necessary.

We evaluated ConServe on Llama-2-7B model [75]. All experiments use FP16 precision and tensor parallelism to align with our baselines.

For offline workloads, we evaluated the document summarization task with the LongBench [6] datasets.

In this section, we evaluated ConServe with real-world workloads and compared its end-to-end performance against the baselines. We used the same trace reported in Figure 1(b) as the online load. We first ran the original vLLM with only online loads to collect its 99^th percentile TTFT and TPOT, and then set them as the SLO targets for all systems (1500ms for TTFT and 110ms for TPOT).

A good result for ConServe would show that it quickly detects any GPU underutilization and judiciously batches offline requests to harvest available GPU resources to achieve good offline throughput while still keeping online serving latency lower than the SLO. In contrast, Online-Only should achieve the optimal online serving latency but fails to harvest idle GPU resources for offline serving. On the contrary, vLLM++ does batch offline requests together with online ones, but it optimizes for the overall serving throughput and does not guarantee online latency. Therefore, we expect vLLM++ to achieve high offline throughput but also experience drastically fluctuating TTFT and TPOT during load bursts.

Figure 5 presents the results. The top two figures present 99^th percentile TTFT and TPOT of all three systems, with red horizontal lines indicating the SLOs for TTFT (1500ms) and TPOT (110ms), respectively. The bottom figure shows the overall (online + offline) serving throughput. As expected, Online-Only achieves lowest TTFT and TPOT. However, during periods of low request rate (from $t=240$s to $620$s), it cannot batch enough requests per inference iteration, resulting in low latency but underutilizing the GPU. On average, it achieves an overall serving throughput of 1999 tokens/s. ConServe maintains low online TTFT and TPOT that are close to optimal ones and consistently lower than SLOs, and it offers 3702 tokens/s overall throughput by harvesting available GPU resources for offline serving. While vLLM++ is also able to batch offline requests and achieves the highest throughput at 4308 tokens/s, frequent swapping significantly increases its 99^th percentile TTFT and TPOT by 84$\times$ (83825ms) and 25$\times$ (2523ms), respectively.

In summary, the experiment demonstrates that ConServe can efficiently utilize available GPU resources for offline serving with negligible impact on online serving latency. As a result, it achieves 2.35$\times$ higher overall throughput compared to Online-Only and 98.8% lower online serving latency compared to vLLM++.

In this section, we conducted a set of experiments to investigate whether ConServe can reactively harvest idle GPU resources—whenever they are available—for offline serving, while still quickly reacting to resource pressure to avoid interfering with online serving performance. For experiments in this section, we set the request input length to 1024 and the output length to 128 as representative values for online loads [78, 91].

We now evaluate specific aspects of ConServe’s design to understand their individual contributions to overall performance.