Distributed Speculative Inference of Large Language Models is Provably Faster

Generative LLMs, such as GPT-4 (OpenAI, 2023), have demonstrated unprecedented results in various applications (Andreas, 2022; Li et al., 2023; Bubeck et al., 2023; Wei et al., 2022). Despite their potential, the inference latency of LLMs presents a significant challenge and a bottleneck for adoption in real-time applications. For example, in algorithmic trading, the model needs to make rapid predictions to execute trades in milliseconds (Tan et al., 2024; Zhang et al., 2023), and in autonomous driving, the model must act quickly to ensure the vehicle’s reliability (Yang et al., 2024). Real-time applications often prioritize low latency over other objectives, and in particular are willing to trade off more computational resources for faster inference. With the growing availability of hardware and decrease in its costs, finding how to effectively utilize more computing power for faster inference becomes increasingly important.

Given their usefulness, speeding up the inference of LLMs is an important area of research. Existing efforts to reduce the inference latency can be classified into two main categories by their approach to hardware utilization. The first category focuses on making LLMs use less computing resources or better utilize the same resources. This includes post-training compression through pruning (e.g., (Frantar and Alistarh, 2023; Sun et al., 2024a; Ma et al., 2023)), knowledge distillation (e.g., Hinton et al. (2015); Gu et al. (2024)), quantization (e.g., (Hubara et al., 2018; Frantar et al., 2023; Lin et al., 2024; Yao et al., 2024; Dettmers et al., 2022)), low-rank factorization (e.g., (Hsu et al., 2022; Xu et al., 2023)), early existing (e.g., (Schuster et al., 2022; Kim et al., 2023; Elbayad et al., 2020; Bapna et al., 2020; Schuster et al., 2021)), and alternative architectures (e.g., (Cai et al., 2024; Li et al., 2024; Zhang et al., 2024b, a; Xiao et al., 2024; Gu and Dao, 2023)). While these solutions are useful in practice, they often require modifications to the model architecture, changes to the training procedure and re-training of the models, without guaranteeing identical outputs. Despite reducing the inference time, these methods often have a significant drawback of degrading the output quality. There are also solutions that preserve the output quality, such as kernel optimizations (Dao et al., 2022; Dao, 2024), but they highly depend on the hardware and therefore are not always available or even feasible. The second category focuses on accelerating the inference by using more computing power. It includes data parallelism (DP) and model parallelism (MP) methods of different types, such as pipeline parallelism (PP), tensor parallelism (TP) (Narayanan et al., 2021), and context parallelism (CP) (Harper et al., 2019). Such partitioning over multiple processors can speed up memory-bounded inference setups, for example, by avoiding memory offloading. They are often effective in increasing the inference throughput and supporting larger context length. However, the typical LLM architectures necessitate autoregressive inference. Such inference is inherently sequential with only limited (if any) MP opportunities, depending on the LLM architecture. Since the portion of the inference that could be parallelized is small (if any) in typical LLMs, the potential speedup of MP methods is limited, by Amdahl’s law. Hence, while DP methods can increase the inference throughput, they remain inherently sequential. Furthermore, as the parallelisms above shard over more processors, the communication overhead increases, which can lead to diminishing returns, limiting the number of processors they can effectively utilize.

A recent line of work (Stern et al., 2018) for accelerating the inference of LLMs is based on speculative inference. The idea is to use speculative execution (Burton, 1985; Hennessy and Patterson, 2012) to predict possible continuations of the input prompt using faster drafter LLMs that approximate the target LLM, then verify the correctness of the predicted continuations simultaneously by utilizing data parallelism capabilities of CUDA-based processors known as batching. They provided empirical evidence that their proposed draft-then-verify approach speeds up the inference. Since the introduction of speculative inference (SI) by Stern et al. (2018), various papers (Leviathan et al., 2023; Chen et al., 2023) have improved this method by introducing novel lossless methods to verify the correctness of token sequences that were generated by the drafter LLMs. Empirically, these approaches lead to speedups in practical use cases, such as 2-3x speedups in decoding LLMs of 11B and 70B parameters in some settings. Following this line of work, Miao et al. (2023) extended the verification algorithm of Leviathan et al. (2023); Chen et al. (2023) and showed that their method increases the probability of accepting draft tokens, and proved its losslessness. Following the success of this approach, research in this area has expanded in various directions (Mamou et al., 2024; Li et al., 2024; Cai et al., 2024; Sun et al., 2024b; Zhou et al., 2024; Liu et al., 2023; Joao Gante, 2023). Most recently, Chen et al. (2024) showed that lossless SI can effectively reduce the inference latency while increasing its throughput, in the multi-user setting. The effectiveness of SI algorithms comes from their ability to reduce the number of target forwards by using GPU batching such that a single target forward is sufficient for generating more than one token. However, existing SI methods implement a sequential process, repeatedly drafting and verifying. They never draft more tokens before the previous verification ends. This limitation implies that SI speeds up the inference if and only if the drafter is sufficiently fast and accurate, as studied in this paper. In cases where the drafter is too slow or inaccurate, SI is slower than non-SI.

Contributions.  Our proposed method, distributed speculative inference (DSI), introduces a novel task parallelism for SI, named speculation parallelism (SP), to make the blocking verifications of SI non-blocking. DSI leverages SP to hide the verification latency of iterations without a draft rejection. Unlike SI, which can run on a single GPU, DSI requires two or more GPUs. DSI is useful in practice because it works given an arbitrary number of GPUs ($\geq 2$) by adjusting its lookahead hyperparameter. Provably, DSI is faster than SI in expectation and always as fast or faster than non-SI (§3.2). DSI can accelerate inference even when drafters slow down SI compared to non-SI, thus enabling the acceleration of more off-the-shelf LLMs. Empirically, DSI on a single node with up to eight GPUs is significantly faster than both SI and non-SI (§4).

We begin by describing autoregressive language models, next-token prediction, speculative inference and how to measure latency.

Autoregressive language models (LMs) are deterministic, real-valued multivariate functions. An input to an LM is a sequence of vectors of dimension $n_{\textnormal{vocab}}$. We call these vectors tokens, and the sequence a prompt. LMs output a real-valued vector of dimension $n_{\textnormal{vocab}}$, also known as the logits. Since prompts may vary in length, we simplify the notation of the forward pass as follows: $f:\mathbb{R}^{*\times n_{\textnormal{vocab}}}\to\mathbb{R}^{n_{\textnormal{vocab}}}$.

Self-Attention LMs are LMs with a pre-defined context length $n_{\textnormal{ctx}}$ (Vaswani et al., 2017). Hence, we represent the forward pass of such LMs in the following manner: $f:\mathbb{R}^{n_{\textnormal{ctx}}\times n_{\textnormal{vocab}}}\to\mathbb{R}^{n_{\textnormal{vocab}}}$. For example, GPT-2 and GPT-3 are Transformers with $n_{\textnormal{vocab}}=50257$, and context lengths $n_{\textnormal{ctx}}=1024$ and $n_{\textnormal{ctx}}=2048$, respectively (Radford et al., 2019; Brown et al., 2020). In this paper, all LMs are Self-Attention ones with pre-set (frozen) parameters.

We extend the prompt notation such that prompts can have length $l\leq n_{\textnormal{ctx}}$. Self-Attention LMs handle prompts of length $l<n_{\textnormal{ctx}}$ by starting the input sequence with a prefix of $n_{\textnormal{ctx}}-l$ tokens, followed by the $l$ given tokens. LMs ignore the prefix, either by zeroing (masking) the Attention parts corresponding to the prefix or by left-padding with dedicated tokens. In this paper, prompts of length $l<n_{\textnormal{ctx}}$ are the non-masked, non-padded suffix of the input sequence of length $n_{\textnormal{ctx}}$.

Generating the next token is the primary application of autoregressive LMs. This process consists of two steps: computing the forward pass of the LM and then selecting the next token based on the output. The selection can be deterministic or non-deterministic. Non-deterministic selection procedures apply the softmax function after the forward pass of LMs and sample from the resulting probability vector:

For convenience, we denote the output probability vector by $f\left({x_{\leq i}}\right)$:

where $a\oplus b=(a,b)$ is the concatenation of the vectors $a$ and $b$ and $x_{\leq i}:=x_{\leq 0}\oplus x_{1}\oplus\dots\oplus x_{i}$. For deterministic selection procedures, composing monotonic functions, such as softmax, is usually unnecessary. For example, the most likely next token is the $\arg\max$ of both the logits and the output of the softmax. Still, for convenience, we assume that LMs always output probability vectors. The sampling process in (2) is either deterministic (i.e., $x_{i+1}$ is a token with maximal probability) or random (achieved by randomly selecting $x_{i+1}$ from the distribution $\text{softmax}\left({f\left({x_{\leq i}}\right)}\right)$).

Speculative Inference (SI) is an approach for accelerating the inference of a target LM (e.g., a member of the GPT series) $f_{m}$. Such methods use faster LMs $f_{1},\dots,f_{m-1}$ that approximate the target model $f_{m}$ in order to reduce the total inference time. For example, Leviathan et al. (2023); Chen et al. (2023) may reduce the amount of time to infer a target model $f_{2}$ on a given prompt $x_{\leq 0}$ by using batching as follows. The inference starts by drafting $k$ tokens $x^{\prime}_{i}:=f_{1}(x^{\prime}_{\leq i-1}):=f_{1}(x_{\leq 0}\oplus x^{\prime}_{1}\oplus\dots\oplus x^{\prime}_{i-1})$ for $i\in[k]$ using a faster drafter model $f_{1}$. Then, the algorithm sends the prompts $\{x^{\prime}_{\leq i}\}^{k}_{i=0}$ altogether as one input batch to the target model $f_{2}$. The idea is to take advantage of the data parallelism that modern GPUs offer to compute the logits corresponding to the prompts $\{x^{\prime}_{\leq i}\}^{k}_{i=0}$ in parallel, hence faster than computing these $k+1$ individual logits sequentially. Given the logits, the algorithm generates $[1,k+1]$ tokens without additional forward passes. By repeating this process, the algorithm can generate $N>k+1$ tokens.

Straightforward algorithms of speculative inference are typically lossless in expectation, i.e., they generate tokens from the same distributions as the target would generate without speculation. Naive algorithms of speculation guarantee returning the same tokens as the target (Joao Gante, 2023; Spector and Re, 2023). More sophisticated algorithms of speculation might generate different tokens, but their generated tokens follow the distribution of the target (Leviathan et al., 2023; Chen et al., 2023).

To implement distributed algorithms for speculative inference, we use multiple processors or servers, which are hardware components capable of executing threads. Processors can compute forward passes and sample tokens from the output probability vectors and we assume that threads can run in parallel. When using DSI we will run sequences of drafter models $f_{j_{1}},f_{j_{2}},\dots,f_{j_{k}}$, where the first model takes $x_{\leq 0}$ and returns some token $x^{j_{1}}_{1}$, the second takes $x_{\leq 0}\oplus x^{j_{1}}_{1}$ as a prompt and returns $x^{j_{1},j_{2}}_{2}$, and so on. As such, in order to denote that a given thread is computing the output of $f_{j_{k}}$ on a sequence $x^{j_{1},\dots,j_{k-1}}_{\leq k-1}:=x_{\leq 0}\oplus x^{j_{1}}_{1}\oplus\dots\oplus x^{j_{1},\dots,j_{k-1}}_{k-1}$, we denote $C_{J}$, where $J=(j_{1},\dots,j_{k})$. When a thread $C_{J}$ computes an LM, we denote the output probability vector by $C_{J}\textnormal{[prob]}$. If $C_{J}$ samples a new token from $C_{J}\textnormal{[prob]}$, we denote this token by $C_{J}\textnormal{[new]}$. For example, thread $C_{J}$ implementing (2) above will have

Once a thread $C_{J}$ finishes sampling a new token, the thread outputs the concatenation of $C_{J}\textnormal{[prompt]}$ and $C_{J}\textnormal{[new]}$. Following the example in (2), we have

A new thread that was initiated by $C_{J}$ is denoted by $C_{J\oplus(j)}$, where $J\oplus(j)$ is the concatenation of $J$ and $(j)$. The set of all the threads that originate from $C_{J}$ is $\{C_{J\oplus J^{\prime}}:J^{\prime}\text{ is a nonempty tuple}\}$. We assume that terminating a concurrent thread terminates all the threads that originate from it.

Time in this paper is the wall time. We measure the time that passes from the initiation of a task until its termination. A task is a nonempty set of threads, denoted by $\left\{{C_{J}:J\in\mathfrak{J}}\right\}$. Its time is

When a task consists of a single thread, we omit the curly brackets, namely,

Note that two threads, denoted by $C_{J}$ and $C_{J^{\prime}}$, may run concurrently and overlap in time. Hence, it is possible that $\max\left\{{T_{\text{wall}}\left[C_{J}\right],T_{\text{wall}}\left[C_{J^{\prime}}\right]}\right\}\leq T_{\text{wall}}\left[\left\{{C_{J},C_{J^{\prime}}}\right\}\right]<T_{\text{wall}}\left[C_{J}\right]+T_{\text{wall}}\left[C_{J^{\prime}}\right]$. However, if $C_{J}$ and $C_{J^{\prime}}$ do not overlap in time, then $T_{\text{wall}}\left[\left\{{C_{J},C_{J^{\prime}}}\right\}\right]\geq T_{\text{wall}}\left[C_{J}\right]+T_{\text{wall}}\left[C_{J^{\prime}}\right]$.

This section lays out a theoretically sound framework to parallelize SI (Leviathan et al., 2023; Chen et al., 2023; Miao et al., 2023). The naive version of our method assumes access to a sufficiently large number of processors so that threads never have to wait. Later, we discuss implementing our method with a fixed number of processors.

Before presenting our algorithm, we first discuss the limitations of existing SI methods. SI reduces target forwards whenever a draft token is accepted. With an accurate and fast drafter, SI can significantly cut the number of target forwards, potentially speeding up the inference compared to non-SI. For instance, if on average one draft is accepted per iteration, the number of target forwards drops to half since the average target forward generates two tokens. As long as the total drafting latency is less than the saved latency from reduced target forwards, SI offers a speedup over non-SI. The main limitation of existing SI methods is their sequential nature. Each SI iteration requires a target forward, and the next iteration only begins after the current one is completed. However, we observe that the verification of each SI iteration is not inherently sequential and could be parallelized.

Previous works on SI exemplified speedups using drafters that run within 1-5% of the time (compared to the target LLM), and iterations of 2-5 draft tokens (namely, $\texttt{lookahead}\in[2,5]$). We can calculate the maximum speedup of our proposed method compared to SI by Amdahl’s law, as follows. Assume the drafter is perfectly accurate. In that case, our method hides all the verification latency such that the overall end-to-end latency remains only the drafting latency. For drafters of 1-5% latency and $\texttt{lookahead}\in[2,5]$, our proposed parallelization leads to a theoretical speedup of 4x-50x compared to SI. Figure 1 and Table 1 illustrate potential speedups compared to SI and non-SI, given a drafter of  14% latency.

DSI can orchestrate any given number of servers. In its simplest nontrivial setup, DSI orchestrates a single node with two GPUs, implementing a target server on one GPU and a drafter server on the other. More advanced setups involve more GPUs, including those connected to other nodes, or combine DSI with data parallelism techniques so that servers can utilize multiple underlying GPUs. For example, a node with eight GPUs can run eight servers, each using one GPU, or four servers, each using two GPUs, employing model parallelism (MP) with tensor parallelism (TP) or pipeline parallelism (TP).

While the theoretical guarantees in section 3.2 hold for both single- and multi-node setups, our evaluations are confined to single-node scenarios with at most eight GPUs. Therefore, we implemented DSI as a multithreading system. For multi-node setups, implementing DSI as a multiprocessing system with a Message Passing Interface (MPI) would be more appropriate.

In both single- and multi-node setups, DSI can employ a thread pool design pattern, where verification tasks are sent to a pool of servers computing the target model. The size of this target pool is, by definition, the SP degree. Our implementation is based on a thread pool of targets and a single drafter server. In all our experiments, DSI has an SP degree of less than or equal to seven and one drafter server, where each server employs a single GPU.

Due to budget constraints, all experiments were conducted on a single NVIDIA A100 80GB GPU. To evaluate DSI over a node with eight GPUs without access to such hardware, we adjusted the DSI implementation accordingly. While DSI incurs all the real-world latencies typical of multithreading systems (e.g., context switching), each call to compute the forward pass of an LM was replaced by a wait command. The wait command blocks the thread for a duration that matches the actual latency.

To ensure realistic wait times, we conducted a separate experiment to estimate the Time to First Token (TTFT) and Time Per Output Token (TPOT) for each model and dataset. These TTFT and TPOT values were then used to set the wait times in the experiment. We also performed another separate experiment to estimate the acceptance rate for each combination of target, drafter, and dataset. For detailed descriptions of how these quantities are estimated, see Appendix A.

The reported speedup of DSI relative to SI ("Speedup DSI vs. SI") is the ratio between their estimated end-to-end latencies, including wall time, prefilling, and decoding latency, but excluding tokenization latency. The end-to-end latency is estimated as follows: we generate 50 tokens using each target-drafter pair on each dataset, employing real-world forward latencies and acceptance rate values from independent experiments, as detailed below. Each combination of target-drafter and dataset is run on multiple lookahead values (specifically, 1, 5, and 10). For the DSI run, we conduct a grid search over the lookahead and SP values where SP $\leq 7$, ensuring that DSI can be deployed on a single node with up to 8 GPUs. We implemented DSI in Python, using Python’s built-in thread pool to manage threads.

The results of our experiments, detailed in Table 2, affirm Theorems 1 and 2, demonstrating that DSI outperforms SI (Leviathan et al., 2023; Chen et al., 2023) in practical settings across various LLMs and well-known datasets. Overall, DSI consistently outperforms SI across all models and tasks. Table 2 also specifies the latency and acceptance rate estimates used in each configuration.

All LLMs were downloaded from the Hugging Face Hub and used as-is. The four well-established datasets used to estimate forward latencies and acceptance rates span various tasks: text summarization using CNN Daily Mail (Hermann et al., 2015); instruction-following using Alpaca (Taori et al., 2023a); code generation using MBPP (Austin et al., 2021) and HumanEval (Chen et al., 2021). Appendix D and Appendix C provide a complete description of the models, datasets, and examples of prompts.

We independently estimated the forward latencies and acceptance rates on a single NVIDIA A100 80GB GPU. Our implementation of DSI and the code for the experiments have been open-sourced.

Acceptance rate. To ensure our evaluation is realistic, we used real-world acceptance rates, calculated as follows. For any given combination of target, drafter, and dataset, we estimate their acceptance rate independently. We generate 256 tokens using the drafter given the same prompt used by the target. For each prompt, we consider the lengths of the longest sequences of exact token matches between the target and the drafter. Below is a simplified example where tokens are counted as English words. If the target generates “We can only see a short distance ahead, but we can see plenty there that needs to be done. […]” and the drafter generates “We can only see a short distance ahead, we done. […]”, then the longest sequence of exact matches is 8 tokens long. The expected number of accepted drafts is $\bar{n}:=\frac{1}{N}\sum_{i=1}^{N}n_{i}$ where $n_{i}$ is the number of accepted draft tokens for the $i$th prompt. The acceptance rate is then calculated as $\texttt{acceptance\_rate}:=1-\frac{1}{1+\bar{n}}$.

This experiment suggests that off-the-shelf LLM “families” like StarCoder (Li et al., 2023) or Vicuna (Zheng et al., 2024) can form good pairs of target and drafter. These families consist of LLM versions of different sizes that were trained similarly and on similar datasets. We observe that even relatively small drafters demonstrate good alignment with larger LLMs from the same family. For example, Starcoder-168M (drafter) and Starcoder-15B (target) yield an acceptance rate of 93%.

Ablation.  Figure 2 presents the results of a complementary simulation measuring the pairwise speedups (or slowdowns): DSI compared to SI, DSI compared to non-SI, and SI compared to non-SI. Unlike the experiment above (Table 2) that estimates the end-to-end wall time latencies and runs an implementation of DSI that is a multithreading system incurring all real-world latencies of such systems, in the simulation below, DSI is implemented without thread pools. Instead of measuring wall time, the simulation estimates the end-to-end latency by summing all the corresponding forward pass latencies. Here, as in the real-world experiment, every call to a target or a drafter is replaced with adding the realistic wait time to the total summed latency. This simulation is insensitive to the real-world latencies of multithreading systems, such as context switching, and therefore could be run in parallel among other simulations, allowing us to scale up the number of configurations explored and provide a comprehensive heatmap of the expected speedups. Otherwise, each of the 2,000,000 configurations presented in the heatmap requires a separate run. Our parallelized (rather than sequential) evaluation method makes it feasible to explore such a large number of configurations within our constrained computational budget. Since DSI can successfully hide the latency of almost all the target forwards, naive counting of the forward pass latencies cannot estimate the end-to-end latency accurately. Instead, the simulation is based on an analysis that sums the forward pass latencies only if they are not hidden by the speculative parallelism. The only randomness in the simulation is the acceptance rate of the drafter. Since SI is slower than non-SI in some configurations, we have included an additional comparison that shows DSI speedups relative to the faster of the two algorithms—SI or non-SI—for any given configuration. This experiment helps identify configurations where DSI achieves the highest speedup. It demonstrates that, unlike SI, our method introduces no slowdown compared to non-SI and consistently accelerates inference. In all runs, DSI uses no more than eight GPUs, ensuring that it remains useful in practical single-node settings. Therefore, to control the required number of processors that are capable of computing the target, ensuring SP $\leq 7$, we set the lookahead hyperparameter such that $\texttt{ceil}(\frac{\texttt{target\_latency}}{\texttt{lookahead}\cdot\texttt{drafter\_latency}})\leq 7$. We say that such a lookahead is feasible. In all the simulations, we only used feasible lookahead values. For every configuration, we consider the latency of SI with lookahead $\in\left\{{1,2,\ldots,200}\right\}$ that minimizes its latency for that configuration. The configurations considered are the cartesian product of drafter latency ($0.01,0.02,\ldots,1$), acceptance rate ($0,0.01,0.02,\ldots,1$), and lookahead ($1,2,\ldots,200$). For each combination of (drafter latency, acceptance rate, and lookahead ), we run 5 repeats of SI and average the results to estimate the expected latency of SI (Appendix E provides the code for the simulation of SI). Then, for each combination of (drafter latency, acceptance rate), we consider the minimal latency (letting SI select its optimal lookahead ). Since the lookahead is a tunable parameter, our experiment assumes that it will be optimized by the user so that SI is optimized. It is known (and trivial) that SI is highly sensitive to the choice of the lookahead . To calculate the speedup of algorithm A over algorithm B per (drafter_latency, acceptance_rate), we divide the latency of B by the latency of A. The speedups are not smooth for drafter latencies $<20\%$ due to the discretization of the lookahead hyperparameter. Theoretically, if we fix the lookahead and decrease the drafter latency to 0, the number of servers required by DSI grows to infinity. However, given an arbitrary number of available servers, it is possible to tune the lookahead hyperparameter such that DSI orchestrates only available servers (as we did, with eight servers where seven of which are target servers and one is a drafter server). For example, the speedups for $\text{{lookahead} }=5$ are smooth for both algorithms as seen in Appendix E.3. As shown in Figure 2(a), to achieve a speedup with SI compared to non-SI, the acceptance rate of the drafter must at least match the latency of the drafter model (the region above the pink region in the figure). This means that the SI algorithm cannot speed up the inference if the acceptance rate of the drafter is not sufficiently high for a given latency. Conversely, in Figure 2(b), we observe that DSI consistently speeds up the inference time, regardless of the latency and acceptance rates of the drafter. This provides our method with much greater flexibility and robustness. In Figure 2(c), we observe that DSI is faster than non-SI for all the configurations for which non-SI is faster than SI. Finally, to obtain a comprehensive view of the inference speedup achieved by DSI, in Figure 2(d), we compare the performance of DSI with the minimal runtime of both SI and non-SI.

This work studies how to reduce the run time of speculative inference algorithms by taking advantage of an arbitrary number of additional multiple processing units (e.g., GPUs). We have shown that in contrast to their empirical success, traditional SI algorithms can end up slowing the inference of LMs in various practical settings. For instance, when the drafters are insufficiently accurate or the drafter is too slow. We showed that by taking advantage of at least one additional GPU, we can design a speculatively inference algorithm that provably reduces the inference time of both non-SI and SI algorithms. Our simulations affirm our theory, indicating significant speedups over both SI and non-SI for all possible configurations given a single node with up to eight GPUs. In essence, this work paves the way to additional SI algorithms that can orchestrate multiple processing units at the same time via speculation parallelism (SP).

Limitations.  We introduce DSI and show it is faster than SI and non-SI for all possible configurations by theoretical analysis and experiments. While the theoretical guarantees hold for both single- and multi-node setups, our experiments cover only single-node scenarios with up to eight GPUs with SP $\leq 7$. Due to budget constraints, we adjusted our implementation of DSI to simulate an access to such a node rather than running on a physical node with eight GPUs. Nevertheless, the empirical results are realistic because DSI is implemented as a multithreading system, incurring all real-world latencies of such systems (e.g., context switching), and all the real-world GPU-related latencies are based on independent experiments accessing a GPU.

Broader Impacts.  DSI introduces a new tradeoff: reducing the inference latency by utilizing more computing resources. Hence, adopting DSI as an alternative to SI or non-SI algorithms will increase the overall computing resources consumption of LLM-based applications.

We thank Intel Labs for funding this research.