CopySpec: Accelerating LLMs with Speculative Copy-and-Paste
Without Compromising Quality

Speculative decoding is an effective approach for accelerating inference in LLMs by parallelizing token generation and verification. Leviathan et al. (2023) introduced the foundational framework, employing a small draft model to propose multiple tokens that a larger model verifies, significantly reducing inference latency. Medusa (Cai et al., 2024) expanded this idea by leveraging multi-head decoding to enable simultaneous token generation and verification, improving throughput.

Dynamic verification pipelines balance speed and accuracy by adjusting verification depth based on output quality (Liu et al., 2024). Token tree verification accelerates serving (Miao et al., 2023), while pipelined exact decoding handles compute-latency trade-offs (Yang et al., 2023). Knowledge distillation enhances draft–target model interaction (Zhou et al., 2023), and retrieval-based token validation improves efficiency (He et al., 2023). Speculative decoding has been further optimized in recent works. SpecHub (Sun et al., 2024) uses optimal transport to improve draft token acceptance rates, and SPEED (He & Wang, 2023) leverages early-layer hidden states for parallel token execution.

While speculative decoding enables efficient token generation, our work addresses a distinct challenge: leveraging predictable token patterns without introducing significant additional computation. CopySpec acts as an intelligent copying mechanism within the speculative decoding framework, reducing redundancy and improving efficiency across various tasks. By identifying and reusing repeated patterns in the context, CopySpec not only accelerates inference but also complements speculative decoding by extending its applicability to scenarios with high redundancy, such as multi-turn interactions and tasks with self-correction. This integration demonstrates the potential of combining these techniques to achieve greater efficiency in large-scale language models.

Copying mechanisms are widely adopted in NLP to handle tasks that require replicating predictable patterns or segments. Gu et al. (2016) introduced CopyNet, a method that enables RNN sequence-to-sequence models to predict words based on a mixed probabilistic model of two modes, where one selects words from the source sequence. Similarly, in summarization tasks, Pointer Networks (Vinyals et al., 2015) and Pointer-Generator Networks (See et al., 2017) demonstrated the effectiveness of combining copying and generation to improve output fidelity and handle out-of-vocabulary tokens.

More recently, McCoy et al. (2023) analyzed the extent to which transformers copy from their training data, providing insights into copying behaviors in modern LLMs. Jelassi et al. (2024) showed that transformers outperform state space models in copying repetitive patterns.

Lastly, in a different domain, Andronov et al. (2024) introduced a copying mechanism into a transformer-based encoder-decoder that models chemical reactions by observing that portions of the input chemicals often remain unchanged in the output.

While previous works have emphasized the importance of copying mechanisms in various applications, our work is the first to explore this concept in the specific context of LLM inference. CopySpec integrates a copying mechanism into speculative decoding, effectively reducing redundancy and enhancing efficiency across a wide range of tasks. By leveraging repeated patterns in the model’s context, CopySpec introduces a novel approach to accelerate inference while maintaining high performance.

Fill-in-the-Middle (FIM) enables language models to generate text segments within a given context, enhancing flexibility in tasks such as text and code infilling. Bavarian et al. (2022) introduced a data transformation approach for autoregressive models to learn infilling without sacrificing left-to-right generative performance, while Shen et al. (2023) proposed FiLM, enabling flexible generation by masking arbitrary positions.

In code generation, FIM techniques are crucial for editing and repair tasks. Models like Code Llama (Roziere et al., 2023) and InCoder (Fried et al., 2023), a utilize bidirectional context for structured prompts, achieving state-of-the-art results on benchmarks such as HumanEval. Frameworks such as Self-Infilling (Zheng et al., 2024) and benchmarks like SAFIM further enhance these methods with backward generation and syntax-aware metrics (Wang et al., 2023; Gong et al., 2024). Recent advancements, models like Codestral and CodeGemma, refine FIM techniques to improve alignment (Mistral AI, 2024; Team et al., 2024).

However, it is important to emphasize the distinct advantages of our method compared to the FIM approach. Unlike FIM, which relies on labeled tokens such as <prefix> and <suffix> to guide the model in fixing a specific section of code bidirectionally. Our method operates label-free, enabling a more flexible and generalizable approach. Additionally, while FIM is constrained to modifying a single code segment (typically the middle), CopySpec extends this capability by allowing modifications in multiple regions, such as quite a few distinct places within the input. Furthermore, we maintain the architectural simplicity of a left-to-right LLM, ensuring that our method remains compatible with existing LLM frameworks while offering significant improvements in efficiency and versatility.

To efficiently detect when the model begins generating a block that has already been produced earlier, we maintain a hash map containing all subsequences of $\gamma$ tokens from the context. During the generation process, we search this hash map for matches to the last $\gamma$ tokens generated. Adding a new tuple of tokens to the hash map and searching for a match after each generated token has a time complexity of $O(\gamma)$. Since $\gamma$ is typically set to a small value (e.g., $3$ or $5$), the computational overhead for processing new tokens and finding matches is minimal and independent of the context size. This stands in contrast to alternative approaches that require searching the entire context for the last substring, which can become computationally expensive as the context grows.

Our technique efficiently leverages larger contexts, allowing inference to become faster as the context size increases. By keeping $\gamma$ fixed, we ensure a balance between efficiency and precision. Additionally, we explored methods to utilize partial outputs without revealing the complete results and investigated how the semantic relationship between the preceding $\gamma$ tokens and the subsequent token can guide the optimal choice of $\gamma$. Further details are provided in Appendix A.

After identifying a match of $\gamma$ tokens in the context, we extract the subsequent tokens from the context, as shown in Figure 2. These extracted tokens, which we call $S_{speculate}$, essentially simulate the behavior of a draft model where the probability for each token in $S_{speculate}$ is treated as 100%.¹¹1In cases where multiple matches exist for the last $\gamma$ tokens, we simplify the process by selecting the first match, though we acknowledge that alternative strategies could improve efficiency.

$S_{speculate}$ is then verified directly by the LLM. Each verification yields $\tau$ tokens that align with the LLM’s ongoing generation, along with one additional guaranteed token. This approach mirrors vanilla speculative decoding (Leviathan et al., 2023), where speculative tokens are appended to the context, and the longest prefix matching the LLM’s output is accepted. In Figure 2, $S_{speculate}$ is highlighted in blue. The output shows the $\tau$ accepted tokens in green, the extra guaranteed token in gold, and any rejected tokens in red. This process effectively treats the copied tokens as a “perfect prediction,” ensuring efficient token generation when patterns are detected.

After each newly generated token or copying attempt, we re-evaluate the last $\gamma$ tokens in the context to identify a new match, allowing the model to utilize longer copyable blocks whenever possible. This eliminates the need for manual token generation between copying steps.

If any tokens in $S_{speculate}$ fail the verification step, the model generates a new token that diverges from the previously matched tokens. This ensures that the next copying attempt yields a different match, preventing the model from getting stuck in repetitive loops. Furthermore, we always use a temperature of 0 to maintain the original output distribution of the model and ensure that our technique does not introduce any stochasticity into the generation process.

To further enhance our technique, we have integrated it within a vanilla Speculative Decoding framework. At each step of the generation process, we attempt to find matches in the context. If a match for the last $\gamma$ tokens is found, we use $S_{speculate}$ as draft tokens, effectively simulating a draft model with perfect confidence in those tokens. If no match is identified, we rely on a smaller draft model to generate $\tau_{2}$ draft tokens. This dual approach allows us to dynamically choose between leveraging repetitive patterns through CopySpec and utilizing speculative decoding for efficient token generation in contexts with little or no redundancy.

This integration provides the best of both worlds: Speculative Decoding accelerates inference when the context size is small or lacks redundancy, while CopySpec builds on this speed-up in subsequent steps by taking advantage of repetitive patterns as the context size increases. As a result, the combined approach significantly enhances model efficiency across diverse scenarios.

It is also worth noting that when used as a stand-alone method, CopySpec does not require a draft model. This eliminates the need for additional GPU memory or modifications to the model, making it lightweight and easy to deploy. We explore the interplay between these techniques in Section 6, while Appendix B provides a detailed account of the full implementation, including key-value caching.

We evaluated our copying technique on five instruction-tuned LLMs: Qwen2.5-72B, Qwen2.5-32B, Qwen2.5-7B (Qwen et al., 2025), LLaMa3.1-70B, and LLaMa3.1-8B (Grattafiori et al., 2024), using 4 A100 GPUs with a batch size of 1. Unless stated otherwise, we set $\gamma$ to 3, $|S_{speculate}|$ to 10, the max generation length to 1024, and the temperature to 0.

We evaluated our technique on five datasets, each targeting specific aspects of model performance: MT-Redundant, CNN/DM, GSM-8K, MT-Bench, and HumanEval. MT-Redundant was designed to emphasize prompts requiring small variations to previous outputs, while CNN/DM focuses on extractive summarization. GSM-8K evaluates the model’s self-correction capabilities, MT-Bench highlights scenarios with minimal copying potential to measure the technique’s overhead, and HumanEval assesses coding capabilities. To accommodate the increased computational demands of GSM-8K and CNN/DM and our limited GPU resources, we restricted these datasets to 100 samples, ensuring they were of comparable size to the other datasets. For HumanEval, we employed the same instruction format as presented in EvalPlus (Liu et al., 2023). Detailed descriptions of all prompts used in our experiments are provided in Appendixes G and F.

Most existing NLP datasets focus on tasks involving either single-turn interactions or scenarios where the model must entirely change its response in the second turn. These setups fail to capture realistic use cases where a user might request slight variations or refinements to a previous answer. To address this gap and highlight the capabilities of our technique, we introduce a new dataset, MT-Redundant.

MT-Redundant is derived by modifying the second turn of MT-Bench (Zheng et al., 2023). In our dataset, the second turn replaces the original question with a prompt asking the model to review its previous answer and make specific adjustments or variations. This modification simulates real-world scenarios where incremental refinement or elaboration is required. Example prompts from the dataset are provided in Appendix F. For questions with reference answers, we retained the original reference for the first turn and created a new reference answer for the second turn to align with the revised prompts.

Our dataset spans a diverse range of practical use cases, categorized into eight groups: Coding, Extraction, Humanities, Math, Reasoning, Roleplay, STEM, and Writing. These categories reflect realistic tasks encountered in various domains. Additionally, we adopted the same evaluation procedure from MT-Bench to ensure consistency and comparability of results.

By creating MT-Redundant, we aim to bridge the gap between artificial benchmarks and practical applications, providing a more representative evaluation for techniques like CopySpec in multi-turn interactions with reptitive information.

We begin our analysis by examining the speedups achieved on MT-Redundant for both the first and second turns, as summarized in Table 2. The results indicate a substantial average speedup of 2.04$\times$ for the second turn, compared to a more modest speedup of 1.08$\times$ for the first turn. Notably, the performance in tokens per second (TPS) achieved by the model increases for the second turn, which features a larger context size. In contrast, the baseline model experiences a decline in TPS as the context size increases. Another notable aspect is that the observed speedup is highly dependent on the specific use case. For instance, we observe speedups as low as 1.2$\times$ in the Humanities category and as high as 3.08$\times$ for Roleplay. However, regardless of the use case, the speedup for the second turn remains consistently positive across all models for both MT-Redundant and MT-Bench.

The results for all five models on MT-Redundant and MT-Bench are detailed in Appendix C.2 and D.2 respectively. On average, the second round of MT-Redundant achieves a significant 91% speedup across all models, compared to 31% for MT-Bench. Notably, even on MT-Bench, which has less redundancy, the TPS achieved by CopySpec in the second turn is almost always higher than the baseline model’s TPS in the first turn. These findings highlight how our approach effectively leverages increased context sizes to enhance performance, even in less favorable scenarios.

We begin our analysis with Figure 3, which illustrates the tokens per second as a red line, alongside the percentage of tokens copied out of the total tokens generated, represented by a blue line for the LLaMa3.1-8B model on HumanEval. The numbers adjacent to the dots indicate the number of attempts made to copy tokens. The figure demonstrates that as $\gamma$ decreases, a higher percentage of tokens is accepted, but the number of copying attempts increases exponentially, leading to a significantly larger overhead. This results in a decline in overall TPS performance. A similar pattern is observed for MT-Redundant and MT-Bench, as presented in Figure 6 and Figure 7 in the appendix.

Empirically, the optimal value of $\gamma$ across datasets is three, with two yielding similar performance. It is also worth noting that all $\gamma$ values ranging from 2 to 10, consistently results in significantly higher overall TPS, even across both turns on MT-Redundant and MT-Bench.

Furthermore, we examine the effect of $\gamma$ on $\tau$ (the average number of tokens accepted). Figure 4 illustrates the average number of tokens accepted per attempt on HumanEval using the LLaMA3.1-8B model. We observe an interesting pattern: as $\gamma$ increases, the average number of tokens accepted per copying attempt also increases, indicating that each attempt becomes more precise. However, this comes at the cost of fewer overall copying attempts, as demonstrated in Figure 3.

This finding is particularly relevant for integrating our technique into various speculative decoding frameworks. If a framework already accepts a high number of tokens per attempt, our technique remains advantageous by increasing $\gamma$, enabling more tokens to be copied with each attempt.

We evaluate the impact of the number of tokens copied on performance and estimate CopySpec’s overhead by setting the number of copied tokens to zero, isolating the cost of token searching. Results in Table 4 show minimal overhead with differences from the base model nearly within the margin of error. Among the hyperparameters studied, setting $|S_{speculate}=10|$ delivers the best performance, while larger values, such as 50 or 100, increase overhead and reduce tokens-per-second (TPS) efficiency.

We followed the steps outlined in Section 3.3 to integrate our technique into a vanilla speculative decoding framework, as described in (Leviathan et al., 2023). Based on our observations from Section 5.2, we experimented with two different values of $\gamma$ (3 and 5) to analyze their impact on performance when used alongside speculative decoding.

We integrated CopySpec into a vanilla speculative decoding framework, following the steps in Section 3.3 and the approach described in (Leviathan et al., 2023). Experiments with $\gamma$ values of 3 and 5, summarized in Table 3, show significant efficiency improvements in the second turn of MT-Redundant, with marginal speedups in the first turn. A $\gamma$ value of 5 achieves higher speedups in the first turn, while $\gamma=3$ provides better TPS in the second turn, highlighting the need for task-specific tuning.

We also evaluated CopySpec with speculative decoding using drafts of 5 tokens instead of 3, with similar experiments conducted on MT-Redundant (Table 10, in Appendix) and with 3 and 5 draft tokens on MT-Bench (Table 16 and Table 17, in Appendix). These results confirm that $\gamma=5$ often outperforms $\gamma=3$ when combined with Spec. Dec., emphasizing the importance of tuning $\gamma$ for optimal performance. The results also show that adding CopySpec to Spec. Dec. almost never leads to a decrease in performance even if there is little redundancy in the data, as seen in MT-Bench.

CopySpec is also compatible with newer speculative decoding frameworks (Cai et al., 2024; Li et al., 2024), where tuning $\gamma$ ensures copying occurs when confidence is high, showing the method’s adaptability to various scenarios.

An important aspect of our analysis is evaluating the impact of our technique on the efficiency of self-correction. To this end, we implemented a self-refine framework, where the model generates Python code and iteratively refines it in two steps, following a process similar to (Madaan et al., 2023). Details of the prompts and example outputs used in our experiments are provided in Appendix G.1. Table 5 presents the results of combining self-correction with copying and Speculative Decoding (SD).

Our technique becomes more effective in later turns as the model iterates over its prior reasoning, allowing progressively more tokens to be copied. This is reflected in a significant rise in the percentage of copied tokens, tokens per second (TPS), and $\tau_{1}$, the average number of tokens accepted. Each copying attempt also becomes more precise as the model refines its reasoning and the context grows.

When combined with SD using $\gamma=5$, our approach achieves better results across all three turns, as shown in the table. The first turn benefits most from SD due to minimal copying, while later turns gain greater advantages from copying. This highlights the complementary nature of the two techniques and their combined effectiveness in improving efficiency and performance. Notably, while the TPS of the base model decreases by 0.85$\times$ as context size grows, our technique reverses this trend, increasing the TPS in the last turn by 2.52$\times$ , showcasing its ability to leverage larger contexts for enhanced efficiency.

We also extended our analysis to cases where the draft model generates 5 tokens at a time, as shown in Table 18 in the appendix. Additionally, Table 19 confirms that the tested models improve their final accuracy, validating the effectiveness of our self-correction implementation. Note that accuracy is not reported for the second round, as it focuses solely on critiquing the model’s prior implementation. Across the entire self-correction process, we achieve TPS improvements of 63%, 52%, and 54% for the Qwen2.5-7B, Qwen2.5-32B, and Qwen2.5-72B instruct models, respectively.