DFRot: Achieving Outlier-Free and Massive Activation-Free for Rotated LLMs with Refined Rotation

The initial idea behind suppressing outliers through scale invariance stems from the observation that weights are easier to quantize than activations, and outliers in activations often appear in a few fixed channels Dettmers et al., 2022. Based on this, SmoothQuant (Xiao et al., 2023) first proposes that we can offline migrate the quantization difficulty from activations to weights via scale invariance. SmoothQuant enables an INT8 quantization of both weights and activations for all the matrix multiplications in LLMs. Furthermore, Outlier Suppression+ (Wei et al., 2023) proposes a fast and stable scheme to effectively calculate scaling values, achieving a better balance in quantization burden. To reduce manual design and further enhance quantization performance in extremely low-bit quantization, OmniQuant (Shao et al., 2023) introduces Learnable Weight Clipping and Learnable Equivalent Transformation, efficiently optimizing the quantization process for both weight-only and weight-activation quantization. In the clipping W4A8 quantization, QQQ (Zhang et al., 2024) proposes to dynamically handle outliers through adaptive smoothing. QServe (Lin et al., 2024b) proposes SmoothAttention to effectively mitigate the accuracy degradation caused by 4-bit KV quantization. Both QQQ and QServe have effectively enhanced the performance of LLMs in W4A8 quantization.

Although scale invariance can reduce outliers and improve quantization performance, it merely transfers the outliers from activations to weights and has not eliminated them fundamentally. When the magnitude of the outliers is large, scaling struggles to achieve an effective balance between weights and activations. Recently, researchers have found that applying rotation matrices to networks can effectively reduce outliers without increasing the complexity of LLMs. QuIP Chee et al. (2024) is the first to suggest that quantization can benefit from the incoherence between weight and Hessian matrices. It employed randomized orthogonal matrices generated by Kronecker product to enhance their incoherence. QuIP# (Tseng et al., 2024) replaces the randomized orthogonal matrices with randomized Hadamard matrices, which are faster and possess better theoretical properties. QuaRot (Ashkboos et al., 2024b) is the first work to apply rotational invariance (Ashkboos et al., 2024a) for model quantization. QuaRot finds that randomized Hadamard transformations yield better results compared to randomized orthogonal transformations. SpinQuant (Liu et al., 2024) further extends the rotation matrices to a trainable space and applied Cayley optimization (Li et al., 2020) to refine them, achieving significant improvements across diverse datasets.

To remove outliers in the input activations ${\bm{X}}_{1}$, a rotation matrix ${\bm{R}}_{1}$ is applied to the input matrix ${\bm{X}}^{1}$, resulting in a new input activation ${\bm{X}}_{1}{\bm{R}}_{1}$. ${\bm{R}}_{1}$ satisfies ${\bm{R}}_{1}{\bm{R}}_{1}^{T}={\bm{R}}_{1}^{T}{\bm{R}}_{1}={\bm{I}}$ and $|{\bm{R}}_{1}|=1$. Using the LLaMA architecture as an example, ${\bm{X}}_{1}{\bm{R}}_{1}$ is then passed to the RMSNorm, which satisfies the commutation property: $\text{RMSNorm}({\bm{X}}_{1}{\bm{R}}_{1})=\text{RMSNorm}({\bm{X}}_{1}){\bm{R}}_{1}$ (Ashkboos et al., 2024a). Here, we assume that RMSNorm operates on each row $i$ of the activations ${\bm{X}}_{1}$ as ${\bm{X}}_{1,i}\leftarrow{\bm{X}}_{1,i}/\left|{\bm{X}}_{1,i}\right|$. This commutation property implies that multiplying the input of RMSNorm by ${\bm{R}}_{1}$ is equivalent to multiplying the RMSNorm output by ${\bm{R}}_{1}$ as well.

The output of LayerNorm is then passed into the subsequent linear blocks. With the introduction of ${\bm{R}}_{1}$, the input to these linear layers is altered. To ensure that the output from the linear layers remains unchanged, ${\bm{R}}_{1}^{T}$ is multiplied by the weight matrix ${\bm{W}}$, resulting in a new weight matrix ${\bm{R}}_{1}^{T}{\bm{W}}$, which can be calculated offline. Since ${\bm{R}}_{1}^{T}{\bm{R}}_{1}={\bm{I}}$, the output from the linear layer remains unaffected. This computational invariance property of LLMs ensure the introduction of the rotation matrices without changing the original results.

A similar approach can be applied to rest layers within an LLM block. As shown in Figure 1, by transforming the weight matrices in the Multi-Head Attention (MHA) as ${\bm{R}}_{1}^{T}{\bm{W}}_{q}$, ${\bm{R}}_{1}^{T}{\bm{W}}_{k}$, ${\bm{R}}_{1}^{T}{\bm{W}}_{v}$, and ${\bm{W}}_{o}{\bm{R}}_{1}$, and the weights in the Feed-Forward Network (FFN) as ${\bm{R}}_{1}^{T}{\bm{W}}_{up}$, ${\bm{R}}_{1}^{T}{\bm{W}}_{gate}$, and ${\bm{W}}_{down}{\bm{R}}_{1}$, the hidden features within both MHA and FFN remain unchanged. Consequently, the output feature ${\bm{Y}}_{1}$ is transformed into ${\bm{Y}}_{1}{\bm{R}}_{1}$, which will sum with the residual input ${\bm{X}}_{1}{\bm{R}}_{1}$ satisfies ${\bm{X}}_{1}{\bm{R}}_{1}+{\bm{Y}}_{1}{\bm{R}}_{1}=({\bm{X}}_{1}+{\bm{Y}}_{1}){\bm{R}}_{1}={\bm{X}}_{2}{\bm{R}}_{1}$. The output will serve as the input for the next LLM block. Similarly, by transforming ${\bm{W}}_{lm\_head}$ to ${\bm{R}}_{1}^{T}{\bm{W}}_{lm\_head}$, the network output will remain unchanged.

Moreover, we can introduce additional rotation matrices to further mitigate outliers between layers. As illustrated in Figure 1, head-wise rotation matrices ${\bm{R}}_{2}$ and ${\bm{R}}_{2}^{T}$ can be applied to ${\bm{W}}_{v}$ and ${\bm{W}}_{o}$, while ${\bm{R}}_{3}$ can be inserted for Query and Key after RoPE. Additionally, ${\bm{R}}_{4}$ and ${\bm{R}}_{4}^{T}$ can be placed between the Swish activation and ${\bm{W}}_{down}$. These strategies help further suppress outliers and reduce quantization error without affecting the block’s output. In this paper, we focus exclusively on ${\bm{R}}_{1}$. For ${\bm{R}}_{2}$, ${\bm{R}}_{3}$, and ${\bm{R}}_{4}$, we adopt the settings from QuaRot (Ashkboos et al., 2024b) by setting them to random Hadamard matrices.

Based on the computational invariance described in Section 3.1, it is evident that the choice of rotation matrices is critical for ensuring the accuracy performance of the quantized model. Therefore, a natural question arises: What type of rotation matrix offers the most advantageous properties? We begin by focusing on $\mathrm{RO}$ and $\mathrm{RH}$, as both QuaRot (Ashkboos et al., 2024b) and SpinQuant (Liu et al., 2024) have shown that the latter delivers substantial improvements over the former in LLMs. We conducted experiments by applying $\mathrm{RO}$ and $\mathrm{RH}$ to the LLaMA and Mistral models, followed by weight quantization using GPTQ under various settings. The results are shown in Table 1, benefiting from the outlier elimination through rotation, we find that for 8-bit activation quantization, both $\mathrm{RO}$ and $\mathrm{RH}$ lead to significant performance improvements compared to standard quantization. Additionally, no substantial difference is observed between the two methods. However, under 4-bit token-wise activation quantization, RH significantly outperforms RO.

To investigate the performance differences between $\mathrm{RH}$ and $\mathrm{RO}$ under 4-bit activation setting, we plot the corresponding quantization error after applying 4-bit quantization to the multiple tokens. We also display the quantization error for the baseline setting where quantization is applied without rotating the activation to better understand the impact of using the rotation matrix. As shown in Figure 4, compared to the no rotation ($\mathrm{NR}$), both $\mathrm{RO}$ and $\mathrm{RH}$ effectively reduce the quantization error for most tokens across different models. While $\mathrm{RH}$ slightly lowers the quantization error, the difference between the two methods is minimal for the majority of tokens. This leads to the question: What explains the significant difference in accuracy during quantization when their quantization errors are so similar?

To answer this question, we turn our attention to massive activation (Sun et al., 2024), a rare but significant feature in LLMs. Since each token has a fixed $L_{2}$ norm after RMSNorm processing, tokens with massive activation naturally exhibit smaller quantization errors when quantized to 4-bit. As shown in Figure 4, the red points represent tokens with massive activation. While most tokens show large quantization errors under NR, these special tokens display significantly smaller errors, which can be observed from Figure 4. Figure 4 presents the quantization error distribution for tokens with massive activation after applying $\mathrm{RO}$, $\mathrm{RH}$, and $\mathrm{NR}$. Surprisingly, the rotation operations do not significantly reduce quantization errors for these tokens. In fact, compared to NR, $\mathrm{RO}$ greatly increases their quantization error, while $\mathrm{RH}$ only marginally reduces it. This leads us to question whether tokens with massive activation are the primary cause of the significant accuracy discrepancies between $\mathrm{RH}$ and $\mathrm{RO}$.

To investigate this further, we build upon QuaRot by retaining tokens with massive activations in FP16 format for both $\mathrm{RO}$ and $\mathrm{RH}$, while applying 4-bit quantization to the remaining input tokens. Therefore, we can conclude that the fundamental reason for the performance disparity between $\mathrm{RO}$ and $\mathrm{RH}$ is that RH more effectively reduces the quantization error for tokens with massive activations in 4-bit activation quantization.

The evaluation results in Section 3.2 show that applying 4-bit quantization to activations leads to significant quantization errors due to the large volume of activations, ultimately causing accuracy degradation. While encoding these activations more precisely could alleviate the issue, it results in a mixed quantization approach that is not well-suited for current GPU platforms. A good rotation matrix ${\bm{R}}_{1}$ should minimize the different between the original input ${\bm{x}}$ and its quantized version, namely:

where ${\bm{x}}\in\mathcal{R}^{C}$ is the token vector from a calibration dataset ${\bm{X}}^{cal}$, $C$ is the number of channels. ${\bm{R}}_{1}$ satisfies ${\bm{R}}_{1}{\bm{R}}_{1}^{T}={\bm{I}}$, ${\bm{g}}$ is the quantization parameters and $\mathcal{Q}_{{\bm{g}}}(\bm{x})$ is the quantization representation of the ${\bm{x}}$. The size of $\mathcal{Q}_{g}(\bm{x})$ is the same to the ${\bm{x}}$. The expectation $\mathbb{E}\left[\cdot\right]$ is taken over the token distribution. For simplicity in analysis, we utilize the mean squared error, denoted as $\|\cdot\|_{2}$.

To better adapt ${\bm{R}}_{1}$ to the massive activations, we adjust it by optimizing the following loss function:

where ${\bm{X}}^{m}\subseteq{\bm{X}}^{cal}$ denotes the subset of tokens with massive activations, while ${\bm{X}}^{cal}\setminus{\bm{X}}^{m}$ represents the remaining tokens. During calibration, we apply a weighted loss to prioritize the quantization error on tokens with massive activations, with $\gamma$ representing the weight.

The motivation behind this principle stems from the observations in Table 1. Since ${\bm{X}}^{m}$ is the key factor contributing to the performance gap between $\mathrm{RO}$ and $\mathrm{RH}$. Simply optimizing ${\bm{R}}_{1}$ over the entire ${\bm{X}}^{cal}$ fails to specifically target ${\bm{X}}^{m}$. Additionally, compared to the $\mathrm{NR}$ approach in Table 1, $\mathrm{RO}$ also significantly improves performance, indicating that reducing the outliers on ${\bm{X}}^{cal}\setminus{\bm{X}}^{m}$ can enhance the performance of the quantization method. However, optimizing only for ${\bm{X}}^{m}$ risks overfitting, which could increase the quantization error for ${\bm{X}}\setminus{\bm{X}}^{m}$, ultimately degrading the model’s overall performance. Hence, it is crucial to optimize both ${\bm{X}}^{m}$ and ${\bm{X}}\setminus{\bm{X}}^{m}$. Using a weighted approach to optimize the quantization loss is a straightforward yet highly effective method. Ablation studies in Section 4.2 further demonstrate the advantages of this strategy.

Optimizing ${\bm{R}}_{1}$ is a challenging task. Since ${\bm{R}}_{1}$ influences every MHA and FFN in the network, adjusting the activation distribution in one layer impacts the quantization outcomes across all layers. This makes it difficult to optimize layer by layer or block by block (Shao et al., 2023; Wei et al., 2023). A straightforward approach is to use training methods for quantization-aware fine-tuning of the rotation matrix across the entire network (Liu et al., 2024). However, this approach necessitates fine-tuning the entire network. Although it does not require retaining the gradients of the weights or the corresponding states in the optimizer, it still demands substantial computational resources during the quantization process.

In this paper, we focus on improving the effectiveness of rotation matrices in mitigating outliers in activation values. Intuitively, we hypothesize that a rotation matrix that minimizes quantization error will lead to fewer activation outliers and, consequently, better performance. Drawing inspiration from Simsiam (Chen & He, 2021), we propose to regard quantization representation $\mathcal{Q}\bm{{}_{{\bm{g}}}(x{\bm{R}}_{1})}$ as cluster centroids $\bm{\eta_{x}}$. In the context, optimizing ${\bm{R}}_{1}$ and $\bm{g}$ is equivalent to optimizing ${\bm{R}}_{1}$ and $\bm{\eta}$, which can be viewed as an implementation of an Expectation-Maximization (EM)-like algorithm, as shown in the following equation:

where $\bm{\eta_{x}}=\mathcal{Q}\bm{{}_{{\bm{g}}}({\bm{x}}{\bm{R}}_{1})}$. This formulation is analogous to k-means clustering (Macqueen, 1967), and ${\bm{R}}_{1}$ acts like the kernel function, representing the learnable rotation matrix. Similar to k-means clustering, the problem described in Eq 3 can be approached using an alternating algorithm, where one set of variables is fixed while solving for the other. Formally, we can alternate between solving these two subproblems:

where $t$ represents the iteration index of the alternating rounds, and $\bm{\eta}^{t}$ and ${\bm{R}}_{1}^{t}$ denote the values of $\bm{\eta}$ and ${\bm{R}}_{1}$ at round $t$.

Choice of $\bm{\gamma}$. To further understand the effect of hyperparameters in DFRot, we conducted an ablation study on Wikitext-2 PPL to investigate the impact of different $\gamma$ settings for W4A4KV4 and W4A4KV16. As seen in Figures 8 and 8, when $\gamma$ ranges between 50 and 200, DFRot achieves significant improvements across various LLaMA models using $\mathrm{RH}$. Notably, on the LLaMA3-8B model, known for its quantization challenges, we observed a PPL improvement of over 0.2. If we set $\gamma=1$ and treat ${\bm{X}}^{m}$ and ${\bm{X}}\setminus{\bm{X}}^{m}$ equally to minimize their quantization errors, it may reduce the quantization loss of ${\bm{X}}\setminus{\bm{X}}^{m}$ but increase the quantization loss of ${\bm{X}}^{m}$, ultimately resulting in a performance decline on the LLaMA2-13B. Conversely, if we set $\gamma\to\infty$ and only optimize the quantization error for ${\bm{X}}^{m}$, it will increase the quantization error of ${\bm{X}}\setminus{\bm{X}}^{m}$, resulting in an accuracy drop across the LLaMA2-7B, LLaMA2-13B, and LLaMA3-8B. It is also worth mentioning that the trend observed in the Mistral-7B-v0.3 model significantly differs from that of the LLaMA models. We believe this is primarily because, compared to the LLaMA models, the $\mathrm{RH}$ has effective in reducing the quantization error on ${\bm{X}}^{m}$ as shown in Figure 13. Therefore, optimizing the quantization error of ${\bm{X}}^{m}$ does not have a noticeable impact on the Mistral-7B-v0.3.