PSAQ-ViT V2: Towards Accurate and General Data-Free Quantization for Vision Transformers

With the success in natural language processing applications, transformers, which enjoy the global receptive field offered by the self-attention mechanism, have recently achieved significant performance gains on a range of computer vision applications [28, 29, 30, 31, 32]. By reshaping the image into a sequence of flattened 2D patches as the input, ViT [33] is the first work to apply the transformer-based models to the computer vision community, which boosts the performance baseline on the image classification task. DeiT [34] employs a distillation token to allow the student to learn from the teacher through attention, which can achieve competitive results on ImageNet with low training data cost. Swin [35] constructs hierarchical feature maps using the shift of the window partition between consecutive self-attention layers and has linear computational complexity to image size, further demonstrating the potential of transformer-based models as vision backbone. In addition to the image classification task, ViTs are also emerging as popular solutions for other high-level vision tasks, such as object detection [36, 37], semantic segmentation [38], and video recognition [39].

Unfortunately, despite the excellent performance, ViTs typically have complex model architectures and their high number of parameters and computation is intolerable for real-world applications [5, 8]. To this end, several works recently focus on lightweight ViTs, such as MobileViT [40] and TinyViT [41], whose model complexity, however, is still far from satisfactory, especially for the deployment on the resource-constrained edge devices. As a result, model compression techniques for ViTs are regarded as necessary and promising solutions.

Model quantization, which replaces 32-bit floating-point parameters with low-bit values, is a popular and promising way to reduce the complexity of neural networks [10, 11], thus allowing their deployment and real-time inference on edge devices. To date, there have been considerable researches attempting to quantize CNNs. A selection of notable works performs quantization-aware training (QAT) to improve the accuracy of the quantized model [12, 42, 43, 44], where the straight-through estimator (STE) [45] is used to approximate the gradient back-propagation of discrete parameters. To reduce the computational cost of QAT, other efforts propose post-training quantization (PTQ) methods [46, 47, 48, 49, 50, 51], which focus on the calibration of quantization parameters.

In addition, several quantization schemes designed for ViTs have been recently presented. Ranking-Aware [8] introduces a ranking loss into the conventional quantization objective to keep the relative order of the self-attention results after quantization. PTQ4ViT [52] proposes the twin uniform quantization method to reduce the quantization error on the activation values after Softmax and GELU functions. FQ-ViT [7] designs specific quantization strategies for Softmax and LN components to obtain the fully quantized ViTs. I-ViT [9] utilizes integer bit-shifting to approximate Softmax and GELU operations to achieve efficient integer-only inference for ViTs.

However, all the above methods rely heavily on the original training data, making them inapplicable in data-sensitive scenarios where the original data is not available.

With the ability to compress models without access to any real data, data-free quantization can potentially address data privacy and security issues and is thus receiving increasing attention [23, 27]. ZeroQ [21] proposes BN regularization to facilitate the distribution matching between the generated samples and the real data, and then uses the generated samples to calibrate the quantization parameters. Based on BN regularization, DSG [22] utilizes slack distribution alignment and layerwise sample enhancement to improve the diversity of the generated samples. GDFQ [25] and IntraQ [26] combine the category prior with BN statistics to generate class-conditional samples while matching the real-data distribution, powering the performance of data-free quantization. These methods, however, are only applicable to CNNs and not to ViTs, because LN in ViTs, unlike BN, does not store any prior information of the original training data.

To address the above challenge, PSAQ-ViT [20], as the previous version of this work, carries out an insight into the general difference in the self-attention module’s processing of Gaussian noise and real images, and thus proposes a relative value metric, patch similarity, to optimize the Gaussian noise to approximate real images. In this work, building on patch similarity, we are interested in making the scheme more accurate and general by introducing advanced quantization learning strategies, which further advances the data-free quantization community for ViTs.

Standard ViTs consist of an embedding layer and several stacked transformer blocks. First, the embedding layer reshapes the input image into a sequence of flatted 2D patches, which are then linearly projected into a $d$-dimensional space to obtain the token embeddings $X\in\mathbb{R}^{N\times d}$. Here, $N$ is the number of patches.

The token embeddings $X$ are then fed into a series of transformer blocks, where each block is composed of a multi-head self-attention (MSA) module and a multi-layer perceptron (MLP) module as follows:

MSA calculates the attention between different patches to learn inter-patch representations from a global view as follows:

where $H$ is the number of attention heads. Here, query $Q_{i}$, key $K_{i}$, and value $V_{i}$ are computed by linear projections using matrix multiplication, $i$.$e$., $Q_{i}=XW_{i}^{Q}$, $K_{i}=XW_{i}^{K}$, $V_{i}=XW_{i}^{V}$. Then, MLP employs two dense layers to process the obtained attention for high-dimensional feature mapping and information fusion as follows:

Finally, the blocks are followed by the classification/detection/segmentation heads depending on the vision task to get the final result.

In this paper, we work on quantizing all the parameters (weights and activations) of the operations ($e$.$g$., large matrix multiplication) in the embedding layer, transformer blocks, and subsequent vision heads, including the input and output. For the quantization strategy, we employ the simple uniform quantization, which is the most popular and hardware-friendly method and is defined as follows:

where $\theta^{p}$ and $\theta^{q}$ denote the parameters of the FP model and the quantized model, respectively. Here, $\lfloor\cdot\rceil$ is the round operator, $q_{0}$ and $q_{2^{k}-1}$ are the quantization clipping values, and $k$ is the quantization bit-precision.

As stated before, ViTs do not have BN layers that store information about the original training data, leaving no available absolute value prior information to generate the samples for performing quantization, which is the major challenge of data-free quantization for ViTs. Therefore, the previous version of this work, PASQ-ViT, is to mine deeper into the prior information of the pre-trained ViTs and thus explore a reliable relative value metric that can well describe the general difference between Gaussian noise and real images, so that we can reduce this difference to make the Gaussian noise approximate the real images.

As the unique structure in ViTs, the self-attention module learns inter-patch feature representations, which are believed to contain a certain amount of original data information. Consequently, we provide an in-depth analysis of the inference process of the self-attention module, and then we observe that the reason the model can make good decisions is that the self-attention module can distinguish the foreground from the background of the training data, thus allocating more attention to the foreground that is more important for the decision. Since the input of ViTs are independent vectors mapped by 2D patches, the responses of the self-attention module to different patches are significantly different, $i$.$e$., the foreground patches receive more attention. More specifically, when the pre-trained model executes inference, real images consistently produce the above features, while Gaussian noise, whose foreground is not easily extracted, does not have a similar capability and inevitably leads to homogeneous responses, as shown in Fig. 2. Note that the real images here are only used to verify the general difference ($e$.$g$., a certain metric of the real images is always larger than that of Gaussian noise), and they will not be involved in any subsequent process. Therefore, this general difference can indirectly represent the prior information of ViTs and thus can be used to design the relative value metric to guide the sample generation.

Based on the above insights, we work on designing a reliable metric that can measure the diversity of the self-attention module’s responses. For the $l$-th block in ViTs, the output of the MSA module is defined as $O_{l}\in\mathbb{R}^{B\times H\times N\times d}$ ($l\in\{1,\cdots,L\}$), where each dimension denotes the batch size, number of heads, number of patches, and hidden size, respectively. To simplify the expression, we ignore the batch dimension, $i$.$e$., $O_{l}\in\mathbb{R}^{H\times N\times d}$.

Due to the relative value metric, it is necessary to first normalize $O_{l}$ to ensure the fairness of the comparison. We accomplish this by calculating the cosine similarity between each subspace vector in the patch dimension, specifying the data range at [-1, 1], as follows:

where the numerator represents the inner product of the vectors, and $||\cdot||$ denotes the $l_{2}$ norm. Here, $u_{i},u_{j}\in\mathbb{R}^{H\times d}$ ($i,j\in\{1,\cdots,N\}$) is the $i$-th/$j$-th vector in the patch dimension of $O_{l}$, and $\Gamma_{l}(u_{i},u_{j})$ represents the cosine similarity between $u_{i}$ and $u_{j}$. After pairwise calculations, we obtain the $l$-th block’s cosine similarity matrix $\bm{\Gamma}_{l}=[\Gamma_{l}(u_{i},u_{j})]_{N\times N}$, which is a symmetric matrix and is termed as patch similarity. The diversity of patch similarity can potentially represent the diversity of the original data, which not only elegantly achieves data normalization, but also has the additional advantage of achieving reasonable $\frac{Hd}{N}$-fold dimensionality reduction ($\mathbb{R}^{H\times N\times d}\to\mathbb{R}^{N\times N}$). For instance, for the DeiT-B model, the amount of data is reduced by a factor of 3.92, which can greatly improve the subsequent computational efficiency.

Then, the diversity of patch similarity is measured by the information entropy, which can represent the amount of information expressed by the input data. To ensure gradient back-propagation, we employ the differential entropy that has a continuous nature. More intuitively, the $l$-th block’s differential entropy of patch similarity $\bm{\Gamma}_{l}$ at the input generated data $\mathcal{G}$ is calculated as follows:

where $f_{h}(x)$ is the continuous probability density function of $\bm{\Gamma}_{l}$, which is obtained using kernel density estimation as follows:

where $K(\cdot)$ is the kernel ($e$.$g$. normal kernel), $h$ is the bandwidth, $x_{m}$ ($m\in\{1,\cdots,M\}$) is a training point drawn from $\bm{\Gamma}_{l}$ and is the center of a kernel, and $x$ is the given test point.

Finally, we sum the differential entropy of each block to account for the diversity of patch similarity across all blocks, and the Patch Similarity Entropy Loss is defined as follows:

To enhance the performance of data-free quantization in PSAQ-ViT, we introduce an adaptive teacher-student strategy, which drives the quantized model (student) to imitate the FP model (teacher) with the help of the generated samples. Here, adaptivity is reflected in the fact that instead of a one-time synthesis, the generated samples evolve alternately with the quantized model in a cycle. This competitive and interactive nature forces the gradual emergence of new features in the generated samples, and thus facilitates the quantized model to constantly learn new knowledge. Specifically, they cyclically play a two-player minimax game with respect to the model discrepancy $\mathcal{D}(\mathcal{O}_{\mathcal{Q}},\mathcal{O}_{\mathcal{P}};\mathcal{G})$ as follows:

where $\mathcal{O}_{\mathcal{Q}}$ and $\mathcal{O}_{\mathcal{P}}$ are the output logits of the quantized model and the FP model when the input is $\mathcal{G}$, respectively.

In the typical real-data-driven knowledge transfer or distillation applications, Kullback-Leibler (KL) divergence is a popular metric for estimating model discrepancy. However, our minimax game is different from these cases where the student can greedily perform one-time learning from the soft targets produced by the teacher, thus it has unsatisfactory contributions. Specifically, the distance described by KL divergence is loose, which makes the optimization space become small after the quantized model converges to the currently generated samples, $i$.$e$., the results of KL divergence are similar when inputting samples with new features, which causes gradient decay and thus hinders the competition and constant evolution of the generated samples and the quantized model. Thus, we employ a more robust function, Mean Absolute Error (MAE), to construct a better search space, and the Discrepancy Loss is defined as follows:

where $n$ is the length of the output logits, $e$.$g$., category number for classification and prediction map size for detection and segmentation.

In practice, each cycle in the above minimax game can be divided into two stages, each of which is discussed in detail below. In stage 1, we fix the FP model $\mathcal{P}$ and the quantized model $\mathcal{Q}$ and only update the generated samples $\mathcal{G}$ by maximizing the model discrepancy $\mathcal{D}(\mathcal{O}_{\mathcal{Q}},\mathcal{O}_{\mathcal{P}};\mathcal{G})$. Intuitively, the purpose of this action is to encourage generating more confusing samples. Thus, for this purpose, new features are forced to emerge gradually in the generated samples $\mathcal{G}$, which can drive the samples to discover missing representations and potentially increase their informativeness and diversity. In stage 2, the FP model $\mathcal{P}$ and the generated samples $\mathcal{G}$ are fixed, and the model discrepancy $\mathcal{D}(\mathcal{O}_{\mathcal{Q}},\mathcal{O}_{\mathcal{P}};\mathcal{G})$ is minimized to update the quantized model $\mathcal{Q}$, which drives the quantized model to imitate the FP model to obtain the promising performance. In this manner, the two stages are executed alternately, allowing for the cyclic evolution of the generated samples and the quantized model.

We fuse the patch similarity metric and the adaptive teacher-student strategy to obtain the overall scheme, which is summarized in Algorithm 1. Following the teacher-student strategy, the overall pipeline is also performed in a two-stage cycle consisting of sample generation and quantization learning. The procedure and loss functions of each stage are described individually below.

Sample generation In this stage, both the diversity of patch similarity and the model discrepancy are considered to search for a satisfactory sample space. Specifically, we combine the Patch Similarity Entropy Loss $\mathcal{L}_{PSE}$ and the Discrepancy Loss $\mathcal{L}_{\mathcal{D}}$, and since they are to be maximized, the loss $\mathcal{L}_{\mathcal{G}}$ of sample generation is defined as follows:

where $\alpha$ is a balance coefficient. Note that in contrast to PSAQ-ViT, there is no auxiliary category prior in the above equation, which ensures the generality of the scheme.

Quantization learning In this stage, we utilize the samples generated in the former stage to facilitate the learning of the quantized model under the supervision of the FP model. However, since the batch size of the generated samples is small ($e$.$g$., 32), the fearful overfitting cases typically appear in quantization learning, which poses a challenge to the constant and effective learning of the quantized model. A naive approach is to simply increase the batch size in the former stages, but this brings unexpected computational overhead. Therefore, we employ image augmentation to cost-effectively expand the sample capacity, following the method in SimCLR [53] whose augmentation policy includes random crop (with flip and resize), color distortion, and Gaussian blur.

Then, we input the augmented samples $\hat{\mathcal{G}}$ into the FP model and the quantized model, calibrate the quantization clipping values $q_{0}$ and $q_{2^{k}-1}$, and update the quantized model by minimizing the model discrepancy via the following loss:

After several iterations of learning in the above two stages, the generated samples and the quantized model will reach an ideal balance point, where the model discrepancy no longer changes after iterations, $i$.$e$., the generated samples cannot pull apart the FP model and the quantized model, and the quantized model have learned the full transferred knowledge. In this case, the quantized model is considered to be functionally identical to the FP model.

It is also worth noting that all the objective functions are task-independent, thus the proposed scheme is general-purpose and compatible with a broad range of vision tasks, and they are also model-independent, which allows for different teacher and student model structures, $e$.$g$., the learning of quantized DeiT-T can be performed under the supervision of DeiT-B.

Models and datasets We evaluate PSAQ-ViT V2 on various popular models for image classification, object detection, and semantic segmentation with the following models and datasets.

• •

  Image classification: We perform classification experiments on ImageNet [54] dataset, which has 1.28M training images and 50K validation images from 1,000 classes. DeiT-T/S/B [34] and Swin-T/B [35] are adopted as the baseline model. Since ViT [33] and DeiT have the same model structure, we do not evaluate ViT in this work. The pre-trained models are obtained from the timm [55] framework.

• •

  Object detection: Object detection and instance segmentation experiments are conducted on COCO 2017 [56] dataset, which contains 118K training, 5K validation, and 20K test-dev images. The baseline is the Cascade Mask R-CNN [57] framework in mmdetection [58] with DeiT-T/S and Swin-T as the backbones.

• •

  Semantic segmentation: We adopt ADE20K [59] dataset to evaluate the performance for semantic segmentation. It has 25K images in total, with 20K for training, 2K for validation, and another 3K for testing, covering a broad range of 150 semantic categories. With DeiT-T/S and Swin-T as the backbones, we take the UperNet [60] framework in mmsegmentation [61] as the baseline.

Comparison methods It is naturally necessary to make a comparison with the previous version PSAQ-ViT, however, it is unfair and inadequate to compare with only PSAQ-ViT, because PSAQ-ViT only performs quantization parameter calibration without a learning process, and it contains the category prior, which is not applicable to detection and segmentation tasks. Therefore, we have to build a reasonable comparison method, called Standard V2, on our own. Standard V2 is distinguished from Standard in the previous version. Standard V2 introduces the model interaction and directly uses real training data with the Discrepancy Loss $\mathcal{L}_{\mathcal{D}}$ for teacher-student learning, and it traverses the training set once to complete convergence. For a fair comparison, we make sure to use the same number of samples, more intuitively, in PSAQ-ViT V2, the result of multiplying the number of iterations, q_step, and batch size is equal to the size of the training set. All other settings for Standard V2 and PSAQ-ViT V2 are the same.

Experimental settings All experiments in this work are implemented in Pytorch. For a fair demonstration of effectiveness, we employ the most basic quantization method, $i$.$e$., for weights, symmetric uniform quantization with vanilla MinMax strategy is applied; for activations, asymmetric uniform quantization is applied, and the default strategy is vanilla MinMax if not specifically declared. We adopt Adam [62] optimizer in both sample generation and quantization learning stages, where the search space of learning rate is {0.2, 0.25} in sample generation and {5e-7, 1e-6, 2e-6} in quantization learning. Although further hyperparameter tuning may achieve better accuracy, for uniformity, all our experiments are conducted using batch size 32 and weight decay 1e-4. The hyperparameter $\alpha$ is set to 1.0 after a simple grid search, and its selection has little effect on the final performance. All experiments are done on a single NVIDIA GeForce RTX 3090 GPU.

We start by discussing the quantization results on ImageNet image classification, which is compared with the previous version PSAQ-ViT and the real-data-driven Standard V2. Interestingly, our proposed PSAQ-ViT V2 without access to the original dataset can even slightly outperforms Standard V2. This is partly thanks to the Patch Similarity Entropy Loss $\mathcal{L}_{PSE}$. It forces the distinction between foreground and background in the generated samples according to the response of the self-attention module, and then when these samples are utilized to calibrate the quantization parameters, they in turn reinforce the functionality of the self-attention module, thus acting as positive feedback that can reduce the activation outliers to some extent and therefore improve the tolerance to parameter clipping. The Discrepancy Loss $\mathcal{L}_{\mathcal{D}}$ also contributes to the surprising performance. It encourages increasing the discrepancy between the FP model and the quantized model to force the emergence of new hard-to-discriminate features in the samples, and these features can be regarded as support vectors (aka hard instances) that contain more typical and instructive knowledge, thus promoting more effective learning of the quantized model.

Table I shows the quantization results for various models at different quantization bit-precisions. For the quantization of DeiT-T, our proposed PSAQ-ViT V2 improves 3.04% over the previous version at W4/A8 precision, and achieves 72.17% accuracy at W8/A8 precision with only 5MB memory footprint. DeiT-S quantized with our method improves 0.38% and 0.32% over the real-data-driven Standard V2 at W4/A8 and W8/A8 precisions, respectively. Similar to the previous models, the quantization results of DeiT-B also show that PSAQ-ViT V2 achieves the best performance, with 2.44% and 0.32% improvements over PSAQ-ViT and Standard V2 at W4/A8 precision, respectively, and only 0.33% accuracy reduction compared to the FP model at W8/A8 precision with 4-fold compression. When using the vanilla MinMax quantization strategy, the parameter distribution of Swin is highly sensitive to discretization; for instance, PSAQ-ViT produces 9.56% and 6.00% accuracy degradation in the W4/A8 and W8/A8 quantization of Swin-T, respectively. Fortunately, the proposed teacher-student strategy updates the parameter distribution in the model to make it quantization-friendly, and thus achieves significant accuracy compensation of 4.49% and 4.86% at W4/A8 and W8/A8 precisions, respectively. The adaptive nature of our method is also more evident in Swin; for instance, for the W4/A8 and W8/A8 quantization of Swin-S, our method achieves performance improvements of 0.64% and 0.71% over Standard V2, respectively.

Results of combining with advanced strategies Since our method is orthogonal to the quantization strategies, we combine PSAQ-ViT V2 with advanced quantization strategies to further improve the performance, as shown in Table II. First, we employ exponential moving average (EMA), which smooths the quantization clipping values, to replace vanilla MinMax, achieving an accuracy gain of 0.97% and 1.05% in the W4/A8 quantization of DeiT-T and Swin-T, respectively. Moreover, as mentioned before, the model prior and loss functions in PSAQ-ViT V2 are model-independent, thus allowing for different teacher and student model structures. Here, we employ the base model to supervise the quantization of the corresponding tiny model. For instance, thanks to the guidance of the powerful FP DeiT-B with 81.85%, the quantization performance of DeiT-T at W4/A8 precision achieves an inspiring 1.25% improvement.

Analysis of generated samples Fig. 3 shows the visualization results of the generated images (224$\times$224 pixels) using PSAQ-ViT and PSAQ-ViT V2 (including mid and final iterations). Note that these images require only a pre-trained FP model, and not any additional information, especially the original data or any absolute value metrics. Thanks to the joint guidance of the patch similar metric and the category prior, the generated images in PSAQ-ViT can clearly distinguish the foreground from the background, and the foreground is rich in semantic information with a specified category, $e$.$g$., the top left pseudo-label is “teddy bear”. However, the background information in these samples is singular and homogeneous. In contrast, there is no category guidance in this work, resulting in weakly intuitive semantic features of the images, but this in turn helps the sample to improve informativeness and diversity in both foreground and background without limitations, thus more closely matching the real data in the overall statistics. In addition, during the cyclic iterations, the samples are progressively evolving in competition and interaction, with continuously emerging new features and increasingly evident semantic information, thus facilitating constant and effective learning of the quantized model.

To demonstrate the validity of the Patch Similarity Entropy Loss $\mathcal{L}_{PSE}$ more obviously, we also conduct the comparison experiments of the kernel density curves of the patch similarity for each block in DeiT-B model when the input is the real image, Gaussian noise, and the generated image, as shown in Fig. 4. For the responses to Gaussian noise, the kernel density curves all show a concentrated unimodal shape and the central value of the curve is high, indicating a high degree of similarity between each patch of Gaussian noise and thus a full classification as background or foreground. Fortunately, the kernel density curves corresponding to our generated images are very approximate to those corresponding to the real images. They all show a dispersed bimodal shape, indicating a high diversity of responses, and the left and right peaks of curves describe inter- and intra-category similarity, respectively, which is in line with the expectation that the images can easily be distinguished between foreground and background.

Unlike PSAQ-ViT which is only applicable to the image classification task, PSAQ-ViT V2 is general and compatible with a broad range of vision tasks. Here, the Cascade Mask R-CNN framework with Swin-S/DeiT-S as the backbone, which achieves state-of-the-art (SOTA) performance on object detection and instance segmentation tasks on COCO dataset, is used to perform quantization experiments and the results are reported in Table III. To the best of our knowledge, this is the first attempt to perform data-free quantization of ViTs for object detection and instance segmentation.

When using DeiT-S as the backbone, PSAQ-ViT V2 achieves consistently better performance than Standard V2, with 0.3 higher box Average Precision (AP) and 0.4 higher mask AP at W4/A8 precision, and it achieves 47.3 box AP and 40.8 mask AP at W8/A8 precision, enabling a 4-fold almost lossless compression. In the case of applying Swin-S as the backbone, our method also shows promising performance, outperforming Standard V2 by 0.6 box AP and 0.6 mask AP at W8/A8 precision, which is only 0.9 box AP and 0.6 mask AP lower than the FP model.

For a more intuitive view of the performance of the quantized model, we visualize the detection results and compare them with those of the FP model, as illustrated in the first row in Fig. 5. The backbone we employ is Swin-S. It can be seen that the quantized model at W8/A8 precision achieves similar performance to the FP model, with slight differences in partial classification scores, $e$.$g$., the score for ’baseball glove’ changes from 0.97 to 0.92. And when performing W4/A8 quantization, the hard small object “baseball bat” is misclassified into “tennis racket”, while the results of easy objects such as “person” are unaffected.

Our proposed PSAQ-ViT V2 can also be applied to the semantic segmentation task. We employ the UperNet framework with Swin-S/DeiT-S as the backbone, which is the SOTA scheme on the semantic segmentation task on ADE20K dataset, and the quantization results are shown in Table IV. To the best of our knowledge, this is the first attempt to quantize ViTs for semantic segmentation without any real data.

The discretization of the model parameters has a large impact on the pixel-level semantic segmentation, making the accuracy of the quantized model not as encouraging as the results on previous tasks. For instance, W8/A8 quantization cannot achieve lossless compression, but rather produces an accuracy degradation greater than 1 mean Intersection over Union (mIoU). This effect also acts on Standard V2 using real data, thus the performance advantage of our method over Standard V2 does not change. For the quantization of DeiT-S/UperNet, our proposed PSAQ-ViT V2 achieves 39.9 mIoU and 42.7 mIoU at W4/A8 and W8/A8 precisions, respectively, which are 0.3 mIoU and 0.2 mIoU higher than Standard V2. Similarly, in the W4/A8 and W8/A8 quantization of Swin-S/UperNet, our method also consistently has performance benefits over Standard V2.

In addition, we also perform visualizations of the segmentation results, as illustrated in the second row in Fig. 5. The backbone we employ is Swin-S. As we can see, the quantized model at W8/A8 precision is comparable to the FP model for the overall segmentation performance, while there is a slight difference in the recognition of a few small regions. In the case of compressing the model to W4/A8 precision, the segmentation results are not good for semantically weak regions, for instance, the object in the lower left corner of the image is not well recognized.

Here, we perform two ablation studies to verify the effectiveness of each component of the proposed method. First, we explore the contributions of the Patch Similarity Entropy Loss $\mathcal{L}_{PSE}$ and the Discrepancy Loss $\mathcal{L}_{\mathcal{D}}$ in sample generation, as shown in Table V. Without any optimization, inputting Gaussian noise directly to guide the learning of the quantized model can produce severe performance degradation. When only $\mathcal{L}_{\mathcal{D}}$ is utilized on Gaussian noise to perform the minimax game, there is a slight improvement in accuracy, but it is still far from satisfactory. In summary, in the absence of real data, the quantization performance obtained directly with Gaussian noise is unacceptable. Thus, $\mathcal{L}_{PSE}$-guided optimization to make the Gaussian noise approximate the real images is highly necessary. For instance, when performing W4/A8 quantization of DeiT-S, only $\mathcal{L}_{PSE}$-guided optimization achieves 74.69% accuracy, which is 59.97% higher than Gaussian noise. Further, we add $\mathcal{L}_{\mathcal{D}}$ to introduce the model interaction so that the quantized model enjoys a more adequate knowledge transfer, resulting in a better performance with a 1.67% improvement. Similar comparative results are also obtained in the W4/A8 quantization of Swin-S, where using $\mathcal{L}_{PSE}$ improves the accuracy substantially to 77.03% and combining it with $\mathcal{L}_{\mathcal{D}}$ further improves to a satisfactory 78.86%. The above results fully demonstrate the critical role of $\mathcal{L}_{PSE}$ and $\mathcal{L}_{\mathcal{D}}$ in improving the quantization performance.

Second, we compare the performance of different estimation functions for the model discrepancy $\mathcal{D}(\mathcal{O}_{\mathcal{Q}},\mathcal{O}_{\mathcal{P}};\mathcal{G})$, and the results are reported in Table VI. The experimental results verify the previous analysis that KL divergence leads to a dying minimax game since it produces decaying gradients after converging to the current generated samples. The reason is that in our minimax game, the student greedily performs one-time learning from the teacher, which leaves little room for optimizing KL divergence, so that the KL divergence results are similar when inputting samples with new features, thus hinders the competition and constant evolution of the generated samples and the quantized model. More specifically, we evaluate the performance of KL divergence for multiple temperature settings $T\in$ {20, 10, 5, 1} and find that as the temperature preliminarily decreases ($e$.$g$., from 20 to 10), the constraint on the soft label is tightened and the above issue is slightly alleviated. Intuitively, in the W4/A8 quantization of DeiT-S, the accuracy is improved from 74.56% to 75.23%. Whereas, after the temperature is small, further reduction does not improve the performance any further, for example, similar performance (75.57% and 75.51%) is achieved at temperatures of 5 and 1. This indicates that the above issues are not radically ameliorated. In addition, we introduce an advanced dynamic temperature strategy [64], which offers some improvements over the scheme with the fixed temperature, however, it is still weaker than the scheme with the MAE function, with the latter being 0.46% and 0.31% higher in the W4/A8 quantization of DeiT-S and Swin-S, respectively. The above results fully demonstrate the validity of the MAE function, which provides a better search space for the minimax game and thus can effectively solve the gradient decay issue.