Benchmarking the Robustness of Quantized Models

Network quantization compresses DNN models by reducing the number of bits required to represent each weight to save memory usage and speed up hardware inference. A classic quantization process (quantization and de-quantization) can be formulated by:

$$Q(r)=\mathbf{Int}(r/S)-Z,\quad r^{{}^{\prime}}=S\cdot(Q(r)+Z)$$

(1)

where $Q$ is the quantization operator, $r$ and $r^{{}^{\prime}}$ is real value and de-quantized real value respectively, $S$ and $Z$ denote scale and zero-point respectively. Given $t$ bits, the range after quantization is determined by $[-2^{t-1},2^{t-1}-1]$.

We here provide a brief review of the commonly used quantization methods. One string of research designed rules to fit the quantizer to the data distribution. For example, DoReFa-Net [29] simply clips the activation to $[0,1]$ and then quantizes it, due to the observation that most activation falls into this range in many network architectures. Other notable work focused on learning appropriate quantization parameters during the backpropagation process. PACT [2] clip the activation by a handcrafted parameter and optimize the clipping threshold. Notice that PACT has no gradient below the clip point, LSQ [4] learns the scale alongside network parameters by estimating the gradient at each weight and activation layer.

Adversarial examples are inputs with small perturbations that could easily mislead the DNNs [5]. Formally, given a DNN $f_{\Theta}$ and an input $\mathbf{x}$ with the ground truth label $\mathbf{y}$, an adversarial example $\mathbf{x}_{adv}$ satisfies

$$f_{\Theta}(\mathbf{x}_{adv})\neq\mathbf{y}\quad s.t.\quad\|\mathbf{x}-\mathbf{x}_{adv}\|\leq\epsilon,$$

(2)

where $\|\cdot\|$ is a distance metric and commonly measured by the $\ell_{p}$-norm ($p\in${1,2,$\infty$}).

A long line of work has been dedicated to performing adversarial attacks [5, 19, 3, 17, 15, 14, 24], which can be mainly divided into white-box and black-box manners based on access to the target model. For white-box attacks, adversaries have complete knowledge of the target model and can fully access it; while for black-box attacks, adversaries have limited or even without any knowledge of the target model and can not directly access it. This paper primarily employs white-box attacks to evaluate the adversarial robustness of target models, as they offer stronger attack capabilities.

A number of studies have been proposed to evaluate the robustness of floating-point networks [27, 23, 16, 25].

However, the robustness of quantized networks has been relatively underexplored. Lin et al. [12] proposed a defensive quantization method to suppress the amplification of adversarial noise during propagation by controlling the Lipschitz constant of the network during quantization. Similarly, Alizadeh et al. [1] also designed a regularization scheme to improve the robustness of the quantized model by controlling the magnitude of adversarial gradients. In this paper, we aim to thoroughly evaluate the robustness of quantized models against multiple noises for several quantization methods, architectures, and quantization bits.

Dataset. We conduct evaluations on the large-scale ImageNet dataset with 1,000 classes, which comprises 1.2 million training images and 50,000 validation images.

Architectures. We consider four architectures, including ResNet18[8], ResNet50[8], RegNetX600M [21], and MobileNetV2 [22]. (1) ResNet18 and ResNet50 are classical backbone architectures that are widely used in various computer vision tasks. (2) RegNetX600M is an advanced architecture with group convolution discovered through model structure search. (3) MobileNetV2 is a lightweight network designed for efficient deployment on edge devices, featuring depthwise separable convolutions.

Quantization Methods. We focus on three popular quantization methods, including DoReFa [29], PACT [2], and LSQ [4]. For the choice of quantization bits, we adopt the commonly used set in deployments (i.e., 2, 4, 6, and 8).

For each architecture, we quantize models on the ImageNet training set starting from the same floating-point model, then evaluate their robustness against perturbations generated on the ImageNet validation set.

Quantized models are vulnerable to various perturbations in real-world scenarios. We classify these perturbations into adversarial attacks, natural corruptions, and systematic noises, following the guidelines proposed in [23].

Adversarial attacks. To craft adversarial perturbations with progressively increasing attack capabilities, we employ FGSM-$\ell_{\infty}$, PGD-$\ell_{1}$, PGD-$\ell_{2}$, PGD-$\ell_{\infty}$ and AutoAttack-$\ell_{\infty}$. For each attack, we set three different perturbation magnitudes (small, middle, and large).

Natural corruptions. To simulate natural corruptions, we utilize 15 distinct perturbation methods from the ImageNet-C benchmark [9] that fall into four categories: noise, blur, weather, and digital.

Systematic noises. Moreover, system noises are always present when models are deployed in edge devices due to changes in hardware or software. To evaluate the impact of system noises on quantized models, we utilize pre-processing operations from ImageNet-S [26], including three frequently used decoders and seven commonly used resize modes.

Adversarial robustness. For specific adversarial attacks, we measure adversarial robustness ($AR$) using model accuracy, where higher $AR$ indicates a stronger model. For the union of different attacks, we adopt the Worst-Case Adversarial Robustness ($WCAR$) to measure adversarial robustness (higher indicates a stronger model):

$$WCAR=1-P_{(\mathbf{x},\mathbf{y})\sim\mathcal{D}}{\mathrm{Any}(f(\mathcal{A}_{\epsilon,p}^{f}(\mathbf{x}))\neq\mathbf{y})},$$

(3)

where $\mathcal{A}_{\epsilon,p}$ represents the adversary, $\mathcal{D}$ is the validation set, and $\mathrm{Any}(\cdot)$ is a function that returns true if any of the adversaries attacks successfully.

Natural robustness. We adopt the average accuracy of the quantized model on all corruptions (denoted as $C$) to measure natural robustness, denoted as $NR$:

$$NR=\mathbb{E}_{c\sim C}({P_{(\mathbf{x},\mathbf{y})\sim\mathcal{D}}{(f(c(\mathbf{x}))=\mathbf{y})}}),$$

(4)

where $c$ denotes a corruption method. A higher value of $NR$ means better natural robustness.

Systematic Robustness. We adopt the model stability on different systematic noises (denoted as $S$) to measure systematic robustness ($SR$). Specifically, we calculate the standard deviation of accuracy as $SR$:

$$SR=\mathbb{D}_{s\sim S}({P_{(\mathbf{x},\mathbf{y})\sim\mathcal{D}}{(f(s(\mathbf{x}))=\mathbf{y})}}),$$

(5)

where $s$ denotes a decode or resize method. A lower value of $SR$ means better stability towards systematic noises.

Tab. 1 reports the clean accuracies of quantized models. Most of the quantized models maintain comparable accuracy to the 32-bit ResNet18. However, 2-bit PACT fails to converge, resulting in a mere 2.61% accuracy. Therefore, we exclude it from the subsequent robustness evaluations.

Since $WCAR$ degrades to 0 under medium and high perturbation magnitudes, we here only present the results under small magnitudes. From the results shown in Tab. 2, we could make the following observations: (1) Unlike the decrease in clean accuracy, lower-bit models exhibit higher worst-case adversarial robustness and are almost better than floating-point network; (2) At the same quantization bit, PACT presents the best adversarial robustness compared to other quantization methods. Furthermore, we compare the adversarial robustness ($AR$) of quantized models against specific attacks. As depicted in Fig. 1, we can identify that LSQ performs better under FGSM and PGD attacks, yet is vulnerable to AutoAttack (the strongest of these attacks).

We present the Natural Robustness of ResNet18 models in Tab. 3 and show the detailed results of LSQ in Fig. 2. Though performing similar clean accuracy with the 32-bit model, quantized models are more vulnerable under natural corruptions, especially in the 2-bit models. And we can find that impulse noise shows the highest impact on the model’s robustness (about 50% average decrease), while brightness is the least harmful (about 10% average decrease).

On the systematic noises, we also observe that lower-bit models present less robustness (i.e., lower stability), as shown in Tab. 4. Moreover, we find that among 14 systematic noises, the nearest neighbor interpolation methods in Pillow and OpenCV have the greatest impact on the model performance, which induce nearly a 6% decrease in performance for the 2-bit models. It indicates that maintaining consistency between the deployment and training process is crucial to avoid unnecessary accuracy loss.