Towards Building More Robust Models with Frequency Bias

Szegedy et al. [30] found the vulnerability of DNNs against adversarial examples and proposed L-BGFS based attack. Subsequently, Goodfellow et al. [12] argued that the main reason for the vulnerability of neural networks to adversarial perturbations is their linear nature, and used Fast Sign Gradient Method (FGSM) to generate adversarial examples efficiently. Kurakin et al. [21] extended FGSM to a more effective iteration-based attack as Basic Iterative Method (BIM). Subsequently, by adding random initialization and extending to more iterative steps, Madry et al. [22] propose PGD as a universal first-order adversary. Boundary-based attack DeepFool [24] and optimization-based attacks like C&W [4].

In addition to the thriving white-box attack methods based on the assumption of model transparency, researchers are also especially interested in black-box attacks, where the structure and parameters of the DNN are unknown to the attackers. (i.e., only a few queries or training data are available). There are two types of black-box attacks: query-based and transfer-based. To produce adversarial instances, query-based algorithms estimate the gradient information via queries [3, 17, 1]. While transfer-based attacks [9, 35, 27] employ white-box attacks on a local surrogate model to produce transferable adversarial perturbations.

AutoAttack [6] is a combination of four complementary attack methods (i.e., APGD-CE, APGD-DLR, FAB [5], and Square Attack [1], which has become a very popular adversarial robustness benchmark today.

Adversarial training is one of the notable defense methods as an empirical defense, which is also adopted in this paper to build robust models. And its basic idea can be expressed as a min-max optimization problem:

$$\theta^{*}=\mathop{\arg\min}\limits_{\theta}\mathbb{E}_{(x,y)\in\mathcal{X}}\left[\mathop{\max}\limits_{\delta\in[-\epsilon,\epsilon]}\ell(f_{\theta}(x+\delta);y)\right]$$

where the $f_{\theta}$ is a DNN parameterized by $\theta$, $\mathcal{X}$ stands for the training dataset and $\ell$ represents the loss function. And perturbations $\delta$ are bounded into the $\epsilon$-ball.

To solve the inner maximization problem, projected gradient descent (PGD) [22] is a prevailing and effective method to generate perturbations. Many promising methods then sprang up. Some of them provide more efficient loss function designs [39, 32], some methods design stronger regularization methods [34, 26], and some focus on solving the problem of the excessive computational overhead of adversarial training [28, 33].

In engineering applications, the vast majority of Fourier transform applications use the discrete Fourier transform (DFT), where the Fourier transform takes a discrete form in both the time and frequency domains. Applying DFT to a flattened image signal $x$ is equal to left multiplying a DFT matrix, the rows of which are the Fourier basis $f_{k}=[e^{2\pi j(k-1)\cdot 0},...,e^{2\pi j(k-1)\cdot(n-1)}]/\sqrt{n}\in\mathbb{R}^{n}$, where $k$ represents the $k$th row of the DFT matrix and $j$ represents the imaginary unit. The Fast Fourier Transform (FFT) is usually used in practical applications to efficiently compute the DFT.

Recent work [31, 25] studied and revealed the low-pass characteristic of basic ViT blocks. ViT tends to reduce high-frequency signals in the feature map and thus almost only the DC component is preserved with the model going deeper. Additionally, MSAs spatially smoothen feature maps with self-attention weights, acting like a natural denoising module for better generalization and robustness. With rigid theoretical analysis, Wang et al. [31] try to avoid over-smoothing by giving higher weights to high-frequency signals in deep ViTs. On the contrary, convolutional networks go the opposite way in terms of frequency characteristics and amplify high-frequency signals. As per the aforementioned findings and the underlying connection between Vit’s low-frequency characteristics and its robustness, it is promising to supplement the low-frequency features that convolutional networks are missing.

We first introduce the proposed FPCM (shown in Fig.1), which reconfigures low- and high-frequency components in features. Given an input $x$, we transform it into a frequency domain via DFT. We then suppress the high-frequency signal with an exponential low-pass filter, followed by the iDFT transformation to restore spatial domain features.

Formally, denote $x\in\mathbb{R}^{C\times H\times W}$ as the input feature, and $\mathcal{F}\in\mathbb{C}^{HW\times C}$ as its frequency representation. We have:

$$\mathcal{F}(u,v)=\sum_{a=0}^{H-1}\sum_{b=0}^{W-1}x_{a,b}e^{-j2\pi(\frac{v}{H}a+\frac{u}{W}b)}$$

where $H$ and $W$ represent the height and width of the input feature respectively, and $j$ is the imaginary unit. In practice, the transformation from the discrete spatial domain to frequency domain can be done by FFT with an optimized computational complexity of $O(HWlog(HW))$. Subsequently, the filtered frequency representation $\hat{\mathcal{F}}$ is formulated by:

$$\hat{\mathcal{F}}(u,v)=F(u,v)H(u,v)$$

where $H(u,v)$ denotes a low-pass filter in frequency domain. In our work, we mainly study the Gaussian low-pass filter (GLPF), which is a common choice in the field of image processing, causing no ringing effect:

$$H(u,v)=e^{-(\frac{D(u,v)}{2D_{0}})^{2}}=e^{-(\frac{D(u,v)}{2(HW*\beta)})^{2}}$$

where $D(u,v)$ represents the distance from the pixel $(u,v)$ to the centre of the two-dimensional spectrum, and $D_{0}$ is the cutoff frequency. For any feature map with height $H$ and width $W$, the total sequence length of the corresponding Fourier spectrum would be $HW$. Correspondingly, We set $\beta$ as a hyper-parameter controlling the cut-off frequency. Having obtained the filtered frequency, we then apply the Fourier inverse transform to restore the feature map with only the low-frequency components $\hat{x}$, and derive the final re-weighted low-frequency features from the element-wise production of $\hat{x}$ and the inter-channel weights $\alpha\in\mathcal{R}^{C}$ learned from the original input:

$$\alpha=0.5*sigmoid(W_{\alpha}^{T}x)+0.5$$

by default, we push the value interval of $\alpha$ to $[0.5,1]$ to make the model focus more on the low-frequency features, inspired mainly by previous studies on robust learning frequency characteristics and success attempts [25]. Overall, the FPCM can be formulated as the weighted summation of low-frequency and high-frequency features:

$$FPCM(x)=\alpha\hat{x}+(1-\alpha)(x-\hat{x})$$

we also studied FPCM with fixed weights $\alpha$, where our method will become completely parameter-free. The design of FPCM follows the following principles:

• •

  Efficiency: the extra parameter and computational overhead it introduces are marginal compared to standard DNNs

• •

  Differentiability: FFT and the subsequent low-pass filtering are essentially linear transformations that do not mask the gradient, which allows joint optimization with DNNs.

• •

  Adaptability: learnable weights capture the inter-channel frequency characteristics and bring the possibility of adapting the frequency differences of the output feature maps between different stages (layers) of a model, various models, and datasets.

In light of prior research demonstrating that convolutional layers typically increase feature map variance, with the variance tending to increase with depth and peaking at the ends of each stage, our investigation focuses on analyzing the feature map frequency characteristics of both a vanilla ResNet18 model and an adversarially trained one. We present the results in Fig.3, which illustrate that high-frequency feature components tend to aggregate as the model goes deeper. The learned high-frequency features of each stage peak at the stage’s final layer, as indicated by a greater quantity of high-frequency components at the layer marked by the dashed line when compared to the previous layer. Our findings also demonstrate that, in comparison to the vanilla model, the adversarially trained model prioritizes low-frequency features, with an overall reduction in the number of high-frequency components, consistent with the findings outlined in [16] regarding the smoothing of convolutional kernels by adversarial training. Based on these observations, we hypothesize that incorporating FPCMs closer to the end of each stage can effectively reduce high-frequency signals and further enhance model robustness. To this end, we present an overview of our meta-model in Fig.2, utilizing the ResNet18 architecture as an example. We insert FPCM ”painlessly” without destroying the original model architecture, which also works for other kinds of models like WideResNets.

Recently, Xu et al. [36] studied the F-principle of deep learning both empirically and theoretically, revealing that DNNs first model the low-frequency features in data. The F-Principle is a salient and intuitive notion, as it aligns with the principles governing human visual perception. Specifically, when humans encounter unfamiliar stimuli, they tend to process and retain coarse, high-level information about its shape and structure prior to encoding more detailed features. Empirically, the weight loss landscape is a widely used indicator to characterize the standard generalization gap in a standard training scenario. For robustness learning, it is also indicated in previous work that a flatter loss landscape leads to higher test robustness as long as the training is sufficient. Also in line with the frequency principle, early stopping can improve the generalization ability of DNNs in practice.

Taking these mentioned aspects into consideration, we endeavor to enhance the concordance between the learning process of the original model and FPCM by modifying the cutoff frequency. Intuitively, we adjust the cutoff frequency linearly as follows:

$$D_{0_{t}}=\frac{S}{2}+\frac{t*(\frac{S}{8}-\frac{S}{2})}{T}$$

where $t$ is the current model training epoch, $T$ represents the total number of training epochs, and $S$ is the length of the frequency spectrum. Therefore, in the process, we gradually decrease the cutoff frequency from $S/2$ to $S/8$. The same setting is used for the perturbations generated during training, but the cutoff frequency is fixed to $S/8$ during evaluation. The impetus behind our approach is to expedite the process of convergence while simultaneously enabling the model to attend to information with complete frequency during the initial training phase. However, we aim to employ more low-frequency signals in the terminal stage, which we believe is to be beneficial for robust learning.

Our method is a plug-and-play component with no interventions in adversarial training. Therefore, it can be easily combined with other AT methods like TRADES which provides optimized loss design, and LAS which introduces an automatic strategy for generating adversaries.

In order to provide additional evidence of the efficacy of our proposed methodology, we endeavored to create a model using WRN-70-16 architecture and compared its performance against state-of-the-art robust models without additional training data. The results are presented in Table. 5. Our approach, build upon AWP, was found to yield greater robust accuracy under AA. Moreover, the performance enhancement yielded by our method is on par or outperforms that observed with the potent LAS technique. Notably, we also can build a more robust model based on LAS-AWP.

We utilize two hyper-parameters that merit further investigation, where $\alpha$ governs the trade-off between low- and high-frequency components and $\beta$ regulates the cutoff frequency of the low-pass filter.