Amplitude-Phase Recombination: Rethinking Robustness of Convolutional Neural Networks in Frequency Domain

Beyond the examples in Figure 2 and 3, here more qualitative analyses are given to measure the contributions of amplitude and phase. Several experiments are conducted on CIFAR-10 [28] to evaluate the performances of the CNNs which are trained with the inversed images by various types of amplitude and phase spectra. For the image $x$, its frequency domain $\mathcal{F}_{x}$ is composed by amplitude $\mathcal{A}_{x}$ and phase $\mathcal{P}_{x}$ as:

where $\otimes$ indicates the element-wise multiplication of two matrices. Here, four types of amplitude spectra, $\mathcal{P}_{x}$, $\mathcal{P}_{x}^{L}$, $\mathcal{P}_{x}^{I}$, and $\mathcal{P}_{x}^{H}$ are combined with four types of amplitude spectra, including $\mathcal{A}_{x}$, $\mathcal{A}_{x}^{L}$, $\mathcal{A}_{x}^{I}$, and $\mathcal{A}_{x}^{H}$, respectively. Here $\mathcal{A}_{x}^{L}$, $\mathcal{A}_{x}^{I}$, $\mathcal{A}_{x}^{H}$ and $\mathcal{P}_{x}^{L}$, $\mathcal{P}_{x}^{I}$, $\mathcal{P}_{x}^{H}$ represent the amplitude spectrum and phase of low-frequency, intermediate-frequency and high-frequency by low-pass $\mathcal{H}_{l}$, high-pass $\mathcal{H}_{h}$, and band-pass $\mathcal{H}_{b}$ filters, respectively. Noted in Eq.(1), if one element of $\mathcal{A}_{x}$ is zero, then the corresponding element of $\mathcal{F}_{x}$ would be zero, and the phase spectrum $\mathcal{P}_{x}$ is not able to be considered. To alleviate the influence of this, we define the transfer function as:

Finally, $\mathcal{P}_{x}$, $\hat{\mathcal{P}_{x}^{L}}$, $\hat{\mathcal{P}_{x}^{I}}$, and $\hat{\mathcal{P}_{x}^{H}}$ are combined with $\mathcal{A}_{x}$, $\hat{\mathcal{A}_{x}^{L}}$, $\hat{\mathcal{A}_{x}^{I}}$, and $\hat{\mathcal{A}_{x}^{H}}$, respectively.

For quantitative evaluation, we trained the ResNet-18 with the inversed images by the above each pair of amplitude and phase spectra:

where $iDFT$ is the inverse Discrete Fourier Transform (DFT), and $f(\cdot)$ means the CNN model with the learnable parameters $\theta$.

The test accuracies of the model trained by each pair are shown in Figure 4. It is clear that the combination of phase and amplitude in the corresponding frequency domain achieves better performance in their various combinations, which indicates the CNN can capture effective information from both amplitude and phase spectrum. Moreover, when fixing the amplitude spectrum and phase spectrum respectively, the range of change without amplitude is larger than the case without phase according to the two directions of the arrow. It indicates that the convergence of the CNN more relies on the amplitude spectrum and neglects the phase spectrum.

Furthermore, we randomly select $1000$ samples from CIFAR-10. Firstly, we generate 1000 corrupted samples by Gaussian noise and show the distribution of corrupted samples and original samples as shown in Figure 5(a). We could observe the amplitude spectrum in high-frequency of two types of samples is so different while the corrupted sample is just added invisible noise. Hence, CNN would make the wrong prediction when the amplitude spectrum is changed. This is also consistent with the conclusion that CNN captured high-frequency information in [49]. Therefore, we propose an assumption (referred to as A1) that presumes:

Secondly, we randomly select $1000$ OOD samples from CIFAR-100. As shown in Figure 5(b), it is not able to distinguish the amplitude spectrum in high-frequency of in-distribution and out-of-distribution, even these samples are from different categories. As a result, CNN would be overconfident for some distributions when similar amplitude information appears. Therefore, we first attempt to provide an assumption (referred to as A2) for the behaviors of the robustness to common perturbations and the overconfidence of OOD:

The proof is a direct outcome of the previous discussion and thus omitted. The Corollary 1 has been proved in previous works [49, 42, 44] and Corollary 2 can also be verified empirically (e.g., in Figure 2 and 3), therefore we can safely state that these two corollaries can serve as the alternative explanations to the generalization behavior of CNN. Meanwhile, we provide more examples for proof in Appendix.

Previous works [34, 11] have shown many of the important features of a signal are preserved if only the phase spectrum is retained. Meanwhile, several works of image saliency [14, 30] shown the connection of phase spectrum with the fixation of the human visual system. Further, we wish to explore why this important information of the image is retained in the phase spectrum. Here, we reinterpret the concept of discrete Fourier transforms from the perspective of template-based contrast computation [30].

Give a gray image $x$ with resolution $N\times N$, its complex-valued Fourier coefficient at $(u,v)$ can be computed as:

where $\theta=-2\pi(un+vm)/N$. Then, the real and the imaginary parts of $\mathcal{F}_{x}(u,v)$ can be rewritten as:

The frequency in $(u,v)$ by Fourier transform can be interpreted as computing by four template-based contrasts:

Moreover, we can define $4\times N\times N$ templates for an image $x$ based on the signs of the real-part and the imaginary-part. A template-based example is shown in Figure 6. More examples for templates are shown in Appendix.

Meanwhile, the phase spectrum $\mathcal{P}_{x}(u,v)$ for the image $x$ is equal to $\arctan(\frac{\mathcal{I}_{x}(u,v)}{\mathcal{R}_{x}(u,v)})$, which can be reinterpreted as:

In Eq.(4), first, we can observe that the above four templates are encoded in the spectral phase. Hence, all $4\times N\times N$ templates are contained in the phase spectrum. This template-based contrast can help to explain the importance of the phase spectrum. Once the templates containing more targets without distractors are correctly estimated, the model can highly effectively locate the target objects [30]. On the other hand, these templates in the phase spectrum could help to recover the structural information of the original image even without the original amplitude spectrum as shown in Figure 3. The robustness human visual system can also rely on this visible structured information for recognition.

Training Setup. For a model’s resilience to common corruptions and surface variations, we adopt various architectures including an All Convolutional Network [40], a DenseNet-BC ($k=2,d=100$) [25], a 40-2 Wide ResNet [54], and a ResNeXt-29 (32x4) [51]. All networks use an initial learning rate of 0.1 which decay every 60 epochs. All models require 200 epochs for convergence. We optimize with stochastic gradient descent using Nesterov momentum [46]. All input images are processed with ”Standard” random left-right flipping and cropping prior to any augmentations. For the data augmentations of APR-S, we adopt those used in [21] which is shown in Appendix. For the OOD detection, we use ResNet-18 [17] with the same training strategies above. The data augmentations are set up the same as the above. We report the Area Under the Receiver Operating Characteristic curve (AUROC) [20] as a threshold-free evaluation metric for a detection score. We divide all methods into two categories, one is to add one augmentation on the basis of standard augmentations (random left-right flipping, and cropping), and the other is to add a combination of multiple augmentations as [5, 21].

Common Corruptions and Surface Variations. We first evaluate all methods with common corruptions and surface variations, such as noise, blur, weather, and digital. Compared to the Mixup or CutMix based on pair images, our APR-P with exchanging amplitude spectrum in pair images achieves 6% lower absolute corruption error for CIFAR-100 as shown in Table 1. For methods based on a combination of multiple augmentations, our APR-S of the single image with just Cross-Entropy loss (CE) performs better than AugMix with simply mixing random augmentations and using the Jensen-Shannon loss substantially. When combining our method for single and pair images, the APR-SP achieves 5% performance improvement compared with AugMix in CIFAR-100. In addition to surpassing numerous other data augmentation techniques, Table 1 also demonstrates that these gains come from simple recombination of amplitude and phase without a complex mixup strategy. More comparisons and results about test accuracy are shown in Appendix.

Out-of-Distribution Detection. We compare APR with those augmentations (Cutout, and Mixup) and those several training methods, the cross-entropy, supervised contrastive learning (SupCLR) [27], and state-of-the-art method contrasting shifted instances (CSI) [47]. Since our goal is to calibrate the confidence, the maximum softmax probability is used to detect OOD samples. Table 2 shows the results. Firstly, APR-P consistently improves 2% AUROC than Cutout on CIFAR-10 while maintaining test accuracy. Then, after combining APR based on single and pair images, APR-SP exceeds CSI and gains in almost all OOD tasks. APR promotes CNN to pay more attention to the phase spectrum so that some OOD samples that affect CNN’s decision-making in amplitude spectrum could be detected effectively.

Adversarial Attack. Moreover, the phenomenon of CNN focusing on amplitude spectrum leads to a question of whether APR can improve the adversarial robustness of models. Here, we evaluate several augmentations against one adversarial attack, AutoAttack [4]. Table 3 shows the AutoAttack [4] performance by combining different methods with revisiting adversarial training method of FSGM [50] on CIFAR10. The cutout is not able to effectively against adversarial attacks compared with the baseline with revisiting adversarial training method of FSGM [50]. On the contrary, APR could effectively against AutoAttack while maintaining test accuracy. Compared with APR-P, APR-S for single images achieves more improvement on AutoAttack. Furthermore, the combination of these two strategies achieves better performance. It is evident that APR-SP improves the ability of the original model not only on clean images but also against adversarial attacks.

Training Setup. ResNet-50 [17] is trained with an initial learning rate of 0.1 which decay every 30 epochs. It is optimized with stochastic gradient descent using momentum $0.9$ [35], and requires 100 epochs for convergence. All input images are pre-processed with standard random cropping horizontal mirroring. For the data augmentations of APR-S, we adopt those used in [21] without augmentations such as contrast, color, brightness, sharpness, and Cutout, which may overlap with the corruptions of ImageNet-C. Following [21], we utilize the convention of normalizing the corruption error by the corruption err of AlexNet [29]. Corruption Error ($CE_{c}$) is computed as $CE_{c}=\sum_{s=1}^{5}E_{c,s}/\sum_{s=1}^{5}E_{c,s}^{AlexNet}$. The average of the 15 corruption errors is as the Mean Corruption Error (${mCE}_{c}$).

Results. Our method APR-SP achieves 15% improvement than the baseline $80.6\%$ ${mCE}_{c}$ while maintaining test accuracy. Other methods such as AutoAugment and AugMix require a more complex combination strategy, while ours does not. Meanwhile, APR improves corruption robustness [21] and uncertainty estimates across almost every individual corruption and severity level while the performance of zoom blur is comparable with most methods. APR-SP gets about 5% improvement than APR-S and APR-P, and APR-SP with DeepAugment improves 6% than the reproduced DeepAugment [19]. As shown in Figure 8, the CNN trained with APR-SP is able to focus on the parts of the target object for classification even in a heavy fog. These results demonstrate that scaling up APR from CIFAR to ImageNet also leads to state-of-the-art results in robustness and uncertainty estimation.

For our proposed APR-P, we utilize the labels of phase spectrum in the pair samples. Naturally, we wish to explore the impact of using labels amplitude and phase separately. Here, we add a linear classifier layer in ResNet-18 to predict the labels of the amplitude spectrum. The model is trained for the sample $\bar{x}$ combined by the phase spectrum $\mathcal{P}_{x_{i}}$ and the amplitude spectrum $\mathcal{A}_{x_{j}}$ by optimizing:

Then, the final prediction is defined as $\bar{y}=\lambda f_{\mathcal{P}}+(1-\lambda)f_{\mathcal{A}}$. The recognition ability of the model to different distribution changes with $\lambda$ as shown in Figure 9. With the enhancement of the weight of phase prediction, the accuracy of the model is improved, especially for common corruptions and surface variations, and OOD detection. Meanwhile, the detection ability of the model for OOD samples becomes stronger with the increase of phase attention. This result could further prove the correctness of our corollaries.

In order to better understand the reliance of our methods on different frequencies, here we measure the model sensitivity to the additive noise at differing frequencies. We add a total of $33\times 33$ Fourier basis vector to the CIFAR-10 test set, one at a time, and record the resulting error rate after adding each Fourier basis vector. Each point in the $33\times 33$ sensitivity heatmap shows the error rate on the CIFAR-10 test set after it has been perturbed by a single Fourier basis vector. Points corresponding to the low-frequency vectors are shown in the center of the heatmap, whereas the high-frequency vectors are farther than the center.

In Figure 13, we observe that the standard model is robust to the low-frequency perturbations but severely lacks robustness to the high-frequency perturbations, where the error rates exceed $80\%$. Then, the model trained by APR-P is more robust to all frequencies, especially to the low and intermediate frequencies. Moreover, the model trained by APR-S maintains robustness to low-frequency perturbations and improves robustness to the high-frequency perturbations, but is still sensitive to the additive noise in the intermediate frequencies. This further explains the experiments in Section 5.1.2 (main text) that APR-P improves performances of both OOD detection and defense adversarial attacks tasks. From Appendix A, the adversarial samples are more different from original samples in intermediate and high frequencies, while the OOD samples may share similarities with the original samples in any frequencies. The gains of APR-P and APR-S to different frequency domains bring the gains to different tasks.

Furthermore, APR-SP (combining APR-S and APR-P) could substantially improve robustness to most frequency perturbations. The weak sensitivity to the intermediate frequencies is reasonable because of the gains for target prediction from intermediate frequencies in Appendix B.

Table 5 reports clean error [21] of CIFAR-10 by different methods, and the proposed method achieves the best performances on various backbone networks. APR-SP not only improves the model adaptability to the common corruptions, surface variations and OOD detection, but also improves the classification accuracy of the clean images.

We conduct OOD experiments on the larger and more difficult ImageNet-1K dataset [38]. ImageNet-O [22] is adopted as the out-of-distribution dataset of ImageNet-1K. ImageNet-O includes 2K examples from ImageNet-22K [38] excluding ImageNet-1K. The ResNet 50 [17] is trained on ImageNet-1K and tested on both ImageNet-1K and ImageNet-O.

In order to evaluate the accuracy of in-distribution and the ability of OOD detection simultaneously, we introduce Open Set Classification Rate (OSCR) [9, 2] as an evaluation metric. Let $\delta$ is a score threshold. The Correct Classification Rate (CCR) is the fraction of the samples where the correct class $k$ has maximum probability and has a probability greater than $\delta$:

where $\mathcal{D}_{\mathcal{I}}^{k}$ is the interest in-distribution classes that the neural network shall identify. The False Positive Rate (FPR) is the fraction of samples from OOD data $\mathcal{D}_{\mathcal{O}}$ that are classified as any in-distribution class $k$ with a probability greater than $\delta$:

A larger value of OSCR indicates a better detection performance. As shown in Table 6, APR-SP performs better than the standard augmentations even on the large and difficult dataset. Especially, APR-SP achieves about 22% improvement on AUROC. From the OSCR, APR-SP improves the performances of the OOD detection while maintaining test accuracy. These results indicate the excellent scalability of APR in larger-scale datasets.

We also list more visualization examples with various corruptions in Figure 14 and 15, the CNN trained by APR-SP is able to focus on the target objects for classification even with different common corruptions and surface variations.