Robust and Accurate Object Detection via Self-Knowledge Distillation

Vanilla-FA proposes Knowledge-Distilled Feature Alignment (KDFA) and Self-Supervised Feature Alignment (SSFA) to guide the output of intermediate feature layer. Without loss of generality, the norm $\|\cdot\|$ is used to measure the distance between two feature vectors, which can be formulated as follows:

where $x_{clean}$, $x_{adv}$, $S(\cdot)$, $T(\cdot)$ are clean images, adversarial images, feature extractor of student and teacher. And $p$ specifies a particular norm. $\mathcal{L}_{\text{fea1}}$ denotes KDFA branch, which guides $S(x_{adv})$ to imitate $T(x_{clean})$. $\mathcal{L}_{\text{fea2}}$ denotes SSFA branch between $S(x_{adv})$ and $S(x_{clean})$, which are generated from student detector.

$S(x_{clean})$ and $T(x_{clean})$ are served as soft labels to guide the learning of $S(x_{adv})$. Nevertheless, KDFA and SSFA are two independent branches and $S(x_{clean})$ is mutable during the training process. Therefore we can not ensure the credibility of $S(x_{clean})$ in the guiding process. Intuitively, mutable $S(x_{clean})$ will make SSFA task easier to achieve than KDFA. We conjecture the inconsistency of KDFA and SSFA can bring imbalanced competitions during training, which is the main reason why clean precision is lower than standard training in Vanilla-FA. Moreover, inconsistent soft labels might hurt the distillation performance, leading to further potential conflicts and sub-optimal results.

We propose decoupled knowledge-distilled feature alignment branches to avoid potential conflict between vanilla KDFA and SSFA branch. Following the triangle inequality, we can disentangle vanilla KDFA branch into two sub-branches, which can be expressed as follows:

where $\mathcal{L}_{\text{fea3}}$ denotes a new KDFA branch between $S(x_{clean})$ and $T(x_{clean})$. In feature alignment framework, initial state can be expressed as subfigure t0 in Figure 3 when we use the teacher detector to initialize student detector. In that case $S(x_{clean})$ and $T(x_{clean})$ are completely coincident in feature space. When we minimize $\mathcal{L}_{\text{fea2}}+\mathcal{L}_{\text{fea3}}$, the optimal directions of $S(x_{clean})$ and $S(x_{adv})$ can be determined, as shown in subfigure t0 and subfigure t1 of Figure 3. Finally, $\mathcal{L}_{\text{fea2}}+\mathcal{L}_{\text{fea3}}$ will converge in a balanced state in total loss function, as shown in subfigure t2 of Figure 3. During the training process, $S(x_{adv})-S(x_{clean})$ and $S(x_{clean})-T(x_{clean})$ are always the same direction in feature space. Specifically, $\mathcal{L}_{\text{fea1}}$ and $\mathcal{L}_{\text{fea2}}+\mathcal{L}_{\text{fea3}}$ are exactly equivalent when $p=1$.

Theoretically, minimizing $\mathcal{L}_{\text{fea1}}$ and minimizing $\mathcal{L}_{\text{fea2}}+\mathcal{L}_{\text{fea3}}$ have the same optimal solutions, namely $S(x_{adv})$, $S(x_{clean})$ and $T(x_{clean})$ coinciding in feature space. However, the parameters of student model are different from teacher during the training process. Therefore the optimal solutions are almost impossible to obtain under total loss function in detection task. However, when $\mathcal{L}_{\text{fea2}}+\mathcal{L}_{\text{fea3}}$ or $\mathcal{L}_{\text{fea1}}$ converges in a balanced state, $S(x_{clean})$ and $S(x_{adv})$ will be constrained to the manifold of $T(x_{clean})$ in feature space. [41] demonstrates that on-manifold adversarial samples can boost generalization capability in classification. Also we argue that decoupled knowledge-distilled feature alignment branches constrains $S(x_{adv})$, $S(x_{clean})$ and $T(x_{clean})$ in the same manifold, which can boost generalization capability across adversarial samples and clean samples in object detection.

Based on the above analysis, UDFA is designed by adding a new KDFA branch between $S(x_{clean})$ and $T(x_{clean})$ based on Vanilla-FA. The overall architecture of UDFA is illustrated in Figure 2. The teacher detector is trained on clean datasets with standard training setting, and served as pretrained model to initialize student in adversarial training. For fair comparisons, we do not use more powerful teacher in our UDFA framework. Therefore, UDFA is essentially a self-distillation process, which aims at exploring more information from pretrained detector. Unlike traditional knowledge distillation, our framework also involves self-supervised learning branch to improve robustness, and aims at addressing the conflict between original knowledge distillation task and self-supervised learning task. $S(x_{clean})$ in $\mathcal{L}_{\text{fea2}}$ will be conducted stop-grad operation inspired by SimSiam [7]. In order to improve the effectiveness of distillation, we adopt decoupled fore/back-ground features [16] instead of global pooling feature used in Vanilla-FA. Compare with global pooling feature, decoupled features contains location information by distinguishing foreground and background feature, more friendly to object detection task. The overall training targets can be summarized as:

where $\mathcal{L}_{\text{loc}}(\cdot)$ is the location loss of detector (e.g., GIoU loss [38]) and $\mathcal{L}_{\text{cls}}(\cdot)$ is the classification loss (e.g., focal loss [29]). $\alpha$ is the weight to balance the detection loss of clean images and adversarial images. $\mathcal{L}_{\text{fea}}(\cdot)$ is the feature alignment loss in UDFA. $\beta$ controls the trade-off between detection loss and feature alignment loss.

We conduct experiments across one-stage detectors (e.g. GFLV1 [26] and GFLV2 [25]) and two-stage detectors (e.g. Faster-RCNN [28]) with ResNet [19] backbone of different scales. We use MMDetection [3] as the codebase and provide a modified version for various adversarial training algorithms. All the pre-trained detectors (also served as teacher detectors in our work) fine-tuned from ImageNet [11] pre-trained classifiers. The experiments of standard training and various adversarial training methods use the same hyper-parameter settings. To be more specific, we use SGD optimizer with initial learning rate, $10^{-2}$, momentum, $0.9$, weight decay $0.0001$. For adversarial training, $\alpha$, $\beta$ are set to $0.5$, $1.0$ respectively. In order to reduce the catastrophic overfitting phenomenon [1, 46], we use a small step size in PGD-1 training following Vanilla-FA [51]. The step size is set to $2/255$ for training. In particular, original TOD-AT uses a large step size ($8/255$) and only uses adversarial samples for training. Experiments in [51] demonstrate that adding clean samples and using a small step size can largely improve performance over results in the work of Zhang et al. [53]. For PASCAL-VOC and minitrain2017 of MS-COCO datasets, we repeat sampling each image three times during each epoch. Unless otherwise stated, we apply standard 1x learning schedule (12 epochs), which decays at $9th$ and $12th$ epochs respectively with decay factor $0.1$.

We first verify the effectiveness of our proposed UDFA on typical one-stage detector GFLV1 and GFLV2. Comparison of the results on PASCAL-VOC benchmark are shown in Table 1. For adversarial robustness, we use different PGD steps to generate adversarial samples with a standard perturbation budget setting (8/255) for test, i.e., $\varepsilon=8$. Without loss of generality, we also report the average adversarial AP under different steps (1,2,4), which is denoted as adv AP. See Appendix A for the corresponding performance under different PGD steps. We also use standard settings (without adversarial training) to fine-tune pretrained detector for a fair comparison, and denote it as STD. It is observed that STD obtains lower clean AP and higher adv AP than pretrained detector. According to our previous experience, we will attribute the decline in clean precision to the overfitting phenomenon. However, the improvement on robustness demonstrates that there might be a tradeoff between clean accuracy and robustness even without adversarial training settings. For adversarial training algorithms, similar to the previous observations, Vanilla-AT, TOD-AT and Vanilla-FA will hurt clean average precision than standard training. Det-AdvProp can improve clean average precision and robustness than standard training. We can find that for the $AP$ and $AP50$ metric, our UDFA achieves the best performance across clean dataset and adversarial dataset. For more strict $AP75$ metric, UDFA can achieve better or comparable performance compared with existing methods. In particular, UDFA can improve student by about 1.0 clean AP than teacher detector across GFLV1 and GFLV2. Also UDFA can improve adv AP by at least 6.0 points over teacher.

We further examine the impacts of different hyper-parameters on clean precision and robustness. There are three main hyper-parameters in the UDFA, $\lambda$, $\gamma_{fg}$ and $\gamma_{bg}$. In order to verify the effectiveness of decoupled features and decoupled branches, we explore different combinations of hyper-parameters for ablation study. Considering that the magnitude of gradients from foreground regions are larger than that from background regions, $\gamma_{bg}$ is larger than $\gamma_{fg}$ to balance gradients in our experiments.

As shown in Table 2, we can find that decoupled fore/back-ground features ($\gamma_{fg}>0$ or $\gamma_{bg}>0$) and the decoupled knowledge-distilled feature alignment branches ($\lambda=1$) can both improve the clean precision than Vanilla-FA. More concretely, when we use default setting $\gamma_{fg}=1$ and $\gamma_{bg}=6$, decoupled features boosts GFLV1 and GFLV2 by 1.0 to 1.4 clean AP, and decoupled branches can further improve 2.1 to 2.8 clean AP than Vanilla-FA. Specifically, if we only use background feature to distill knowledge, there are also an improvement in clean AP than Vanilla-FA. But combining background feature with foreground feature can get better results. We can also see that the clean AP is not sensitive to different $(\gamma_{fg},\gamma_{bg})$ pairs in a certain range.

For robustness, the result of adv AP is shown is Table 2, and performance under different PGD steps is shown in Appendix A. On GFLV1-Res18, decoupled features can obtain comparable robustness than Vanilla-FA, and overall UDFA can improve robustness by 0.2 adv AP. On GFLV2-Res18, decoupled features and decoupled branches can consistently improve the robustness than Vanilla-FA. On GFLV2-Res34, although adding decoupled branches lowers accuracy by 0.2 adv AP than only using decoupled features in the default setting $\gamma_{fg}=1$ and $\gamma_{bg}=6$, overall UDFA still improve 0.7 adv AP than Vanilla-FA baseline.

Based on UDFA framework, we further explore the effectiveness of combining UDFA with AdvProp. To verify this, we conduct more experiments on GFLV2 and Faster-RCNN with various backbones. The results of GFLV2 are presented in Table 3. It is observed that the version of UDFA with AdvProp improves the clean precision by $+1.5\sim+1.6$ AP than SOTA algorithm Det-AdvProp. For robustness, our approach improves by $+0.3\sim+1.0$ AP.

For Faster-RCNN, considering that Det-AdvProp does not conduct experiments on two-stage detectors, we first extend Det-AdvProp to Faster-RCNN. Det-AdvProp disentangles classification and localization loss to generate adversarial samples for training. Intuitively, we set $\mathcal{L}_{\text{loc}}$ consisting of $\mathcal{L}_{\text{roi-loc}}$ and $\mathcal{L}_{\text{rpn-loc}}$, $\mathcal{L}_{\text{cls}}$ consisting of $\mathcal{L}_{\text{roi-cls}}$ and $\mathcal{L}_{\text{rpn-cls}}$. As shown in Table 4, we can find that our approach can improve at least 2.1 clean AP while obtain comparable adversarial robustness.

We further conduct experiments on MS-COCO to verify our approach. The pretrained detector is trained with 1x schedule, and other methods are trained with 3x (36 epochs) schedule. The results of GFLV2-Res34 trained on train2017 are summarized in Table 5. It is observed that UDFA obtains consistent improvement on val2017 and test-dev2017. For example, UDFA improves clean precision by 1.7 AP over Det-AdvProp on val2017 set, and improves clean precision by 1.9 AP on test-dev2017. Fore ablation studies, we conduct more experiments on GFLV2 trained with minitrain2017. In Table 6, we report the performance on val2017 under different attack strengths. Although there are still large gaps between clean precision and adversarial robustness, our approach achieves better clean precision and robustness than exiting methods.

To push the limit of our approach, we further evaluate models on natural corruptions dataset. We apply the same corrupted augmentation scheme in all methods during training. Specifically, at each iteration, we randomly select one corrupted augmentation from 15 types of corruptions. The severity level is set to 1 during training augmentation. Corrupted augmentation can improve clean AP on standard training and adversarial training methods as shown in Table 3 and Table 7. However, evaluation results of clean AP reveal that Det-AdvProp fails to obtain consistent improvements than pretrained detector with corrupted augmentation. Our approach performs generally well across different backbones and metrics than existing methods even with corrupted augmentation, which further proves the generalization ability of our approach. To further break down the improvement into different types of corruptions, severity levels and classes, the performance gain achieved by UDFA is shown in Figure 4 and Appendix B. We can find that our approach outperforms the standard training on all 15 corruptions and 20 classes on PASCAL-VOC benchmark. We can observe that the largest improvement when the samples are corrupted by jpeg compression (+5.69 AP).