Double Reverse Regularization Network Based on Self-Knowledge Distillation for SAR Object Classification

In the offline distillation that uses the pre-trained model as its offline teacher [14], the student in the early stage of training contains limited knowledge about the data. Meanwhile, the teacher has abundant knowledge about the data and can provide high-quality soft targets for the student to learn. As student continue to learn more, the fixed soft targets provided by teacher become less beneficial to student training compared to earlier stage. Similar analysis can also be conducted in self-knowledge distillation. Thus, it can be inferred that the weight of offline distillation should decrease and the weight of self-distillation should increase as student performance grows.

We design the experiment to verify the inferences. The VGG-11 is trained for 100 epochs on OpenSARShip via DLB [16] and Tf-KD [14], respectively. The other experimental settings are the same as those in section 3. Due to the slow convergence of VGG-11, the strategy for DLB is setting a fixed weight of 0.5 for the first 50 epochs. Finally, the weights of the $t^{th}$ epoch for DLB and Tf-KD are calculated respectively according to the following formula:

$$\alpha_{t}=\frac{t}{T},\alpha_{t}^{\prime}=1-\frac{t}{T},$$

(1)

where $T$ is the total number of epoch.

The results are shown as Table 1. The best improvement of 5.07% and 2.08% is achieved by DLB and Tf-KD when the fixed weight is replaced by $\alpha_{t}$ and $\alpha_{t}^{\prime}$, respectively. Moreover, the significantly reduced accuracy variance indicates that the dynamic weight can enhance the stability and generalization of classification model. But the adoption of dynamic weight also brings some new challenges. Each of them presents a high risk of overfitting when the distillation weight is small. And the weight update strategy based on the number of epoch may not be reasonable enough due to the network performance not always improving with the increase of epoch.

Inspired by the multi-teacher offline distillation weight assignment [17], we exploit the cross-entropy between the softened teacher predictions and true labels to assign adaptive weight as follows:

$$\mathcal{L}_{ON}=-\sum_{c=1}^{C}y^{c}\log\left(\varphi\left(\boldsymbol{p}_{\theta_{t-1}}/\alpha_{\tau}\right)\right),$$

(2)

$$\mathcal{L}_{OF}=-\sum_{c=1}^{C}y^{c}\log\left(\varphi\left(\boldsymbol{p}^{T}/\alpha_{\tau}\right)\right),$$

(3)

where $\boldsymbol{p}_{\theta_{t-1}}$ and $\boldsymbol{p}^{T}$ are the predictive distributions of ${t-1}^{th}$ iteration student and offline student, respectively. The temperature-like parameter $\alpha_{\tau}$ softens the distribution to smooth the values of the two equations. As the student continues to learn more, the value of Eq. 2 will be close to the value of Eq. 3. So, the online and offline distillation weights at the $t^{th}$ iteration can be calculated by:

$$w^{t}_{LB}=exp(\mathcal{L}_{OF}-\mathcal{L}_{ON}),$$

(4)

$$w^{t}_{KD}=\alpha-w^{t}_{LB},\alpha\geq w^{t}_{LB}.$$

(5)

The value of Eq. 4 will gradually increase and finally fluctuate around a certain number (near 1). Conversely, the value of Eq. 5 will gradually decrease but never be less than zero. $\alpha$ controls the distillation weight range of offline student to represent its importance during the distillation process. So the AWA module can achieve adaptive assignment of distillation weight based on network performance.

In this paper, we focus on single-objective fully-supervised classification tasks. We denote $\boldsymbol{z}_{\theta}=\left[z^{1},\ldots,z^{C}\right]$ as the logits outputted by the network $\theta$, where $C$ is the class number. The predicted probability of $c$-th class is calculated by a softmax function:

$$p_{i}\left(c\right)=\varphi\left(z^{c}\right)=\frac{\exp\left(z^{c}\right)}{\sum_{j=1}^{C}\exp\left(z^{j}\right)},c\in\{1,2,\cdots,C\}.$$

(6)

The student is trained by optimizing the Kullback-Leibler (KL) divergence loss between the softened outputs from the last mini-batch and the current mini-batch:

$$\mathcal{L}_{LB}=\tau^{2}\cdot KL\left(\varphi\left(\boldsymbol{z}_{\theta_{t-1}}/\tau\right)\|\varphi\left(\boldsymbol{z}_{\theta_{t}}/\tau\right)\right),$$

(7)

where $\boldsymbol{z}_{\theta_{t-1}}$ is the output logits of the $t-1^{th}$ iteration student. Then they could be expressed as teacher and student, respectively. $\tau$ represents the distillation temperature, which regulates the degree of smoothing of the network outputs. Similarly, the KL divergence loss between the softened outputs from the offline student and the current mini-batch is

$$\mathcal{L}_{KD}=\tau^{2}\cdot KL\left(\varphi\left(\boldsymbol{z}^{T}/\tau\right)\|\varphi\left(\boldsymbol{z}_{\theta_{t}}/\tau\right)\right),$$

(8)

where $\boldsymbol{z}^{T}$ is the output logits of offline student. To train a multi-class classification model, we typically adopt the Cross-Entropy (CE) loss between the predicted and ground-truth label distributions as the loss function:

$$\mathcal{L}_{CE}=-\sum_{c=1}^{C}y^{c}\log\left(\varphi\left(z_{\theta}^{c}\right)\right).$$

(9)

Eventually, the overall loss function of our DRRNet-SKD is summarized as:

$$\mathcal{L}=\mathcal{L}_{CE}+w_{LB}\cdot\mathcal{L}_{LB}+w_{KD}\cdot\mathcal{L}_{KD}.$$

(10)

Data Description. The OpenSARShip Dataset has been widely used to evaluate SAR ship classification methods since 2018. Three primary ship categories are utilized for training and testing. The FUSAR-Ship dataset has a higher resolution and a larger number of samples compared to OpenSARShip. The seven types of ships used in this experiment, i.e., fishing, bulk, container, tanker, other cargo, general cargo, and others. Experiment settings. All models are trained for 100 epochs using Adam optimizer with a learning rate of 0.0002, and the learning rate is decayed by 20% every seven epochs. The batchsize is set to 16 on OpenSARShip and 32 on FUSAR-Ship. Since Batch Normalization(BN) [19] has been widely used in VGG, Pytorch provides a version with BN layers that will be employed in our experiments.

As shown in Table 2, the results show that our proposed method can significantly improve the performance of all the backbone networks, even on the larger and higher-resolution SAR ship dataset. It can be found that our framework is more effective in the VGG series. VGG-11 obtains the highest improvement among the backbones on OpenSARShip, achieving an accuracy gain of 6.96% over the baseline. VGG-19 achieves the most improvement on FUSAR-Ship, increasing the accuracy by 4.46% over the baseline. The results sufficiently demonstrate the effectiveness of our method.

To demonstrate the superiority of our framework, DRRNet-SKD is compared against classical LSR and the best regularization methods based on self-knowledge distillation. As shown in Table 2, we observe that the first three regularization methods, LSR, Tf-KD, and DLB, can improve the performance of almost all backbones. However, the same regularization method does not always work best for different backbones. For instance, on FUSAR-Ship, LSR improves the accuracy of AlexNet and VGG-16 by 0.95% and 1.90%, respectively, surpassing the gains achieved by Tf-KD and DLB. But this improvement is not observed on the other three networks. Our DRRNet-SKD consistently can achieve the best results for different networks on both datasets, demonstrating its superior effectiveness and universality compared to other regularization methods. To provide a clearer comparison, we plotted the training performance of the four networks for baseline, DLB, and our method on FUSAR-Ship, as shown in Fig. LABEL:Fig:training_performance. Compared to DLB, our method can further accelerate the convergence speed and improve the accuracy.