A Light-weight Deep Learning Model for Remote Sensing Image Classification

We assessed various convolutional neural network (CNN) based architectures for both the teacher and the student models by evaluating twelve different benchmark deep convolutional neural networks: MobileNet, MobileNetV2, ResNet50, Resnet151V2, InceptionV3, InceptionResNetV2, DenseNet121, DenseNet201, EfficientNetB0, EfficientNetB7, ConvNeXtTiny, and ConvNeXtLarge, all available in the Keras library [26]. As the top of Figure 1 shows, the benchmark networks are first trained with the ImageNet dataset [27], referred to as the up-stream task. Then, the first layer to the global pooling layer of these pre-trained networks are extracted and combined with a Dense Layers block to perform the down-stream RSIC task as shown at the bottom of Figure 1. In other words, we apply a transfer learning method in which the first layer to the global pooling layer, trained from the up-stream task using the ImageNet dataset [27], are transferred to the down-stream RSIC task. The Dense Layers block is considered to house the adapting layers for the down-stream RSIC task.

We also apply data augmentation for the RSIC down-stream task, namely Image Rotation [28] and Mixup [29], performed on the remote sensing image input dataset. In particular, all images in an original RSIC dataset are rotated using three different angles: 90, 180, and 270°. Since three angles are used, the augmented dataset is four times larger than the original. Next, batches of 60 images are randomly selected from the new dataset. For each batch, we apply the Mixup [29] method to mix the images within one batch with random ratios. Both Uniform and Beta distributions are used to generate the mixup ratios, and we make use of both the rotation augmented image database in addition to the new mixup images; as a consequence the batch size increases by three times from 60 to 180 images.

Thanks to the use of Mixup [29] for data augmentation, the labels will no longer be in one-hot encoding format. Therefore we apply Kullback-Leibler divergence (KL) loss [30] instead of Entropy loss to train the evaluating models, as in equation 1:

where $\Theta$ presents trainable parameters, the constant $\lambda$ is empirically set to $0.0001$, the batch size $B$, the number of classes C, $\mathbf{y_{bc}}$ and $\mathbf{\hat{y}_{bc}}$ denote expected and predicted probabilities of an input image, respectively. Note that we set the low learning rate to be 0.0001 and none of trainable parameters are frozen during the training process.

Given $N$ high-performance models selected from Phase I, we then develop and train the teacher architecture during this phase. Again, we leverage parameter based transfer learning techniques to develop the teacher as shown in Figure 2. In particular, the first layer to the Dense Layer 01 of Dense Layers block from all $N$ high-performance networks described in Phase I are reused and then combined to generate a composite high-level feature. If we consider $N$ vectors $\mathbf{e_{n}}\in R^{512}$ as the output of the Dense Layer 01, the Combination block used to generate the composite high-level feature in Figure 2 by,

where $\mathbf{w_{i}},\mathbf{b}\in R^{512}$ are weight and bias trainable parameters. The high-level feature is finally transferred into a Fully Connected layer followed by a Softmax for classifying to target classes. When we finish training the teacher model, the high-level features are then extracted and used for the knowledge distillation process to train the student in Phase III which follows.

Data augmentation is used when training the teacher network, however only Image Rotation [28] is applied at this and thus the labels can remain in one-hot format, and Entropy loss can be used to train the teacher model as in equation 3:

where $\Theta$ are trainable parameters, the constant $\lambda$ is set to $0.0001$, the batch size $B$ and the number of classes C, $\mathbf{y_{bc}}$ and $\mathbf{\hat{y}_{bc}}$ denote expected and predicted probabilities of a particular image, respectively.

We again set the low learning rate to 0.0001, and the trainable parameters of the first layer to the Dense Layer 01 are frozen when training the teacher. In other words, only trainable parameters in the Combination block and in the finally Fully Connected layer are updated during the training process.

From the results in Phase I, a network architecture, which not only performs well but also presents a low footprint, is selected and considered as the base student model. We then train the student with the high-level features extracted from the teacher mentioned in Phase II. As Figure 3 shows, the student is trained with two loss functions. While the first Entropy loss is used for the classification task, the Euclidean Distance loss helps to ensure the high-level features of the student become closer to the high-level features extracted from the teacher, effectively guiding the feature discrimination ability of the student. The ratio between both losses is empirically set to 0.5/0.5.

Regarding the data augmentation used to train the student, only Image Rotation [28] is applied. We also set the low learning rate of 0.0001 and no trainable parameters are frozen during the student training phase.

In this paper, the benchmark dataset of NWPU-RESISC45[10] is used to evaluate all state of the art and proposed models. The dataset, which was collected from more than 100 different countries and regions around the world, consists of 31,500 remote sensing images separated into 45 scene classes. Each class comprises 700 RGB images with a resolution of $256\times 256\times 3$. To compare with state-of-the-art systems, we comply with the original settings mentioned in [10]. We then split the NWPU-RESISC45 dataset into Training and Testing sets with two different ratios: 10%-90% and 20%-80%, respectively.

As the Accuracy (Acc.%) has been used as the main metric to compare performance among the RSIC systems, we also apply the metric in this paper. Additionally, as we aim to achieve a low complexity model for the RSIC task, we compute the number of trainable parameters (M) to compare against state-of-the-art RSIC systems.

We constructed our proposed deep learning networks with Tensorflow using the Adam method [42] for optimization. The training and evaluating processes are conducted on two Titan RTX 24GB GPUs. The results presented in this paper are all the average scores from 10 individual experimental runs.

As experimental results show in Table I, we can see that ConvNeXt, EfficientNet, DenseNet based models are competitive and outperform MobileNet, ResNet and Inception based models. Particularly, the best network architectures of ConvNeXtLarge and ConvNeXtTiny achieve 95.3% and 93.0% accuracy, respectively. Around 2% worse than ConvNeXtLarge, the performance of EfficientNetB7 and DenseNet201 on the NWPU-RESSIC45 task are 93.6% and 93.3%, respectively. Meanwhile, their smaller variants named DenseNet121 and EfficientNetB0 achieve over 92% accuracy.

Although ConvNeXt, EfficientNet and DenseNet based models perform well among the evaluating network architectures, these involve large footprints. In particular, the three best variants, namely ConvNeXtLarge, EfficientNetB7, and DensNet201 have some of the largest parameter set sizes of 196.6, 65.1, and 19.1 M, respectively. Among the ConvNeXt, EfficientNet and DenseNet variants, only EfficientNetB0 combines a relatively good accuracy of 92.3% with a low complexity footprint (4.7 M parameters). As a result, we select EfficientNetB0 as the foundation network for the student model required in Phase III. We also note that DenseNet201, EfficientNetB7 and ConvNeXtLarge perform better than 93% and their general architectures are dissimilar to each other. We therefore, select these three network architectures to generate the teacher, as required in Phase II.

As Table II shows, the teacher (i.e. a combination of DenseNet201, EfficientNetB7 and ConvNeXtLarge) achieves an accuracy of 96.2%, but with a very large footprint of 280.8 M parameters. Knowledge distillation from this capable teacher into student EfficientNetB0 allows it to achieve an accuracy of 94.8% while maintaining a low complexity of 4.7 M parameters. To propose a wide range of low complexity models, we further evaluate variants of the student EfficientNetB0 model. In particular, variants of the student are generated by removing certain convolutional blocks in the EfficientNetB0 backbone architecture to reduce complexity further. EfficientNetB0-6B to EfficientNetB0-3B are variants of EfficientNetB0 obtained by removing convolutional block 7 only, removing convolutional blocks 6 and 7, removing all convolutional blocks from 5 to 7 and removing all convolutional blocks 4 to 7 inclusive. Experimental results in Table II indicate that when the footprint of EfficientB0 based students is reduced, the accuracy performance also tends to decrease. However, we can achieve a very low complexity model of 0.37 M parameters with a performance of 91.3% from EfficientNetB0-4B, which opens the potential for RSIC applications on a very wide range of edge devices.

Finally, we compare our proposed models to the state-of-the-art RSIC systems basing on two criteria: (1) accuracy performance without any model complexity constraint and (2) accuracy performance with a constraint of 5 M trainable parameters maximum. As Table III shows, RSIC performance with the first criterion reveals that our proposed teacher (i.e. a combination of ConvNeXtLarge, DenseNet201, and EfficientNetB7) outperforms the state-of-the-art systems, achieving 94.6% and 96.2% for the training/testing settings of 10/90 and 20/80, respectively. For the second criteria, i,e, low-complexity RSIC models ($<$ 5 M trainable parameters) shown in Table IV, our proposed student with knowledge distillation also outperforms the state-of-the-art systems on both training/testing split arrangements, yielding results of 93.3% for a 10/90 split ratio and 94.8% for a 20/80 split ratio.