SwinV2DNet: Pyramid and Self-Supervision Compounded Feature Learning for Remote Sensing Images Change Detection

Transformer first has been stand out in natural language processing (NLP) due to its ability of modeling global information [43]. Vision transformer (ViT) is proposed by [44] to bring the spirit of self-attention to image classification, but this huge model cannot be directly applied to other computer vision detection tasks. Swin transformer adopts the inspiration of architectural hierarchy refinement and local information interaction, which reduces the heavy computation involved in modeling global information [45]. Furthermore, [46] proposes Swin V2 that stabilizes the training process caused by the increase of model parameters and mitigates the resolution difference between the upstream and downstream tasks. At the same time, various transformer models are developing rapidly in the field of remote sensing images processing [47, 48, 49, 50].

Due to the lack of low-level details in the pure transformer model [20], transformer is generally combined with CNN for change detection. On the one hand, in the serial combination methods, Chen et al. proposes bitemporal image transformer (BiT) that embeds transformer to CNN to enhance the ability of modeling contexts within the spatial-temporal domain [26]. TransUNetCD mades up for the former omission of the high-resolution shallow information by cascading the up-sampling decoder to achieve the more detailed change map [51]. On the other hand, in the parallel associative approachs, [52] uses transformer and CNN respectively to extract the pair change features for feature fusion, but it also ignores the supplement of low-level details at the decoder stage. Our study adopts a new parallel approach with the high and low level features association to avoid the defects of the above models and obtain the best detection result.

Multi-scale information is always an important tool in image processing. In deep learning, multi-scale features can be divided into three categories: multi-scale features between different layers, multi-scale interaction features between different layers and multi-scale features from different convolution units. The first category is often already implicit in a variety of classic networks, such as ResNet, FCN and U-Net [53, 54, 55]. Feature pyramid transformer (FPT) uses two different transformers to interfuse the features between different layers to generate the second multi-scale features [56]. The third type of multi-scale features is provided by various convolution units, such as inception, dilated convolution, Res2Net-Conv, SK-Conv and so on.

In VHRRS images change detection, [57] extracts the multilevel intertemporal features through the double branches of shared weights, and then performs the information fusion and features difference to obtain the robust change features. The multiscale decoupled convolution is constructed by using atrous convolutions with different dilation rates. The researchers embed several of these convolutions in different layers to acquire the two multiscale features between and within layers [58]. Based on FPT, we propose mixed feature pyramid that provides inter-layer multi-scale interaction information and intra-layer multi-scale information to supplement the main network only with inter-layer multi-scale information.

Since the labeled data requires precious labor costs, self-supervised learning (SSL), in which a large amount of unlabeled data can be used to train networks and extract knowledge that are then transferred to downstream tasks, has become a flourishing deep learning technology recently. SSL is first used in NLP. [59] designs the automatic regression task for words, which provides more contextual information than the task of matching a single sentence to a single label. Then SSL is introduced into computer vision. Contrast learning is one of the representative models [60], that is, the image sample pairs are generated through the network, and the cost function is set to shorten the distance between the two positive samples as far as possible and expand the distance between the positive and negative samples. Mask autoencoder (MAE) is another important branch of SSL [61].

In change detection, Chen and Bruzzone [62] use bootstrap your own latent (BYOL) framework to pretrain the heterogeneous remote sensing images, and then transfer it to the downstream change detection task. The use of ViT encoder and random masking as SSL data enhancement techniques further advances this architecture [63]. SSL has been shown to provide the more discriminative semantic information when combined with the supervised techniques to train the network [40]. Different from [40], we design a new encoder-decoder architecture with VGG16 as the encoder, and apply SSL to this branch architecture to provide the self-supervised semantic features for the main network.

The overall architecture of SwinV2DNet is shown in Fig. 2. In conjunction with Algorithm 1, we elaborate on the details of the network. First of all, we initialize the parameters ${i}$ and ${j}$ to satisfy the following conditions:

where ${N}$ is the set of natural numbers. ${i}$ is the number of layers of SwinV2DNet or VGG16, and ${j}$ is the number of columns of SwinV2DNet. Then we obtain the encoder features of CNN branch network to compensate for the lack of low-level details and single image convolution features of main network:

${CN{N_{SSL}}}$ refers to CNN branch network trained using SSL strategy. ${V_{1}^{\rm{i}},V_{2}^{\rm{i}}}$ and ${I_{1},{I_{2}}}$ represent the encoder feature pair and the original image pair respectively in this paper. On another path in parallel, the features in the first column of main network are generated:

${SwinV{2_{backbone}}}$, ${DConv}$ and ${Cat}$ refer to Swin V2 backbone [46], double convolution module (DConv) and the concatenation in the channel dimension, respectively. ${C}$ denotes the feature from DConv. ${S^{\prime}}$ denotes the feature in column 1 from Swin V2 backbone. The features are estimated by MFP that provides inter-layer multi-scale interaction information and intra-layer multi-scale information:

${S}$ refers to the feature that comes from MFP or Swin V2 block.

At this point, the rest of Swin V2 and FF features in rows 1 to 4 of main network can be generated:

${FF()}$ represents feature fusion module (FF), and ${FF}$ represents the FF features. ${SwinV2}$ represents Swin V2 block. ${Conv}$ and ${Up}$ are used to adjust the spatial and channel resolutions of the features. Then other DConv features are acquired:

Finally, we generate change map ${CM}$ via CBAM module [33] and sigmoid function:

It should also be noted that dense connectivity allows the network to provide more diverse features. And deep supervision solves the problem of features shift in large network during training. The joint of these two techniques allows the network to train stably and converge quickly. FF concatenates, nonlinearizes, and finally uses CBAM to process the heterogeneous features for effective fusion. DConv is the double union of convolution, batch normalization (BN) and ReLU function.

To summarize, the superior detection performance of SwinV2DNet, a parallel compounded architecture, can be attributed to the following reasons. Firstly, main network consisting of Swin V2 blocks is responsible for extracting the change relationship features which are crucial for the change detection task. Secondly, CNN branch network complements main network by providing the low-level pre-changed and post-changed features necessary for accurate detection. Moreover, the incorporation of the multi-scale features and SSL semantic features further enhances the overall feature learning of the network. These combined factors contribute to SwinV2DNet achieving the SOTA detection performance.

Swin-type transformers greatly reduce the number of model parameters while modeling global information through shifted window and hierarchical mechanism [45]. Swin V2 [46] further employs the post-normalization and scaled cosine attention techniques to improve the stability of the large vision model. At the same time, the log-spaced continuous position bias method is used to alleviate the problem of transferring the model trained on low-resolution images to high-resolution images. So we use Swin V2 as the base block to build the main network. It is shown in Fig. 3.

Swin V2 splits the image into the patchs and then models the patchs to generate the features with global information. It is generally pairwise. Its input and output are ${S^{l-1}}$ and ${S^{l+1}}$, respectively. We place layer normalization (LN) after multi-head self-attention (MSA) or multilayer perceptron (MLP) to improve the stability of the network. MLP is to enhance the nonlinear ability of the block. MSA includes window MSA (WMSA) and shifted window MSA (SWMSA): the former extracts the image features in the windows to reduce the computational cost, and the latter maintains the interaction of global information by sliding windows. The residual connection ensures the effective dissemination of information in the large network. The specific cooperation of these components is as follows:

Here, ${S}$ is the image feature and ${l}$ is the number of Swin V2 block layers.

Attention is the core of the whole block. The image features are firstly mapped into the three vectors: query (Q), key (K), and value (V). Then we use Sim function (namely Eq. (14)) to calculate the correlation weight matrix coefficients of Q and K, and normalize these weight matrix by softmax. Finally, the dot product between the weight coefficients and V is finished to form the self-attention features:

where ${Q}$, ${K}$ and ${V}$ are N${\times}$d matrices. N and d severally represent the number of patches and the dimension of single head self-attention. Instead of applying a single head self-attention, MSA of Swin V2 computes each head self-attention separately and concatenates these head self-attentions that represent different subspaces:

where ${W_{p}^{Q}\in{{}^{D\times{d_{k}}}}}$, ${W_{p}^{K}\in{{}^{D\times{d_{k}}}}}$, ${W_{p}^{V}\in{{}^{D\times{d_{v}}}}}$, and ${{W^{O}}\in{{}^{h{d_{v}}\times D}}}$ are parameter matrices. ${h}$ and ${D}$ respectively represent the number of self-attention heads and the dimension of embedding layers. We set ${{d_{k}}={d_{v}}=D/h}$.

Swin V2 computes the attention logit of a pixel pair ${m}$ and ${n}$ by a scaled cosine function:

Here, ${B_{mn}}$ is the relative position bias between pixel ${m}$ and ${n}$. ${r}$ is a learnable scalar, non-shared across heads and layers. Furthermore, Swin V2 proposes the log-spaced coordinates instead of the original linear-spaced ones to facilitate the transferring of the model between different resolution images. The log-spaced coordinates is then used as the input of ${\Phi}$ to generate the bias values:

${\Phi}$ is the 2-layer MLP with a ReLU activation. ${\Delta x}$, ${\Delta y}$ and ${\widehat{\Delta x}}$, ${\widehat{\Delta y}}$ are the linear-scaled and log-spaced coordinates, respectively.

FPT utilizes different transformers to provide the self-attention features of different layers and the interaction features between layers [56]. Self-transformer provides the self-attention features of three layers. Rendering transformer provides a down-top interaction feature between layers, and grounding transformer provides a top-down interaction feature. Inspired by FPT and the requirements of the existing task, we construct a novel multi-scale module, MFP, by improving FPT.

Since the basic blocks of current networks have evolved from convolutions to transformers, we remove self-transformer. Moreover, we delete rendering transformer due to it does not experimentally play a role on the VHRRS images change detection. Thus we only impose grounding transformer to provide the interaction information between layers. However, in addition to multi-scale features of different layers and multi-scale interaction features between layers, multi-scale information also includes multi-scale features modeling within layers. So we use SK-Conv [37] and Res2Net-Conv [36] to provide the multi-scale features inside different layers, respectively. Of course, other multi-scale convolutions, for example, inception, dilated convolution, can be used here. We only use SK-Conv and Res2Net-Conv as a example to experimentally validate our idea. MFP, FPT, SK-Conv and Res2Net-Conv are shown in Fig. 4.

Firstly, SK-Conv and Res2Net-Conv are severally used to mine the intra-layer multi-scale information of the input features ${a_{f}}$, ${b_{f}}$, and ${c_{f}}$. At the same time, grounding transformer is utilized to provide the multi-scale interaction features between different layers. These features are then rearranged according to different spatial resolutions. After residual blocks are added to the different layers, feature fusion is performed. Finally, using convolution, CBAM, and dropout in turn, we get the output features with intra-layer multi-scale information and multi-scale interaction information between layers. The roles of CBAM and dropout are to enhance the effective fusion of different features and prevent overfitting, respectively. Since the spatial and spectral resolution of the input features are consistent with the output, MFP is a plug-and-play module. MFP has also been shown to be effective for a variety of VHRRS images change detection models.

Finally, we introduce the three multi-scale feature extractors used. The core operation of grounding transformer (GT) is:

Here, ${Q_{g}}$, ${K_{g}}$ and ${V_{g}}$ are the nonlinear transformations of ${X_{g}}$. ${K_{g}^{T}}$ is the transposition matrix of ${K_{g}}$. And we define that ${X_{g}}$, ${\bar{X}_{g}}$ and ${\tilde{X}_{g}}$ are the input feature, intermediate variable and output feature of GT, respectively. Take the layer ${a_{m}}$ and ${b_{m}}$ in Fig. 4 as an example to explain the complete operation of GT:

Both SK-Conv and Res2Net-Conv are the multi-scale convolutions. SK-Conv obtains the features ${U_{s}}$ and ${V_{s}}$ by a 3${\times}$3 convolution and a 5${\times}$5 convolution, respectively. A joint SE attention is then applied for ${U_{s}}$ and ${V_{s}}$ to obtain ${\bar{U}_{s}}$ and ${\bar{V}_{s}}$. In the end, ${\bar{U}_{s}}$ plus ${\bar{V}_{s}}$ gives the SK-Conv feature ${\tilde{X}_{s}}$. The multi-scale component of Res2Net-Conv can be expressed:

We pass the input ${X_{r}}$ through 1${\times}$1 convolution, Eq. (19), and 1${\times}$1 convolution in turn, to get the Res2Net-Conv feature ${\tilde{X}_{r}}$.

To solve the problem that the VGG16 encoder [64] is untrainable, we design a new decoder and impose the SSL strategy [40] to optimize the low-level features of the VGG16 encoder. The detailed implementation is shown in Fig. 5. We compose the decoder base block with convolution, BN and ReLU. The five base blocks are arranged in order of the increasing spatial resolutions. Further more, the decoder features and the corresponding encoder features are fused using upsampling and addition. Finally, 1${\times}$1 convolution is used to output the probability map of change detection.

For SSL strategy, we firstly generate the pseudo-labels based on the probability maps through CNN branch network:

Here, ${u}$ refers to each pixel of the probability maps or pseudo-labels. ${P{M_{1,{\rm{u}}}}}$ and ${P{M_{2,{\rm{u}}}}}$ are the two probability maps that are the outputs from the branch network. Since ${P{M_{1,{\rm{u}}}}}$ and ${P{M_{2,{\rm{u}}}}}$ are normalized to [0,1] by sigmoid, we can obtain the pseudo-labels ${P{L_{1,{\rm{u}}}}}$ and ${P{L_{2,{\rm{u}}}}}$ via Eq. (20). After that, we determine the changed and unchanged regions based on the labels from the change detection datasets. In the unchanged regions, we use the pseudo-labels generated by the one branch to supervise the other branch. In the changed regions, we use the opposite results of the pseudo-labels from the one branch, to supervise the other branch. The goal is to keep the unchanged features as close as possible and the changed features as far away as possible. Therefore, the encoder features provided by CNN branch network are rich in semantic information and have more discrimination ability for change detection. The SSL-CD loss is designed as follows:

where ${{U_{\alpha}}}$ and ${{C_{\alpha}}}$ represent respectively the unchanged and changed regions. ${{L_{SSL1}}}$ and ${{L_{SSL2}}}$ are the two SSL-CD loss. ${F()}$ is a metric function, and we choose ${{L_{BCE}}}$ as it.

First, bi-temporal change detection is fundamentally a binary classification task, so binary cross entropy (BCE) loss is usually used:

In our paper, ${t}$ and ${\hat{t}}$ denote the predicted change confidence and the label in the corresponding position, respectively. For change detection tasks, however, the changed regions are far less than the unchanged regions, so there is a serious class imbalance problem in change detection. For example, the ratio of changed pixels to unchanged pixels in CDD is 0.147 [65]. To mitigate this problem, Dice loss is often used:

Here, adding ${\sigma}$ avoids the case where the denominator is zero.

The loss function used by our model is a combination of BCE and Dice loss. Moreover, to address the features shift during the training of the large network, we use the deep supervision strategy. Specifically, the deep supervision strategy uses the same labels as those used for the network output. The labels are replicated in four copies for the four deep supervision interfaces. And DConvs ${C_{0,1}}$, ${C_{0,2}}$, ${C_{0,3}}$, ${C_{0,4}}$ output the probability maps through convolution and sigmoid function. ${\sum\limits_{m=1}^{4}{L_{BCE}^{m}}+{\lambda_{1}}L_{Dice}^{m}}$, at last, is used to measure and optimize these probability maps by the deep supervision labels. The total loss function is expressed as follows:

where ${\sum\limits_{m=1}^{5}{L_{BCE}^{m}}+{\lambda_{1}}L_{Dice}^{m}}$ represents the output loss and the four deep supervision losses of main network. ${L_{SSL1}}$ and ${L_{SSL2}}$ are the self-supervision losses of CNN branch network. ${\lambda_{1}}$ and ${\lambda_{2}}$ are the weight coefficients. ${\lambda_{1}}$ is set to 0.5, and ${\lambda_{2}}$ is 0.25.

To further elucidate the practical effect of each network module, we conduct the analysis of network visualization. As depicted in Fig. 9, we broadly segment the network into three components: CNN branch network, main network, and feature fusion modules. Given that CNN branch network is a dual-branch network with shared weights, we only present the activation maps for one branch. It is distinctly seen that the encoder layers furnish attention to the low-level detail information for main network. Conversely, main network provides the structured abstract features. Feature fusion modules concentrate on detailing the differential specifics of remote sensing objects while preserving structural information. The amalgamation and retention of these two types of information are pivotal for change detection in VHRRS images. By contrasting the label and change activation map acquired through CBAM, it becomes evident that our model exhibits the robustness in the intricate scene and different lighting condition.