FCCDN: Feature Constraint Network for VHR Image Change Detection

FCN is an end-to-end architecture and can make full use of semantic information to obtain pixelwise results. It is naturally suitable for change detection on high-resolution optical images. FCN was first used in street view change detection tasks [24]. Rodrigo et al. [14] were some of the first researchers to fit this structure into remote sensing image change detection. They presented three FCNs for change detection tasks and achieved state-of-the-art results on several datasets.

In recent years, many change detection networks based on FCN have been proposed. FCN-based change detection architectures can be roughly divided into single-stream networks [15, 24, 30, 18, 31] and double-stream networks [16, 27, 26, 17, 25, 20].

Single-stream networks are usually semantic segmentation networks that take concatenated or differential images of two bitemporal images as input. Peng et al. [15] use a single-path network to obtain change mask. They first concatenated the coregistered image pairs at the channel dimension. Then, the new multispectral data were fed into a modified UNet++ [32]. Liu et al. [31] inputted concatenated images to a modified UNet [33], which was built with depthwise separable convolution. Zheng et al. [18] proposed a cross-layer convolutional neural network for change detection tasks under the structure of UNet. They designed a cross-layer block to aggregate the multiscale features and multilevel context information.

Double-stream networks are commonly made of two weight-sharing feature extraction streams that directly take bitemporal images as the input. Compared with single-stream backbones, most recent works prefer a double-stream network, owing to the Siamese structure and weight-sharing strategy. Zhang et al. [16] proposed a fully convolutional two-stream network in which bitemporal features are extracted in a Siamese manner. Then, the paired features are sent to a feature fusion subnetwork to reconstruct the change map. Zhang et al. [20] proposed a Siamese network for change detection tasks with a hierarchical fusion strategy. Bitemporal features are hierarchically fused with concatenating options. Fang et al. [25] extracted bitemporal features with a dual encoder. The bitemporal features are fed into UNet++ to generate change masks. Although bitemporal features are generated by a weight-sharing encoder, features of unchanged objects may vary greatly in bitemporal feature maps. Therefore, how to fuse these features effectively remains a problem. Some researchers fused bitemporal features by concatenation or difference [14, 27, 20, 25], while others sought better methods. Chen et al. [34] proposed a dual attention mechanism that can capture long-range dependencies in feature pairs. The dual attention mechanism is applied before feature fusion to obtain more discriminant feature representations. However, the attention module is only applied to the last stage features of the weight-shared backbone. Since the features are downsampled eight times at the last stage, their method may lose much spatial information. Zhang et al. [16] introduced an attention module to effectively fuse features in different domains. The attention module they built is a combination of a channel attention module and a spatial attention module. Diakogiannis et al. [26] designed a new attention module to calculate change features. The module uses the fractal Tanimoto similarity to compare queries with keys inside the attention module. Zhang et al. [17] proposed an object-level change detection network with a dual correlation attention-guided detector. They built a correlation attention-guided feature fusion neck that uses a position-correlated attention module and channel-correlated attention module to guide the generation of change features. These methods use a series of attention modules that fuse bitemporal features hierarchically. Although they can remarkably boost the performance of networks, most attention-based fusion methods introduce a large amount of computation and a large number of parameters. Moreover, there still exist several issues that need to be considered. On the one hand, most existing works fuse shallow features to obtain the change information, which may introduce considerable noise in change maps [17]. On the other hand, existing change detection networks mainly rely on the supervision of labels. They tend to ignore the relationship among bitemporal features. Inspired by the above issues, a feature constraint architecture for change detection tasks is proposed in this literature.

SSL is a subset unsupervised learning method that can learn general image features without using any human-annotated labels [35]. It is widely used in the computer vision field. Unlike other unsupervised learning methods, SSL trains with annotations, which are obtained by mining the internal information of images through certain pretext tasks. There have been a variety of pretext tasks for SSL, such as colorizing grayscale images [36], image inpainting [37], image rotation [38], and image jigsaw puzzles [39].

Recently, several researchers in the field of remote sensing have carried out their research with SSL. Tao et al. [40] introduced a new instance discrimination-based SSL mechanism to pretrain a feature exactor for remote sensing scene classification tasks. They also investigated the impacts of several factors in the SSL-based classification task. Vincenzi et al. [41] took advantage of SSL and trained a promising feature extractor for the downstream landcover classification task. Their pretext task was to reconstruct the visible colors with high-dimensionality spectral bands. For change detection, Dong et al. [42] an SSL-based change detection algorithm. They built a network to identify different sample patches between two temporal images, namely, temporal prediction. With this pretext task, the network can encode input images to more consistent feature representations, which can be used to generate binary change maps by clustering. Chen et al. [43] proposed an SSL-based change detection approach based on an unlabeled multiview setting. They used SSL by pretraining a Siamese network with a contrastive loss for heterogeneous images and a regression loss for homogeneous images. Although these change detection tasks benefit from the SSL-based strategy, SSL is only used as a pretraining strategy. There is still a lack of research on the introduction of an SSL strategy to change detection tasks.

The overall structure of the proposed feature constraint change detection network (FCCDN) is shown in Figure 2. FCCDN uses a DED backbone (Figure 3). At the center of DED, an NL-FPN (Figure 5) is proposed to enhance multiscale features in a nonlocal way. At the decoding stage, a series of DFMs (Figure 6) are applied to fuse bitemporal features and generate change features. Then, the change features are hierarchically fed to a change decoder to obtain the final change map. FCCDN contains three output branches: one change branch and two segmentation branches Figure 8). The change branch produces the change confidence, which is used to calculate the change loss with labels in the training process. The two segmentation branches output segmentation scores and are supervised by the SSL-based strategy.

Since bitemporal data are usually used in change detection tasks, a fully convolutional Siamese (FCS) network has been a common structure for change detection in recent years. Conventional FCS (also referred to as FCS) generally uses a dual encoder with weight sharing to extract bitemporal features. After the encoding process, a series of feature fusion blocks are applied to obtain change features based on the extracted features. At the decoding stage, a single-stream change decoder is used to combine the change features mentioned above in a similar way as UNet [33].

Figure 3 shows the overall structure of our baseline FCS. We build the dual encoder with a series of SE-ResNet modules (Figure 3) [44], which uses the squeeze and excitation (SE) block (Figure 3) in the residual branch. The SE block uses global average pooling (GAP) to aggregate the feature maps across spatial dimensions and captures channelwise feature responses. After GAP, two fully connected (FC) layers are applied to learn a weight vector for each channel. The vector is then normalized and multiplied by the original feature. The dual encoder is built with four SE-ResNet modules. The output of the four modules are features with sizes of (h/2, w/2), (h/4, w/4), (h/8, w/8), and (h/16, w/16) (h represents the height of the input image and w represents the width of the input image). The encoder features are then fed to four feature fusion modules to fuse bitemporal features and generate four change features. In the decoding process, several change decoder blocks are employed to fuse features generated by bitemporal feature fusion blocks hierarchically. Since the bitemporal feature fusion block and the change decoder block are both aimed at fusing features, we build them with the same module, as shown in Figure 3. This fusion module is a combination of an upsampling layer (if necessary), a concatenate operation, a convolution layer with kernel size = 3, a batch normalization (BN) layer [45], and a rectified linear unit (ReLU) function. The output of the last change decoder block (C-block3) is then fed into a simple segmentation head, which is meant to reduce the channels and resize the change mask to the input size.

The conventional FCS network uses low-level features in the dual encoder to calculate the change features. However, low-level features pay more attention to general information, which contains much information that is useless for change detection [17]. We validate this conjecture with the change detection task on buildings and visualize some features in Figure 4. The feature maps are visualized in the form of heatmaps with Grad-CAM [46]. In the heatmap, the warmer the color is, the more attention the network pays to that area. That is, the degree of warmth represents the contribution of each pixel. As shown in Figure 4, the low-level features ($f_{\rm 1}$ and $f_{\rm 2}$) from the dual encoder contain not only the edges of buildings but also the edges of background objects. In this case, there is much noise in the change features, which is not conducive to the change detection task.

In this paper, we propose a novel DED neural network backbone for change detection tasks (see Figure 3). This architecture has the same dual encoder and change decoder as FCS. Nevertheless, unlike FCS, DED has an extra dual decoder with weight sharing, which is used to filter out invalid information and reconstruct useful information. The dual decoder is also built with the feature fusion block shown in Figure 3. Change features are generated by fusing the corresponding features from the dual decoder. Then, these change features are fed into a change decoder in the same way as in FCS. The bitemporal features reconstructed by the dual decoder are shown in Figure 4. Compared with the bitemporal features from the dual encoder of FCS, the bitemporal features from the dual decoder can better highlight the borders of buildings and suppress background features. Therefore, the change features generated by DED are more accurate.

It is well known that ground objects vary on different scales. Therefore, it is necessary to extract and fuse features at multiple scales and with large receptive fields. Some researchers try to tackle the above issue with pyramid pooling (PSP) [47, 48]] or atrous spatial pyramid pooling (ASPP) [49]. However, these methods use large-scale pooling or convolution layers with a large dilate rate, resulting in the absence of local information [50]. In addition, there are also some solutions based on a feature pyramid network (FPN) [51, 52], which can effectively fuse multiscale features. However, conventional FPN methods cannot obtain long-range dependencies, which makes it hard to extract large object features.

Here, we propose an NL-FPN module to extract and fuse features in a nonlocal way. The module is added to the center of FCCDN (Figure 2) to augment the feature maps by considering similarities between any pairs of pixels. Because similarities between any two positions is considered, we can further strengthen the intraclass correlations and increase interclass separation. Figure 5 shows the details of our NL-FPN. Nonlocal blocks (NL-blocks) are added to the upsampling stage of FPN, which consists of six convolution layers. We build the NL-block inspired by the spatial self-attention mechanism [53, 54]. The self-attention mechanism uses the dot product between the key vector and the query vector to obtain the similarity between different pixels. Since the features are normalized to unit lengths, the distance between vectors can be computed using the dot product [55]. We can formulate this as

The NL-block we build is shown in Figure 5. The input feature map is fed into three branches: a reshape branch, a reshape and transpose branch, and a convolution branch. The features in the first two branches are fused by a dot product followed by a softmax function. Here, a similarity map is generated. The feature in the convolution branch is fed into a convolution layer. Then, the output of the convolution branch is reshaped and transposed so that we can further fuse it with the similarity map mentioned before by using a dot product. At the end of this block, the fusion result is reshaped and fed into a convolution branch to obtain the final weight map. The weight map is used to augment the features in FPN by a Hadamard product.

For change detection with Siamese networks, bitemporal feature fusion is the most critical part. It is difficult for two reasons: (1) Bitemporal images fed into Siamese networks are often offset in spatial position and color. (2) The background objects are complex and diverse. Conventional methods use direct subtraction or concatenation to fuse features [14]. Unfortunately, although Siamese networks extract features by dual blocks, there is still much misalignment among the bitemporal features. There have also been many researchers who have tried to tackle this problem with attention mechanisms [16, 34, 26, 17]. Nevertheless, most existing attention-based feature modules introduce many calculations and consume considerable memory.

Here, we propose a simple yet effective feature fusion module based on dense connections. We name this module DFM. DFM consists of two branches, the sum branch and the difference branch. The sum branch is used to enhance the edge information, and the difference branch is used to generate change regions. Each branch is built with two densely connected streams with weight sharing. Figure 6 illustrates the details of DFM. All convolution operations use 3×3 kernels. We should note that we do not use BN in each branch until the final convolution layer.

We build DFM for the purpose of ensembling multiple features in each stream and making better decisions. This structure can increase the robustness of the model and prevent pseudochanges caused by feature misalignment. Additionally, owing to the rich residual connections in dense connections, the last two features in each stream can be regarded as the residual of the previous feature, which is to some extent the correction of the previous feature and makes the new feature map more aligned. We validate this module by visualizing features in the difference branch (Figure 7). From the visualization of the feature maps, we can see that DFM indeed reduces feature misalignment and calculates more accurate change features.

Change detection is aimed at finding changes of interest in bitemporal images. Currently, most change detection tasks are inter-class change detection, in which the categories of bitemporal images should differ in changed areas, and the categories in the unchanged areas should be the same. That is, the bitemporal features in unchanged areas should be as close as possible, and the bitemporal features in changed areas should be as far away as possible. To this end, we use the novel idea of applying SSL to the task of change detection. With this SSL-based strategy, we can further constrain feature learning.

As shown in Figure 8, we add two auxiliary branches to the dual decoder of the proposed DED backbone. The two auxiliary branches are meant to obtain the semantic segmentation results of target objects. For the learning of the auxiliary branches, we use a novel SSL strategy to generate pseudolabels for each branch. It should be noted that we take the binary change detection task as an example to describe our idea. We discuss the application of the SSL-based feature constraint strategy for multiclass change detection tasks in the discussion section(Section 5). The details of the SSL strategy are as follows: (1) According to the change detection label, we split the semantic segmentation results of the auxiliary branches ($Seg_{\rm 1}$ and $Seg_{\rm 2}$ in Figure 8) into two parts: the changed area and unchanged area. (2) In the unchanged area, the semantic segmentation result of one branch is used as the label of the other branch. (3) In the changed area, the opposite semantic segmentation result of one branch is used as the label of the other branch.

The way we generate pseudolabels can be expressed by the following formulas. We assume that the output of the two auxiliary branches is $S_{\rm 1}$, $S_{\rm 2}\in R^{x\times y\times 1}$. Since $S_{\rm 1}$ and $S_{\rm 2}$ are normalized to [0, 1], we can obtain the semantic segmentation results of the target object ($P_{\rm 1}$ and $P_{\rm 2}$) with:

Both the change loss and auxiliary loss are calculated with the same loss function, which is a combination of the binary cross-entropy (BCE) loss and dice coefficient loss functions [56]. The BCE loss can be formulated as follow:

where N represents the total number of pixels in a label patch $x_{\rm n}$ and $y_{\rm n}$ denote the predicted change confidence and the label in the corresponding position, respectively. The dice coefficient loss is calculated as:

where X and Y represent the predicted change confidence and the label $\cap$ denotes the intersection of X and Y. Hence, the BCE + dice loss we use can be expressed as:

There is a simple way to validate our SSL-based feature constraint strategy. That is, we can validate the improvement by checking the inference results of the two auxiliary semantic segmentation branches. The two auxiliary branches are meant to guide DED to extract better target object features and suppress background object features. Therefore, the accuracy of the semantic segmentation results directly represents the performance of our SSL-based feature constraint. We show several results in Figure 9. From left to right: pretemporal images, posttemporal images, the pretemporal semantic segmentation results, the posttemporal semantic segmentation results, and the change detection results. Apparently, the two semantic segmentation branches achieve good performance and further improve the change task.

Note that we only test the SSL-based strategy on inter-class change detection tasks since almost all the available change detection datasets are inter-class change detection. It is hard to tell whether this strategy can boost performance on intra-class change detection tasks. So, we can only conclude that the SSL-based strategy can effectively constrain the feature learning of inter-class change detection tasks.

We describe the experimental datasets in this subsection. We offer a brief view of the LEVIR-CD dataset and WHU dataset in Table 1. More details are shown in Section 4.1.1 and Section 4.1.2.

In this paper, we use the intersection over union (IoU) and F1 score as the evaluation metrics. The two metrics are commonly used to quantify the performance of the change detection task. The values of IoU and F1 range from 0 to 1, and the higher the value is, the better the performance. The IoU and F1 score are calculated as:

In this subsection, we discuss the application of the proposed SSL-based feature constraint strategy to multiclass change detection tasks. The discussion is addresses three popular multiclass change detection datasets: 1) the season-varying dataset [63], which is used to identify multiclass changes with binary change labels (Figure 14); 2) SECOND [64], which is used to identify multiclass changes with multiclass semantic segmentation labels in the bitemporal changed area (Figure 14); 2) HRSCD [65], which is used to identify multiclass changes with multiclass semantic segmentation labels in each temporal image (Figure 14).

Since both the SSL-based strategy and contrastive loss strategy are used for constraining feature learning in an unsupervised way, we discuss the differences between these two methods in this subsection. The main difference lies in the fact that the SSL-based strategy uses a stricter constraint. For the contrastive loss strategy, the network needs to learn discriminative features. Unchanged features are pulled together, and changed features are pushed apart. However, ground objects in the bitemporal images are not distinguished. Compared with the contrastive loss strategy, the SSL-based strategy introduces an additional constraint that assigns bitemporal features to different semantics classes. With the supervision of additional semantic segmentation branches, changed features and unchanged features can be easily acquired by comparing bitemporal features.

We test the contrastive loss strategy and SSL-based strategy on the LEVIR-CD dataset and the SECON dataset. The comparison results are shown in Table 8. From the table, we can learn that both the contrastive loss strategy and SSL-based strategy can boost model performance. Compared with the contrastive loss strategy, the SSL-based strategy performs better. On the LEVIR-CD dataset, SSL improves 0.29% in terms of the IoU and 0.16% in terms of the F1 score. On the SECOND dataset, SSL improves 0.2% in terms of the mIoU and 0.24% in terms of the SeK.