Joint Spatio-Temporal Modeling for Semantic Change Detection in Remote Sensing Images

In the pre-Deep-Learning era, one possible way to classify change detection methods is to divide them into three categories based on the types of analyzed change features, including texture features, object-based features and angular features [9]. The CNNs can better capture the context in RSIs, thus they have been widely used for BCD in recent years. Early works on deep-CNN-based CD treats CD as a segmentation task, and employ UNet-like CNNs to directly segment the changes [10]. In [11] a paradigm for BCD is established, which uses two Siamese CNN encoders (i.e., weight-sharing CNNs) to extract temporal features, and transforms the features into change representations through a CNN decoder.

One of the objectives of CNN-based BCD methods is to discriminate the semantic changes from seasonal changes and spatial misalignment. To extract the change features, feature difference operations are commonly used in the CNNs methods [11, 12]. Since there is redundant spatial information in high-resolution (HR) RSIs, multi-scale features are often utilized to filter the spatial noise [13]. The attention mechanism is also effective to detect semantic changes. In [14] the channel attention is employed to refine the multi-scale features. In [15] and [16] hybrid spatial and channel attentions are used to embed discriminative change representations.

Different from BCD which only detects ’where’ the changes are, SCD provides information on ’what’ the changes are, which is more valuable in many RS applications. In [17] the post-classification comparison method is introduced, which compares the multi-temporal results to classify the LCLU transitions. However, the temporal correlations are not considered in this method. In [4, 5] the compound classification method is proposed, which applies Bayes rules to iteratively maximize the posterior joint probabilities of multi-temporal LCLU classes. In [6] the posterior joint probabilities are considered to integrate the change probabilities, which are obtained through slow feature analysis.

The method in [18] is an early attempt to apply neural networks for SCD. It is a joint CNN-RNN network where the CNN extracts semantic features, while the RNN models temporal dependencies to classify multi-class changes. In [11] CNN architectures for SCD are elaborately analyzed and a triple-branch CNN architecture is suggested. Two of the branches are deployed to exploit temporal semantics, while another branch is deployed to model the binary change information. In [7] a well-annotated benchmark dataset for SCD is released and several evaluation metrics are proposed. In [8] a SSCDl (disentangled semantic segmentation and CD, late fusion) architecture is proposed, which has been demonstrated to be more effective for the SCD.

One of the challenges in the CNN-based methods is the learning of temporal dependencies. While the multi-temporal semantic features are separately embedded, it is important to bridge the connections between different temporal branches. In [8] a temporal spatial attention design is proposed to model the bi-temporal semantic correlations. In [19] the channel attention is used to embed change information into the temporal features. In [20] a learnable symmetric transform is proposed to align the temporal features and to learn the change maps.

Although these modules to some extent improve the feature representations, the modeling of spatio-temporal dependencies is still partial, one-sided, or shallow. We argue that to learn the intrinsic mechanism in SCD, it is important to jointly model the semantic-change correlations, which is the objective of this study.

Vision transformer is a recent emerging research topic since its application in visual recognition tasks [21]. Differently from CNNs whose computations are limited to sliding kernels, vision transformers leverage self-attention to model global dependencies. The vision transformers have been first used in high-level recognition tasks such as image classification [22, 23] and object detection [24]. Then they have been applied to down-stream tasks such as semantic segmentation[25] and image synthesis [26]. Several literature works propose pure-CNN backbone networks for general vision tasks. They arrange hierarchical transformer blocks in CNN-like manner and limit the range of self-attention in local areas to reduce computation[27, 28, 29].

There are also investigations to apply transformers to better recognize the complex ground objects in RSIs. In [30] a context transformer is proposed to exploit wide-range context information through a context window. In [31] two siamese transformers are organized into a SegFormer-like [25] architecture for BCD in RSIs. Instead of using transformer as backbone networks feature extraction, the method in [32] employs transformer to model the bi-temporal context correlations. In a recent study, transformer is also used in the SCD task [33], which is also a variant of the SegFormer. However, the transformers are simply employed as feature extractors to replace the CNN counterparts. Differently, in this work, we investigate leveraging the transformers to deeply model the long-range spatio-temporal correlations that are coherent in the SCD task.

Given a pair of input images $(I_{1},I_{2})$, the task of SCD is to generate a pair of semantic change maps $(Y_{1},Y_{2})$ that present the changed areas and their semantic categories. An ideal SCD function can be formulated as:

where $p$ denotes a spatial position on ${I_{1},I_{2}}$ and $L_{1}^{p},L_{2}^{p}$ are the bi-temporal semantic classes at $p$.

Recent works pointed out that compared to directly learning the semantic changes, separate embedding of the semantic features and change features obtains better accuracy [2, 7, 8]. In our previous work[8], an SCD framework named SSCDl has been introduced for SCD. As shown in Fig.2(a), the SSCDl includes two temporal encoders to exploit the semantic information and a CNN module to detect the changes. There are also optional network modules (presented as ’heads’) to enhance the feature representations and to segment the results. One of the remaining problems is that the obtained results are coarse-grained and lack a precise representation of spatial information. Here we improve the SSCDl by adding decoder modules. The resulting TED SCD framework has three feature representation branches organized in an encoder-decoder manner. Fig.2(b) presents the TED framework, where the ’Necks’ are decoding modules that are attached with lower layers in the encoders.

Let us consider images $I_{1},I_{2}\in\mathbb{R}^{c\times W\times H}$ input into the TED. First, the encoder networks embed and reshape them into multi-scale features $[X_{1}^{u},X_{1}^{v}],[X_{2}^{u},X_{2}^{v}]$ where $X^{u}\in\mathbb{R}^{C_{u}\times W/4\times H/4},X^{v}\in\mathbb{R}^{C_{v}\times W/8\times H/8}$. Then, the necks further enlarge and concatenate $[X_{1}^{u},X_{1}^{v}],[X_{2}^{u},X_{2}^{v}]$ before forwarding them to the heads. Concurrently, there is also a change branch that detects changes by using bi-temporal features. Its inputs are semantic features $[X_{1}^{v},X_{2}^{v}]$ obtained from the highest encoding layers in the temporal branches, while its output is the change feature $X_{c}^{v}\in\mathbb{R}^{C_{v}\times W/8\times H/8}$. The neck in the change branch spatially aligns $X_{c}$ with temporal features $[X_{1}^{u},X_{2}^{u}]$. Finally, outputs of the triple embedding branches are $X_{1},X_{2},X_{c}\in\mathbb{R}^{C_{v}\times W/4\times H/4}$. Compared with the SSCDl, the TED can better retain spatial details.

In the TED framework, separate embedding of the temporal and change features disentangles the learning of semantic change into Semantic Exploitation (SE) and CD. Although this separate learning enables more dedicated exploitation of the semantic and change information, their correlation is not considered. Intuitively, the bi-temporal semantic differences/consistency helps to discriminate the changes/no-change classes, while the LCLU transition pattern allows to better recognize the semantic categories. To model this ’semantic-change’ dependency, we resort to the recently emerging transformers and introduce the SCanFormer.

As depicted in Fig.3, the SCanFormer can be plugged in the TED framework as the heads. The resulting network, i.e. the SCanNet, is a hybrid ’CNN-Transformer’ architecture. The CNN parts serve as feature extractors due to their efficiency and preservation of spatial information, which is the common practice in segmentation tasks [30]. Meanwhile, the SCanFormer is attentive to correlations in the embedded semantic space. This allows deep modeling of the ’semantic-change’ dependencies in the whole spatio-temporal domain, which differs from previous methods that only model the bi-temporal correlations [8] or one-way (change-to-temporal [19]) attention operations.

To jointly model the spatio-temporal correlations, we concatenate features $[X_{1},X_{2},X_{c}]$ and flatten them into a semantic token $\textbf{x}\in\mathbb{R}^{hw\times d}$ where $d$ is the depth of the token (equal to $3C_{v}$). The SCanFormer consists of $L$ layers of attention blocks. Each of the blocks consists of a Self-Attention (SA) unit and an MLP unit. The units are organized in a residual manner and contain a normalization layer. Mathematically, the calculations inside each attention block are:

where $\textbf{x}^{l}$ represents the output token of the $l$-th block.

The original ViT [21] that uses full SA is calculation-intensive. Its computation complexity is quadratic to the image size, which is expensive for handling HR RSIs. Considering this, we adopt the Cross-Shaped Window (CSWin) SA [29] to model long-range context more efficiently. In the CSWin-SA unit, the input features are spatially partitioned into $M$ vertical and horizontal stripes (in two separate groups), each with the width $s$. There are $2K$ heads to perform SA, $K$ of the heads for the horizontal group and another $K$ ones for the vertical group. The calculations in the horizontal group are:

where $x^{i}\in\mathbb{R}^{sw\times d}$, $A_{k}$ represents the SA operation of the $k$-th unit, $Q,K,V\in\mathbb{R}^{d\times d_{k}}$ are the projecting matrix for the query, key, value tokens, respectively. The calculations in the vertical group are similar, thus are omitted for simplicity. The CSWin-SA calculations for the $i$-th stripe are:

where $\sigma$ is the softmax operation, $\textbf{q}_{i},\textbf{k}_{i},\textbf{v}_{i}$ are linear projections of $\textbf{x}_{i}$, $B_{i}$ is a learnable parameter matrix to encode the relative positions. Finally, the outputs of the $2K$ heads are concatenated and projected:

where $W^{o}\in\mathbb{R}^{d\times C^{o}}$ is a projection matrix to adjust the token dimension.

In the common settings of the SCD task, the changed areas are provided with semantic labels, whereas the unchanged areas are annotated as no-change. In other words, the number of semantic labels is very limited. This leads to the challenge of learning semantic information with limited samples. However, considering the intrinsic mechanism in the SCD, it is possible to utilize the bi-temporal consistency as prior information to guide the exploitation of semantic information. Considering the different cases in change and no-change areas, we propose a semantic learning scheme with two task-specific learning objectives. In the following, we separately discuss the two cases.

First, let us consider the changed areas that are provided with semantic labels. The semantic loss can be calculated with the commonly used cross-entropy loss, defined as:

where $i$ is the temporal index. To disentangle SE with CD, this loss is only calculated in the changed areas (i.e. ignoring the no-change class).

Meanwhile, the unchanged areas are annotated as no-change but usually are not provided with the specific semantic classes. However, it is possible to infer the semantic labels with bi-temporal predictions. Suppose that a place $p$ is known to be unchanged, if its bi-temporal semantic probabilities ${Y_{s}^{1}}^{p}$ and ${Y_{s}^{2}}^{p}$ are similar, we can assume with high confidence that its semantic class should be ${Y_{s}^{1}}^{p}$ (or ${Y_{s}^{2}}^{p}$). Therefore, we adopt the pseudo-labeling method to generate a pseudo semantic label $\widetilde{L}_{u}$ for $no-change$ areas. Mathematically, the calculation is as follows:

where $cos()$ is a Cosine function calculated in the vector space to measure the semantic similarity, and $T$ is a threshold. We use annotation 0 to exclude the change areas from the loss calculation. A pseudo semantic objective $\mathcal{L}_{psd}$ can be calculated as:

The calculation of this objective function is illustrated in Fig.4(a).

Last but not least, according to the intrinsic logic in the SCD, there should be semantic consistency in the no-change areas on $Y_{1}$ and $Y_{2}$, whereas there is a difference in the changed areas. Using this temporal constraint as prior information, a semantic consistency learning objective $\mathcal{L}_{sc}$ can be constructed to guide the network training. $\mathcal{L}_{sc}$ is calculated with the ground truth change label $L_{c}$, which can be easily derived by binarizing $L_{1}$ or $L_{2}$ (setting the annotation of changed regions to $0$). The calculation of$\mathcal{L}_{sc}$ is as follow:

The calculation of this learning objective function is depicted in Fig.4(b). Note that this function is calculated in both no-change and change areas. It encourages the network to generate the same semantic predictions in change areas, whereas generating different predictions in no-change areas.

Incorporating these learning objective functions, the overall loss $\mathcal{L}$ is given by:

By adding $\mathcal{L}_{sc}$, the temporal semantic information contained in the two images is jointly considered, which improves the discrimination of critical areas.

In the following, we report the parameter settings and the training details in the implementation of SCanNet.

1) TED. The CNN encoders in TED are constructed following the practice in [8]. We adopt the ResNet blocks which are commonly used in segmentation tasks [34]. Since the pre and after-change input images are in the same domain, the temporal encoders are made Siamese (i.e., weight-sharing) to avoid over-fitting and to better align the extracted features. The convolutional layer to embed change features is a ResNet-like block with 6 residual layers. The necks are decoder blocks constructed as those in [35].

2) ScanFormer. The important parameters in the SCanFormer are $s$, $L$, and $K$. $s$ determines the context modeling range, which is set to 2 considering the input feature size ($H/4\times W/4$). To avoid heavy computations, $L$ (the number of layers in the SA blocks) is set to 2. $K$ is set to 2 considering the feature dimensions.

3) Training Settings. The proposed methods are implemented with the PyTorch toolbox. The training is performed for 50 epochs. The batch size and initial learning rate are 8 and 0.1, respectively. The gradient descent optimization method is Stochastic Gradient Descent with Nesterov momentum. The learning rate $lr$ is updated at each iteration as: $0.1*(1-iterations/total\_iterations)^{1.5}$. The augmentation operations consist of random flipping and rotation before loading the input images.

For more details, readers are encouraged to visit our codes released at: https://github.com/ggsDing/SCanNet.

We conduct experiments on two openly-accessible benchmark datasets for the SCD: the SECOND and the Landsat-SCD datasets. The former is an HR dataset collected in city regions, whereas the latter is a mid-resolution dataset at the margin of a desert area. These differences in the spatial resolution and the observed scenes make it possible to assess the effectiveness of the tested methods in different scenarios.

1) SECOND[7]. The SEmantic Change detectiON Dataset (SECOND) is a large-scale and well-annotated benchmark dataset for the SCD in HR RSIs. This dataset includes 4662 pairs of RSIs acquired by different sensors and platforms. They are obtained in various cities including Hangzhou, Chengdu, and Shanghai in China. Each image has $512\times 512$ pixels. The ground sampling distance (GSD) in this dataset is in the range between 0.5 and 3m. Each pair of RSIs are spatially matched, recording the changes in the considered region. The manual annotations are the semantic change maps associated with each image. In the annotated labels, there are 1 no-change class and 5 LC classes, including non-vegetated ground surface, tree, low vegetation, water, buildings and playgrounds (only the changed areas are annotated). Comparing the pre-change and after-change classes, a total of 30 types of semantic changes can be derived. The change pixels account for 19.87% of the total image pixels. We further split 1/5 of the data (593 pairs of RSIs) as the test set and use the remaining ones for training (2375 pairs) following the practice in [8].

2) Landsat-SCD Dataset[33]. This dataset is constructed with Landsat images collected between 1990 and 2020. The observed region is in Tumushuke, Xinjiang, China, which is at the margin of the Taklimakan Desert. The ground sampling distance is 30m. Changed pixels account for 18.89% of the total pixels. The dataset is annotated into the no-change class and 4 LC classes, including farmland, desert, building and water (only the changed areas are annotated). These LC classes are associated with 10 types of semantic changes.

The Landsat-SCD dataset consists of 8468 pairs of images, each having the spatial resolution of $416\times 416$. Excluding those samples obtained by spatial augmentation operations (including flipping, masking and resizing), there are 2425 pairs of original data. We further split them into training, validation and test sets with the numeric ratio of 3:1:1 (i.e., 1455, 485 and 485, respectively).

To evaluate the segmentation accuracy, we adopt 4 metrics commonly used in the SCD task, including overall accuracy (OA), mean Intersection over Union (mIoU), Separated Kappa (SeK) coefficient and SCD-targeted F1 Score ($F_{scd}$). OA is a common metric in segmentation tasks[36], which represents the numeric ratio between the correctly classified pixels and the total image pixels. Let $Q=\{q_{i,j}\}$ be the confusion matrix, where $q_{i,j}$ represents the number of pixels that are classified into class $i$ while their GT index is $j$ ($i,j\in\{0,1,...,N\}$ ($0$ represents no-change). OA is calculated as:

In the SCD task, since no-change class is the majority, OA cannot well-describe the discrimination of semantic classes. In [7] mIoU and SeK are introduced to evaluate the performance CD and SE, respectively. The mIoU in SCD is calculated based on the discrimination of change/no-change classes:

Meanwhile, SeK evaluates the segmentation of semantic classes especially in the changed areas. It is calculated based on the confusion matrix $\hat{Q}=\{\hat{q}_{ij}\}$, where $\hat{q}_{ij}=q_{ij}$ except that $\hat{q}_{00}=0$. This excludes the true positive non-changed pixels whose number is dominant. It is calculated as:

In [8] the $F_{scd}$ is introduced to evaluate the segmentation accuracy in change areas. It is calculated deriving the $Precision$ ($P_{scd}$) and $Recall$ ($R_{scd}$) in the areas annotated as change:

1) Quantitative Results. The techniques proposed in this work include i) the TED framework; ii) the semantic learning scheme and iii) the SCanNet (incorporating SCanFormer into the TED). Table.I reports the quantitative results after using the proposed techniques. The TED framework outperforms the SSCDl framework by around 0.3% in mIoU and $F_{scd}$, which demonstrates its advantages in both SE and CD. After applying the semantic learning scheme to the TED, there are significant improvements in all the metrics (around 0.7% in mIoU, SeK and $F_{scd}$). Compared to the plain TED, the SCanNet (adding the SCanFormer) has significant improvements in SE (demonstrated by improvements of around 0.5% in Sek and 0.9% in $F_{scd}$).

2) Effects of the Semantic Learning Scheme. The $\mathcal{L}_{psd}$ in the proposed semantic learning scheme is calculated with pseudo labels generated by bi-temporal semantic predictions. To visually assess the quality of generated pseudo labels, we present some examples in Fig.5. The pseudo labels cover the pixels with no semantic labels, thus their correctness can only be visually assessed together with the input images. One can observe that the generated labels are generally correct and multiple semantic categories are included. This supervision function improves the learning of semantic information in unchanged regions where the bi-temporal prediction confidence is high.

Fig.6 presents qualitative comparisons of the results obtained w. and w./o using the semantic learning scheme. The comparisons are made on the plain TED framework where there is no other explicit spatio-temporal modeling design, so that all the differences can be attributed to the proposed learning scheme. After using the semantic learning scheme, some non-salient changes are captured (e.g., the emergence of a playground is detected in Fig.6(b)), while the discrimination of semantic classes is also improved (e.g., the error of segmenting ground as low vegetation in Fig.6(a)). These results indicate that the proposed learning scheme brings overall improvements to the SCD task.

3) Effects of the SCanFormer. Fig.7 compares the results obtained w. and w/o. using the proposed SCanFormer. While the detected changes are mostly the same, the SCanNet, leveraging the joint spatio-temporal information, can better discriminate the semantic categories. Specifically, in the results of the SCanNet there are less errors in the discrimination of ground and building (see Fig.7(a)), as well as in the discrimination of low vegetation and ground (see Fig.7(b)).

To conclude, this ablation study demonstrates that: i) the proposed TED framework and the learning scheme bring improvements to the detection of changes and the extraction of their semantic information; ii) the SCanFormer significantly improves the learning of temporal semantic information.

To comprehensively evaluate the performance of the proposed method, we compare it with literature methods that are provided with accessible codes. The compared methods include i) the CD and SCD architectures proposed in [11, 2], ii) the hybrid CNN-RNN methods for SCD [18] and iii) other recent literature methods [19, 8].

1) Quantitative Results. We report the quantitative results obtained on the two benchmark datasets in Table.II. Among the compared methods, those SCD-targeted methods obtain higher accuracy metrics. Specifically, the ResNet-LSTM[18], which contains temporal-modeling recurrent modules, has a relatively high $F_{scd}$. The HRSCD-str.4, which integrates CD and LCLU mapping, obtains higher mIoU and $F_{scd}$ on the SECOND. Among the literature methods, the SCDNet[19] that uses change-to-temporal attention operations obtains significantly higher accuracy. The Bi-SRNet[8] that enables interactions between the temporal semantic branches obtains the second-best and third-best accuracy on the SECOND and the Landsat-SCD dataset, respectively.

Without bells and whistles, plain TED outperforms most of the literature methods. Its advantages are more obvious in the Landsat-SCD dataset, which can be attributed to its spatial-preserving designs. The major difference between the TED and the HRSCD-str.4 is that the former reuses the large amount of semantic information presented in features from the temporal branches, therefore is more sensitive to the semantic changes. The proposed SCanNet enables deep and intrinsic modeling of the spatio-temporal dependencies in the SCD task. Thus it has significant advantages over the compared methods in all the metrics. Notably, its advantages over the second-best results in SeK are around 0.7% in $SeK$ and around 1% in $F_{s}cd$. This demonstrates its superior capabilities in discriminating the semantic classes with a limited number of change samples.

2) Qualitative Results. To visually compare the results, we present the segmentation maps obtained by different methods in Fig.8. The first 4 rows (Fig.8(a1)-(b2)) are results obtained on the SECOND. One can observe that the literature methods have difficulty in detecting the non-salient changes, e.g., the emergence of a playground in Fig.8(a) and the removal of small buildings in Fig.8(b). There are also discrepancies in the results. For example, in the SCD results of the HRSCD-str.4 and the SCDNet, there are areas that are segmented as building on both of the bi-temporal segmentation maps, which is contradictory with respect to the represented change information. These issues are mostly addressed in the results of the proposed methods. With the guidance of semantic learning objectives, the detection of non-salient changes is improved and there are much fewer discrepancies in the bi-temporal results. The SCanNet further outperforms the plain TED in recognizing the critical areas, e.g., discrimination between low vegetation and water in Fig.8(a).

Fig.8(c1)-(d2) present SCD results obtained on the Landsat-SCD dataset. Since the GSD of this dataset is relatively low, it requires the SCD model to better preserve the spatial details. One can observe that the proposed methods based on the TED architecture can precisely capture fine-grained changes in LU types, such as the drying of a river in Fig.8(c), and the emergence of small farmlands in Fig.8(d). The SCanNet shows advantages in discriminating the semantic categories of the small objects.

3) Change Analysis. In Fig.9 we present the confusion matrices obtained on the bi-temporal SCD results, which intuitively present the ’from-to’ change types in the observed regions. The GT labels in the SECOND (see Fig.9(a)) indicate that the major semantic changes include ground$\rightarrow{}$building, low vegetation$\rightarrow{}$ground, ground$\rightarrow{}$low vegetation and low vegetation$\rightarrow{}$building, which account for 25.86%, 15.58%, 12.09% and 11.26% of the change proportions, respectively. This change pattern is generally consistent with the results of the SCanNet. To compare the semantic changes detected by the different methods, Table.III sorts the top-6 major changes derived from the change confusion matrices. One can observe that in the results of the SCanNet, there are fewer errors in terms of the sequence of the major semantic changes. Meanwhile, in the Landsat-SCD dataset, the major changes include desert$\rightarrow{}$farmland, desert$\rightarrow{}$water, farmland$\rightarrow{}$desert and water$\rightarrow{}$desert, accounting for 61.74%, 10.97%, 10.45% and 5.05% of the total changes, respectively. This dataset is less challenging, thus most of the methods successfully detected these changes.

It is worth noting that the results of the HRSCD.str4 (in the SECOND and in the Landsat) and the SCDNet (in the SECOND) show some discrepancies (i.e., false changes such as ground$\rightarrow{}$ground and farmland$\rightarrow{}$farmland). Meanwhile, in the results of the SCDNet, the false changes are limited to only a small proportion. There are a total of 0.2% false changes in the SECOND (excluding the building$\rightarrow{}$building changes which are affected by wrong annotations), and 0.26% in the Landsat-SCD dataset. In another word, most of the discrepancies in the results are eliminated.