Semantic decoupled representation learning for remote sensing image change detection

Fig. 1 illustrates the proposed Semantic Decoupled Representation Learning. Our architecture consists of a Siamese encoder/projector and a predictor. The Siamese encoder converts the input image views to semantic disentangled representations by leveraging semantic masks. Such intermediate representations are then projected into an embedding space by the projector. The predictor is employed to predict the representation of a view from that of another view.

Views generation. Given an input sample consisting of a bitemporal image $\mathbf{I}^{t_{i}},i\in\{1,2\}$ and a semantic mask $\mathbf{M}$, we randomly generate two views $\mathbf{I}_{j}^{t_{i}},j\in\{1,2\}$ from each temporal image via synthetic augmentations. The semantic mask $\mathbf{M}_{j}^{t_{i}}$ is also generated by applying the same geometric augmentation as that in $\mathbf{I}_{j}^{t_{i}}$.

Disentangling semantic representations. Two views $\mathbf{I}_{1}^{t_{i}},\mathbf{I}_{2}^{t_{i}}$ are then encoded by a Siamese encoder $f$ into dense representations $\mathbf{X}_{1}^{t_{i}},\mathbf{X}_{2}^{t_{i}}\in\mathbb{R}^{H\times W\times C}$. Instead of pooling the per-pixel features into one vector, we decouple the global representation into several representation vectors by separately pooling features in each semantic region. Formally, the disentangled representations $\mathbf{x}_{jk}^{t_{i}}\in\mathbb{R}^{C},k\in\{1,2,...,\text{K}\}$ are calculated by

$$\mathbf{x}_{j1}^{t_{i}},...,\mathbf{x}_{jK}^{t_{i}}=\text{MaskedPool}(\mathbf{X}_{j}^{t_{i}},\mathbf{M}_{j}^{t_{i}}),$$

(1)

where K denotes the number of semantic categories. Specially, we use the binary mask to disentangle representations of foreground objects (i.e., buildings) and background. Therefore, we set $\text{K}=2$. Note that semantic masks are resized to the same shape as the feature map before performing masked pooling.

Semantic dissimilarity. Although representations of different semantic regions are decoupled, the relationship between them is not utilized. We employ a simple cosine-similarity-based loss to push away representations of different semantic regions. Our intuition is to enlarge the angle between the two vectors (i.e., representations of the foreground and the background) in the embedding space by minimizing their cosine similarity. Given the two embeddings $\mathbf{x}_{j1}^{t_{i}},\mathbf{x}_{j2}^{t_{i}}$ of a view $j$ of a temporal image $i$, the semantic dissimilar (SE) loss $\mathcal{L}_{sd,j}^{t_{i}}$ is:

$$\mathcal{L}_{sd,j}^{t_{i}}=\mathcal{D}(\mathbf{x}_{j1}^{t_{i}},\mathbf{x}_{j2}^{t_{i}})+1,$$

(2)

where $\mathcal{D}(\cdot,\cdot)$ denotes the cosine similarity. 1 is supplemented to ensure a non-negative value.

In our data configuration, the semantic mask is paired with $\mathbf{I}^{t_{1}}$, but may not perfectly match $\mathbf{I}^{t_{2}}$ due to the inherent real changes. Therefore, we only use views from $\mathbf{I}^{t_{1}}$ to calculate the SE loss for one sample: $\mathcal{L}_{sd}=\frac{1}{2}\sum_{j=1}^{2}\mathcal{L}_{sd,j}^{t_{1}}$.

Cross-view similarity. We follow SimSiam [6] to implement the cross-view similarity. Each semantic representation $\mathbf{x}_{jk}^{t_{i}}$ is projected into a space $\mathcal{Z}\in\mathbb{R}^{C^{\prime}}$ by a MLP head (projector) $g$ to projection vectors $\mathbf{z}_{jk}^{t_{i}}$. Another MLP head (predictor) $h:\mathcal{Z}\mapsto\mathcal{Z}$ transforms one view (e.g., $\mathbf{z}_{1k}^{t_{i}}$) to a prediction vector (e.g., $\mathbf{p}_{1k}^{t_{i}}\in\mathbb{R}^{C^{\prime}}$) and matches it to the other view (e.g., $\mathbf{z}_{2k}^{t_{i}}$). We minimize the negative cosine similarity of cross-view representations. A symmetric similarity loss is defined as:

$$\mathcal{L}_{s,k}^{t_{i}}=1-\frac{1}{2}(\mathcal{D}(\mathbf{p}_{1k}^{t_{i}},\text{sg}(\mathbf{z}_{2k}^{t_{i}})+\mathcal{D}(\mathbf{p}_{2k}^{t_{i}},\text{sg}(\mathbf{z}_{1k}^{t_{i}}))),$$

(3)

where sg$(\cdot)$ means a stop-gradient operation. Previous work [6] shows such operation is vital to prevent model collapse. The similarity loss for one sample is given by $\mathcal{L}_{s}=\frac{1}{4}\sum_{i=1}^{2}\sum_{k=1}^{2}\mathcal{L}_{s,k}^{t_{i}}$.

Overall loss for one sample is given by: $\mathcal{L}=\mathcal{L}_{sd}+\mathcal{L}_{s}$. The total loss is averaged over all samples in a batch.

Encoder. We use the ResNet-18 [9] without fully connected layers (fc) and global pooling as $f$. We simply add a bilinear interpolation layer with an upsampling factor of 4 behind the ResNet-18 to reduce the loss of spatial details. The dimension $C$ of output is 512.

Projector/Predictor. $g,h$ are 2-layer MLPs with an output dimension $C^{{}^{\prime}}=1024$. Their hidden dimensions are 1024 and 256. BN and ReLU are added between fc.

Data augmentation. We use similar augmentations in SimSiam, including color jittering, Gaussian blurring, random flip. Note that we do not apply the random crop.

Optimizer. We use SGD with a base learning rate of 0.01 and a poly decay schedule. We set the weight decay to 0.0005 and the SGD momentum to 0.9. The default number of epochs is 20 and the batch size is 64.

Pre-training Dataset. We leverage image-label pairs from the existing Inria building dataset [10], which provides 180 labeled RS images ($5000\times 5000$, 0.3 m/pixel). The data is labeled into the building and non-building classes. For each image ($t_{1}$) in Inria, we obtain a co-registered image as temporal augmentation ($t_{2}$) via Google Earth. In this way, we collect bitemporal images with semantic building masks. We cut images into patches of size $256\times 256$ with no overlap and remove patches not containing building regions. It results in more than 45k patch samples. We randomly split it into training (80%) and validation sets (20%). Note that due to building changes over time, the semantic mask may not perfectly match the image of $t_{2}$.

CD Datasets. We conduct experiments on LEVIR-CD [7] and WHU-CD [8] to evaluate the pre-trained model. We follow [11] to split each dataset into training, validation, and testing sets. Each image is cut into small patches of size $256\times 256$ with no overlap.

CD networks. We employ a simple yet effective change detection model [12]. Differently, we use a more light FPN-based decoder head [13] for the feature extractor. Please refer to [12] for more implementation details.

Evaluation Metrics. We use the F1-score of the change category as the evaluation indices.

Our models are implemented on PyTorch and trained using a single NVIDIA RTX 3090 GPU.

We compare with several baselines, including random initialization, ImageNet pre-training, in-domain supervised pretraining (Sup.), and two SSL methods.

In-domain Sup.: We employ an FCN-based semantic segmentation network with a ResNet-18 backbone and an FPN head [13], supervised by the image-mask pairs from our pre-training dataset. The best model on the validation set is used for the downstream task.

SimSiam [6]: A state-of-the-art contrastive method that does not require negative samples and uses stop-gradient and predictor to prevent model collapse.

SeCo [4]: A MoCo-based [14] method that uses multi-temporal RS images as natural augmentations to learn seasonal invariant/variant representations.

For a fair comparison, we implement these SSL methods using their public codes with default hyperparameters on our pre-training dataset.

We set a variety of data conditions: 1%, 5%, 20%, and 100%, each of which represents a proportion of available labeled training data. From Tab. 1, we can observe that the proposed method consistently outperforms other pre-trained models on the two CD datasets, especially in the small data regimes. Interestingly, we can achieve comparable results with only 20% training data than the baseline (random) using 100% data. It indicates our SDRL can effectively alleviate data insufficiency. Our model also outperforms the strong ImageNet pre-training the recent SeCo. We can observe in-domain sup. shows relatively poorer results on the WHU-CD dataset than on the LEVIR-CD dataset. It is because subdomain diversity may be present in different RS datasets. The traditional supervised pre-training may overfit patterns in a certain domain and lack transferability to another. Our empirical results indicate that our SDRL can learn more transferable features than supervised pre-training.

Fast convergence. Fig. 2 illustrates the validation accuracy (mean F1) for each epoch using 5% LEVIR-CD/WHU-CD training data. Our SDRL outperforms others in terms of accuracy and stability. It indicates that SDRL accelerates fine-tuning convergence and incurs better performance on the downstream CD task.