Semantic-aware Dense Representation Learning for Remote Sensing Image Change Detection

Training DL-based models typically rely on a massive amount of labeled data. Despite a large amount of RS data, manually collecting bitemporal images containing real changes and annotating high-quality labels could be costly. Many efforts in RS CD have been made to tackle the label insufficiency, including applying data augmentation[37, 38, 39, 40, 41, 42, 43], generating pseudo labels for unlabeled data via semi-supervised learning [36, 44, 45], using active learning to select a small number of informative samples [46, 47, 48, 49], and transfer learning from pre-trained models [8, 10, 50, 19, 16, 17].

Data augmentation is an effective solution to enhance the size of the training dataset. The most common way is to use transformation-based augmentations [37, 39, 40, 41, 42, 43], including geometric transformations (e.g., random crop, horizontal flip), color transformations, and Gaussian blur, etc. A recent advance increases the number of the positive samples (change of interest) by blending the GAN-generated instance on the appropriate spatial-temporal position of the bitemporal image [38].

Transfer learning from a pre-trained model allows leveraging the knowledge of a source dataset to improve the data efficiency of a downstream task. ImageNet pre-training is widely used in change detection [7, 8, 9, 10, 11, 12, 13] and shows superior to random initialization, especially in small data regimes. Considering the domain gap between ImageNet and RS images, a new trend is to pre-train on the remote sensing data to learn the in-domain representations [10, 14, 15, 51, 50, 19, 16, 17, 52, 53]. Supervised RS pre-training normally learns discriminative representations from the RS image-level [10, 15] or pixel-level [14] land-cover classification samples, while self-supervised pre-training could learn meaningful representations by making use of a vast amount of unlabeled RS data [19, 16, 17].

Our method falls in the supervised pre-training scope. Different from normal supervised pre-training methods that learn the mapping from image to label, we incorporate the semantic supervision in a contrastive manner in the SSL framework to enhance the feature discrimination ability. Notably, we leverage image-mask pairs freely available in RS society to guide the contrastive pre-training.

Please note that a shorter, preliminary study of our semantic-aware representation learning has been presented in IGRASS 2022 [53]. This paper is a continuation and extension of our previous work by introducing two more pre-training designs related to the use of semantic information as well as supplementing more extensive experimental results, including more comparisons, ablations, and visualizations.

Self-supervised learning has attracted many researchers by virtue of its ability to learn good feature representations from massive unlabeled data to benefit various downstream tasks [54]. Currently, contrastive methods (e.g., SimCLR [55], MoCo [20, 56], BYOL [57], SimSiam [21] and DenseCL [58]) have made significant progress in SSL. The key to contrastive approaches is to perform the instance discrimination pretext task by pulling together representations of similar images (positive pairs) and pushing apart those of dissimilar images (negative pairs). The InfoNCE loss [59] is typically applied to induce similarity between positive pairs and dissimilarity between negative pairs [55, 20, 11]. More recent attempts [57, 21, 60] apply simpler loss functions (e.g., negative cosine similarity and cross-entropy) that only rely on positive samples and does not use negative ones.

Very recently, contrastive SSL methods have been introduced in RS to obtain in-domain representations. Contrastive pre-training have been shown to benefit RS downstream tasks, including land-cover classification [61, 62, 22, 24, 25, 26, 27, 28], semantic segmentation [29, 30, 31, 32], and change detection [16, 17, 18]. Most existing methods apply InfoNCE loss [24, 22, 33, 26, 63, 30, 25, 27, 28, 18] or triple loss [61, 17] on the constructed positive and negative pairs. Positive samples can be obtained by different artificial augmentations (e.g., color and geometric transformations) of the same image [25, 28], spatial augmentations (.i.e., geospatially overlapped images) [61, 24, 27], temporal augmentations (i.e., multi-temporal co-registered images) [22, 26, 63, 16, 17], and modality augmentations (e.g., optical image, SAR, and semantic mask) [29, 64]. Negative pairs can be different samples in a mini-batch or spatially distinct images [61, 63].

Contrastive methods for change detection have seldom been explored. Some attempts apply SSL directly on a small downstream change detection dataset to extract seasonal invariant features for unsupervised change detection [65, 66, 67]. Differently, we employ an additional relatively large-scale RS dataset to learn semantic-aware representations and then transfer knowledge from the pre-trained model to the downstream supervised CD task on a variety of CD datasets. Other more related studies that follow the normal SSL pipeline mostly evaluate the pre-trained model on the medium-resolution change detection dataset [16, 17, 18]. We instead explore pre-trained models suitable for high-resolution RS image change detection. Instead of manipulating global representations, we propose dense points consistency based on spatial correspondence between the positive pairs to better sustain the sensitivity of the spatial information, therefore benefiting the downstream dense prediction task, especially on VHR RS images that contain plenty of spatial details. Instead of manipulating image-level representations, we constrain point-level consistencies according to spatial correspondences of the sampled pixels across positive views to better sustain the sensitivity of the spatial information, therefore benefiting the downstream dense prediction task, especially on VHR RS images that contain plenty of spatial details. Unlike the existing dense SSL method (DenseCL) that calculates similarities between representations of any two pixels to estimate pixel correspondence across views, our design utilizes the exact pixel correspondence calculated according to known geometric relations between views for learning view-invariant representations. Moreover, we leverage the semantic information in a contrastive manner to further improve the model discrimination ability to distinguish land covers of interest and other unconcerned backgrounds. Most existing methods [65, 67, 66, 16, 17] treat co-registered multi-temporal images as positive samples to learn the seasonal invariant features. However, the intrinsic change of interest within the bitemporal image may induce the pre-trained model invariant to real changes, which may be unfavorable to the downstream CD task. Different from existing methods using natural bitemporal images, ours learns consistent foreground representations between one image and its synthetic version whose background is replaced by that from another image. In this way, we leverage synthetic bitemporal images to enable the model to focus more on foreground objects regardless of changes in unconcerned land covers.

The recent few years witness the great success of deep learning techniques, e.g., convolutional neural networks (CNNs), graph convolutional networks (GCNs) [68], and transformers [69], applied in various RS tasks [70], including land cover classification/semantic segmentation [68, 69], multimodal image classification [71, 72], object detection [73], and change detection [7, 37, 38, 2, 39, 40, 41, 42, 43, 74, 75, 76, 77]. CNNs have been demonstrated to be effective to extract meaningful and transferable image representations by multiple levels of abstraction, thus attracting much attention in remote sensing society. Most of the existing deep learning-based CD models [37, 38, 2, 39, 40, 41, 42, 43] are based on CNNs. GCNs originally designed for graph data and transformers for sequential data have also been employed in RS image understanding tasks [68, 69], including change detection [74, 75, 76, 77], for improving model discriminative ability by leveraging their capacity of modeling long-range relations among image pixels. In this paper, instead of exploring sophisticated model structures for the CD task, we focus on the representation learning method to improve the downstream conventional CNN-based CD model.

To alleviate the insufficiency of labeled CD data, we explore a supervised representation learning method by leveraging massively available data in the RS society. We collect large-scale image-mask pairs (see Sec. IV-A) in a painless manner for contrastive pre-training. Specially, we focus on the building category as the objects of interest to perform change detection.

We propose Semantic-aware Dense representation Learning (SaDL) for remote sensing image change detection. The goal of SaDL is to learn a feature extractor that could 1) generate consistent representations invariant to illumination changes as well as changes of unconcerned land covers (i.e., background), 2) distinguish objects of interest from the background, 3) sustain spatially sensitive features to reduce the gap with the downstream dense prediction CD task.

To achieve this, we introduce the semantic mask into a self-supervised learning framework. Fig. 1 illustrates the proposed SaDL. Following SimSiam [21], we use simple Siamese networks as the feature extractor to learn meaningful image representations without using negative samples. Our SaDL is composed of three main steps:

1) Views Generation. Apart from two views of the same image via normal augmentations (i.e., color and geometry transformations), we generate a third view by swapping its background with that from another image using the semantic mask. These views can be used to learn consistent representations invariant to illumination and background changes. More generation details are given in Sec. III-B.

2) Masked Sampling. Instead of obtaining image-level representations, our Siamese encoder converts each image view into dense per-pixel representations. We sample the same number of feature points from each semantic region in the overlapped area of the generated views. For more details see Sec. III-C.

3) Representation Learning. Given the sampled point representations, we employ MLP-based projection/prediction networks to perform feature transformations. We constrain point-level cross-view similarity and semantic dissimilarity to learn representations that are invariant to irrelevant changes and discriminative to foreground/background. For more details see Sec. III-D.

Given each input training sample (including an image $\mathbf{I}$ and a mask $\mathbf{M}$) in a mini-batch, we produce three views of the same image, including two basic transformed versions, and one background swap version. Let $\mathcal{T}=\mathcal{T}_{g}\circ\mathcal{T}_{c}$ denotes a set of artificial augmentations, including color augmentations $\mathcal{T}_{c}$ (e.g., color jetting and Gaussian blur) and geometric augmentations $\mathcal{T}_{g}$ (e.g., random cropping and flipping). Details of our views generations are described in Algorithm 1.

We first generate two views $\mathbf{I}^{i}$, $i\in\{1,2\}$ of $\mathbf{I}$ via different artificial augmentations:

where $\mathcal{T},\mathcal{T^{\prime}}$ denote two random sets of artificial augmentations.

The corresponding semantic masks $\mathbf{M}^{i}$ for each view $i\in\{1,2\}$ are generated via applying the same geometric augmentations as that in the image:

View generation via background swap. The third view of $\mathbf{I}$ is obtained by swapping the background of the first view $\mathbf{I}^{1}$ with that of another image $\mathbf{I}^{{}^{\prime}}$. For simplicity, $\mathbf{I^{\prime}}$ has the inverted index as $\mathbf{I}$ from the same mini-batch. In other words, the summation of the index of $\mathbf{I}$ and that of $\mathbf{I}^{{}^{\prime}}$ equals the mini-batch size. Note that we swap the common background area between $\mathbf{I}^{1^{\prime}}$ and $\mathbf{I}^{1}$ to simulate the changes of unconcerned land covers. We obtain the common background mask $\mathbf{M}_{bg}$ by calculating the intersection of the background regions in $\mathbf{M}^{1}$ and $\mathbf{M}^{1^{\prime}}$.

However, directly pasting the background of $\mathbf{I}^{1^{\prime}}$ on $\mathbf{I}^{1}$ using $\mathbf{M}_{bg}$ may result in undesired synthetic artifacts (see Fig. 2 (d)) that could be detrimental to representation learning. Towards a realistic and effective image composite, we present a simple view synthesis method, including two main steps:

1) Color transfer. To reduce the domain gap between $\mathbf{I}^{1^{\prime}}$ (source image) and $\mathbf{I}^{1}$ (target image), we employ a simple color transfer method [78] to change the color characteristic of one image to accord with another in the image domain. The key idea is to match the mean and variance of each channel of the transformed image to that of the target image. To be formal, the transformed image is given by

where $\mu(\cdot),\sigma(\cdot)$ denote the means and variances computed across spatial dimensions independently for each channel of the image.

2) Context-aware blending. The mere presence of the context surrounding the foreground (e.g., building) is a critical cue for object recognition. For a realistic composite, we preserve pixels near the foreground in image $\mathbf{I}^{1}$ by performing morphological erosion on the background mask. We further blur the eroded background mask using a Gaussian filter towards natural and seamless composition. The composite image $\mathbf{I}^{3}$ (view 3) is calculated by alpha blending using the eroded mask of the blurred version:

Fig. 2 illustrates an example of the third view synthesis. We can observe that using color transfer and context-aware blending could give a more realistic image composite.

Obtaining dense representations. The generated three views $\mathbf{I}^{1},\mathbf{I}^{2},\mathbf{I}^{3}$ are then encoded by a Siamese neural network $f$ into dense representations $\mathbf{X}^{i}=f(\mathbf{I}^{i})\in\mathbb{R}^{H\times W\times C},i\in\{1,2,3\}$. The resulting feature map has a size of $H\times W$ and a channel dimension of $C$. Our encoder has a downsampling factor of 4. For more details see Sec. III-E.

Please note that we do not apply global average pooling to per-pixel image embeddings. Instead of manipulating image-level representations, we perform dense point-level supervision via sampling multiple points in the overlapped area across views. Our intuition is that the learned spatially-sensitive features may better reduce the gap between the pre-training and downstream dense prediction CD task that requires spatially-variant representations. Specifically, we introduce the semantic mask to achieve class-balanced sampling, i.e., an equal number of points is randomly sampled for each semantic region in the overlapped area. The details of our dense representation learning based on masked sampling on the generated views for each sample are described in Algorithm 2.

Overlapped region between views. Different views (view 1 and view 2) may correspond to different spatial areas in the original image due to the difference in applied geometry augmentations. In order to perform cross-view point consistency, we first need to obtain the overlapped region between $\mathbf{I}^{1}$ and $\mathbf{I}^{2}$. Given the known geometry augmentations ($\mathcal{T}_{g}$ and $\mathcal{T}_{g}^{{}^{\prime}}$), we could calculate the corresponding bounding box region ($BB^{i}=(u^{i},v^{i},w^{i},h^{i})$) of view $i,i\in\{1,2\}$ in the original image $\mathbf{I}$ by utilizing the parameters of each geometry augmentation (e.g., random cropping and random flipping) in the inverse order. For example, random cropping gives the upper-left coordinate $(u,v)$, the width $w$, and the height $h$ of the cropped area in the input image. The overlapped region $BB$ is the intersection of the two bounding box regions $BB^{1}$ and $BB^{2}$. The region within $BB$ defines the valid sampling region and other regions outside $BB$ are invalid for the latter point sampling process.

Please note that we do not consider view 3 for calculating the overlapping region because we only take care of the foreground of view 3, which is shared with view 1.

Fig. 3 (a) gives an example of visualization of the view correspondence. View 1 ($\mathbf{I}^{1}$) and view 2 ($\mathbf{I}^{2}$) respectively correspond to the region within the green/yellow bounding box in the original image $\mathbf{I}$. The overlapped area between views in form of bounding boxes (red) is shown in Fig. 3 (b).

Point sampling in the overlapped region. We sample $N$ points $P_{kn}=(u_{kn},v_{kn})$ with replacement for each semantic category $k$ in the overlapping region, where $(u_{kn},v_{kn})$ denotes the upper-left coordinate of point $P_{kn}$ in the original image/mask ($\mathbf{I}/\mathbf{M}$). $N$ is the number of sampled points for each class. Our default $N$ is set to 16 (See Sec. III-E). $k$=1 denotes the background and $k=2$ denotes the foreground class. Let $P=\{P_{kn}=(u_{kn},v_{kn})|n\in\{1,2,...,N\},k\in\{1,2\}\}$ denotes the sampled point set. The corresponding points in view 1 and view 2 can be obtained by coordinate transformation under the given geometry augmentations:

where $P^{i}=\{P^{i}_{kn}=(u^{i}_{kn},v^{i}_{kn})|n\in\{1,2,...,N\},k\in\{1,2\}\}$ denotes the set of coordinates of the sampled point in view $i,i\in\{1,2\}$. Note that we divide $P^{i}$ by ds (i.e., the downsampling factor of the encoder, defaults to 4) to match the coordinates of the per-pixel representations $X^{i}$. The sampled points set for view 3 is given by $P^{3}=\{P^{3}_{kn}=P^{1}_{kn})|n\in\{1,2,...,N\},k=2\}$.

Then for each view $i,i\in\{1,2\}$, we gather $N$ representations $\mathbf{x}^{i}_{kn}$ in the embedding space $\mathcal{V}\in\mathbb{R}^{C}$ for each semantic category $k$ by indexing points on the feature map $\mathbf{x}^{i}$ using coordinates $P^{i}$:

where $\mathbf{x}^{i}=\{{x}^{i}_{kn}|n\in\{1,2,...,N\},k\in\{1,2\}\}$ represents the set of representations of sampled points for view $i$.

Similarly, we gather $N$ representations $\mathbf{x}^{3}_{kn}$ for view 3 using sampled points $P^{3}$ that belong to the region of the foreground class (k=2). Formally, $\mathbf{x}^{3}=\{{x}^{3}_{kn}|n\in\{1,2,...,N\},k=2\}$.

Semantic discrimination. We exploit the semantic supervision in a contrastive manner via pushing apart representations of points that belong to different semantic regions in the same image. We constrain point-level pair-wise semantic dissimilarity. Our motivation is that the learned discriminative dense features could distinguish the objects of interest and other unconcerned backgrounds, which may facilitate the downstream change recognition of the land covers of interest.

To achieve this, we minimize the cosine similarity between the background and foreground vectors (i.e., point representations from the foreground and the background, respectively) in the embedding space $\mathcal{V}$. For simplicity, we construct $N$ pairs of points by one-to-one correspondence between $N$ foreground points $\mathbf{x}_{2n}^{i}$ and $N$ background points $\mathbf{x}_{1n}^{i},n\in\{1,...,N\}$ for each view $i\in\{1,2\}$. Note that we supplement 1 to cosine similarity to ensure a non-negative loss value. Our semantic dissimilar loss $\mathcal{L}_{sd,n}^{i}$ on one constructed point pair indexed $n$ for view $i$ is defined as follows:

where $\mathcal{D}(\cdot,\cdot)$ denotes the cosine similarity, given by:

where $\lVert\cdot\rVert_{2}$ denotes $\ell_{2}$-norm, $a,b$ are two vectors of the same dimension.

The semantic dissimilar loss for one sample is the average of $\mathcal{L}_{sd,n}^{i}$ on all the constructed point pairs in each view:

Cross-view similarity. Given the sampled points from different views, we constrain the pair-wise similarity between point representations by utilizing the spatial correspondences of cross-view points. We aim to learn two kinds of point-level consistencies: 1) consistent representations invariant to illumination changes, and 2) consistent foreground representations regardless of background changes. We employ view 1/2 to achieve the first kind consistency and view 1/3 for the second kind.

MLP-based projection/prediction networks are applied to implement the cross-view consistency. For each view $i\in\{1,2,3\}$, the sampled point representations $\mathbf{x}_{kn}^{i}$ from the encoder are projected into a subspace $\mathcal{Z}\in\mathbb{R}^{C^{\prime}}$ by a projection MLP head (projector) $g:\mathcal{V}\mapsto\mathcal{Z}$ to obtain projection vectors $\mathbf{z}_{kn}^{i}=g(\mathbf{x}_{kn}^{i})$. Then, a prediction MLP head (predictor) $h:\mathcal{Z}\mapsto\mathcal{Z}$ transforms the output of one view (i.e., $\mathbf{z}_{kn}^{i}$) to obtain a prediction vector (i.e., $\mathbf{p}_{kn}^{i}=h(\mathbf{z}_{kn}^{i})\in\mathbb{R}^{C^{\prime}}$) in the same space and matches it to the other view (i.e., $\mathbf{z}_{kn}^{i^{\prime}}$). We minimize the negative cosine similarity of representations of corresponding points between the two views ($i/i^{\prime}$). We define a symmetrized similarity loss between view $i$ and view $i^{\prime}$ for $n$-$th$ point belonging to semantic category $k$ as follows:

where stopgrad$(\cdot)$ denotes the stop-gradient operation. The encoder and projector can not receive gradient from projection vectors. Previous work [21] has empirically demonstrated the vital importance of such an operation to prevent model collapse.

The first consistency loss is the average similarity loss between view 1 and view 2 on all the sampled points $n\in\{1,...,N\}$ for each semantic category $k\in\{1,2\}$:

Similarly, the second consistency loss is the average similarity loss between view 1 and view 3 on all the sampled points $n\in\{1,...,N\}$ that belong to the foreground ($k$=2):

Overall loss function. We calculate the summation of the semantic dissimilar loss and the two kinds of similarity losses to obtain our overall loss for each sample:

Note that the final loss is averaged over all samples in a mini-batch.

Encoder. We use the ResNet-18 [79] backbone (without fully connected layers and global pooling) equipped with Feature Pyramid Networks (FPN) [80] as our default encoder $f$ to generate high-resolution and high-level per-pixel features for each image view. The ResNet-18 has 5 stages, each with downsampling by 2. Through the feedforward computation of the backbone, we obtain a feature hierarchy consisting of feature maps at different scales from the activation outputs of the last four stages. The FPN takes as input the feature hierarchy and progressively upsamples the coarse-resolution feature maps while merging them with the corresponding intermediate feature maps by element-wise addition. Each intermediate feature map undergoes a $1\times 1$ convolutional layer to transform to the same channel dimension $C$ before addition. The resulting feature map has a 1/4 spatial size of the input image. The channel dimension $C$ of output features is set to 256.

Projector. The projector $g$ is a 2-layer MLP with a hidden dimension of 2048 and an output dimension of 1024. BN and ReLU are added between fully connected (fc) layers.

Predictor. The predictor $h$ is a 2-layer MLP with a hidden dimension of 256 and an output dimension of 1024. BN and ReLU are also added between the two fc layers.

Data augmentation. We set probability of color jittering to 0.8, with {brightness, contrast, saturation, hue} strength of {0.4, 0.4, 0.4, 0.1}, and the probability of Gaussian blurring to 0.5 with a kernel standard deviation in [0.1, 2.0]. Geometric augmentations include random vertical flip, random horizontal flip, and random resized crop with a scale in [0.8, 1.0]. Note that we do not follow the common scale [21] for the random crop (i,e., [0.2, 1.0]) because the two cropped regions with a small scale may not contain an overlapped area, which may be detrimental to the effective sampling of spatially-aligned points across views.

Optimizer. We use Stochastic Gradient Descent (SGD) with a momentum hyperparameter of 0.9 and a weight decay of 0.0005. The initial learning rate is 0.01 and decays to zero at the end of training. We follow the ”poly” learning rate schedule [81] with a decay coefficient of 0.9. The default number of pre-training epochs $N_{e}$ is 200 (see Sec. IV-E). Due to capacity limitations of memory, the default batch size is set to 64. Our models are implemented on PyTorch and trained using a single NVIDIA RTX 3090 GPU.

Pre-training Dataset. We leverage image-mask pairs from the existing Inria building segmentation dataset [34], which contains 180 labeled RS images, each size of $5000\times 5000$ and spatial resolution of 0.3 m. It provides pixel-level annotations, including the building and non-building categories. We cut the original samples into small patches of size $256\times 256$ with no overlap and remove those without building regions. The resulting dataset contains more than 45k patch samples. We randomly split it into training (80%) and validation (20%) sets. We additionally obtain the co-registered image patch of the corresponding geospatial region via Google Earth as the temporal augmentation ($t_{2}$) for each image ($t_{1}$) in the dataset. For a fair comparison, the natural augmentations (bitemporal images) are used to train some other related pre-training methods that learn seasonal invariant features. Please note that the semantic mask may not perfectly match the image of $t_{2}$ due to land-cover changes (e.g., building construction and demolition) over time, we do not include the augmented temporal image in our pre-training framework.

To evaluate the effectiveness of the proposed pre-training method, we make a comparison with the random initialization, ImageNet pre-training, in-domain supervised pre-training, and several self-supervised pre-training methods:

• •

  Random initialization. All the parameters of the CD networks are initialized by the zero-mean normal distribution with a deviation of 0.02.

• •

  ImageNet pre-training. The backbone (ResNet-18) of the CD model is initialized with the ImageNet pre-training.

• •

  In-domain supervised pre-training (In-domain Sup.): We employ an FCN-based semantic segmentation network with a ResNet-18 backbone and an FPN head [80], supervised by the image-mask pairs of our pre-training dataset. We follow the same optimization configuration as that for CD networks. We perform the model evaluation in the validation set after each training epoch and save the best model (backbone) for the downstream task.

• •

  SimSiam [21]: A popular SSL method without using negative samples. SimSiam employs the stop-gradient and predictor to prevent model collapse.

• •

  MoCo-v2 [56]: A competitive SSL model that formulates contrastive learning as a dictionary look-up problem where a query should be similar to its matching key and dissimilar to others in a dynamic queue covering a large set of negative samples.

• •

  DenseCL [58]: A dense contrastive learning method that constrains pixel-level cross-view consistencies where the pixel correspondence is computed via Ssing pair-wise similarities between any two pixels from different views.

• •

  CMC [29]: A recent SSL method in RS that takes the semantic mask as an additional view along with the optical/SAR image to learn consistent cross-modal representations under the framework of contrastive multiview coding [82]. Note that we implement multi-modal representation learning with image-mask pairs of our pre-training dataset.

• •

  SeCo [16]: A very recent SSL method that is based on MoCo-v2 and further exploits the use of multi-temporal RS images as natural augmentations to learn seasonal invariant/variant representations. We implement it using the bitemporal images in our pre-training dataset.

For a fair comparison, we use our pre-training dataset to train these supervised/self-supervised methods, i.e., in-domain supervision, SimSiam, MoCo-v2, DenseCL, CMC, and SeCo. The SSL methods are implemented using their public codes with default hyperparameters. Note that all the compared models are initialized with ImageNet pre-trained weights as our SaDL does.

Each pre-trained model is evaluated by the performance of the downstream CD task. Tab. I reports the overall comparisons of our proposed method and other pre-trained models on the LEVIR-CD, WHU-CD, and Guangzhou-CD datasets, respectively. Precision, recall, F1-score, and IoU of the CD networks on the three CD test sets are given. To evaluate the data-efficient performance, we set several data regimes: 1%, 5%, 20%, and 100%, where each number represents the proportion of available labeled data in the original training set.

From Tab. I, we can observe that the proposed method consistently outperforms the compared pre-trained models in terms of the F1-score across different data regimes on the three datasets. Specifically, our SaDL pre-training is significantly superior to random initiation. It is notable that we can achieve comparable or even better results with only 20% training data than the random initiation baseline using 100% training data. It indicates that our pre-training can effectively alleviate the labeled data insufficiency problem.

ImageNet pre-training serves as a strong baseline due to its powerful general representations learned from the large-scale labeled ImageNet dataset. The empirical results demonstrate that our method overachieves ImageNet pre-training across different data conditions on the downstream CD task. For instance, Tab. I (a) shows that compared to ImageNet pre-training, we achieve considerable improvements of more than 20% of the F1-score in the LEVIR-CD test set under small data regimes (1% and 5%).

Comparison to in-domain supervision. Typically, supervised RS pre-training obtains in-domain discriminative representations by learning the mapping from RS images to labels. Tab. I shows the in-domain supervision is consistently inferior to our SaDL under each data configuration of the three datasets, even not comparable to ImageNet pre-training in some data regimes. Note that we use the same amount of pre-training data for SaDL and in-domain supervision. The quantitative results indicate that our SaDL can learn more effective transferable features than the traditional approach. We can observe that in-domain supervision shows relatively poorer results on the WHU-CD/Guangzhou-CD datasets than on the LEVIR-CD dataset. It may be due to the subdomain diversity, i.e., different RS datasets contain land-covers of diverse appearances and different spatial resolutions. The traditional supervised pre-training may overfit patterns of a certain subdomain and lack transferability to another.

Comparison to other SSL pre-training. Tab. I also shows that our method consistently outperforms several SSL methods, including two popular image-level representation learning methods, one dense contrastive learning method, and two recent SSL methods in the RS domain. We can observe that the normal dense contrastive learning (DenseCL) can even be inferior to the image-level counterpart (MoCo) in many data conditions. It may due to the discrepancy between the commonly used ImageNet image and the uncurated RS image. Unlike the ImageNet data that mainly contains one single object in the center of the image, multiple similar foreground objects may exist at different spatial locations in an RS image. The intrinsic characteristic of RS data brings difficulty in computing pixel correspondence across views in DenseCL. The resulting incorrect correspondence may hinder learning meaningful representations. On the contrary, we utilize the exact pixel correspondence for representation learning by leveraging spatial correspondences of the pixels in the overlapped area between views.

The comparison with CMC indicates that we exploit a better approach to using the semantic mask as additional information in a self-supervised learning framework to improve feature representations. We argue that directly using the mask as an augmented view along with the image views may not be an optimal solution. In [29], the mask is encoded into one vector that losses spatial details. Semantic spatial relations have not been fully utilized in such a scenario. Instead, we leverage semantic masks to push away representations of pixels from different semantic regions to learn semantic-aware features that can distinguish the foreground or background pixels, thus benefiting the recognition of objects of interest in the downstream CD task.

Visualization comparisons of the CD predictions on the three datasets for different pre-trained methods are also given in Fig. 4. We use different colors for a better view, where white for true positives, black for true negatives, red for false positives, and green for false negatives. We could observe that the CD model using our pre-trained model achieves more accurate predictions than those of its counterparts.

Better convergence. We further give validation curves on the downstream CD task to show the transferability of each pre-trained model. Fig. 5 depicts the curve of validation accuracy (mean F1-score) during the training phase using 5% LEVIR-CD/WHU-CD/Guangzhou-CD training data. Different colors denote different pre-training methods for a better view. It can be observed that our SaDL outperforms its counterparts in terms of stability and effectiveness. The comparison results indicate that our method provides better model initialization, which accelerates convergence and improves fine-tuning performance on the downstream CD task.

Computational complexity. We also report the computational complexity of our method and other compared SSL methods. For a fair comparison, we test all the methods on a computing server equipped with an NVIDIA RTX 3090 GPU. Table II lists the number of model parameters (Params.), floating-point operations per second (FLOPs), and GPU inference time of each method. We set the size of the image input to the model to $256\times 256\times 3$. The reported time is the average time consumption of each image over $100$ forward iterations given a random input sample of a batch size of $64$. The results show that our method requires relatively higher FLOPs (about $1.5$ times) than counterparts that have two views. It is because our SaDL needs to calculate three views each iteration. We can also observe that our method has fewer FLOPs than SeCo, but takes slightly more time. It may be due to the additional time consumption of our online view synthesis and masked sampling process. Nevertheless, our computational overhead is reasonable and acceptable.

We conduct ablation studies on the three key components of our proposed SaDL: 1) dense representation learning via masked-based point sampling (MS), 2) learning discriminative features via semantic discrimination (SD), and 3) learning consistent foreground features via background swap (BS). We start from the SimSiam [21] with an FPN-based encoder, which learns consistent features between two artificially augmented views of the same image, as our baseline. We incrementally add the three key components to the baseline to evaluate their respective gains to the performance of the model. All the above models are trained on our pre-training dataset for 200 epochs and then transferred to the downstream CD task. Downstream experiments are performed under the 5% data regime of the LEVIR-CD/WHU-CD/Guangzhou-CD datasets.

Ablation on masked sampling (MS). Tab. III shows that our dense representation learning via sampling multiple points for each semantic category brings consistent and significant performance improvements across the three datasets, compared to the baseline. It indicates that our MS introduces much more valuable information to representation learning than the traditional global average pooling-based counterpart that loses spatial details. Moreover, we make a comparison to the masked pooling-based approach [53], which pools features belonging to each semantic region into one semantic vector by leveraging the semantic mask. Simply constraining each semantic vector invariant across views brings small improvement, compared to the global pooling approach (baseline). We can observe that our sampling-based method is far superior to the masked pool one. The multiple pixel correspondences across views through masked sampling provide more diverse supervision of spatial accordance, thus benefiting representation learning.

Ablation on semantic discrimination (SD). To make full use of the per-pixel semantic information, we employ an additional loss term, namely semantic dissimilarity loss, to learn discriminative features that can distinguish pixels from different semantic regions. Tab. III shows that our SD induces considerable improvements in the F1-score of the CD model across the three datasets. The empirical results indicate that our semantic discrimination can significantly improve the model transferability. It may attribute to the role of the semantic mask in our SSL framework. We include the per-pixel semantic supervision into our pre-training in a contrastive manner. Different from directly mapping the image to the label, we learn an embedding space where pixels from the foreground (i.e., buildings) and the background are pushed apart. The resulting model obtains discriminative features that could distinguish land covers of interest from backgrounds, thus benefiting the recognition of change of interest in the downstream CD task.

Ablation on background swap (BS). Apart from two views by artificial augmentations, we propose a third view via swapping the background of the current image from that of another one. This incurs an additional cross-view similarity loss, i.e., learning invariant foreground features between view 1 and view 3. Quantitative results in Tab. III demonstrate that our BS further brings consistent improvements in the performance of the pre-trained CD model on the three CD datasets. It indicates that the learned foreground representations invariant to background changes benefit the downstream change detection for the land covers of interest. Our background swap can be viewed as an approach to synthesizing unconcerned changes in a scene. Such design may enforce the model to focus more on changes of interest related to the foreground, but not irrelevant changes within the background. To further verify the effectiveness of the introduced third view, we also compare the natural augmentation, i.e., augmenting the current view with a registered image from another temporal. For a fair comparison, we apply the same loss for constraining consistent foreground representations between view 1 and view 3. Tab. IV demonstrates our synthetic view consistently outperforms the natural counterpart. While natural augmentation provides rich illumination changes, undesired real foreground changes could also occur, which may hinder representation learning.

Sampled point number. The number of sampled points $N$ for each semantic category is an important hyperparameter. We test different $N\in\{1,4,16,64\}$ to analyze its effect on the performance of the downstream fine-tuning performance. Tab. V reports the F1-score/IoU of the pre-trained CD model trained on a 5% data regime of the LEVIR-CD/WHU-CD/Guangzhou-CD datasets. We can observe an overall improvement in the F1 score of the CD model on the three datasets when increasing the sample number from 1 to 16. It is because that multiple sampled points from different spatial locations introduce more spatial-diverse information for both learning cross-view invariant and semantically discriminative features, thus facilitating the dense change recognition in the downstream CD task. We can also observe a drop in F1-score when further increasing $N$ from 16 to 64. It may be because the over-sampled points nearby the edge of the semantic regions could bring difficulty in training semantically discriminative features. Therefore, we set $N$ to 16.

Pre-training epochs number. The epochs number $N_{e}$ of pre-training affects its fine-tuning performance on the downstream task. We train our SaDL using varying epochs numbers $N_{e}\in\{5,20,100,200,400\}$. Tab. VI reports the F1-score/IoU of the pre-trained CD model on the LEVIR-CD/WHU-CD/Guangzhou-CD datasets under a 5% data regime. Note that $N_{e}=0$ denotes the ImageNet pre-training. We can observe a significant improvement in the F1-score even with a very small number of pre-training epochs (e.g., 5), compared to the baseline ($N_{e}=0$). The performance of the pre-trained CD model further improves with longer training (e.g., from 5 to 200 epochs). We can also obverse a negligible improvement or even some drops in the F1-score of the CD model on the downstream tasks when further increasing the $N_{e}$ from 200 to 400. A pre-training with too many training epochs may overfit the pre-training data and incur poorer model transferability. Therefore, we set $N_{e}$ to 200.