A Survey of Sample-Efficient Deep Learning for Change Detection in Remote Sensing: Tasks, Strategies, and Challenges

Background: BCD has been the most extensively studied CD task in the past few decades. Unless otherwise specified, BCD is also commonly abbreviated as CD in literature. As BCD has been comprehensively reviewed in existing literature, here we only provide a brief summary of the typical paradigms and representative work.

In the initial stages, DL-based BCD was seen as a segmentation task, where UNet-like Convolutional Neural Networks (CNNs) are employed to directly segment changes [8]. Let $I_{1}$ and $I_{2}$ denote a pair of RSIs obtained on the dates $t_{1}$ and $t_{2}$, respectively. The general function of CD can be represented as:

$$\mathcal{F}_{\theta}(I_{1},I_{2})=Y_{c},$$

(1)

where $Y_{c}$ is the predicted change map, $\mathcal{F}$ is the mapping function of a DNN with the trained parameters $\theta$. Differently, Daudt et al. [9] proposed to first extract the temporal features, then model the change representations:

$$\nu[\mu_{1}(I_{1}),\mu_{2}(I_{2})]=Y_{c},$$

(2)

where $\mu_{1}$ and $\mu_{2}$ are two DNN encoders, $\nu$ is a DNN decoder. Under the circumstance that $I_{1}$ and $I_{2}$ exhibit homogeneity (e.g., collected by the same sensor or have similar spatial and spectral characteristics), $\mu_{1}$ and $\mu_{2}$ can be configured as siamese networks [9], i.e., share the same weight. This approach has been widely accepted as a paradigm for DL-based CD, as it allows effective exploitation of the temporal features.

Techniques: The major challenges in BCD are distinguishing semantic changes between seasonal changes and mitigating spatial misalignment as well as illumination differences. In CNN-based methods, channel-wise feature difference operations are commonly used to extract change features [9, 10]. Another common strategy is to leverage multiscale features to reduce the impact of redundant spatial details [11]. Multiscale binary supervisions are also introduced in [8] to align the embedding of change features. As an effective technique to aggregate global context, the attention mechanism is also widely used in CD of RSIs. Channel-wise attention is often used to improve the change representations [12, 13], while spatial attention is often used to exploit the long-range context dependencies [14, 15].

Another research focus in BCD is to model the temporal dependencies in pairs of RSIs. In [16] a multilayer RNN module is adopted to learn change probabilities. Graph convolutional networks are also an efficient technique to propagate Land Cover Land Use (LCLU) information to identify changes[17]. Recently, Vision Transformers (ViTs) [18, 19] have emerged and gained great research interests in the RS field [20, 21]. There are two strategies to utilize ViTs for CD in RSIs. The first is to replace CNN backbones with ViTs to extract temporal features, such as ChangeFormer[22] and ScratchFormer[23]. Meanwhile, ViTs can also be used to model the temporal dependencies. In BiT[2], a transformer encoder is employed to extract changes of interest, while two siamese transformer decoders are placed to refine the change maps. In CTD-Former[24], a cross-temporal transformer is proposed to interact between the different temporal branches.

Background: In BCD, the results only indicate location of the change, leaving out the detailed change type. This is often not informative enough to support RS applications. In contrast, multi-class CD (MCD) refers to the task of classifying changes into multiple predefined classes or categories [25]. On the other hand, semantic change detection (SCD) is introduced in recent DL-based CD literature to classify and represent the pre-event and after-event change classes [26, 27]. Although there are slight differences in the representation of results, both MCD and SCD enable a detailed analysis of the changed regions, e.g., identifying the major changes and calculating the proportion of each type of change. The results can further be represented in an occurrence matrix indicating pre-event and after-event LCLU classes, such as presented in Fig.2(b).

Architectures: MCD/SCD, with its provision of more detailed information, is indeed a more challenging task compared to BCD due to the need for modeling semantic information in particularly changed areas. According to the order of semantic modeling and CD, conventional methods for MCD can be roughly divided into two types, i.e. the post-classification comparison [28] and compound classification [29, 30]. In DL, it is feasible to perform multi-task learning by jointly using different training objectives. There are two types of deep architectures for MCD/SCD in RSIs. The first architecture applies the common CD architecture in Equation. (2), and fuses bi-temporal information to classify multiple change types [31, 32]. The second approach employs a joint learning paradigm to learn semantic features and change representations simultaneously through different network branches [26, 27]. This can be formulated as follows:

		$\displaystyle\phi_{1}[\mu_{1}(I_{1})]=Y_{1},\phi_{2}[\mu_{1}(I_{1})]=Y_{2},$		(3)
		$\displaystyle\nu[\mu_{1}(I_{1}),\mu_{2}(I_{2})]=Y_{c},$

where $\phi_{1}$, $\phi_{2}$, and $\nu$ are three DNN modules that project the temporal features into semantic maps $Y_{1},Y_{2}$ and change map $Y_{c}$, respectively.

Techniques: The techniques used in MCD/SCD can be categorized into two types: i) spatio-temporal fusion [32, 33] and ii) semantic dependency modeling [34]. In [31], Mou et al. made an early attempt to employ DNNs for MCD. It is a joint CNN-RNN network where the CNN extracts semantic features, while the RNN models temporal dependencies to classify multi-class changes.

Background: Differently from common CD studies that analyze bi-temporal RSIs, Time-Series CD (TSCD) aims to capture changes that have occurred over multiple periods or across a series of temporal images. This can better characterize the dynamics of change [35] and discriminate between transient fluctuations and persistent changes, leading to more reliable and informative CD results. Conventional algorithms analyze the temporal curves to model the change patterns. Among these algorithms, trajectory classification models the trajectory in change regions, statistical boundary detects departure from common variations to detect changes, and regression models the long-term momentum in the observed regions[36]. Since these methods commonly do not consider spatial contexts, they are sensitive to noise and seasonal variations. Moreover, they have difficulty modeling complex or long-term change patterns.

Architectures: Due to the scarcity of training data, DL-based TSCD did not emerge until very recent years. An intuitive approach is to employ RNNs to model temporal variations in time-series observations, as RNNs were originally designed for sequence processing. In [37] Long-Short-Term Memory (LSTM) network, a more delicate type of RNN is first introduced to detect and predict the burned areas in forests. Experimental results reveal that LSTM can better model the nonlinear characteristics in temporal data. In [38] a temporal semantic segmentation method for time-series images is proposed. LSTM is employed to classify the spectral vectors into different LCLU types at different timestamps.

In these LSTM-based methods, the analysis is limited to the temporal dimension. Although the method in [38] involves analysis of the spatial consistency, this is conducted as post-processing to reduce noise and is not end-to-end trainable. To overcome this limitation and to consider the spatial context in time-series RSIs, in [39] LSTM is combined with a CNN for joint spatiotemporal analysis. A CNN is employed to project time-series RSIs into spatial features, followed by an LSTM to model the temporal dependencies. This can be formulated as:

		$\displaystyle x_{i}=\psi(I_{i}),$		(4)
		$\displaystyle\omega[x_{1},x_{2},...,x_{t}]=[h_{1},h_{2},...,h_{t}],$
		$\displaystyle\nu[h_{1},h_{2},...,h_{t}]=Y_{c},$

where $i=1,2,...t$ is the time step, $x_{i}$ and $h_{i}$ are the extracted spatial and temporal features, $\psi$ and $\omega$ are the CNN and RNN units, respectively. $\nu$ can be a $softmax$ operation in multi-date LCLU CD applications[39], or an anomaly detection function in disaster monitoring applications [40].

Semi-supervised learning presupposes the availability of only a limited volume of labeled data for training. In scenarios where labeled samples are extremely scarce, this paradigm transitions into the domain of few-shot change detection. This necessitates intrinsic learning of the change patterns that can be generalized across diverse instances of change.

Pseudo Labeling: Pseudo labeling allows a DNN to generate pseudo labels for unlabeled data based on its predictions, thus effectively augmenting the training dataset. In segmentation-related tasks, pseudo labels can be obtained by thresholding the predictions of DNNs.

Since single DNN predictions may contain many errors, various methods combine multiple predictions to enhance the robustness of pseudo-labeling. In [49] pseudo labels are obtained by composing and voting multi-scale predictions. In [50], historical models are used during training to produce ensemble predictions. By calculating the mean Intersection over Union (IoU) in historical predictions, the reliable results are selected as pseudo labels to train the unlabeled data. The method in [51] utilizes multiple DNNs to produce multiple predictions and also performs IoU calculations to generate reliable labels. In [34] a cross-temporal pseudo-labeling technique is introduced. The semantic similarity between multitemporal predictions is calculated to select the high-confident pixels. In [53], a sophisticated cross-pseudo supervision method is proposed within the Teacher-Student (TS) learning paradigm. The knowledge learned in a teacher model is distilled to supervise the student models, and the predictions of multiple student models are composed to generate reliable pseudo labels. Kondmann et al. [54] employ an unsupervised method as the teacher model, subsequently train and fine-tune different CD models with pseudo labels from the teacher model. In [118], the method employs superpixel segmentation to create objects and enable self-supervised learning through object overlaps in bitemporal images. It produces and integrates multiscale object-level and pixel-level difference images and utilizes temporal prediction for SSCD.

The essence of pseudo-labeling is minimizing the errors and uncertainty in generated labels while enhancing guidance for critical cases. Therefore, it is important to measure the certainty of DNN predictions. If the pseudo labels are generated by multiple methods, the number of votes can be deemed the confidence score [54]. In [51] and [50] the certainty is measured through IoU in multiple predictions. For a single DNN prediction on unlabeled data, low entropy indicates high confidence, and entropy-based objectives are commonly used to filter uncertain predictions [55] [56]. In [57] similarity measures and uncertainty calculations are combined to map the pseudo CD labels. To improve the guidance for minor classes (i.e., changes), Hou et al. [59] cluster the extracted deep features to generate pseudo labels and rebalance the change/non-change instances in pseudo labels to strengthen the learning of minority class (i.e., changes). Furthermore, uncertain predictions also contain potential knowledge. In a contrastive learning paradigm, reliable and unreliable pixels can be sampled as positive and negative samples, thus improving the representation of temporal semantic features [55, 60].

Auxiliary regularization: To facilitate training on unlabeled data, a common strategy is to introduce auxiliary training objectives or regularization. This can constrict the optimization landscape and regularize DNNs to learn noise-resistant change representations. In [43] Ding et al. propose a temporal similarity regularization to optimize learning of temporal semantics in SCD. This objective drives DNNs to embed similar features in unchanged areas and different semantics in changed areas. In [4] it is extended with temperature regularization to model the implicit semantic latent in the BCD. In [52] temporal regularization is implemented in the form of mutual supervision with pseudo labels. In [44] a focal cosine loss is designed to align feature representations in unchanged areas for SSCD of hyperspectral images. It assigns greater weights to hard positive samples to emphasize the learning of critical samples.

In [41] adversarial learning is introduced to align the feature distributions of unlabeled data with the labeled data, thus promoting GT-like results. In [119], adversarial learning is introduced to learn consistent feature representations in bitemporal images. The CD results are then derived by clustering the different features.

Among auxiliary regularization-based approaches, Consistency Regularization (CR) is an effective strategy to enhance the model generalization. CR applies spatial or spectral perturbations to unlabeled data, training the model to reduce discrepancies between varying perturbations of the same image [120].

Bandara et al. [45] first introduce CR to WSCD, and extend perturbations from images to feature differences. A paradigm for CR is proposed in the context of WSCD, which involves different types of perturbations, such as random feature noising, random feature drop, feature cutout, and instance masking. Similarly, Yang et al. [46] extend the CR paradigm with dual stream feature-level perturbations, which greatly improves the generalization even with a very small proportion of training samples. A simplified paradigm of this CR learning under a teacher-student knowledge distillation framework is illustrated in Fig.4.

Building on top of the CR paradigm, many literature methods investigate to improve WSCD through advanced DNN designs and sample selection mechanisms. In [121], rotation augmentation is introduced in CR-based WSCD, and class-wise uncertainties are calculated to alleviate the class imbalance issue. Wang et al. [47] introduce a reliable sample selection mechanism that selects samples with stable historical predictions during training. In [122], a coarse-to-fine CD network with multiscale attention designs is designed as the backbone for CR-based WSCD. In [48] selection, trimming and merging of reliable instances is performed to enhance the robustness of extracted change instances. Hafner et al. consider multi-modal data as different views of the same regions and employ CR across different modalities to learn robust built-up changes [123].

Graph Representation: Graph neural networks (GNNs) are a family of DNNs that are adept at modeling relationships. Since GNNs can be trained with partial labels, they are well suited to semi-supervised learning settings [78, 74]. A crucial step in graph learning is graph construction. The literature methods can be categorized into superpixel-based [74, 75, 76], feature-based [77], and difference-based [78, 58] graph construction.

Liu et al. [78] first introduced graph learning in the context of SMCD. The differences between temporal features are calculated to construct change graphs, while adversarial learning is also introduced to train the graphs constructed with unlabeled data. Saha et al. construct change graphs with multi-temporal parcels, and propagate change information from labeled parcels to unlabeled ones through training iterations [74]. Tang et al. [58] employ a multi-scale Graph Convolutional Network (GCN) to capture long-range change context and generate pseudo labels with similarity metrics. In [75] a method for dynamic graph construction in SAR image CD is presented. It constructs graphs from three-channel pixel blocks and dynamically updates graph edges based on trained features. The method in [76] combines superpixel graph modeling and pixel-level CNN embedding for SMCD in hyperspectral images. It introduces a graph attention network (GAT) to capture temporal-spatial correlations via an affinity matrix and uses CNN layers to merge features to map changes. In [77], GAT is incorporated into a CR learning framework to learn robust multi-temporal graph representations. In [124] graph is employed to represent and cluster the change evolutions for unsupervised TSCD.

While CD is a fine-grained segmentation task that requires pixel-level annotations, in the weakly supervised learning setting, only coarse-grained labels such as points, surrounding boxes, scribbles, and image categories are available. WSCD enables easy construction of a CD training set, as it does not require intensive human annotation. However, it does not mitigate the scarcity of change samples.

Most of the WSCD methods utilize image-level labels. The labels indicate either the image categories [66] or the image pair (change/nonchange [69]). Meanwhile, various types of coarse CD labels are also utilized in literature studies, including point labels [125], low-resolution labels [126], patch-wise labels [73] and box labels [127]. The differences in these supervisions derive different methodologies of utilizing and recovering spatial information. Two major categories of WSCD methodologies that correspond to image-level supervision and coarse CD supervision are change activation mapping and difference refinement, respectively.

Change activation Mapping: This strategy is frequently employed in WSCD to parse image-level label into spatial change representations. First, an image encoder is trained with image-level information, then the feature responses in the late layers, i.e., class activation maps (CAMs), are utilized to generate coarse pseudo labels. However, CAMs contain only coarse feature responses and do not indicate fine-grained change details. To improve the accuracy and robustness of CAMs, Cao et al. [66] ensemble multi-scale CAMs and propose a noise correction strategy to generate reliable pseudo labels. The method in [67] also adopts a multi-scale approach. It extracts more robust and accurate change probability maps through knowledge distillation and multi-scale sigmoid inference, as illustrated in Fig.5. The method in [68] introduces mutual learning between different time phases. It utilizes CAMs derived from the original image and the affine transformed image to improve the certainty of change mapping and incorporates contrastive learning to enlarge the distance between changed representations and unchanged representations. In [69] GradCAM++ is introduced to weight the multi-scale CAMs. It also leverages multi-scale and transformation consistency regularization to improve the quality of CAMs.

Difference Refinement: In comparison to image-level labels, coarse CD labels contain a certain degree of spatial information and thus can be utilized to train a coarse CD model. After mapping the differences, various kinds of techniques are developed to refine and highlight the salient change regions.

Several methods employ conventional machine learning methods to perform the refinement. In [126], the refinement is achieved through bitemporal comparison and morphological filtering operations. In [127], a candidate suppression algorithm is designed to reduce the overlapping box candidates and select the most confident candidate regions that indicate changes. In [70] temporal features are extracted by contrastive learning, and the mapping from difference image to CD result is achieved through PCA and K-Means algorithms.

In contrast to refinement on the CD results, several methods refine the labels to perform fine-grained supervision. The method in [128] first calculates a difference map through edge mapping and superpixel segmentation algorithms, then trains a denoising autoencoder to refine the pre-classification results. Fang et al. apply region growth on point labels and DNN predictions to expand the annotations and propose a consistency alignment objective to align the coarse and fine predictions [125]. In [71], the training of a CD CNN and the refinement of the results are carried out iteratively to reduce the errors in the noisy crowd-sourced labels. A guided anisotropic diffusion algorithm is introduced to filter the wrong predictions while preserving the edges.

Differently from these approaches, the method in [72] utilizes object-level class labels to perform WSCD. It first compares image pairs with a Siamese Unet and then masks the changed object to classify its category. To enable accurate masking of the changed object, a CRF-RNN (Conditional Random Fields as Recurrent Neural Network) layer is employed to integrate spatial details from the original image. Similarly to this object-masking approach, the method in [73] masks and re-segments superpixels as interesting instances (buildings), and utilizes a voting mechanism to classify the changed instances (damaged buildings).

Self-supervised learning exploits the inherent consistency within data to learn sensor-invariant and noise-resilient semantic representations. Leveraging the capability of self-supervised learning, SSCD learns to discriminate temporal variations in unlabeled RSIs. It is worth noting that SSCD can be regarded as a distinct subclass within the broader category of UCD, but they typically require extensive pre-training in the target domain. Additionally, many approaches employ SSCD for pretraining and still require fine-tuning on target datasets.

Contrastive Learning: This strategy constructs and compares positive and negative pairs to exploit the structure and relationships within unlabeled data. In CD, bi-temporal images are often utilized to construct the contrastive pairs. By maximizing the consistency among positive pairs and the difference among negative pairs through contrastive losses, DNNs are trained to exploit feature embeddings that can capture temporal similarities and discrepancies. Fig.6 illustrates a simplified paradigm of contrastive learning, where change pairs are constructed with cropped RSIs at the same and different locations. The mapping of pre-trained representations into CD results further categorizes two major types of methods: fine-tuning-based and thresholding-based.

The fine-tuning-based methods use CD labels to retrain based on the pre-trained model obtained from self-supervised methods. Common methods utilize data augmentation methods for comparative learning. The results of data augmentation based on the same sample are regarded as positive samples. Feng et al. [79] obtain a pre-trained model based on SimSiam and unlabeled samples. Multiple data augmentation methods are often combined to generate positive samples, and then the pre-trained model is directly fine-tuned [80] [81]. In addition to data augmentation, some studies construct contrastive learning by mining multiple clues, such as multi-level contrast and multi-feature contrast. Jiang et al. [82] design global-local contrastive learning, where global and local contrastive learning respectively implement instance-level and pixel-level discrimination tasks. Huang et al. [83] propose a soft contrastive loss function to improve the inadequate feature tolerance. In the downstream CD fine-tuning task, the features of different receptive fields are captured by a multiscale feature fusion module and combined with a two-domain residual attention block to obtain long-range dependencies on spectral and spatial dimensions. The method in [84] proposes a multilevel and multi-granularity feature extraction method and applies contrastive learning to obtain the pretrained model. A multilevel CD is performed by fine-tuning the network with limited samples.

The thresholding-based methods derive the CD map from dual feature maps using thresholding, thus no labeled samples are used for fine-tuning. Contrastive learning based on multiple clues has also been used in these methods. The method in [85] pretrains the model using a pseudo-siamese network and multiview images and then generates binary CD maps through feature distance measurement and thresholding. In [86], shifted RSI pairs are leveraged to train pseudo-siamese networks, performing pixel-level contrastive learning. Kuzu et al. [87] employ instance-level (BYOL, SimSiam) and pixel-level (PixPro, PixContrast) methods to derive pre-trained models and directly produce CD maps using DCVA. In [88], a multicue contrastive self-supervised learning framework is designed. Beyond mere data augmentation, this approach also constructs positive sample pairs from semantically similar local patches and temporally aligned patches. The preliminary change embeddings are then obtained from the affinity matrix. The method in [89] first performs contrastive learning on bitemporal RSIs, and then performs contrastive learning on early fusion and late fusion features. Meanwhile, pseudo label contrast has also been widely explored, which regards samples with the same class as positive pairs and samples with different classes as negative pairs. Saha et al. [90] employ deep clustering and contrastive learning for self-supervised pre-training. Adebayo [91] trains a classifier using land cover labels of available years to identify unchanged regions through post-classification comparisons. The pre-trained model is obtained through the BYOL method based on trusted unchanged regions. He et al. [92] employ clustering to obtain pseudo labels (non-changed, changed, and uncertain). Furthermore, this framework introduces a self-supervised triple loss, including changed and non-changed losses based on contrastive learning and an uncertain loss based on image reconstruction.

Masked Image Modeling: Masked Image Modeling (MIM) is a self-supervised reconstructive approach aims at learning generalized representations from extensive volumes of unlabeled data. Within the MIM paradigm, DNNs are trained to reconstruct masked image pixels or patches based on available unmasked image content. However, MIM does not provide task-specific feature representations and typically requires subsequent fine-tuning.

With large-scale pretraining using MIM, Sun et al. [93] constructed a foundational model for RS scenes and proved its improvements to BCD. Cui et al. [94] pre-train a network using multi-scale MIM and fine-tune it with labeled data. The model first processes images with convolutional structures and then extracts global information using transformers. The method in [95] combines contrastive learning and MIM in a self-distillation way, allowing effective representations with global semantic separability and local spatial perceptibility. Zhang et al. [96] propose a multi-modal pretraining framework. The DNNs learn visual representations through MIM, and align them with multi-modal data through contrastive learning. A temporal fusion transformer is also proposed to transfer the pre-trained model to CD.

Augmentation: Natural changes are infrequent and registered bitemporal RSIs are difficult to collect. To overcome these limitations, in [104] a background augmentation method is introduced for image-level WSCD. It augments samples under the guidance of background-exchanged images, enabling the model to learn intricate environmental variations.

Several literature studies resort to augmenting semantic changes with single-temporal RSIs in segmentation datasets. In [105], pseudo change pairs are constructed by randomly sampling labeled RSIs and mixing their semantic labels. This pseudo supervision is proved to generalize well on CD datasets without fine-tuning. In [107], Chen et al. propose a simple image patch exchange method to generate pseudo-multi-temporal images and pseudo labels from a wide range of single-temporal HR RSIs, facilitating the training of CD DNNs in a self-supervised manner. In [106], patches from other images are cut and pasted to create a pseudo-post-change image.

There are also several literature studies aiming to generate more diverse and realistic change pairs with instance-level augmentations. They commonly utilize instance labels in segmentation datasets to perform the creation or removal of synthetic changes, as illustrated in Fig.7. For example, Seo et al. [109] implement copy-pasting or removal-inpainting operations based on the labels of ground objects. Zheng et al. [111] first synthesize changes by copying or removing objects, then simulate temporal variations using a GAN. Zhu et al. [110] generate object segments with a GAN and employ Poisson blending to fuse them into background images. The resulting approach enables a few-shot CD in forest scenes. Quan et al. [108] generate pseudo-change pairs by masking the instances in labeled building segmentation datasets. After pretraining on these synthesized datasets, high accuracy is yielded with few amount of labeled data for fine-tuning.

UCD eliminates the necessity for prior training, allowing direct deployment of CD algorithms on unlabeled data. This significantly broadens the application scope of DL-based CD, representing a critical objective in the advancement of CD methodologies. However, unsupervised CD presents significant challenges for DL-based frameworks, as the training of DNNs requires task-specific objectives. To address the absence of explicit supervision in UCD, the literature identifies three principal strategies: generative representation, pre-detection supervision, and leveraging external knowledge.

Generative Representation: This approach uses generative models to extract features, eliminating the need for manually labeled data [129].

The model extracts feature maps from the original multi-temporal image for pixel-wise comparison to generate a difference map. A distance metric, such as the Euclidean distance, combined with a threshold segmentation algorithm, derives the final CD results. Prevalent deep generative models include auto-encoders (AE), deep belief networks (DBN), generative adversarial networks (GAN) [130], and denoising diffusion probabilistic models (DDPM) [131].

AEs are unsupervised learning models optimized by minimizing reconstruction errors. However, vanilla AEs tend to learn redundant information (e.g., simply replicating the input data) and encounter difficulties in deriving meaningful representations within a single-layer architecture. Consequently, various variants such as stacked AE (SAE), sparse AE, denoising AE (DAE), and variational AE (VAE) have been adapted for CD tasks.

In [132], an SAE-based algorithm for CD of HR RSIs employs a sparse representative sample selection strategy to reduce time complexity. Liu et al. [98] use an SAE with Fisher’s discriminant criterion for high-resolution SAR image CD to better distinguish between changed and unchanged features. In [133], SAE served as a predictor of hyperspectral anomaly CD. Touati [134] designed a multimodal CD (MMCD) framework based on anomaly detection, noting that changed regions often exhibit significant reconstruction losses in sparse AE. Lv et al. [135] used a contractive AE to minimize noise and extract deep features from superpixels for the SAR image CD. In [133], SAE served as a predictor of hyperspectral anomaly CD. Touati [134] designed an MMCD framework based on anomaly detection, noting that changed regions often exhibit significant reconstruction losses in a sparse AE. Lv et al. [135] employ a contractive AE to minimize noise and extract deep features from superpixels for the SAR image CD. In [136], a cross-resolution difference learning method involving two coupled AEs was developed for CD across images of varying resolutions.

Since DAEs help reduce the impact of noise on original images, they are widely used in SAR and MMCD [97, 137, 138, 139]. To mitigate the loss of spatial contextual information typically associated with vectoring operations in conventional AEs, convolutional layers have been incorporated into AEs, resulting in the development of convolutional AEs (CAEs) for CD. Bergamasco et al. [140] develop a CAE to learn multi-level difference features for multispectral CD. Wu et al. [141] add a commonality constraint to CAE for MMCD applications. Furthermore, to address spatial information loss in fully connected AEs, Wu et al. [142] propose a kernel principal component analysis (KPCA) convolution feature extraction model. A deep KPCA convolutional mapping network is designed following the layer-wise greedy training approach of SAE for both BCD and MCD in HR RSIs. Chen et al. [99] present a graph-based framework to model structural relationships for unsupervised multimodal CD. It employs dual-graph convolutional autoencoders to discern modality-agnostic nodes and edges within multimodal images.

DBNs are another type of classic unsupervised deep model with multiple layers of restricted Boltzmann machines (RBMs). Like SAE, DBNs are trained using a layer-wise greedy approach, enabling them to extract informative features from input images. Despite their potential, DBNs have seen relatively limited application in CD. Gong et al. [100] utilized pre-trained DBN weights as initial weights for a DNN to perform CD on SAR images. Zhao et al. [101] designed a DNN composed of two symmetric DBNs to learn the modality-invariant features for MMCD. Jia et al. [143] introduced a generalized Gamma DBN to learn features from different images, and Zhang et al. [144] compressed features extracted by DBN into a 2D polar domain for MCD on multispectral images.

As a prominent framework for approaching generative AI, GANs have also been widely applied in unsupervised CD. Lei et al. [102] apply GANs to learn representative features from hyperspectral images, achieving robust CD results. Saha et al. [145] develop a GAN-based method to learn deep change hypervectors for CD on multispectral images. Ren et al. [146] developed a GAN-based CD framework to mitigate the issues caused by unregistered objects in paired RSI. Wu et al. [42] propose an end-to-end unsupervised CD framework, jointly training a segmentor and a GAN with L1 constraints. Noh et al. [147] employ GANs for image reconstruction using single temporal images in training and bitemporal images in inference, identifying changed regions by high reconstruction losses. GANs demonstrate exceptional efficacy in MMCD owing to their advanced capabilities in image style transfer. One of the major types of unsupervised MMCD, modality translation methods, predominantly leverages GANs. For instance, Niu et al. [148] use a conditional GAN for modality translation between SAR and optical images, obtaining CD results through direct comparison of transformed images. Subsequent advances include sophisticated GAN architectures and training techniques for improved detection accuracy, such as cycle-consistent GAN [149, 150], CutMix [151], feature space alignment [152], and robust fusion-based CD strategies [153]. These approaches often incorporate pre-detection techniques to isolate changed regions for more stable modality translation results, aligning with the concepts we will discuss in the following subsection.

DDPMs, drawing inspiration from the principles of non-equilibrium thermodynamics, have garnered significant attention in generative artificial intelligence [131]. These models involve a diffusion process that gradually introduces random noise into the data, followed by a reverse diffusion process to reconstruct the desired data distribution from the noise. Training by reconstructing inputs makes DDPMs naturally suitable for feature extraction in CD tasks. Bandara et al. [103] first introduced DDPMs for CD. However, subsequent works focus mainly on fully supervised CD (FSCD) [154], while studies on UCD with DDPMs remain rare.

Pre-detection Supervision: Although unsupervised generative models do not require labeled data to extract features from images for CD, the lack of objectives during the feature extraction process may result in suboptimal and less informative features. Additionally, the absence of labeled data can limit the learning of more advanced DL models. To address these issues, pre-detection-based approaches first generate pseudo labels using traditional unsupervised CD algorithms, then train deep CD models with the pseudo labels. This strategy emulates supervised learning paradigms for training purposes while remaining

entirely unsupervised, as it does not depend on any pre-existing labeled data. Several early DL-based CD methods have adopted this strategy.

The effectiveness of pre-detection supervision depends on the accuracy of pre-detection algorithms. Thus, it is crucial to design or select algorithms that suit the characteristics of input images. Synthetic Aperture Radar (SAR) images, in particular, have been extensively studied due to their unique speckle noise. Gao et al. [62] developed an automatic CD algorithm using PCANet [155], which employs a Gabor wavelet transform and Fuzzy C-means clustering (FCM) to select the most reliable changed and unchanged samples from SAR images. These samples are then used to train the PCANet. Similarly, Gong et al. [100] proposed a deep neural network-based CD algorithm for SAR images that incorporates a pre-detection algorithm based on FCM to select the most representative samples. In another study, Gong et al. [156] introduced an unsupervised ternary CD algorithm where deep feature representations are learned from the difference image using an SAE, effectively suppressing image noise. Geng et al. [65] integrated saliency detection into CD for SAR images by designing a pre-detection algorithm to select representative and reliable samples for training the deep network. Additionally, Yang et al. [157] combined the concept of transfer learning with pre-detection methods to broaden the application scope of CD in SAR images. Liu et al. [158] proposed a locally restricted CNN that adds spatial constraints to the output layer, effectively reducing noise in Polarimetric SAR (PolSAR) images. This model was also supported by a pre-detection algorithm based on the statistical properties of PolSAR images.

Methods tailored for multispectral, hyperspectral, and high-resolution images have also been developed. Gong et al. [159, 160] leveraged the initial difference image generated by the CVA to provide a priori knowledge for sampling training data for GANs. Shi et al. [161] extended this approach to MCD. Du et al. [61] introduced a deep slow feature analysis (DSFA) model combined with a deep neural network to learn nonlinear features and emphasize changes. The authors employed a CVA-based pre-detection method to select samples from multispectral images for training the network. Song et al. [63] utilized PCA and image element unmixing algorithms to select training samples for a recurrent 3D fully convolutional network for binary and multiclass CD. In [162], pseudo-labels from BCD were employed to guide hyperspectral MCD. For high-resolution images, pre-detection algorithms need to focus more on the spatial information within the image. Gong et al. [64] developed a high-resolution CD algorithm based on superpixel segmentation and deep difference representation. This method achieved varying pre-detection results based on different superpixel features and implemented a voting mechanism to select reliable training samples from these results. Xu et al. [163] used SFA as a pre-detection algorithm to select reliable samples to train a stacked DAE for high resolution RSI CD.

Leveraging external knowledge: DNNs pre-trained on natural images are adept at extracting general visual features, which can be highly beneficial for the recognition tasks of RSIs. An early exploration by Saha et al. [114] utilized a CNN encoder pre-trained on natural optical images to extract bitemporal features, which were then pixel-wise compared to classify changes. Subsequently, Saha et al. [115] applied the pre-trained VGG network as a feature extractor for planetary CD. Bandara et al. [116] introduce multiple bitemporal constraints based on metric learning to transfer the inherent knowledge from pre-trained VGG networks to the RS target domain. The approach in [117] initially transfers deep features pre-trained on semantic segmentation datasets, then fine-tunes them with distance constraints and pseudo-change labels to enhance relevance. Furthermore, in [164], object-based image analysis was leveraged to refine feature extraction with a pre-trained CNN. To better tailor the features for the RS domain, a clustering function based on feature distance calculation was introduced in [165]. Yan et al. utilize multi-temporal remote sensing indices as domain knowledge to guide the contrastive learning of change representation [166].

Recently, Visual Foundation Models (VFMs) such as CLIP [167] and Segment Anything Model (SAM) [168] have emerged and gained significant research interest. VFMs, pre-trained on web-scale datasets, are designed to capture universal feature representations that can be generalized to a variety of downstream tasks. However, since these VFMs are generally trained with natural images, they exhibit certain biases in RS applications [169]. Considering spectral and temporal characteristics of RSIs, several RS foundation models (FMs) have been developed, including GFM [170], SpectralGPT [1] and SkySense [171]. These FMs enables training-free feature embedding on multi-spectral, multi-temporal, and multi-modal RS data, thereby supporting a variety of downstream tasks including CD. However, since these FMs are typically trained with the context intrinsic to RSIs, they do not consider the specific application context of CD tasks. Consequently, employing these models for CD still necessitates incorporating CD-specific modules and performing fully supervised fine-tuning.

Considering that FMs contain implicit knowledge of the image content, several recent methods have explored employing FMs to achieve sample-efficient CD. In [4], VFMs are adapted to the RS domain using a semantic latent aligning technique, demonstrating their sample efficiency. Fig.8 presents an overview of this approach, where the latent are aligned via temporal consistency regularization. In [112],

a side-adaption framework is proposed to inject the VFM knowledge into CD models. In [172], SAM is utilized to generate pseudo labels from vague change maps used as prompts. In [113], zero-shot CD is achieved by measuring the similarity of SAM-encoded features. In [173], Chen et al. employed SAM to achieve unsupervised CD between optical images and map data. Dong et al. [174] utilized CLIP to learn visual-language representations to improve CD accuracy.

To elucidate the efficacy of the sample-efficient DL methodologies discussed, Table II presents a comparative analysis of the SOTA accuracy obtained on several benchmark CD datasets. The accuracy metrics include overall accuracy (OA), intersection over union (IoU), and $F_{1}$, which are common in BCD. To facilitate comparison between different types of supervision, we select the most frequently used datasets in various tasks, including Levir [177], WHU [178], and OSCD [179]. It is important to acknowledge that there are significant variations in the experimental configurations of the methods being compared, a concern raised in [180]. Therefore, this table is intended solely to provide an intuitive assessment of the accuracy of SOTA.

To facilitate a comprehensive understanding of the training samples utilized across various methods, Table III presents the metadata of each CD benchmark. Overall, Levir and WHU are two VHR datasets with large image size and rich change samples. In contrast, OSCD has lower resolution and contains less training samples. To present an intuitive comparison of the SOTA accuracy across various learning paradigms, Table III also summarizes the highest $F_{1}$ scores achieved in each dataset.

Tables II and III clearly demonstrate that the accuracy of CD is highly dependent on the level of supervision during the training process. First, the CD accuracy on the Levir and WHU datasets is significantly higher than that on the OSCD dataset. This disparity is attributed to the richer set of change samples and the finer spatial resolution present in the Levir and WHU datasets. Second, the accuracy of FSCD and SSCD with fine-tuning (FT) is higher than that of SMCD, WSCD, UCD and SSCD without FT. Notably, the SSCD with FT marginally surpasses FSCD, which can be attributed to its extensive pre-training that effectively utilizes the image contexts as extra supervisions. This observed accuracy hierarchy aligns with the strong-to-weak supervision level in the different learning paradigms, as illustrated in Fig.3.

SMCD achieves the highest accuracy among sample-efficient CD approaches. Recent advances in SMCD ensure remarkably high accuracy with only a small proportion of training samples. For example, utilizing only 5% of the training data, the SOTA SMCD methods only see a minor $F_{1}$ reduction of 2% on the Levir and 0.6% on the OSCD datasets. However, it is important to note that even with this small portion of training data, SMCD still requires a substantial number of change samples. Based on the number of change instances detailed in Table II and through a rough estimate, SMCD typically requires more than 100 change samples on the Levir and WHU datasets.

The accuracy of WSCD is significantly influenced by the level of supervision applied. Compared to image-level labels, employing spatial labels (such as box or point labels) for training WSCD algorithms generally results in superior accuracy. For example, as tested on the Levir dataset, point label-supervised CD approach [125] has an advantage of exceeding 30% in $F_{1}$ compared to approaches that utilize image labels. Regarding image label-supervised SMCD, while a relatively high accuracy is attained (particularly on VHR datasets), it is important to note that training is carried out using patch labels rather than a complete RSI. As reported in [67] and [69], image labels are assigned to each pair of patches with 256$\times$256 pixels. Therefore, this type of SMCD still necessitates a certain degree of human intervention.

SSCD can be employed as either an approach to achieve label-free learning of change representations, or merely as a pretraining technique to initialize the DNN parameters. SSCD without FT is challenging, since the image contexts utilized in self-supervised learning are independent of the application contexts inherent in CD tasks. Most literature works adopt the latter strategy, that is, pre-training through self-supervision and fine-tuning with all available change samples. This strategy yields substantial accuracy improvements over the vanilla FSCD. The improvements are particularly significant on the OSCD dataset (up to 12% in $F_{1}$ [89]), which can be attributed to the scarcity of training samples within this dataset.

Meanwhile, SSCD without FT can be regarded as a distinct subset of UCD that utilizes self-supervised learning techniques. Most literature studies on UCD and label-free SSCD have been conducted on medium resolution datasets such as OSCD. They are commonly adopted for analysis of satellite images such as those collected by Sentinel and Landsat. Due to the fact that numerous experiments are performed on non-open benchmarks, it is challenging to assess the level of accuracy, and hence, these results are not presented in Table II. The highest metrics obtained on the OSCD dataset are 92.63% in OA and 35.85% in $F_{1}$ [61], exhibiting a reduction exceeding 23% in $F_{1}$ relative to FSCD. UCD (or label-free SSCD) is more challenging when applied to VHR datasets due to the increased spatial complexity. A reduction of approximately 30% in $F_{1}$ is noted when applied to the Levir dataset. One of the zero-shot CD approaches, Anychange [113], obtains an accuracy of 24.5% in $F_{1}$, highlighting a substantial gap for further advancements.

In summary, sample-efficient CD methods have greatly reduced the dependence on a large volume of training samples, thereby achieving relatively high accuracy with a reduced number of samples or the utilization of weak labels. However, training CD algorithms without labels or using a very low level of supervision remains a challenge.

Sample-efficient CD still encounters considerable challenges in mitigating the data dependency and in generalizing insights across diverse datasets without necessitating extensive fine-tuning. In the followings we analyse several principal obstacles.