​${}_{\!}$​    ​ pen-CD: A Comprehensive Toolbox for Change Detection

• •

  FC-EF [9] follows U-Net architecture. It concatenates the bi-temporal images before passing them through the network, treating them as different color channels.

• •

  FC-Siam-Conc [9] is a Siamese network. Compared to FC-EF, it separates the encoder layers into two streams with the same structure and shared weights (i.e., Siamese network) that process the bi-temporal images separately. In the skip connection, it concatenates the bi-temporal features and feeds them to the decoder.

• •

  FC-Siam-Diff [9] is a variant of FC-Siam-Conc. Instead of concatenating both features from the encoding streams, it calculates the absolute value of their difference.

• •

  STANet [4] is a metric-based change detection model. It consists of feature extractor, spatial-temporal attention module and metric module. The spatial-temporal attention module is a variant of self-attention in change detection task, which captures the global spatial–temporal relationships in the whole space-time to obtain more discriminative features. This attention module has two forms, basic spatial-temporal attention module (BAM) and pyramid spatial-temporal attention module (PAM). In addition, to reduce the effect of class imbalance, a class-sensitive loss, batch-balanced contrastive loss (BCL), is designed in STANet, which uses the number of samples in each class within a batch to determine the weight factor in the loss function.

• •

  DSIFN [35] builds on the FC-Siam-conc model. It uses channel and spatial attention to construct stronger basic modules (like CBAM) and also introduces deep supervision for fast convergence and performance improvement of the model.

• •

  SNUNet [11] is a standard encoder-decoder architecture and uses the Siamese network as encoder. To maintain high-resolution features and fine-grained localization information, SNUNet uses the dense skip connection mechanism between the encoder and decoder (like UNet++). For fusing multi-granularity features, the ensemble channel attention module (ECAM) is designed to automatically select and focus on more effective information between different decoder groups. Structurally, ECAM is a natural expansion of plain channel attention module (CAM) in deep supervision and ensemble learning.

• •

  BIT [5] is a CNN-Transformer based change detction model. It abstracts pixel-level features into several visual tokens and models spatio-temporal context information in a compact token space. Compared with directly extracting dense spatio-temporal semantic correlations in pixel-level space, on the one hand, it reduces the interference of redundant information in image space through pixel-level feature aggregation, and uses Transformer encoders to construct spatio-temporal correlations, which significantly reduces computational complexity. The calculation efficiency is improved. On the other hand, the Transformer decoder is used to enhance the original pixel-level features with the learned context information, and the global spatio-temporal information is used to enhance the pixel-level representation, which improves the ability of the model to identify objects of interest and exclude irrelevant changes.

• •

  ChangeStar [37] and ChangeStar2 [40] are simple yet unified change detectors capable of addressing binary change detection, object change detection, and semantic change detection, which is composed of a Siamese dense feature extractor and ChangeMixin or ChangeMixin2. This design is inspired by reusing the modern semantic segmentation architecture because semantic segmentation and change detection are both dense prediction tasks. ChangeMixin and ChangeMixin2 enable any off-the-shelf deep semantic segmentation network to detect changes.

• •

  ChangeFormer [2] is a variant of SegFormer [31] in change detection task. Different from SegFormer, it uses a Siamese customized mix transformer (MiT) as the encoder. The bi-temporal features in the Siamese network are concatenated and then fed into the decoder consisting of multi-layer perceptron (MLP) layers.

• •

  Changer [12] series models emphasize interactions between bi-temporal branches. There are two simple interaction strategies: aggregation-distribution (AD) and feature “exchange.” Specifically, the AD interaction is abstracted from some co-/cross-attention mechanisms. The “exchange” interaction is a completely parameter and computation-free operation, which is achieved by exchanging bi-temporal feature maps in the spatial or channel dimension, and the exchanged features are mixed as they pass through subsequent convolution or token mixer. In addition, the flow-based dual-alignment fusion (FDAF) module is proposed to overcome the problem of side-looking and misalignment in multi-temporal images.

• •

  TinyCD [8] aims to develop a deep learning model that significantly reduces memory consumption and computational complexity. To achieve this, TinyCD extracts low-level features from two images using the early layers of a selected Siamese backbone. This approach minimizes the model’s size and leverages the locality structure of features induced by the inductive bias inherent in the initial stages of a fully convolutional backbone. Subsequently, TinyCD introduces an attention module called MAMB. This module comprises an initial mixing stage that employs grouped convolution to integrate semantically similar features from the two images. Following this, a MLP fuses the mixed features into a single heatmap. This heatmap serves as a skip connection, reweighting the upsampled features during the upsampling phase of the Siamese U-Net. This process functions as a spatio-temporal attention mechanism. In the final stage, TinyCD utilizes a multi-layer perceptron as a pixel-level classifier to produce a probability score indicating change for each pixel. These architectural choices result in a model with a significantly reduced number of parameters and computational complexity while still achieving state-of-the-art performance. Consequently, TinyCD presents a valuable alternative to other leading models, particularly in scenarios where rapid training and deployment on low-end devices are critical.

• •

  HANet [13] is a discriminative Siamese network, hierarchical attention network, which can integrate multiscale features and refine detailed features. HANet has four progressive foreground-balanced sampling strategies based on not adding change information to help the model accurately learn the features of the changed pixels during the early training process and thereby improve detection performance. The main part of HANet is the HAN module, which is a lightweight and effective self-attention mechanism.

• •

  LightCDNet [32] is a lightweight and accurate change detection model designed for real-world application deployment. LightCDNet follows the standard “encoder-decode” structure, but unlike most Siamese network-based models, this model have designed a novel early fusion module to facilitate cross-temporal information fusion, called the deep supervised fusion module (DSFM), which effectively enhances the information preservation capability of the change detection model even with a small number of parameters. On the other hand, LightCDNet uses an improved ShuffleNetV2 as the encoder, coupled with a feature pyramid decoder to achieve high-precision change detection performance. LightCDNet provide versions with different parameter quantities for flexible deployment.

• •

  CGNet [14] is the change guiding network to tackle the insufficient expression problem of change features in the conventional U-Net structure adopted in previous methods. It contains a self-attention module named change guide module, which can effectively capture the long-distance dependency among pixels and effectively overcomes the problem of the insufficient receptive field of traditional convolutional neural networks.

• •

  BAN [17] is a universal foundation model-based change detection adaptation framework aiming to extract the knowledge of foundation models for change detection. It contains three parts, i.e. frozen foundation model (e.g., CLIP), bi-temporal adapter branch (Bi-TAB), and bridging modules between them. Specifically, BAN extracts general features through a frozen foundation model, which are then selected, aligned, and injected into Bi-TAB via the bridging modules. Bi-TAB is designed as a model-agnostic concept to extract task/domain-specific features, which can be either an existing arbitrary change detection model or some hand-crafted stacked blocks.

• •

  TTP [7] enhances high-precision change detection in complex spatio-temporal remote sensing scenarios by integrating the general knowledge of foundational visual models into the change detection task. It leverages the potential of foundational models even with limited annotated change detection data. TTP overcomes domain shift issues during knowledge transfer and addresses the challenge of expressing both homogeneous and heterogeneous features in multi-temporal images. Specifically, TTP utilizes general segmentation knowledge based on the segment anything model (SAM) by introducing low-rank fine-tuning parameters into the SAM backbone, which mitigates spatial semantic domain shifts. Furthermore, TTP proposes a time-travel activation gate, enabling temporal features to permeate the pixel semantic space, thus enhancing the foundational model’s ability to understand both homogeneous and heterogeneous features of bi-temporal images. Lastly, an efficient multi-level change prediction head is designed to decode dense and high-level semantic change features within the foundational model. This method paves the way for more accurate and efficient change detection in remote sensing imagery.

There is no doubt that the construction of change detection models benefits from other fundamental tasks in general vision, especially classification and segmentation. The former provides powerful feature extractors (backbones) and the latter provides extensive change mask generation schemes. Therefore, a core aspect of Open-CD’s design is to allow users to arbitrarily call components from other vision models and to make it easier for change detection to benefit from advanced general modules. This is an important reason why we build Open-CD on top of OpenMMLab toolkits.

Another change detection codebase that allows for custom module combinations is change_detection.pytorch (CDP), which is built on a ”Siamese encoder + feature fusion + decoder” mode and supports some common backbone networks and segmentation heads. However, since CDP relies on developers to frequently code to support the latest models, it is difficult to maintain and update. Open-CD, on the other hand, is able to benefit directly from the constantly updated OpenMMLab Toolkits, and to modify and combine hundreds of models and modules. In brief, Open-CD allows users to play free architecture games.

As shown in Figure 1, Open-CD consists of five parts.

Config Files. In Open-CD, the model, optimizer, dataset, data piprline, etc. are specified in config files. In general, if the user does not need to redesign the model internally, simply modifying the config file is enough.

Model Zoo. Besides change detection models listed in Table 1, Open-CD supports almost all models in OpenMMLab related algorithm codebases, which can be called directly in the config file by simply installing them as dependencies. In addition, there are thousands of pre-trained weights available, including plain ImageNet, foundation models, etc.

Internal Modules. Following the design concept of the OpenMMLab toolkits, the training process of Open-CD is conducted in the MMEngine Runner. We will describe it in Section 3.2.

Data Files. Open-CD supports the current mainstream binary and semantic change detection datasets, as listed in Table 2. And since we define two extensible base classes, _BaseCDDataset and BaseSCDDataset, it is easy for users to integrate custom datasets into Open-CD.

Tools. Open-CD also contains several functional scripts e.g. training tools, inference tools, data analysis tools, result analysis tools, etc.

Similar to other algorithm codebases in OpenMMLab toolkits, Open-CD follows the unified interface of abstract components defined in MMEngine. The training pipeline of Open-CD is shown in Figure 14. First, the Dataloader obtains data from the dataset and transmits it to the model. Outside the model, there is a Wrapper for distributed training, etc. Then, the parameter scheduler is used to adjust Optimizer related parameters e.g. learning rate during training, and the Optimizer is used to update the model. The Optimizer wrapper is capable of gradient accumulation, gradient clipping, mixed precision training, etc. Finally, in the evaluation phase, the data and model outputs are fed to the Evaluator and the corresponding values are returned based on the evaluation metrics.

To make the training pipeline extensible, some hook points are inserted into the pipeline. With these hooks, the data flow in the Runner can be operated and observed to achieve the user-customized requirements. During the whole training pipeline, the Logging component is constantly working, transmitting the required information to the Visualizer.

1. (1)

   Images are randomly cropped to 256 $\times$ 256.

2. (2)

   We use a single RTX 3090 GPU for training and inference, with a training batch size of 8.

3. (3)

   The training schedule is “40k”, meaning 40k iteration.

4. (4)

   Data augmentation including: RandomRotate, RandomFlip and PhotoMetricDistortion.

Main results. We benchmark different methods on LEVIR-CD Test, including FC-EF [9], FC-Siam-Diff [9], FC-Siam-Conc [9], STANet-BAM/PAM [4], DSIFN [35], SNUNet-c16 [11], BIT [5], ChangeStar [37], ChangeFormer-b0/b1 [2], Changer [12], TinyCD [8], HANet [13], LightCDNet-base/large [32], CGNet [14], BAN (ViT-L) [17] and TTP [7]. We report the parameter, GFLOPs, inference speed, $Precision^{c}$, $Recall^{c}$, $F_{1}^{c}$ and $IoU^{c}$ of these methods in Table 3. The inference time is tested on a single RTX 3090 GPU. The GFLOPs is calculated at a size of $256\times 256$. For re-production and comparison in subsequent studies, we will also release all weights and visualization results.

Comparison with official code. We compare the official implementations of some methods with those in Open-CD, as shown in Figure 3. With the exception of STANet and TinyCD, all other implementations in Open-CD gain some degree of performance improvement compared to the original implementations. Especially for BIT, ChangeStar and ChangeFormer, the improvements are significant. Notably, the original ChangeFormer customizes the configuration of MiT (e.g. depth, width of the network, etc.). In Open-CD, we restore the settings of b1$\sim$b5 in SegFormer to make it scalable, and obtain higher $F_{1}^{c}$ score with fewer parameters.

As mentioned above, Open-CD can be used for the implementation of custom change detection algorithms, validation of existing algorithms, and model deployment (relying on the OpenMMLab ecosystem, i.e., MMDeploy¹⁵¹⁵15https://github.com/open-mmlab/mmdeploy. Although the current Open-CD only contains supervised learning pipelines, it can also be applied as a construction tool for models, datasets, etc., promoting the exploration of semi-supervised and unsupervised algorithms.

Recently, researchers tend to explore the general capabilities of foundation models. Within the remote sensing community, foundation models are also actively explored. Some works attempt to train a robust backbone that is capable of extracting general features for a series of downstream perceptual tasks e.g., object detection, semantic segmentation, change detection, etc. For this, Open-CD is an out-of-the-box downstream validation tool for change detection. Typically, MTP [30], a remote sensing foundation model via multi-task pre-training, yields an image encoder for downstream task fine-tuning and uses Open-CD for validation of change detection task.

Although huge amount of remote sensing images are available through Google Earth etc., data annotation remains a challenge, which is time-consuming and costly. One trend is to synthesize simulated image-label pairs through conditional GAN, Diffusion models, etc [38, 29, 39]. and use these data to (pre-)train downstream perception models. In this case, Open-CD can be applied to validate the quality of the simulated change detection data. A typical study is ChangeAnywhere [29], which proposes a diffusion model to generate a simulated change detection dataset ChangeAnywhere-100K and uses Open-CD for subsequent validation.

As an advanced algorithm codebase, Open-CD can be easily used for change detection competitions and challenges. In general, users only need to customize their Dataset Class to enjoy all the advanced algorithms, data analysis tools, inference tools, etc. in Open-CD. Here, some solutions of winning competitions with Open-CD are listed:

• •

  2024 ISPRS TC 1 Contest.¹⁶¹⁶16https://www.gaofen-challenge.com/challenge Winner. ¹⁷¹⁷17https://github.com/DanyangLihhh/2024-ISPRS-TC-I-Contest

• •

  2023 “Jilin-1” Cup.¹⁸¹⁸18https://www.jl1mall.com/contest 5th place. ¹⁹¹⁹19https://github.com/DanyangLihhh/Cultivated-land-change-detection