ChangeViT: Unleashing Plain Vision Transformers for Change Detection

Regarding the network architecture, existing change detection methods employing deep learning can be generally categorized into two groups: CNN-based and transformer-based.

CNN-based Methods. CNN-based change detection approaches have been the mainstream framework in the literature [7, 9, 33, 34, 35, 36, 8, 37, 38, 39] for a long time, known for their hierarchical feature modeling capabilities. These works primarily focus on multi-scale feature extraction, difference modeling, lightweight architecture designing, and foreground-background class imbalance. For instance, methods in [35, 39] utilize fully convolutional networks to capture hierarchical features for learning multi-scale feature representations. For adequate differential feature modeling, approaches in [33, 36] incorporate the attention mechanism to establish relational dependencies among bi-temporal features. In contrast, Changer [34] introduces a parameter-free method, which simply exchanges the characteristics of each phase to capture and perceive each other’s information. Methods in [9, 8] focus on designing efficient and effective network architectures, utilizing lightweight feature extractors [40, 41] as backbones. Several studies [7, 37] address the significant challenge posed by foreground-background class imbalance by developing innovative loss functions that prioritize foreground alterations while minimizing interference from background noise (e.g., seasonal variations, climate changes).

Transformer-based Methods. Recently, Vision Transformer [11] and its variants [30, 42, 31] have surpassed CNN in various visual tasks and became the dominant backbone [12, 14, 15, 43]. Motivated by these achievements, several works [16, 18, 6, 44, 45, 17, 19] have explored the application of transformers in change detection tasks. Some of these methods [45, 18] utilize pure transformers, while others [16, 6, 44, 17] adopt CNN-Transformer hybrid architectures. Methods in [45, 18] introduce hierarchical transformer networks based on the swin transformer [30]. The others typically follow a paradigm in which features extracted by CNN serve as semantic tokens, followed by contextual relation modeling between bi-temporal tokens using transformer blocks. The method introduced in [19] exhibits an efficient tuning strategy that involves freezing the parameters of the Transformer encoder while introducing additional trainable parameters. However, this method fails to deliver optimal results due to an inadequate exploration of the strengths and limitations of the transformers. This precludes a more effective application of the model’s capabilities, thereby capping the potential gains in performance.

Different from the previous approaches mainly using hierarchical networks, the proposed ChangeViT applies the plain ViT as the cornerstone feature extractor, which we find has previously unidentified potential in detecting large-scale changes.

ViT [11] is a plain, non-hierarchical architecture, which is a powerful alternative to standard CNN for image classification. Due to the significant computational overhead of self-attention in ViT, subsequent works focus on designing more efficient architectures, such as Swin [30], PVT [31] and PiT [32]. These works inherit some designs from CNN, including hierarchical structures, sliding windows, and convolutions. Recently, researchers have begun to study the potential of ViT for various downstream tasks motivated by the emergence of large pre-trained models, e.g., DeiT [26], DINO [46], DINOv2 [28], MAE [47] and CLIP [48]. The plain ViT has already made remarkable progress in dense prediction [43, 12, 13], pose estimation [15], image matting [14], etc. ViTDet [12] is the first to employ plain, non-hierarchical ViT as the backbone for object detection with minimal adaptation, i.e., building a simple feature pyramid for single-scale features and aiding a few cross windows for information propagation. ViT-Adapter [43] introduces a pre-training-free adapter that injects prior knowledge to ViT without redesigning its architecture for various dense prediction tasks. Similarly, SimpleClick [13] and ViTPose[15] apply vanilla ViT as the feature extractor to acquire single-scale features. For image matting, ViTMatte [14] is the first work to unleash the potential of ViT with concise adaption.

Inspired by the above works, we aim to unleash the potential of the plain ViT model, enabling it to adapt well to change detection tasks.

The feature extractor is composed of a plain ViT, and a detail-capture module which is described as follows:

Plain ViT. Bi-temporal images $I_{1}$, $I_{2}$ are fed into patch embedding layer, dividing them into non-overlapping $16\times 16$ patches. These patches are then flattened and projected to $D$-dimension tokens, and the feature resolution is reduced to $1/16$ of the original images. Afterwards, position embedding is added to these tokens, which are passed through $L$ transformer layers. Each layer consists of a layer normalization (LN), a multi-head self-attention (MHSA) and a feed-forward network (FFN). The formulation of these layers is given by Eq. 1 and Eq. 2:

where $i$ denotes the output of the $i$th transformer layer. The final output of the ViT backbone is represented as $F_{V}^{t}\in\mathbb{R}^{\frac{H}{16}\times\frac{W}{16}\times C_{4}}$, where $C_{4}$ equals to $D$ .

Detail-capture. As discussed in Sec. I, ViT demonstrates proficiency in detecting large changes but exhibits reduced effectiveness with smaller ones. Addressing this challenge, we introduce a detail-capture module designed to compensate for the absence of fine-grained local cues crucial for change detection. This module comprises three residual convolutional blocks (C2-C4) adapted from ResNet18 [20]. Upon processing the input images through the detail-capture module, three-scale detailed features are generated, i.e., $1/2$, $1/4$ and $1/8$, denoted as $F_{C_{i}}^{t}\in\mathbb{R}^{\frac{H}{2^{i}}\times\frac{W}{2^{i}}\times C_{i}}$ ($i\in\{1,2,3\}$).

In the change detection task, preserving detailed spatial features is crucial as they can help detect small objects. Ensuring the effective transmission of low-level details to high-level semantic features is paramount.

Therefore, we introduce a feature injector, composed of three cross-attention blocks [49], as illustrated in Fig. 3(a). It considers the low-level features as the key and value vectors and the high-level feature as the query vector. Intuitively, this is reasonable as it allows the feature injector to gather the most relevant information based on the provided key information and integrate it into the query. By enabling cross-layer feature propagation, detailed information can be incorporated into the high-level representations of the ViT, denoted as $F_{V_{E}}^{t}$. The $F_{V_{E}}^{t}$ is computed as follows:

where $i\in\{1,2,3\}$ denotes the index of low-level layers, $F_{V}^{t}$ as query and $F_{C_{i}}^{t}$ as key and value, respectively. The $\mathrm{FC}$ is a 2D depth-wise convolution with the kernel size of $1\times 1$ and ⓒ denotes concatenation operation along the channel dimension.

Additionally, we explore an alternative approach to feature injector, as depicted in Fig. 3(b), which considers the low-level features as query, and the ViT’s semantic information as key and value to refine the ViT’s representation according to the characteristics of the hierarchical detailed features.

Compared to existing methods [7, 6, 36, 44], which employ complex techniques to model difference information and predict the change probability map, we chose a simpler decoder to better demonstrate the learning capabilities of ChangeViT. Specifically, we use a straightforward feature fusion layer to capture differences between bi-temporal features. A cascade convolutional layer, followed by an upsampling operation, is employed to progressively aggregate differential features from deep to shallow layers, ultimately restoring them to the original resolution of $H\times W$. The difference modeling is formulated as Eq. 5:

where $F_{i}^{t}\in\{F_{C_{1}}^{t},F_{C_{2}}^{t},F_{C_{3}}^{t},F_{V_{E}}^{t}\}$ ($t\in\{1,2\}$), $\mathrm{MLP}$ is a three-layer 2D convolutional network with kernel size of $3\times 3$ along with ReLU activation function, ⓒ denotes concatenating on channel dimension, and $|\cdot|$ means absolute value operation.

To restore the original resolution of the changed map, we use a simple cascade upsampling operation, which is represented as follows:

where $\mathrm{Conv_{1\times 1}}$ is a 2D convolution with a kernel size of $1\times 1$ to reduce the channel dimension, and $\mathrm{Deconv_{4\times 4}}$ denotes 2D deconvolution to upsample the feature map with a kernel size of $4\times 4$ and stride size of $2\times 2$.

Finally, a classification layer is applied to transform the shallowest features $F_{D_{4}}$ into change maps $P$, which is formulated as Eq. 7:

where $\mathrm{Conv_{3\times 3}}$ is a 2D convolution with the kernel size of $3\times 3$, $\mathrm{Sigmoid}$ function maps the feature map to $(0,1)$ and then transforms to a binary map given a predefined threshold (i.e., 0.5), i.e., $P\in\{0,1\}^{H\times W}$.

As mentioned in prior works [7, 9], the proportion of changed targets is significantly lower than that of unchanged ones. Following the above works, we adopt binary cross-entropy (BCE) and dice loss (Dice) [50] to alleviate the class imbalance problem. The change detection loss $\mathcal{L}_{total}$ is defined as Eq. 8:

The BCE and Dice losses are formulated as follows:

where $i$ denotes the $i$-th pixel, $N$ is the number of total pixels, $Y$ denotes the ground truth and $\epsilon$ (i.e., 1e-5) is a smooth term utilized to avoid zero division.

We adopt vanilla ViT [11] as our primary backbone, specifically incorporating its tiny and small variants, thereby constructing two models named ChangeViT-T and ChangeViT-S. We use DeiT [26] and DINOv2 [28] pre-trained weights for initialization, respectively. Our models are implemented using the PyTorch framework [51] and executed on a computing platform consisting of a single NVIDIA GeForce RTX 3090 GPU paired with an Intel(R) Xeon(R) Gold 6138 CPU. For optimization, we opt for the Adam optimizer [52], with beta values set to (0.9, 0.99) and a weight decay of 1e-4. Initially, the learning rate is 2e-4 and gradually reduces according to a scheduled reduction formula: (1-(curr_iter/max_iter))${}^{\alpha}\times$ lr, where $\alpha$ is set to 0.9 and max_iter is set to 80K iterations on LEVIR-CD and WHU-CD, 40K for CLCD dataset, and 10K for OSCD, respectively. The batch size remains at 16 across all experiments. To augment the training data and bolster the model’s robustness, we apply random flipping and cropping data enhancement approaches. The channel dimensions of $F_{C_{i}}$ are set to 64, 128, and 256, respectively. Furthermore, we ensure consistency and fairness in comparison by meticulously aligning the experimental settings of the compared methods with those specified in the original paper.

As illustrated in Tab. I, we compare ChangeViT with the previous methods on three high-resolution datasets, i.e., LEVIR-CD, WHU-CD, and CLCD. Notably, all the compared methods employed hierarchical backbone as primary feature extractors. Specifically, DTCDSCN, BIT, ICIFNet, DMINet, GASNet and AMTNet apply ResNet [20] or its variants [54] as backbones, SNUNet and EATDer apply nested UNet [22] and stack non-local blocks [55], respectively. In contrast, our approach employs a non-hierarchical, plain ViT as the core backbone which includes ViT-T and ViT-S, coupling with a lightweight detail-capture module which serves as an auxiliary network. From Tab. I, we can summarize the following valuable findings: (1) ChangeViT consistently outperforms the existing works across all datasets and evaluation metrics, despite utilizing the tiny backbone of ViT, which demonstrates its effectiveness. (2) The primary feature extractor in ViTs, despite being non-hierarchical, demonstrates competitive performance when compared to hierarchical-based methods. This underscores the robust feature extraction and representation capabilities that large-scale pre-training ViT can offer, fully realizing its potential. (3) Notably, ChangeViT-T and ChangeViT-S exhibit significant performance gains over the SOTA method (i.e., AMTNet) by 3.99% and 4.54% IoU on the WHU-CD dataset. This finding is sensible given that the changes in WHU-CD vary widely, with fewer medium-sized objects compared to smaller and larger ones. This observation aligns with the results illustrated in the middle of Fig. 1(b), underscoring the efficacy of our proposed method in capturing global features and extracting fine-grained spatial information. (4) With an increase in the size of the primary feature extractor, ChangeViT demonstrates enhanced performance. Notably, the detail-capture module, comprising just 2.7M parameters, stands out for its lightweight nature when compared to the total parameter count of each model (i.e., 11.68M and 32.13M). Our proposed ChangeViT achieves a superior balance between efficiency and effectiveness compared to previous methods, underscoring its superiority.

As shown in Tab. II, we also compare ChangeViT with several existing methods on the low-resolution dataset, i.e., OSCD. The targets in the OSCD are relatively smaller than those in high-resolution datasets, exacerbating the foreground-background imbalance issue and making it challenging to detect smaller targets. From Tab. II, the following key points can be noted: (1) The proposed ChangeViT outperforms all compared methods across three evaluation metrics, despite utilizing tiny or small models of ViT, demonstrating its effectiveness on the low-resolution dataset. (2) GASNet and AMTNet perform poorly on this dataset, likely due to their inefficiency in detecting small targets. Although GASNet introduces a foreground-awareness module to address the category imbalance between the foreground and background, it still underperforms in detecting changes in low-resolution remote sensing images.

Effectiveness with different architectures. In Tab. III, we investigate the effectiveness of the proposed modules with different architectures, including hierarchical (i.e., Swin-S [30], PVT-S [31], PiT-S [32]) and non-hierarchical (i.e., ViT-S [11]) transformers. Key observations from the table include: (1) Without combining with our proposed modules, the non-hierarchical ViT-S underperformers the other hierarchical methods across all metrics on three datasets. (2) When integrated with our proposed modules, all transformers exhibit performance improvements, indicating the efficacy of our approach regardless of transformer architectures. (3) ViT-S achieves significant performance gains over hierarchical transformers when equipped with our proposed modules, suggesting that our modules effectively mitigate ViT-S’s limitations in capturing detailed information to detect smaller objects.

Effectiveness of proposed modules. To investigate the effectiveness of the proposed modules, we conduct comprehensive diagnostic experiments on three datasets. As shown in Tab. IV, we take various combinations of components into account and explore the contribution of each module. We apply ViT as baseline, which consists of a plain ViT and a decoder. Coupling with detail-capture module, ViT can unleash its potential and improve 8.81%, 8.60%, and 6.18% F1 on three datasets compared to the baseline, which indicates that the detail-capture module can supplement the detailed spatial information, which is essential for the change detection task. Furthermore, when combined with the feature injector, there are additional performance gains of 0.78%, 1.54%, and 2.17% in F1, indicating the effectiveness of incorporating detailed information at higher levels. In summary, all of our proposed modules are essential and effective in the ChangeViT framework.

Impact of multiple scales. To investigate the necessity of capturing multiple scales in the detail-capture module, we conduct experiments using multi-scale features, i.e., 1/2, 1/4, 1/8. As shown in Tab. V, we can get the following key observations: (1) Single-scale features often yield subpar results, while the amalgamation of multi-scale features leads to enhanced performance. (2) An interesting finding is that high-level features or their combinations can achieve better performance than low-level features. (3) Furthermore, the inclusion of three-scale features results in mutual improvements, indicating that multi-scale features leverage spatial cues across complementary levels.

Impact of pre-trained weights. To investigate the impact of pre-trained weights on ChangeViT, we apply various model initialization approaches, including random initialization and several publicly available large-scale pre-trained weights derived from both supervised and self-supervised training strategies on various datasets. As illustrated in Tab. VI, we observed the following key points: (1) Both ChangeViT-T and ChangeViT-S exhibit improved detection accuracy when utilizing pre-trained weights compared to random initialization. (2) DINOv2-S provides the most effective pre-trained weights for the ChangeViT-S model, benefiting from large-scale data pre-training. (3) When DMINet, GASNet, AMTNet, and ChangeViT are pre-trained on the same data, i.e., ImageNet-1k, the proposed ChangeViT outperforms all the CNN-based methods, demonstrating the effectiveness of transferring the priorities of large pre-trained ViT models to the change detection task.

Choice of query, key and value. Two experiments are conducted to investigate different modeling approaches in the feature injector, as shown in Tab. VII. In the first experiment, $F_{V}$ serves as query, and $F_{C}$ serves as key and value, yielding the best performance. This result is consistent with the conjecture mentioned in Sec. III-B, suggesting that the feature injector effectively captures low-level value information most relevant to the high-level query and reintegrates it back to the query. Therefore, through cross-attention, high-level fine-grained features can seamlessly merge with low-level features.

Size of changes v.s. performance. As depicted in Fig. 4 (a), we conduct experiments on three datasets using the detail-capture module, ViT-S, and our proposed method to quantitatively analyse the performance of each method under different change sizes. The detail-capture module and ViT-S both integrate with a decoder which is the same as ChangeViT. The results indicate that the detail-capture module excels at detecting smaller changed targets, while the ViT-S demonstrates superiority in detecting larger ones. Our method capitalizes on ViT’s powerful feature expression while leveraging a detail-capture module for fine-detail information mining. This comprehensive approach enables superior performance across targets of all sizes.

Qualitative results. We present representative visualization results on three datasets, comparing the performance of the detail-capture module, ViT-S, and our proposed method to demonstrate the effectiveness of ChangeViT. As shown in Fig. 4 (b), the first row in each dataset presents the test results for smaller targets, while the second row corresponds to larger targets. From the results, we can see that the detail-capture module excels at detecting smaller targets, whereas ViT-S demonstrates superiority in detecting larger targets. The fundamental distinction lies in the local receptive field of CNN, enabling them to extract intricate local features, while ViT possesses a global receptive field, facilitating the extraction of comprehensive global information. The proposed method efficiently integrates global and local information, resulting in superior performance.

To qualitatively compare with previous methods, we provide comprehensive samples encompassing small, large, sparse, and dense targets, as illustrated in Fig. 5. From these samples, several key observations emerge intuitively: (1) Our proposed method consistently outperforms all compared methods across various change sizes. This is attributed to the robust global modeling capabilities of ViT and the detail-capture module’s capacity to extract intricate spatial information. Additionally, a feature injector integrates low-level fine-grained spatial features into ViT’s high-level semantic representations, enhancing ChangeViT’s capability to detect changes of diverse sizes. (2) In detecting dense objects, regardless of their size, ChangeViT consistently delineates clear boundaries compared to prior methods. This underscores ChangeViT’s effectiveness in capturing both global semantic information and local spatial details of neighboring objects.