S²-cGAN: Self-Supervised Adversarial Representation Learning for Binary Change Detection in Multispectral Images

Binary change detection (CD) in multitemporal multispectral remote sensing (RS) images is a key component for monitoring environmental phenomena [1, 2, 3, 4]. In the last years, Deep Neural Networks (DNNs) have achieved remarkable performance in several RS applications, including CD [2, 3]. Most of the DNNs designed for CD problems require a huge amount of multitemporal labeled samples to adjust all parameters during training and reach high performance. However, collecting multitemporal labeled samples is often highly expensive and needs expertise. To address this problem, unsupervised deep learning (DL) based approaches are recently introduced in RS to employ pre-trained networks as a generic feature extractor. El Amin et al. propose to extract deep distance images (DI) using a fully convolutional network trained for semantic segmentation in [1]. In this work, it is shown that the extracted deep feature maps contain semantic information, which can improve the DI analysis. Similarly, Saha et al. demonstrate the effectiveness of employing semantic-aware deep feature maps to achieve a performance gain in Change Vector Analysis (CVA) [5]. Bergamasco et al. propose a multilayer convolutional-autoencoder to reconstruct the input image and learn task-specific features using a deep network [6]. Then, multi-scale features are extracted from the pre and post change images that are analyzed for CD. In recent years, unsupervised CD methods based on deep Generative Adversarial Network (GANs) [7] have come to the light of attention in RS. GANs are deep networks commonly used for generating realistic data (e.g., images), where the supervision is indirectly provided by an adversarial game between two independent networks: a generator ($G$) and a discriminator ($D$), which are both trained with unlabeled samples. Recently, Saha et al. adapt a GAN to solve multisensor CD, where the task is to detect changes across two images obtained by different sensors [4]. In this work, the generative power of GAN is only utilized to transcode images between two domains, and an external classifier is required to detect the changes between two transcoded images. Most of the above-mentioned methods exploit DL models with proven architectures that are pre-trained on large-scale computer vision (CV) datasets (e.g., ImageNet). However, this is not a fully suitable approach in RS, because of the differences in characteristics of CV and RS images.

Unlike the aforementioned deep generative approaches, in this paper we aim at learning an adversarial representation that not only can be learned in a self-supervised fashion but also can be directly used for CD without any need for an external classifier and a pre-trained network. To this end, we propose a self-supervised conditional Generative Adversarial Network (S²-cGAN). The S²-cGAN is trained using a training set containing only pairs of unchanged samples that are reconstructed (generated) from a single image without using any external supervision. During the adversarial training, $G$ learns how to generate unchanged data only, whereas $D$ learns how to detect deviations from it. Such $D$ can be directly used as a change classifier, considering changes as outliers with respect to the sample distribution learned on unchanged data. According to our knowledge, the proposed S²-cGAN is the first method that directly exploits the discriminator GAN to formulate the binary change detection problem.

Let $\boldsymbol{X}_{1}$ and $\boldsymbol{X}_{2}$ be two co-registered RS images acquired using a same sensor over the same geographical area at times $t_{1}$ and $t_{2}$, respectively. Both images are divided into non-overlapping $N$ patches and represented as $\boldsymbol{X}_{1}=\{x_{1}^{i}\}_{i=1,...,N}$ and $\boldsymbol{X}_{2}=\{x_{2}^{i}\}_{i=1,...,N}$, where $x_{1}^{i}$ and $x_{2}^{i}$ show the $i^{th}$ pair of patches associated to $t_{1}$ and $t_{2}$. In order to detect binary changes in $\boldsymbol{X}_{1}$ and $\boldsymbol{X}_{2}$, in this paper a self-supervised conditional Generative Adversarial Network (S²-cGAN) is proposed. The proposed S²-cGAN aims to train networks $G$ and $D$ using only pairs of patches without any land-cover change. Network $G$ learns how to generate only the unchanged pairs of patches. On the other hand, $D$ learns to distinguish pairs of unchanged samples (pixels) from those of changed samples. Fig. 1 summarize the general training strategy of the proposed S²-cGAN. The details are provided in the following sub-sections.

To evaluate the proposed S²-cGAN, in the experiments a bitemporal Very High Spatial Resolution (VHR) multispectral images acquired from Shenzhen, China by the Worldview 2 satellite has been used [8]. The image acquired in 2010 ($t_{1}$) has been considered as $X_{1}$, while the image acquired in 2015 ($t_{2}$) has been taken as $X_{2}$. Both $X_{2}$ and $X_{2}$ have size of $1431\times 1431$ pixels with a spatial resolution of 2 m. In all the experiments, the generator network $G$ was based on a U-net autoencoder for input patches of size $128\times 128$ pixels. The generator was compound by a network with six convolutional layers. For discriminator $D$, a pixel-based network with two convolutional layers was employed. The co-registered images $\boldsymbol{X}_{1}$ and $\boldsymbol{X}_{2}$ were divided into 7646 number of patches of size $128\times 128$ pixels. For the self-supervised adversarial learning of the unchanged sample representations, we construct $\boldsymbol{\tilde{X}}_{2}$ from $\boldsymbol{X}_{1}$ by applying a Gaussian noise over $\boldsymbol{X}_{1}$. This represents the unchanged (generated) image at time $t_{2}$. Half of the pairs of patches of $\boldsymbol{X}_{1}$ and $\boldsymbol{\tilde{X}}_{2}$ were randomly selected for training. The remaining patches of $\boldsymbol{X}_{1}$, together with their pairs in $\boldsymbol{X}_{2}$ were considered as pairs of test patches for the evaluation. The training was based on stochastic gradient descent with momentum 0.5, and the network was trained for 50 epochs. The two score maps $S_{f}$ (the differences map) and $e_{r}$ (the reconstruction error) were computed as explained in Sec. 2.2. Fig. 3 illustrates the score maps for two different pairs of patches. From the figure, one can observe that in most of the cases $e_{r}$ and $S_{f}$ provide complementary information. Similar behaviour has been observed by varying the pairs of patches. For quantitative evaluation, we applied a threshold value $\tau$ over the score map $\mathcal{H}$ to obtain the binary CD map. Due to the strong local variations in the considered VHR images, instead of choosing a single decision boundary for $\tau$ value, we adapted the context-dependent local adaptive decision boundary strategy used in [5]. We compared the CD map obtained by the proposed S²-cGAN with: 1) Fully Convolutional Early Fusion (FC-EF) method, which is one of the best performing supervised methods with a fully convolutional U-Net architecture [2]; and 2) Deep Change Vector Analysis (DCVA) method, which is a powerful unsupervised method using a CNN pre-trained for semantic segmentation to obtain multi-temporal deep features [5]. For the sake of fairness, the same pairs of test patches have been used for all the considered methods. Note that DCVA is fully unsupervised and does not use labeled samples. The fully-supervised FC-EF exploits a set of pairs of training patches that consists of: i) the same pairs of labeled unchanged samples used in the S²-cGAN; and ii) 13,065 pairs of changed samples. The binary CD maps obtained by the FC-EF, DCVA and the proposed S²-cGAN are shown in Fig. 4. From Fig. 4, one can see that the proposed S²-cGAN in most cases is able to identify the central area of the changed objects correctly. Table 1 shows quantitative results in terms of overall accuracy (OA), specificity (SPC), sensitivity (SEN), and overall error rate (ERR) obtained by the FC-EF, the DCVA, and the proposed S²-cGAN. By analysing the table, one can observe that the proposed S²-cGAN is comparable with the fully supervised FC-EF, and outperforms the unsupervised method DCVA mainly in terms of specificity (which shows the accuracy of detecting unchanged samples), sensitivity (which shows the accuracy of detecting changed pixels) and overall accuracy. As an example, sensitivity measure obtained by the proposed method is 4% and 14% higher than those obtained by the unsupervised DCVA and fully supervised FC-EF, respectively. It is worth emphasizing that our method performs better than both methods in terms of sensitivity despite the absence of changed samples during the training. This is due to the impact of adversarial training during GAN optimization.

In this paper, we have introduced a self-supervised conditional Generative Adversarial Network (S²-cGAN) for binary CD problems in RS. The proposed S²-cGAN exploits the mutual supervisory information of the generator and the discriminator networks to train a deep network by using a self-supervised multitemporal training set (which includes only pairs of unchanged samples). Differently from the existing GAN based CD methods, the proposed method directly uses the GAN discriminator as the classifier. Experimental results show that the proposed S²-cGAN leads to a higher performance in terms of sensitivity compared to the state of the art fully supervised and unsupervised methods. This has been achieved without using any pairs of labeled change samples. We underline that this is a very important advantage, since the proposed S²-cGAN suppresses the cost required for reference data collection. As a future work, we plan to extend the proposed learning approach to be integrated in an active learning setup with the human in the loop.

This work was supported by the European Research Council under the ERC Starting Grant BigEarth-759764.