Continual Unsupervised Domain Adaptation for Semantic Segmentation

We start with an idea that the network needs additional capacity in real-world scenarios where the number of target domains keeps increasing. Thus, the added sub-memory needs to be stored and retrieved to reproduce its learned knowledge. There have already been precedents for expanding the network capacity in continual learning for the image classification task [36, 40, 26]. However, UDA for semantic segmentation differs from the image classification task in many aspects such as network structure, training objectives, and training strategies. Thus, adapting this idea to Continual UDA is not trivial.

We first consider the structure of the TM. Above all, each TM should contain domain discrepancy information between the source domain and each target domain. Unlike in the image classification task, the added sub-memory cannot be a simple MLP network. In semantic segmentation, the “accurate localization” information can be contained by using the small receptive field with the small dilation value, and “context assimilation” information can be contained by considering the large receptive field with the large dilation value [2]. From Fig. 1(c), we observe that forgetting occurs for various objects from sidewalk to person. This indicates that there exists the domain discrepancy not only for the localized information but also for the contextual information. To this end, we design two modules for each TM. One for the localized information, and the other for the contextual information.

Moreover, the TM should be lightweight memory for the efficiency of the overall continual learning scheme. Thus, we design the sub-memory structure as follows: As shown in Fig. 2(b), a $1\times 1$ Conv. module ($1\times 1$ convolution layer), which represents the module with the smallest receptive field. An Avg. Pool. module represents the module having the largest receptive field, which consists of an average pooling layer followed by a $1\times 1$ convolution layer and the ReLU nonlinearity. Since the trainable parameters only exist in the $1\times 1$ convolution layers, the TM is a fairly small-capacity network compared to the segmentation network.

The next step is for allowing each TM to contain each domain discrepancy information of the corresponding target domain during training. To do so, the knowledge, which is originally contained only in the segmentation network, should be divided up with the TM. Therefore, we let the TM be trained along with the segmentation network by attaching it to the network. In addition, to take advantage of high-level features, we expanded the TM in the top layer.

To be specific, when the input enters the TM it passes through the $1\times 1$ Conv. and Avg. Pool. modules, respectively, and then the outputs are summed. The output of the TM for arbitrary input $x$ is expressed as $TM(x)$ (Here, $TM$ indicates an arbitrary $i$-th TM, $TM_{i}$. We omit the subscription $i$ for simplicity.). $TM(x)$ and the hidden state value of the segmentation network are added, and then pass through the next layers. For example, the $(k-1)$-th hidden state value passes to the $k$-th layer and the TM, respectively, and then the added output passes through the $(k+1)$-th layer (see Fig. 2(a)). Let us express the output value as $f^{[:m]}(x)$ when the input passes through until the $m$-th layer of the segmentation network. Similarly, the output value when the input passes from the $(n+1)$-th layer of the segmentation network is expressed as $f^{[n:]}(x)$. Then, the added output right before the $(k+1)$-th layer is represented as follows:

In addition, the final output passed through both the segmentation network and the TM is defined as follows:

In our framework, the source and target domain data are aligned via adversarial learning. The layer where the TM is added is decided by the adversarial learning procedure of the backbone UDA method. It can be multiple positions if the backbone method uses various features for a discriminator input. If the TM is added at the $k$-th layer, its hidden state value is used as the input of the discriminator. In other words, $f^{[:k]+}(x)$ in Eq. (1) becomes the input of the discriminator.

Typically, the UDA methods perform the alignment of the source and target domain data through adversarial learning with the discriminator [33, 32, 35, 37] based on GAN [10]. Since the segmentation network in UDA can correspond to the generator of GAN, the adversarial learning scheme has been simply adapted to the UDA field. By minimizing the adversarial loss, the segmentation network is trained to align the source and target domain data, and at the same time, the discriminator is trained to distinguish them as the discriminator loss is minimized. There is a variation of GAN [18] which considers the geometric relations between feature vectors in the hyperplane by using the Support Vector Machine (SVM) [31]. In [18], the discriminator maximizes the minimum margin by value $1$ of the separable data in the feature space. This leads to stable training and solves the mode collapse problem in the image generation task.

However, the generator’s task in GAN and that of the segmentation network in UDA are different, while the discriminator’s role is in line with each other. Thus, we design a new adversarial loss that is customized for UDA. In the feature space, the generator tries to move the fake feature vectors toward the real feature space so that they can be classified as the real feature vectors. This is because the real data are the absolute truth in GAN. On the other hand, the target feature vectors in UDA do not have to imitate the source feature vectors blindly. Thus, the segmentation network can achieve its goal in two ways: by pulling the target features toward the source features or by pushing the source features toward the target features. Our suggestion is to use both ways. Here, we do not update the segmentation network when the source features are already closer to the target side than the target features and vice versa, by using the ReLU nonlinearity (see Eq. (4)).

To this end, we propose the DHA loss for the segmentation network. This loss is minimized when the target features are extracted as if the input is from the source domain, and vice versa. The DHA loss replaces the loss functions for adversarial learning used in the backbone UDA method. Let $\hat{z}^{s}$ and $\hat{z}^{t}$ be the output values when $x^{s}$ and $x^{t}$ pass through the discriminator after the segmentation network, respectively. Then, $\hat{z}^{s},\hat{z}^{t}\in\mathbb{R}^{h\times w}$, where $h$ and $w$ are height and width of each value, respectively. Then, our DHA loss is expressed as

where $(\cdot)_{+}=max(\,\cdot\,,0)$.

In short, we propose an adversarial learning procedure customized for the UDA task.

Even if the TM for the previous target domain remains frozen, however, the significant change in the segmentation network may aggravate the forgetting on the previous target domain. Thus, we add the distillation loss [17] that conducts knowledge distillation from the previous segmentation network:

with

where $T^{\prime}$ is the temperature which softens the weight distributions, and $y_{old}^{s}$ is obtained by passing the source image through the segmentation network before training on the current target domain.

Let $f(x)$ and $D(x)$ be the outputs from the segmentation network and the discriminator for input $x$, respectively. Then, the training objective of the proposed ETM framework is as follows:

with

where $L_{seg}$ is the cross-entropy loss. $\lambda_{seg}$, $\lambda_{adv}$ and $\lambda_{distill}$ are hyperparameters which are weights for each loss term. Once the segmentation network and the TM are updated by Eq. (7), the discriminator is updated by Eq. (8), alternately. Here, we assume that the source domain data are accessible in every stage since the purpose of UDA is to perform semantic segmentation for the target domains.

Algorithm 1 summarizes the learning procedure of our framework. Here, $\phi_{f}$, $\phi_{TM_{i}}$, and $\phi_{D_{i}}$ indicate the parameters of the segmentation network, the $i$-th TM, and the $i$-th discriminator, respectively. In short, by applying the proposed ETM framework to the existing UDA methods, the segmentation network and all TMs learn knowledge about all the sequential target domains while effectively keeping the knowledge about the previous target domains.

We used two synthetic road image datasets (GTA5, SYNTHIA) as the source domains and three real-world road image datasets (CityScapes, IDD, Cross-City) as the target domains. We used three types of baseline models: (1) Source Only, which indicates the model trained on the source domain data and tested on the target domains, (2) the UDA methods containing FCN-W [14], AdaptSegNet [32], AdvEnt [35], SIM [37], and (3) the Continual UDA method, ACE [38]. For a fair comparison, DeepLab-V2 [2], which is one of the representative semantic segmentation networks, was used as the segmentation network in all methods. For Ours, we applied the ETM framework on AdaptSegNet [32]. The evaluation metric is mIoU (mean Intersection over Union), i.e., the average of the IoU values for each object. We implemented our experiments using Pytorch of version 1.4.0 on a Ubuntu 16.04 workstation. Single RTX 2080ti GPU was used with CUDA version 10.0.

To validate the effectiveness of the proposed ETM framework, we compared the model trained with our framework with the state-of-the-art methods. In our experiments, two Continual UDA scenarios were considered: a two-target scenario and a four-target scenario. Following the prior works on UDA [14, 32, 35, 44], the synthetic datasets, the GTA5 and SYNTHIA datasets, were used as the source domains. In the first experiment, we first performed UDA to the CityScapes dataset, followed by Continual UDA to the IDD dataset. The CityScapes dataset is the most widely used real-world image dataset in the UDA field. To maximize the catastrophic forgetting problem for the CityScapes dataset collected in European cities, we then performed UDA to the IDD dataset collected in entirely different Indian cities. In the second experiment, we performed Continual UDA sequentially for four different cities in the Cross-City dataset (Rio, Rome, Taipei, and Tokyo). Through a longer target domain sequence, we analyzed the catastrophic forgetting problem in depth.

We experimented with two different input sizes. If the original images were used as the inputs by converting into high-definition images of 1024 $\times$ 512, we specified them as H, and L if the images were converted into low-definition images of 512 $\times$ 256. We conducted the experiments using both input sizes for all methods except ACE [38]. For the ACE, experiments on high-quality images (H) were not possible due to a problem with the network capacity.

We conducted in-depth analyses of our framework internally by checking how the TM and the DHA loss affect the performance of the framework. All experiments in this section were conducted in an environment performing Continual UDA on the CityScapes and IDD datasets, with an input size of H.

First of all, we analyzed the effectiveness of each component of our framework. In Table 3, the first row indicates our model without both the TM and the DHA loss since we applied our framework to the AdaptSegNet. When comparing the results in the second row and the first row, the forgetting is alleviated as well as the mIoU values for the CityScapes dataset are increased. Before adapting to the IDD dataset, the performance on the CityScapes dataset is similar to each other in both cases. That is, adding TM results in mitigation of the catastrophic forgetting problem. Moreover, when comparing the third row to the second row, the overall UDA performance is improved. It means that the UDA performance is enhanced due to the DHA loss.

Moreover, we further analyzed how each module of the TM affects the overall performance. The differences in performance depending on the presence or absence of each module are summarized in Table 4. For a fair comparison, we set all the other conditions same except the TM architecture. To verify the effect of each module clearly, the DHA loss was not used. Firstly, note that even using one module as the TM leads to alleviating catastrophic forgetting. Also, the catastrophic forgetting problem is more mitigated when the $1\times 1$ Conv. module is used compared to when the Avg. Pool. module is used. This indicates that localized information extracted by the $1\times 1$ Conv. module contributes more than contextual information extracted by the Avg. Pool. module for the domain discrepancy. Finally, when both modules are used, the least forgetting occurs. From this, it can be claimed that there exists a synergy between the two modules, and the validity of the TM’s design is demonstrated.

Previously, we claimed that the TM stores domain discrepancy information of each target domain by attaching to the segmentation network and being trained together; however, it is vague that exactly which information is stored in the TM. To clarify this, we carried out semantic segmentation using the network with the ETM framework. By comparing the results of semantic segmentation with the segmentation network only and with the integration of the segmentation network and the TM, it would be revealed which information is contained in the TM.

Fig. 3 shows the semantic maps predicted by the network when Continual UDA is performed to the CityScapes dataset and the IDD dataset, respectively, with the GTA5 dataset as the source domain. For each of the four images extracted from the CityScapes and IDD dataset, the ground truth of each image and the semantic maps predicted with the TM (w/ TM), and without the TM (w/o TM) are compared with each other. Fig. 3(a) is an example for the CityScapes dataset. In the first image of 3(a), a traffic sign (expressed in yellow) and a person (expressed in red) riding bicycles (expressed in brown) are well shown inside the sky-blue circle when the TM is used, but disappeared when the TM is not used. Likewise, in other images, people, traffic lights (expressed in orange), and bicycles are properly predicted with the TM, but disappeared without the TM. From Fig. 3(b) for the IDD dataset, we can verify that people and motorcycles (expressed in blue) are entirely missed when without the TM.

In summary, the TM contains information about all objects in general, but especially more about the types of objects that are likely to differ from one instance to another. As stated in Section 2.1, the $1\times 1$ Conv. module of the TM is added to extract more localized information and the Avg. Pool. module is added for broader contextual information. Through the experiment, it can be claimed that the $1\times 1$ Conv. module plays a more vital role and such specific information extracted by the module contributes more in overcoming the domain discrepancy.