Domain Adaptive Semantic Segmentation without Source Data

Unsupervised domain adaptation (UDA) has witnessed significant development in both theory and various applications recently (Ben-David et al., 2010; Zhao et al., 2019; Luo et al., 2019b; Li et al., 2019b, 2018). Domain adaptation aims at learning a general model from both labeled source domain and unlabeled target domain, where different domain has distinctive distributions (Long et al., 2015; Sun and Saenko, 2016).

By considering the privacy and data transmission, source-free domain adaptation (SFDA) attracts a lot of attention lately (Liang et al., 2020; Kim et al., 2020). SFDA divides the training process into two steps: training the model on labeled source domain, and then adapt to the unlabeled target domain. Although SFDA is proposed very recently, it has been studied in many fields. For object recognition, 3C-GAN (Li et al., 2020c) generated target-style samples and leverages other regularizations to moderately retrain the source model. Hou et al. (Hou and Zheng, 2020) converted the target-style images to the source-style based on the mean and variance stored in BN layers of the source model. SFDA (Kim et al., 2020) proposed to use the samples with low entropy to refine the pseudo labels for training. SHOT (Liang et al., 2020) proposed two strategies: information maximum loss and weighed deep clustering, and achieved state-of-the-art performance. For object detection, SFOD (Li et al., 2020b) proposed a metric named self-entropy decent to select the appropriate threshold for pseudo label generation.

To the best of our knowledge, we are at the very early attempt to investigate SFDA in semantic segmentation. In Sec. 4.4, we compare our method with other state-of-the-art SFDA methods in detail.

Domain Adaptive semantic segmentation (DASS) is an important application of UDA. Existing DASS methods can be roughly divided into two categories: adversarial training and self-training.

Adversarial training mainly includes two strategies: (1) Employing a style transfer across domains through adversarial training (Chang et al., 2019; Zhang et al., 2018). (2) Learning the domain-invariant feature representations across domains (Tsai et al., 2018; Vu et al., 2019; Tsai et al., 2019; Luo et al., 2019b; Saito et al., 2018). For example, Vu et al. (Vu et al., 2019) match the entropy of output predictions in source and target domains via adversarial training. Recently, Luo et al. (Luo et al., 2021) proposed two modules to purify the features and perform the category-wise adversarial training based on it. However, these methods cannot be implemented directly in SFSS setting since the source data are unknown during training. Furthermore, adversarial based methods usually take more training time, while our method provides an efficient cross-domain semantic segmentation model (see Sec. 5.4).

Another mainstream line of work for DASS leverages the self-training. Some methods (Chen et al., 2019b; Zou et al., 2019; Vu et al., 2019) utilizes prediction entropy to guarantee a distinct decision boundary in target domain. Recently, many methods product pseudo-labels in target domain based on confidence or uncertainty estimation (Iqbal and Ali, 2020; Lian et al., 2019; Pan et al., 2020; Sakaridis et al., 2019; Zheng and Yang, 2021; Zou et al., 2018; Li et al., 2020d; Zhang et al., 2019). For example, Zou et al. (Zou et al., 2018) proposed a class-balanced selection strategy and selected the high-confidence pseudo-labeled target samples, while Zheng et al. (Zheng and Yang, 2021) estimated the uncertainty via an auxiliary classifier and selected the rectified pseudo labels through the dynamic threshold. Unlike the aforementioned methods, Li et al. (Li et al., 2020d) proposed to select the source images that share similar distribution with the real ones in the target domain to alleviate the domain gap in another perspective.

The most related work is CBST (Zou et al., 2018), which proposed class-balanced pseudo-labeling. In our proposed label-denoising, intra-class threshold selection is also adopted. However, we have significant different motivations and claimed contributions. In SFSS, without any supervision from source domain, the class-imbalance issue becomes more severe than DASS. Once the category is a majority, it will be gradually replaced and totally dominated if there is no external intervention. In DASS, the supervision from source domain data preserve the minority classes, thus alleviating this issue. We summarize this phenomenon as “winner-takes-all”. Therefore, based on our original findings on the challenges of SFSS, class-balanced selection is a simple yet significant way to address it and it is not a direct re-use. Besides, we also propose negative learning to address other challenges.

In domain adaptive semantic segmentation, we have a source domain $\mathcal{X}_{S}\subset\mathbb{R}^{H\times W\times 3}$ with corresponding ground-truth segmentation maps $\mathcal{Y}_{S}\subset\{(0,1)\}^{H\times W\times C}$, where $H,W,C$ denote the height, width and the number of classes, respectively. The target domain $\mathcal{X}_{T}\subset\mathbb{R}^{H\times W\times 3}$ is unlabeled. Let $S$ be a segmentation network with parameters $\theta$ , which takes an image $x$ and gives a prediction $p^{(h,w,c)}$. Especially, in SFSS, the training procedure has two stages: training a supervised loss on the labeled source domain and adapting to the unlabeled target domain. For simplicity, we optimize the cross-entropy loss over the source domain:

However, directly applying the model trained on source domain to the target domain often results in significant performance degradation (Ben-David et al., 2010; Li et al., 2020d). Therefore, we present our method and show how to enhance the performance on target domain without accessing source data.

Pseudo labeling is a widely used strategy in semi-supervised learning, and attracts great attention in DASS recently. Intuitively, obtaining the pseudo labels through applying argmax operation on the softmax predictions is unreliable due to the domain gap between source and target domains. A trivial solution is to select the high-confidence pseudo labels, which effectively removes the wrong labels but neglects the “winner-takes-all” dilemma, where the model would be bias towards the majority classes and ignore the minority classes (see Fig. 2 (a)). Therefore, we propose to select the pixels with intra-class confidence. To avoid the imbalanced selection, the intra-class threshold is defined as:

where $\top_{\alpha}$ means the top $\alpha$ (%) value in the softmax prediction value set $\mathcal{P}_{T}^{(C)}$ with respect to class $c\in C$. Then, for each class $c\in C$, we select the samples whose softmax confidence is larger than $\delta^{(c)}$. It is worth noting that our selection strategy addresses the aforementioned two challenges: (1) The most noise labels are filtered due to the high softmax prediction confidence selection. (2) The intra-class confidence selection addresses the dilemma where selecting high-confidence pseudo labels always results in the biased segmentation network. We update the pseudo labels at the beginning of each epoch to avoid additional running time. After selection, we optimize following cross-entropy loss at the selected pixels:

where $(\mathcal{X}^{\prime}_{T},\mathcal{\hat{Y}}^{\prime}_{T})$ are the selected pixels on target domain and the pseudo labels are obtained by the argmax operation based on softmax prediction. Furthermore, we optimize the pixels not be selected by entropy minimization. Entropy minimization has been shown to be effective in semi-supervised learning (Grandvalet et al., 2005; Springenberg, 2015) and domain adaptation (Vu et al., 2019; You et al., 2021). Vu et al. (Vu et al., 2019) argues that entropy minimization can be recognized as a soft-assignment version of cross-entropy loss. Therefore, to balance the unselected pixels, we optimize the following entropy minimization loss:

where $\mathcal{P}_{T}$ is the set of softmax prediction over each pixels on target domain images. Now, we summarize the formulation from positive learning perspective:

In order to avoid the incorrect pseudo labels, we propose to address SFSS from another perspective, i.e., negative learning. Our idea is easy to be understood: instead of determining which category a pixel belongs to, it is easier to infer which category it does not belong to, which is more feasible in SFSS where no supervised information is provided. Let $x\in\mathcal{X}_{T}$ be an input, and $\hat{y}\in\mathcal{\hat{Y}}_{T}$ be its corresponding pseudo label. We generate the complementary label $\overline{y}\in\mathcal{\overline{Y}}_{T}$($\overline{y}\neq\hat{y}$), and optimize following loss function:

To the best of our knowledge, we are the first to perform negative learning in pixel-level. In previous work, Kim et al (Kim et al., 2019) perform negative learning in classification task and the complementary label is generated randomly from the labels of all classes except for the pseudo label $\hat{y}$. However, we further raise following question: which soft-label deserves to be optimized? To answer this question, we illustrate a toy example in Fig. 2 (b). Firstly, if we sort the softmax output $\mathcal{P}_{T}$, the class having the max softmax value is selected as the pseudo label. However, in SFSS, there are lots of noises in the pseudo labels due to the domain gap. Therefore, intuitively, the “head” classes (except the pseudo label) have a high probability of being the correct label and performing negative learning on these complementary labels causes an accumulation of errors. And the loss on the complementary labels inside the “tail” classes converges to 0. Therefore, we propose a heuristic complementary label selection (HCLS) to select the more accurate and valuable complementary labels, i.e., the “body” classes, which avoids the risks mentioned above. The pseudo code of HCLS in presented in Algorithm 1.

Actually, performing positive learning or negative learning individually improves the performance of the source pre-trained model on target domain (see Table 1, 2). Furthermore, incorporating them achieves better performance. On one hand, positive learning provides the class-balanced pseudo labels and helps correcting the complementary labels, thus improving the negative learning. On the other hand, negative learning denoises the pseudo labels, preventing the accumulation of errors in pseudo-labeling. Therefore, these two components are not mutually exclusive or independent, but reinforcing. The overall objective function for segmentation networks $S$ becomes:

where $\lambda_{ent}$ and $\lambda_{neg}$ are the hyper-parameters. And it is worth noting that the $\mathcal{L}_{sce}$ is calculated on the selected pixels, and the selected pixels are updated at the beginning of each epoch out of saving training time. And $\mathcal{L}_{ent}$ and $\mathcal{L}_{neg}$ are calculated on all pixels since both of them not explicitly provide supervision and can balance the unselected pixels. The details of training process are described in Algorithm 2.

GTA5 is a synthetic dataset, which contains 24966 high-resolution images collected from game video and the corresponding ground-truth segmentation map can be generated by computer graphics. We train the 19 common classes with CityScapes dataset.

SYNTHIA is also a synthetic dataset, which contains 9400 images. And it shares 16 common classes with Cityscapes dataset.

Cityscapes is a real-world dataset collected for autonomous driving scenario from 50 cities around the world. It contains 2975 and 500 images for training and validation, respectively. In SFSS, the model is trained on the unlabeled training set and evaluated on the validation set with manual annotations.

We keep the same network architecture as in previous methods for fair comparison. Specifically, we use Deeplab-V2 (Chen et al., 2017) with pre-trained ResNet-101 (He et al., 2016) as the segmentation network $S$ and the Atrous Spatial Pyramid Pooling (ASPP) module is applied on the last layer’s output. Similar to  (Chen et al., 2017), the sampling rates are fixed as {6,12,18,24}. Then, an up-sampling layer and a softmax operator are applied to obtain the prediction (segmentation map) with the matched size of input image. We implement our methods with Pytorch on a single NVIDIA RTX 2080Ti GPU. The segmentation network $S$ is trained using the Stochastic Gradient Descent optimizer with momentum 0.9 and weight decay $5\times 10^{-4}$. We adopt the polynomial learning rate scheduling with the power of 0.9, the initial learning rate is set as $1\times 10^{-3}$ and the batch size is 3. We adopt the same data augmentations as (Li et al., 2020d): we resize the images’ short side to 720 and crop the images into 600$\times$600 randomly. And horizontal flip and random scale between 0.5 and 1.5 are performed. The evaluation metrics is the mean intersection-over-union (mIoU).

To evaluate the proposed LD, we compare it with some widely-used self-training methods in domain adaptive semantic segmentation (Vu et al., 2019) and the state-of-the-arts SFDA method SHOT (Liang et al., 2020). Here, we shortly introduce the compared methods and our implementation since some methods cannot be directly applied to source-free domain adaptive semantic segmentation. EntMin. We minimize the entropy of softmax output (Vu et al., 2019). Pseudo. We directly obtain the pseudo label with argmax operation. Then, we minimize the cross-entropy loss on each pixel (Lee et al., 2013) . Pse.+Ent. We combine the aforementioned two methods, and the trade-off hyper-parameter is set as 1. Pse.+Sel. Since directly applying the pseudo label leads to many noisy predictions, many methods proposed various selection strategies (Zheng and Yang, 2021; Zou et al., 2019, 2018). A recent work (Li et al., 2020b), which investigates the SFDA in object detection, proposed to generate the pseudo labels through prediction confidence. In our implementation, we empirically fix the confidence threshold as 0.9, and minimize the cross-entropy loss on the selected pixels. SHOT. SHOT contains information maximization and deep clustering. However, the latter one cannot be applied to the semantic segmentation scenario directly. For instance, in semantic segmentation, a 600$\times$600 image contains 360000 pixels with 2048-dim (ResNet) feature representations and a dataset contains large amounts of images. Therefore, performing clustering on the dataset in semantic segmentation is impractical and we implement information maximization.

We verify the effectiveness of our method in two synthetic-to-real benchmarks: GTA5 $\rightarrow$ Cityscapes and SYNTHIA$\rightarrow$Cityscapes. To ensure a fair comparison, all reported results share the same backbone: DeepLab-v2 with pre-trained ResNet-101. Since existing DASS methods cannot be implemented directly in the source-free settings, we then re-implement some widely-used self-training approaches in DASS (Vu et al., 2019) and the state-of-the-art SFDA method SHOT (Liang et al., 2020). The details of implementation are elaborated in Sec. 4.3. The results on GTA5$\rightarrow$Cityscapes and SYNTHIA$\rightarrow$Cityscapes are shown in Table 1 and Table 2, respectively. From Table 1 and 2, we have following findings:

1. 1)

   Entropy minimization and direct pseudo-labeling leads to “winner-takes-all” under SFSS setting. Compare to Source-only, it has $9.7\%$, $12.3\%$ drop in GTA5$\rightarrow$Cityscapes, and $4.5\%$, $8.8\%$ drop in SYNTHIA$\rightarrow$Cityscapes. Both in DASS and SFSS, the model is biased towards the majority classes. However, in DASS, the labeled source domain data preserve the minority classes, while in SFSS, once the class is at a disadvantage, it will be gradually replaced if there is no external intervention. We illustrate the results of SYNTHIA$\rightarrow$Cityscapes in Fig. 2, which vividly demonstrates our claims.

2. 2)

   Selecting the pseudo-labeled samples based on the confidence threshold (Pse.+Sel.) effectively removes the wrongly pseudo-labeled samples, but still suffers from the “winner-takes-all”. Pse.+Sel. gets $-3.2\%$ in GTA5$\rightarrow$Cityscapes, and $+0.3\%$ in SYNTHIA. However, in the minority classes (e.g., fence, pole, light, sign and etc.), Pse.+Sel. has a clear drop compared to “Source only”, revealing that the selected pixels are extremely class-imbalanced and the segmentation networks suffer from the “winner-tasks-all” dilemma. Existing state-of-the-art SFDA approach SHOT also gets poor performance in semantic segmentation scenario (i.e., +2.8% mIoU in GTA5$\rightarrow$Cityscapes and -4.8% mIoU in SYNTHIA$\rightarrow$Cityscapes).

3. 3)

   Our approach LD surpasses the strong baseline “Source only” by a large margin under the challenging SFSS setting. We achieve a promising mIoU of 45.5% in GTA5$\rightarrow$Cityscapes (+9.8%) and mIoU of 50.1% in SYNTHIA$\rightarrow$Cityscapes (+12.4%), which highlights the effectiveness of our proposed LD. Compared to other methods under SFSS setting, our method maintains (e.g., wall, fence, pole, light, sign and etc.) or upgrades (e.g., side., rider, truck, bus and etc.) the performance of minority classes and consistently improves the performance of majority classes.

4. 4)

   Our LD even achieves competitive performance compare to the DASS methods, which access the labeled source domain data during adapting to the target domain. Our method outperforms the recent state-of-the-art adversarial based methods. For example, we surpass the representative adversarial-based work CLAN with a 1.3% mIoU in SYNTHIA$\rightarrow$Cityscapes. Compared to state-of-the-art self-training based methods, our performance is also competitive. Notably, in SYNTHIA$\rightarrow$Cityscapes, we achieves the best or the second performance at classes “bus”, “vege.”, “motor” and “bike”, and all of them belong to the minority classes except for “vege.”. Furthermore, our method only applies two self-training components, while many DASS methods combine many methods and design complex network architectures or sampling strategies (e.g., DPR, IntraDA and DAST). i.e., incorporating with other methods (e.g., attention module) will further enhance the performance of LD, which revealing the simplicity and expandability of the proposed framework.

Hyper-parameters $\lambda_{ent}$, $\lambda_{neg}$. The overall objective function Eq. (7) contains two trade-off hyper-parameters $\lambda_{ent}$ and $\lambda_{neg}$. We fix $\lambda_{ent}$ as 1 and $\lambda_{neg}$ as 1 for all tasks, respectively. Furthermore, we conduct parameter sensitivity analysis to evaluate LD on tasks GTA5$\rightarrow$Cityscapes and SYNTHIA$\rightarrow$Cityscapes. As shown in Fig. 3 (a) and (b), the performance steadily improves with the increasing parameters $\lambda_{ent}$ and $\lambda_{neg}$ from 0 to 1, demonstrating the effectiveness of each components in LD. And we observe that the performance would not be greatly influenced by the value of $\lambda_{ent}$ and $\lambda_{neg}$, indicating that our LD is not quite sensitive to $\lambda_{ent}$ and $\lambda_{neg}$.

Positive learning: selection range $\alpha$. In Eq. (2), the parameter $\alpha$ controls the selection range of pseudo labels and we fix $\alpha$ as 0.2 for all tasks. To fully investigate the influence of different value of $\alpha$, we select the balance weights from 0.1 to 0.9, since $\alpha=0$ indicates applying negative learning only and $\alpha=1$ indicates applying pseudo-labeling directly. From Fig. 3 (c), we find that the performance is not affected much by different values of $\alpha$ in GTA5$\rightarrow$Cityscapes. In the more challenging task SYNTHIA$\rightarrow$Cityscapes, the performance begins to drop after $\alpha>0.3$ since it contains more and more wrong labels. It is worth noting that even with a large value $\alpha$, the performance of LD is also acceptable, indicating the negative learning is effective in learning with noisy labels.

Negative learning: selection range $\epsilon$. In Algorithm 1, $\epsilon$ represents the selection width of complementary label and we fix $\epsilon=3$ for all tasks. To better reflect the influence of different value of $\epsilon$, we apply negative learning only and modify the Eq. (1) to $K=\lfloor C/2\rfloor$ + rand ($-\epsilon,0$). In GTA5$\rightarrow$Cityscapes, the number of classes $C=19$, and $\lfloor C/2\rfloor=9$, while in SYNTHIA$\rightarrow$Cityscapes, $C=16$ and $K=\lfloor C/2\rfloor=8$. Therefore, we select the balance weights from 0 to 6, since $\epsilon=7$ indicates that the complementary label may have the Top 1 value of the softmax output in SYNTHIA$\rightarrow$Cityscapes. As shown in Fig. 3 (d), we observe that the performance begins to drop when $\epsilon>=4$ in GTA5$\rightarrow$Cityscapes, which is consistent with our claims in Sec. 3.3 and Fig. 2: generating the complementary labels from the “head” classes results in the degeneration of model. Specifically, the empirical results indicate that our method outperforms the mainstream complementary strategy (Ishida et al., 2017; Kim et al., 2019), who generating the complementary label randomly from the labels of all classes except for the pseudo label, since our method generates more accurate and reliable complementary labels to perform negative learning.

For the sake of examining the key components of LD, we perform ablation study by removing each component from LD at a time. In LD, we have two simple components: positive learning and negative learning. Therefore, we obtain two variants: (1) “LD w/o $\mathcal{L}_{pos}$” and (2) “LD w/o $\mathcal{L}_{neg}$”, which denote removing the corresponding self-training module, respectively. The results of ablation study are shown in Table 1 and 2. It is obvious that LD outperforms other variants and achieves significant improvement under source-free setting. “LD w/o $\mathcal{L}_{pos}$” achieves the mIoU of 41.3 in GTA5$\rightarrow$Cityscapes and 43.7 in SYNTHIA$\rightarrow$Cityscapes. “LD w/o $\mathcal{L}_{pos}$” does not explicitly provide any supervision but tells the model “which category does this pixel not belong to”. “LD w/o $\mathcal{L}_{neg}$” achieves the mIoU of 43.0 in GTA5$\rightarrow$Cityscapes and 47.6 in SYNTHIA$\rightarrow$Cityscapes, revealing that our pseudo-labeling effectively alleviate the “winner-takes-all” and enhance the performance of model in source-free setting.

We illustrate some qualitative segmentation examples in Fig. 4. Obviously, “Source only” model gives confused predictions due to the domain gap between GTA5 and Cityscapes. And many self-training methods performs well on the majority classes (e.g., “road”, “car”), but tends to over-fit the majority classes, e.g., “road” tends to occupy the “sidewalk”. Furthermore, they totally ignore the minority classes. For instance, the “bike” class is totally disappeared and replaced with “road” class, which creates a potential risk for real-world applications like automatic pilot. Unlike them, the proposed LD significantly enhances the performance of semantic segmentation. Specifically, LD stably improves the performance of majority classes, making the segmentation cleaner compared to “Source only” model. Furthermore, from the key areas highlighted with white boxes, our method yields better scalability to the minority classes, like confused classes (e.g., “bike”) and small-scale objectives (e.g., “traffic sign”). The qualitative results vividly confirms our claims and demonstrates the superiority of our method.

Without accessing source data, our method shows superiority on privacy-preserving, data transmission and training time. In this section, we directly measure the size of training data and the training time of our proposed LD and other methods on task GTA5$\rightarrow$Cityscapes and SYNTHIA$\rightarrow$Cityscapes. We run the experiments on the same running environment (same hardware, same OS, same background applications, etc.). The results are reported in Table 3. In Table 3, “Training data” indicates the size of data when adapting to the target domain. “Training time” represents the time taken by the model to reach the reported mIoU. “Speedup” is calculated by: (The training time of the compared method - The training time of LD)/(The training time of LD). From the results, we can find that our method only takes 6.9GB training data during adapting to the target domain and has a significant speedup compared to the source-accessible methods. Meanwhile, our performance is also competitive to these methods, and even better than CLAN and IntraDA on SYNTHIA$\rightarrow$Cityscapes. Therefore, our method achieves efficient domain adaptation.