Exploring High-quality Target Domain Information for Unsupervised Domain Adaptive Semantic Segmentation

Early unsupervised domain adaptive (UDA) semantic segmentation methods (Hoffman et al., 2018; Huang et al., 2020; Tsai et al., 2018; Wang et al., 2020; Yang and Soatto, 2020; Zhang and Wang, 2020), usually use adversarial training to align source and target domain images. However, these methods often suffer from instability in the training process (Chen et al., 2020c; Gulrajani et al., 2017). Different from adversarial-based methods, there are methods utilizing pseudo-labels (Tranheden et al., 2021; Melas-Kyriazi and Manrai, 2021; Zhou et al., 2021) and contrastive learning (Zhou et al., 2021; Xie et al., 2021) to improve the performance. These methods are more concise than methods based on adversarial training. In addition to the above technical routes, ProDA (Zhang et al., 2021b) surpasses the above methods by using distillation technology. Subsequent works which outperform ProDA methods (Zhang et al., 2021a; Wang et al., 2021a; Huang et al., 2022), generally employ distillation techniques and use much more sophisticated training techniques. For example, MFA (Zhang et al., 2021a) proposes multiple fusion adaptation to fuse three kinds of information and combine distillation technology to further improve the performance of ProDA. CRA (Wang et al., 2021a), based on ProDA, aligns the distribution of confident pixels and unconfident pixels in the target domain through an additional cross-region adaptation operation.

In this paper, we propose an efficient and non-distillation method that leverages well-designed pseudo-labeling techniques and contrastive learning techniques to reach or even surpass distillation-based methods.

The workflow of the pseudo-label methods can be roughly divided into the following two steps: first, generating pseudo-labels in the unlabeled data, and second, finetuning the model on these pseudo-labels. Early methods (Zou et al., 2018, 2019; Feng et al., 2020b, a) were usually iteratively performed in an offline manner. Therefore, these methods often require multiple training stages and are very cumbersome. Recently, some works (Tranheden et al., 2021; Melas-Kyriazi and Manrai, 2021; Zhou et al., 2021; Xu et al., 2019) have greatly simplified the training process through online pseudo-label techniques. For example, Pixmatch (Melas-Kyriazi and Manrai, 2021) uses random transformations to obtain a pair of strongly and weakly transformed images. Then the strongly transformed images can then enjoy the ”supervised-form” of the training process with the pseudo-labels produced by the weaker transformations. DACS (Tranheden et al., 2021) proposes to construct a mixed domain by copying image patches from the source domain to the target domain and learn the consistency between the student model and the teacher model under different contexts. In this way, it explicitly learns the cross-domain consistency of source boundaries. The DSP (Gao et al., 2021) additionally introduces a mix up strategy between source domain images on the basis of DACS to further improve the performance. BAPA (Liu et al., 2021) increases the loss weight of boundary pixels generated by DACS to improve classification accuracy of boundary pixels. The above methods use the mix up strategy of DACS to strengthen the learning of boundary samples, but DACS only explicitly generates source domain boundaries and does not explicitly construct target domain boundaries, resulting in insufficient learning of the boundary pixels of the target domain.

Different from these previous methods, we improve the performance of the model by explicitly constructing high-quality target domain boundary pixels.

The goal of contrastive learning is to learn a model that can extract semantically compact features from raw images by encouraging positive pairs to be close and pulling negative pairs. Many works (Chen et al., 2020a; He et al., 2020; Grill et al., 2020; Caron et al., 2020; Zhang et al., 2022; Li et al., 2021) have demonstrated the effectiveness of contrastive learning. In these works, an InfoNCE (Oord et al., 2018) loss is usually adopted to implement contrastive learning.

Some recent works introduce contrastive learning into UDA semantic segmentation. RCCR (Zhou et al., 2021) builds different contexts for a target domain region through cutmix between the target domain and the source domain. Then they use pixel-to-pixel contrastive learning to maximize the difference of inter-region pixels and minimize the inconsistency of intra-region pixels. SPCL (Xie et al., 2021) propose a novel semantic prototype-based contrastive learning framework to achieve pixel and prototype alignment.

Different from these methods, we design a multi-level contrastive learning loss, which both considers pixel-to-pixel and prototype-to-prototype contrastive learning losses, to explore more discriminative target domain feature representations.

In general, only the inner pixels of the target image can obtain the correct pseudo-labels, while it is difficult to obtain accurate labels for the boundary pixels of the target image. As a result, when training target domain images directly with pseudo-labels, only the inner pixels can be effectively trained with supervision, while the boundary pixels of the target image are difficult to be effectively trained. To alleviate this problem, we need to obtain target boundaries with correct labels, strengthening the learning of target boundary pixels.

In this section, we propose a simple yet effective high-confidence target domain pixel cross-domain mixing strategy (HTCM) to generate high-quality target domain boundary pixels. Generating high-quality target boundary pixels means constructing target pixels located in semantic mutation regions with correct pseudo-labels. In the pseudo-labels of the target domain pixels, the labels of high-confidence pixels tend to be correct, but these pixels are usually located inside objects rather than boundaries. We want to have these correctly labeled target domain pixels at boundary locations so that we can get the target domain boundary pixels with the correct labels. To do this, we paste these pixels into a source image so that some of the target domain pixels are at the border. Specifically, for a target domain image $x_{t}$, we can get its corresponding predicted probability map $p_{t}^{i}$. Then we get a high-confidence template $M_{ht}$, which is the selected indicator for the pixels whose predicted probability is greater than a threshold $\tau$. Finally, we use the obtained templates $M_{ht}$ to construct high-quality boundary samples as follows:

Similarly, the labels of the mix images can be obtained:

The segmentation loss of the mixed image $x_{hts}$ is defined as follows:

Due to the lack of explicit supervision signals in the target domain data, the features of the target domain are not sufficiently discriminative. Fortunately, contrastive learning can improve the distinguishability of unlabeled data (Chen et al., 2020a; He et al., 2020; Grill et al., 2020; Caron et al., 2020), so we employ contrastive learning to further improve our model. In particular, we design a multi-level contrastive learning loss to improve feature distinguishability from both prototype-level and pixel-level. Since we want the features to be invariant across domains, we choose to perform contrastive learning on the mixed images instead of directly on the target image. Because the prototype of the mixed image will contain the information of the source domain and the target domain, the features of the same category in the source domain and the target domain will be as consistent as possible during prototype-level contrastive learning. At the same time, because the target pixels in the mixed image $x_{hts}$ are all high-confidence samples, they have the correct pseudo-labels. It means that all the pixels of the mixed image $x_{hts}$ can be trained like the source domain image. At this time, using contrastive learning to improve the discriminativeness of features will not have much effect. Therefore, we choose to perform contrastive learning on mixed images $x_{st}$.

Specifically, for the mixed image $x_{st}$, we first obtain the transformed mixed images $x^{1}_{st}$ and $x^{2}_{st}$ through two different random transformations. Then, we use the feature extractor $E$ to obtain the feature maps $\mathbf{f}^{1}_{st}$, $\mathbf{f}^{2}_{st}$ corresponding to $x^{1}_{st}$ and $x^{2}_{st}$, whose resolution $H^{{}^{\prime}}\times W^{{}^{\prime}}$ is $\frac{1}{8}$ of the native resolution $H\times W$. In order to alleviate the interference of wrong samples on the prototype, we only use the features with high confidence to calculate the prototype. Therefore, we can get the prototype of each category as follows.

$M^{c}_{st}$ is a binary mask, $1$ indicates that the predicted category of this position is $c$ and the predicted probability is greater than $\tau$, 0 indicates that the predicted category is other categories or predicted probability is less than $\tau$. For a query sample $\mathbf{p}^{1,c}_{st}$, the prototype $\mathbf{p}^{2,c}_{st}$ is the positive and all other prototype are the negative. In domain adaptation tasks, it is not enough to improve the discriminativeness of features, and we also need to narrow the distance between features and classifier weights (Li et al., 2021). Therefore, following PCL (Li et al., 2021), we compute the contrastive loss using the probabilities instead of features and then the prototype-level contrastive loss (ProCL) is defined as:

In addition to prototype-level contrastive learning, we further consider pixel-level contrastive learning. For a query sample $\mathbf{f}^{1,i}_{st}$, the feature $\mathbf{f}^{2,i}_{st}$ at the corresponding position of another transformed image is taken as a positive sample, and all the other features are taken as negative samples. The pixel-level contrastive learning loss (PiCL) is defined as follows:

where $s_{1}$ and $s_{2}$ are the scaling factors, and we set $s_{1}$ to 7 and set $s_{2}$ to 20. $H(x,y)=\sigma(G(x))^{\top}\sigma(G(y))$ calculates the inner product of the classification probabilities corresponding to the two vectors and $\sigma$ represents the softmax function.

Our loss function is defined as:

Here $\mathbb{L}_{pro}=\sum_{c}(\ell_{\mathbf{p}^{1,c}_{st}}+\ell_{\mathbf{p}^{2,c}_{st}})$ and $\mathbb{L}_{pixel}=\sum_{i}(\ell_{\mathbf{f}^{1,i}_{st}}+\ell_{\mathbf{f}^{2,i}_{st}})$. $\lambda_{1}$ and $\lambda_{2}$ are the hyper-parameters to balance different losses.

We compare our method with the state-of-the-art UDA methods on two common UDA benchmarks. Depending on whether the distillation technique is used, we divide these methods into two broad categories. Table 1 and Table 2 give the results. From the results, we have the following observations.

First, our method outperforms all non-distillation methods, especially under the SYNTHIA$\rightarrow$Cityscapes setting, where our method outperforms the second-best method by 4.5$\%$ mIoU and 3.4$\%$ mIoU on 16 classes and 13 classes. Second, we found that ProDA, the first method using distillation technology, outperforms all previous non-distillation methods. However, our method not only outperforms ProDA, but also a series of subsequent methods based on distillation techniques. For example, our method is better than CRA (Wang et al., 2021a) by $0.2\%$ in GTA5$\rightarrow$Cityscapes and $0.9\%$ in SYNTHIA$\rightarrow$Cityscapes. Third, it should be noted that our method not only outperforms the distillation-based methods, but also simplifies the training process. In particular, our method does not require an additional two-step distillation process, nor does it require special training techniques.

In addition, we qualitatively compare our model with the source only model ($\mathbb{L}_{s}$) and baseline model ($\mathbb{L}_{s}$+$\mathbb{L}_{st}$) by showing the segmentation results in Figure 6. It can be seen that baseline significantly improves the segmentation results compared to the source only model. But for the thin objects such as ”bicycle” (yellow box) and ”pole”(green box), almost all pixels are boundary samples, resulting in an incorrect prediction by the baseline model. After using our proposed HTCM and multi-level contrastive loss, many of these mispredicted pixels are corrected back. This demonstrates the effectiveness of our method.

In this section, we conduct experiments to reveal the effectiveness of our proposed method. All experiments are particularly conducted on GTA5$\rightarrow$Cityscapes benchmark.