ML-BPM: Multi-teacher Learning with Bidirectional Photometric Mixing for Open Compound Domain Adaptation in Semantic Segmentation

Our work assumes that the domain-specific property of images comes from their styles. Existing works adopt a predefined parameter to decide the number of subdomains, which might lead to a nonoptimal domain adaptation performance; furthermore, they rely on a pre-trained CNN-based encoder to extract the style information for the subdomain discovery. However, we propose an automatic domain separation (ADS) to effectively separate the target domain using the distribution of pixel values of the target images. The proposed ADS is capable of predicting the optimal number of subdomains without relying on any predefined parameters and extracting the image style information without relying on any pre-trained CNN models. We denote the source domain as $\mathcal{S}$, and the unlabeled compound target domain as $\mathcal{T}$. We also assume compound target domain contains $k$ latent subdomains: $\{T_{1},\dots,T_{k}\}$, which lack of clear prior knowledge to distinguish themselves. The goal of ADS is to find the optimal number of subdomains $k^{*}$ and separate $\mathcal{T}$ into several subdomains accordingly.

Current work [14] suggests a simple yet effective style translation method by matching the distribution of pixel values on LAB color space. Thus, we adopt LAB space into ADS to extract the style information of the target image. Given a target RGB image $x_{t}\in\mathcal{T}$ as input, we convert it into LAB color space $rgb2lab(x_{t})$. The three channels in LAB color space are represented as $l,a$, and $b$. Then, we compute the histograms of the pixel values for all three channels in LAB color space: $H^{l}(x_{t}),H^{a}(x_{t})$, and $H^{b}(x_{t})$. The histograms are concatenated and represented as the style information of $x_{t}$. Let $s(x_{t})={H^{l}(x_{t})}^{\frown}{H^{a}(x_{t})}^{\frown}{H^{b}(x_{t})}$ denote the concatenated histograms of $x_{t}$, and we take $s(x_{t})$ as input to ADS for domain separation. However, most existing clustering algorithms require a hyperparameter to determine the number of clusters. Directly applying a naive clustering might lead to a nonoptimal adaptation performance. Thus, we propose to find the optimal number $k^{*}$ of the subdomains using Silhouette Coefficient (SC) [23]. Suppose the target domain $\mathcal{T}$ is separated into $k$ subdomains, $\{T_{1},\dots,T_{k}\}$. For each target image $x_{t}$, we denote $\gamma(x_{t})$ as the average distance between $x_{t}$ and all other target images in the target subdomain to which $x_{t}$ belongs. Additionally, we use $\delta(x_{t})$ to represent the minimum average distance from $x_{t}$ to all other target subdomains to which $x_{t}$ does not belong. Let us assume $x_{t}$ belongs to the $m\textsuperscript{th}$ target subdomain $T_{m}$, then $\gamma(x_{t})$ and $\delta(x_{t})$ are written as

where $L(s({x_{t}}^{\prime}),s(x_{t}))$ represents the euclidean distance of $s({x_{t}}^{\prime})$ and $s(x_{t})$, and $|T_{m}|$ is the number of the target images in $T_{m}$. The SC score for $k$ number of the target subdomains is given by

Hence, the goal of the proposed ADS is to find $k^{*}$ for

With the help fo automatic domain separation, the number of abnormal samples with different styles is small inside each target subdomain. Though these abnormal samples might be useful for the model’s generalization, they could also lead to a negative transfer, which further hinders the model from learning domain invariant features in a specific subdomain. To cope with it, we propose to purify the style distribution of the target images inside each subdomain. We design a subdomain style purification (SSP) module to effectively make similar styles for the images within the same subdomain. Given the $m\textsuperscript{th}$ target subdomain $T_{m}$, we adopt the histograms of LAB color space $\{(H^{l}(x_{t}),H^{a}(x_{t}),H^{b}(x_{t}));\forall x_{t}\in T_{m}\}$ (mentioned in 3.1), and then we compute the mean of the histograms for all the three channels, represented by $\widetilde{H}_{m}^{l},\widetilde{H}_{m}^{a}$, and $\widetilde{H}_{m}^{b}$, and this process is achieved by

where $|T_{m}|$ represents the number of the target images in $T_{m}$. We take $\{\widetilde{H}_{m}^{l},\widetilde{H}_{m}^{a},\widetilde{H}_{m}^{b}\}$ as the standard style for $T_{m}$. For each target RGB image $x_{t}\in T_{m}$, we change the style of $x_{t}$ to generate the RGB new image $\tilde{x}_{t}$ by the histogram matching [20] on $\widetilde{H}_{m}^{l},\widetilde{H}_{m}^{a}$, and $\widetilde{H}_{m}^{b}$ on the LAB color space. The process of SSP is done for all the subdomains $\{T_{1},\dots,T_{k^{*}}\}$. We denote the purified subdomains after SSP as $\{\widetilde{T}_{1},\dots,\widetilde{T}_{k^{*}}\}$.

Through automatic domain separation and subdomain style purification (mentioned in 3.1 and 3.2), the compound domain $\mathcal{T}$ is automatically separated into multiple subdomains $\{\widetilde{T}_{1},\dots,\widetilde{T}_{k^{*}}\}$, where $k^{*}$ represents the optimal number of the subdomains. Our next plan is to minimize the domain gap between the source domain and each target subdomain. A recent UDA work DACS [25] presents a mixing-based UDA technique for semantic segmentation. Inspired by DACS, we propose bidirectional photometric mixing (BPM) to minimize the domain gap between the source domain and each target subdomain separately. Compared with DACS, the proposed BPM adopts a photometric transform to decrease the style inconsistency of the mixed images to reduce the pixel-level domain gap. On this basis, BPM applies a bidirectional mixing scheme to provide a more robust regularization for training. The architecture of BPM is shown in Figure 2(a). The proposed BPM contains a domain adaptive segmentation network $G_{m}$ and a momentum network $M_{m}$ that improves the stability of pseudo labels. Let $(x_{s},y_{s})\in\mathcal{S}$ denote the source RGB image and its pixel-wise annotation map, $x_{s}\in\mathbb{R}^{H\times W\times 3}$, $y_{s}\in\mathbb{R}^{H\times W}$. And $(\tilde{x}_{t})\in\widetilde{T}_{m}$ represent a purified target RGB image from the $m\textsuperscript{th}$ purified subdomain $\widetilde{T}_{m}$, $\tilde{x}_{t}\in\mathbb{R}^{H\times W\times 3}$. Note that $H$ and $W$ represent the size of height and width. Our BPM applies the mixing in two directions: $\mathcal{S}\to\widetilde{T}_{m}$ and $\widetilde{T}_{m}\to\mathcal{S}$.

On the direction of mixing from $\mathcal{S}\to\widetilde{T}_{m}$, we choose ClassMix [15] because the source image $x_{s}$ has the pixel-wise annotation map $y_{s}$. We first randomly select some classes from $y_{s}$. Then, we define $\Psi\in\{0,1\}^{H\times W}$ as a binary mask in which $\Psi(h,w)=1$ when the pixel position $(h,w)$ of $x_{s}$ belongs to the selected classes, and $\Psi(h,w)=0$ otherwise. While ClassMix suggests directly copying the corresponding pixels of selected classes of $x_{s}$ onto $\tilde{x}_{t}$, the mixed image generated by ClassMix contains inconsistent style distribution which might hinder the adaptation performance. To cope with the limitation, the proposed BPM applies photometric transform $\Gamma$ on the selected source pixels to the style of target image before directly copying them onto it. Let $\Psi\odot x_{s}$ represent the selected source pixels by the mask $\Psi$, and $\odot$ is element-wise multiplication. We first calculate the histograms of selected source pixels in LAB color space, and match them with $\{\widetilde{H}_{m}^{l},\widetilde{H}_{m}^{a},\widetilde{H}_{m}^{b}\}$. The translated source pixels is represented as $\Gamma(\Psi\odot x_{s})$. Then, we copy the translated source pixels onto $\tilde{x}_{t}$. We present some qualitative results in Figure 4. Note that no ground-truth annotation is available for $\tilde{x}_{t}$. Thus, we send the purified target image $\tilde{x}_{t}$ to the momentum network $M_{m}$ to generate a stable prediction map $\tilde{y}^{\prime}_{t}$ as the pseudo label. The mixing process on the direction of $\mathcal{S}\to\widetilde{T}_{m}$ by BPM is shown as

where $x_{\psi}$ is the generated mixed image, $y_{\psi}$ is the corresponding mixed pseudo label, and $\Gamma(\cdot)$ is the photometric transform of the source selected pixels by histogram matching on LAB color space.

On the direction of mixing from $\widetilde{T}_{m}\to\mathcal{S}$, however, it is impossible to choose ClassMix since no ground-truth annotation is available for $\tilde{x}_{t}$. Inspired by CutMix [31], we generate another binary mask $\Phi\in\{0,1\}^{H\times W}$ by sampling rectangular bounding box $(d_{x},d_{y},d_{w},d_{h})$ according to the uniform distribution; $d_{x}\sim U(0,W),d_{y}\sim U(0,H),d_{w}=W\sqrt{1-\eta},d_{h}=H\sqrt{1-\eta}$, where $\eta\sim U(0,1)$, $(H,W)$ are the height and width of the image. The binary mask $\Phi$ is formed by filling with $1$ the pixel positions inside the bounding box, and filling with $0$ other positions. With the help of $\Phi$, we select the target pixels $\Phi\odot\tilde{x}_{t}$ and transform them into the source style. The transformed target pixel is represented by $\Delta(\Phi\odot\tilde{x}_{t})$. Then we paste them onto the source image $x_{s}$. We present the mixing of $\widetilde{T}_{m}\to\mathcal{S}$ at

where $x_{\phi}$ is the other generated mixed image, $y_{\phi}$ is the corresponding mixed pseudo label, and $\Delta(\cdot)$ is the photometric transform of the target selected pixels by histogram matching on LAB color space.

we $(x_{\psi},y_{\psi})$ $(x_{\phi},y_{\phi})$ and $(x_{s},y_{s})$ to train the segmentation network $G_{m}$ and the momentum network $M_{m}$. We first optimize the parameters of $G_{m}$ through

where $\theta_{m}$ represent the parameters of $G_{m}$, $\mathcal{L}_{CE}$ is the cross-entropy loss for the predicted segmentation maps and the ground-truth or pseudo labels, $\alpha$ and $\beta$ are the hyper-parameters to control the effect of the mixing of both the directions for the loss function. To help the momentum network $M_{m}$ provide stable pseudo labels, we update the parameters of $M_{m}$, represented by ${\theta}^{\prime}_{m}$, using an exponential moving average (EMA) with a momentum $\lambda\in[0,1]$. After finishing the training iteration $t$, ${\theta}^{\prime}_{m}$ is updated by

To demonstrate the efficacy of our approach, we conduct experiments on the benchmark datasets of GTA5$\to$C-Driving and SYNTHIA$\to$C-Driving. We first compare our approach with the existing state-of-the-art OCDA approaches: CDAS [13], DHA [19], and CSFU [8]. Furthermore, we compare the proposed approach with the current state-of-the-art UDA approaches SAC [2] and DACS [25].