Amplitude Spectrum Transformation for
Open Compound Domain Adaptive Semantic Segmentation

Broadly, AST involves an encoder mapping and a decoder mapping. As shown in Fig. 2, the encoder mapping involves a Fourier transform $\mathcal{F}$ of the input feature $h$ to obtain the amplitude spectrum $\mathcal{F}_{A}(h)$ and the phase spectrum $\mathcal{F}_{P}(h)$. The amplitude spectrum is passed through a transformation $\mathcal{T}$ and a fully-connected encoder network $Q_{e}$ to obtain the AST-latent $z=Q_{e}\circ\mathcal{T}\circ\mathcal{F}_{A}(h)$. Consequently, the decoder mapping involves the inverse transformations to obtain $\hat{h}=\mathcal{F}^{-1}(\mathcal{T}^{-1}\circ Q_{d}(z),\mathcal{F}_{P}(h))$. It is important to note that the encoding side phase spectrum is directly forwarded to the decoding side to be recombined with the reconstructed amplitude spectrum via inverse Fourier transform $\mathcal{F}^{-1}$.

Here, the vectorization ($\mathcal{T}$) and inverse-vectorization ($\mathcal{T}^{-1}$) of the amplitude spectrum must adhere to its symmetric about origin (mid-point) property. To this end, $\mathcal{T}$ only vectorizes the first and fourth quadrants and $\mathcal{T}^{-1}$ reconstructs the full spectrum from the reshaped first and fourth quadrants by mirroring about the origin (see Fig. 2).

Training. Let $\theta_{Q}^{(k)}=\{\theta_{Q_{e}},\theta_{Q_{d}}\}$ denote the trainable parameters of AST at the $k^{\textit{th}}$ layer. Note that, AST operates independently for each channel of $h_{k}$ with the same parameters. Effectively, $z_{k}$ is obtained as a concatenation of $z$’s from each channel of $h_{k}$. After obtaining the frozen ERM-network, AST, at a given layer, is trained to auto-encode the amplitude spectrum using the following objective,

For simplicity, we denote the overall AST as a function AST and simplify the reconstruction as $\hat{h}_{k}=\texttt{AST}(h_{k},z_{k})$. Note that we omit the $z_{k}$ term in the auto-encoding phase i.e. $\hat{h}_{k}=\texttt{AST}^{(\textit{ae})}(h_{k})$ as $z_{k}$ is passed through unmodified.

One of the major components in prior OCDA approaches is to discover sub-target clusters as it is used either for domain-translation of source samples (Park et al. 2020) or to realize the domain-specific batch-normalization (Gong et al. 2021). In order to obtain reliable sub-target clusters, both works rely on unsupervised clustering of the compound target data. In contrast, we aspire to avoid such discovery and do away with introducing an additional hyperparameter (no. of clusters) in order to realize a clustering-free OCDA algorithm.

Cross-domain pairing. How to select reliable source-target pairs for cross-domain simulation? In the absence of sub-target clusters, a naive approach would be to form random pairs of source and target instances. Aspiring to formalize a better source-target pairing strategy, we propose to pair instance with the most distinct domain style. This is motivated from the hard negative mining as used in deep metric learning approaches (Suh et al. 2019). Consider a source dataset with instances $\{s_{i}\}_{i=1}^{N_{s}}$ and a target dataset with instances $\{t_{j}\}_{j=1}^{N_{t}}$. For each target instance $t_{j}$, we mine a source instance $s_{i}$ such that these are maximally separated in the AST-latent space of the $l^{\text{th}}$ CNN layer. Formally, we obtain a set of cross-domain pairs as,

Cross-domain simulation. Utilizing Insight 1, we simulate the style of $t_{j}$ in the source $s_{i}$ by replacing its AST-latent i.e., $\hat{h}_{l,s_{i\hskip 0.56905pt\scalebox{0.8}{\begin{picture}(0.0,0.0)\roundcap\put(0.0,0.0){\leavevmode\hbox{\set@color\char 41\relax}} \put(0.0,0.0){\line(1,0){missing}} \end{picture}}\hskip 0.56905ptj}}=\texttt{AST}(h_{l,s_{i}},z_{l,t_{j}})$. Similarly, $\hat{h}_{l,t_{j\hskip 0.56905pt\scalebox{0.8}{\begin{picture}(0.0,0.0)\roundcap\put(0.0,0.0){\leavevmode\hbox{\set@color\char 41\relax}} \put(0.0,0.0){\line(1,0){missing}} \end{picture}}\hskip 0.56905pti}}$ represents the source stylized target feature. We illustrate the cross-domain simulation in Fig. 1C. For notation simplification, $\hat{h}_{l,s_{i\hskip 0.56905pt\scalebox{0.8}{\begin{picture}(0.0,0.0)\roundcap\put(0.0,0.0){\leavevmode\hbox{\set@color\char 41\relax}} \put(0.0,0.0){\line(1,0){missing}} \end{picture}}\hskip 0.56905ptj}}$ and $\hat{h}_{l,s_{i\hskip 0.56905pt\scalebox{0.8}{\begin{picture}(0.0,0.0)\roundcap\put(0.0,0.0){\leavevmode\hbox{\set@color\char 41\relax}} \put(0.0,0.0){\line(1,0){missing}} \end{picture}}\hskip 0.56905ptj}}$ are denoted as $h_{l,s^{\hskip 0.56905pt\scalebox{0.8}{\begin{picture}(0.0,0.0)\roundcap\put(0.0,0.0){\leavevmode\hbox{\set@color\char 41\relax}} \put(0.0,0.0){\line(1,0){missing}} \end{picture}}\hskip 0.56905pt}}$ and $h_{l,t^{\hskip 0.56905pt\scalebox{0.8}{\begin{picture}(0.0,0.0)\roundcap\put(0.0,0.0){\leavevmode\hbox{\set@color\char 41\relax}} \put(0.0,0.0){\line(1,0){missing}} \end{picture}}\hskip 0.56905pt}}$ respectively. Note that, $h_{l,s^{\hskip 0.56905pt\scalebox{0.8}{\begin{picture}(0.0,0.0)\roundcap\put(0.0,0.0){\leavevmode\hbox{\set@color\char 41\relax}} \put(0.0,0.0){\line(1,0){missing}} \end{picture}}\hskip 0.56905pt}}$ and $h_{l,t^{\hskip 0.56905pt\scalebox{0.8}{\begin{picture}(0.0,0.0)\roundcap\put(0.0,0.0){\leavevmode\hbox{\set@color\char 41\relax}} \put(0.0,0.0){\line(1,0){missing}} \end{picture}}\hskip 0.56905pt}}$ hold the same task-related content as $h_{l,s}$ and $h_{l,t}$ respectively. Thus, both original and stylized features are expected to have the same segmentation label.

Following Insight 1, the domain normalization is performed as $h_{l^{\prime}\!,s_{g}}=\texttt{AST}(h_{l^{\prime}\!,s},z_{l^{\prime}\!,g})$. We illustrate the domain normalization in Fig. 1D. The normalized versions of $h_{l^{\prime}\!,t},h_{l^{\prime}\!,s^{\hskip 0.56905pt\scalebox{0.8}{\begin{picture}(0.0,0.0)\roundcap\put(0.0,0.0){\leavevmode\hbox{\set@color\char 41\relax}} \put(0.0,0.0){\line(1,0){missing}} \end{picture}}\hskip 0.56905pt}},h_{l^{\prime}\!,t^{\hskip 0.56905pt\scalebox{0.8}{\begin{picture}(0.0,0.0)\roundcap\put(0.0,0.0){\leavevmode\hbox{\set@color\char 41\relax}} \put(0.0,0.0){\line(1,0){missing}} \end{picture}}\hskip 0.56905pt}}$ are $h_{l^{\prime}\!,t_{g}},h_{l^{\prime}\!,s^{\hskip 0.56905pt\scalebox{0.8}{\begin{picture}(0.0,0.0)\roundcap\put(0.0,0.0){\leavevmode\hbox{\set@color\char 41\relax}} \put(0.0,0.0){\line(1,0){missing}} \end{picture}}\hskip 0.56905pt}_{g}},h_{l^{\prime}\!,t^{\hskip 0.56905pt\scalebox{0.8}{\begin{picture}(0.0,0.0)\roundcap\put(0.0,0.0){\leavevmode\hbox{\set@color\char 41\relax}} \put(0.0,0.0){\line(1,0){missing}} \end{picture}}\hskip 0.56905pt}_{g}}$ respectively.

In this stage, we aim to prepare (or initialize) the network components prior to commencing the proposed Simulate-then-Normalize procedure (i.e. adaptation). We start from the pre-trained standard source model and the pre-trained auto-encoder based AST-networks at layer $l$ and $l^{\prime}$, i.e. $\texttt{AST}_{\textit{cs}}^{\textit{(ae)}}$, and $\texttt{AST}_{\textit{dn}}^{\textit{(ae)}}$. Here, the superscript (ae) indicates that these networks are not yet manipulated as AST-Sim, or AST-Norm (i.e., without any manipulation of the AST-latent).

Following this, the pre-adaptation finetuning involves two pathways. First, $x_{s}$ is forwarded to obtain $y_{s}=C\circ\texttt{AST}_{\textit{dn}}^{\textit{(ae)}}\circ\Psi\circ\texttt{AST}_{\textit{cs}}^{\textit{(ae)}}\circ\Phi(x_{s})$. The following source-side supervised cross-entropy objective is formed as;

Here, $\theta$ denotes the collection of parameters of $\Phi$, $\Psi$, and $C$. Note that, after integrating ASTs into the CNN segmentor, the content-related gradients flow unobstructed through the frozen AST networks while finetuning $\Phi$ and $\Psi$.

In the second pathway, both $x_{s}$ and $x_{t}$ are forwarded to obtain input-output pairs to finetune $\theta_{\textit{cs}}$, and $\theta_{\textit{dn}}$, i.e.

Here, $(h_{l,s},\hat{h}_{l,s})$ and $(h_{l,t},\hat{h}_{l,t})$ are the input-output pairs corresponding to $x_{s}$ and $x_{t}$ to update $\texttt{AST}_{\textit{cs}}^{\textit{(ae)}}$. Similarly, $(h_{l^{\prime}\!,s},\hat{h}_{l^{\prime}\!,s})$ and $(h_{l^{\prime}\!,t},\hat{h}_{l^{\prime}\!,t})$ are the input-output pairs corresponding to $x_{s}$ and $x_{t}$ to update $\texttt{AST}_{\textit{dn}}^{\textit{(ae)}}$.

The finetuned networks from the pre-adaptation stage are now subjected to adaptation via the simulate-then-normalize procedure. We manipulate the AST-latent thus denoting $\texttt{AST}_{\textit{cs}}^{\textit{(ae)}}$, and $\texttt{AST}_{\textit{dn}}^{\textit{(ae)}}$ as $\texttt{AST}_{\textit{cs}}$, and $\texttt{AST}_{\textit{dn}}$ respectively. Note that, latent manipulation is independent of the network weights, $\theta_{\textit{cs}}$, and $\theta_{\textit{dn}}$. Thus, we freeze $\theta_{\textit{cs}}$ and $\theta_{\textit{dn}}$ from the pre-adaptation training to preserve their auto-encoding behaviour, and only update $\theta$ during the adaptation stage.

The adaptation training involves four data-flow pathways as shown in Fig 3. The Simulate step at layer-$l$ outputs the following features; $(h_{l,s},h_{l,s^{\hskip 0.56905pt\scalebox{0.8}{\begin{picture}(0.0,0.0)\roundcap\put(0.0,0.0){\leavevmode\hbox{\set@color\char 41\relax}} \put(0.0,0.0){\line(1,0){missing}} \end{picture}}\hskip 0.56905pt}},h_{l,t^{\hskip 0.56905pt\scalebox{0.8}{\begin{picture}(0.0,0.0)\roundcap\put(0.0,0.0){\leavevmode\hbox{\set@color\char 41\relax}} \put(0.0,0.0){\line(1,0){missing}} \end{picture}}\hskip 0.56905pt}},h_{l,t})$. Moving forward, the Normalize step at layer-$l^{\prime}$ outputs the following: $(h_{l^{\prime}\!,s_{g}},h_{l^{\prime}\!,s_{g}^{\hskip 0.56905pt\scalebox{0.8}{\begin{picture}(0.0,0.0)\roundcap\put(0.0,0.0){\leavevmode\hbox{\set@color\char 41\relax}} \put(0.0,0.0){\line(1,0){missing}} \end{picture}}\hskip 0.56905pt}},h_{l^{\prime}\!,t_{g}^{\hskip 0.56905pt\scalebox{0.8}{\begin{picture}(0.0,0.0)\roundcap\put(0.0,0.0){\leavevmode\hbox{\set@color\char 41\relax}} \put(0.0,0.0){\line(1,0){missing}} \end{picture}}\hskip 0.56905pt}},h_{l^{\prime}\!,t_{g}})$. Finally, we obtain the following four predictions; $(y_{s_{g}},y_{s_{g}^{\hskip 0.56905pt\scalebox{0.8}{\begin{picture}(0.0,0.0)\roundcap\put(0.0,0.0){\leavevmode\hbox{\set@color\char 41\relax}} \put(0.0,0.0){\line(1,0){missing}} \end{picture}}\hskip 0.56905pt}},y_{t_{g}^{\hskip 0.56905pt\scalebox{0.8}{\begin{picture}(0.0,0.0)\roundcap\put(0.0,0.0){\leavevmode\hbox{\set@color\char 41\relax}} \put(0.0,0.0){\line(1,0){missing}} \end{picture}}\hskip 0.56905pt}},y_{t_{g}})$.

As shown in Fig. 3F, we apply supervised losses against ground-truth (GT) $y_{\textit{gt}}$ for the source-side predictions i.e.

Following the self-training based adaptation works (Zou et al. 2019), we propose to use pseudo-labels (pseudo-GT) $y_{\textit{pgt}}$ for the target-side predictions i.e.

Pseudo-label extraction. In order to obtain reliable pseudo-labels $y_{\textit{pgt}}$, we prune the target predictions by performing a consistency check between $y_{t_{g}}$ and $y_{t_{g}^{\hskip 0.56905pt\scalebox{0.8}{\begin{picture}(0.0,0.0)\roundcap\put(0.0,0.0){\leavevmode\hbox{\set@color\char 41\relax}} \put(0.0,0.0){\line(1,0){missing}} \end{picture}}\hskip 0.56905pt}}$. Thus, $y_{\textit{pgt}}=\operatorname*{arg\,max}_{c}y_{t_{g}}$ is obtained only for reliable pixels i.e. for pixels $x_{t}$ which satisfy the following criteria,