Learning Feature Decomposition for Domain Adaptive Monocular Depth Estimation

An overview of the proposed LFDA is shown in Fig. 2. The entire framework consists of eight sub-networks: shared content encoder $E_{con}$, source-specific style encoder $E^{s}_{sty}$, target-specific style encoder $E^{t}_{sty}$, MDE task decoder $D$, generator $G$, domain discriminator $Disc$, source-to-target translation discriminator $Disc^{s\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}t}$, and target-to-source translation discriminator $Disc^{t\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}s}$. $\{E_{con},D\}$ composes as a MDE primary task network, which is a standard encoder-decoder architecture.

First, we propose to estimate the feature distributions of $I^{s}$ and $I^{t}$ individually using a separate BN structure [4, 29, 43]. Specifically, two BN branches [19], denoted as $BN^{s}$ and $BN^{t}$, are deployed after each convolutional layer in $E_{con}$ (see Fig. 2c). Each BN branch works individually for its own domain. To elaborate, $BN^{s}$ and $BN^{t}$ learn domain-specific affine parameters $\{\gamma^{s},\beta^{s}\}$/$\{\gamma^{t},\beta^{t}\}$, and distribution statistics $\{\mu^{s},\sigma^{s}\}$/$\{\mu^{t},\sigma^{t}\}$ for the source and target data, respectively. Note that all the layers other than BNs are still shared (e.g., convolution and ReLU). Suppose that $\ddot{z}^{d}_{con}$ is the content feature of domain $d$, where $d\in\{s,t\}$, the separate BN structure at an arbitrary layer in $E_{con}$ is formulated as:

$$BN^{d}(\ddot{z}^{d}_{con};\,\gamma^{d},\beta^{d})=\gamma^{d}\,\bigg{(}\frac{\ddot{z}^{d}_{con}-\mu^{d}}{\sqrt{(\sigma^{d})^{2}+k}}\bigg{)}+\beta^{d},$$

(1)

where $k$ is a tiny constant for numerical stability. With this design, the domain gap between $z^{s}_{con}$ and $z^{t}_{con}$ is acquired by the domain-specific parameters $\{\mu^{d},\sigma^{d},\gamma^{d},\beta^{d}\}$, and their domain-invariant part passes through each BN layer.

Second, inspired by GRL [12], we employ adversarial learning [16] to align the features $z^{s}_{con}$ and $z^{t}_{con}$ (see Fig. 2a). Details are discussed in Sec. III-B.

After feature decomposition, $z^{t}_{sty}$ and $z^{t}_{con}$ have different feature characteristics though they are from the same target domain. Hence, directly fusing them in the task decoder $D$ would cause potential accuracy degradation. To address this issue, as shown in Fig. 2c, we also deploy separate BNs in $D$. There are three BN branches: $BN^{s}_{con}$, $BN^{t}_{con}$ and $BN^{t}_{sty}$, each of which works as Eq. (1). $BN^{s}_{con}$ and $BN^{t}_{con}$ are used for the same purpose as discussed before, and $BN^{t}_{sty}$ is responsible for characterizing the feature distribution of $z^{t}_{sty}$ exclusively. Since the content and style features have different underlying distributions, simply leveraging a single set of BN parameters for $z^{t}_{con}$ and $z^{t}_{sty}$ would estimate an inaccurate mixture. Therefore, the additional $BN^{t}_{sty}$ is used to disentangle such mixture distribution, allowing proper integration of $z^{t}_{con}$ and $z^{t}_{sty}$ for decoding target features. Because the content and style components may have different importance for MDE, we employ a $1\times 1$ convolution and a residual connection to combine $z^{t}_{con}$ and $z^{t}_{sty}$ right before the output layer of $D$. This weighted fusion helps to adjust the balance between these two features of target data. Finally, $D$ outputs predicted depth maps, $\tilde{Y}^{s}=D(z^{s}_{con})$ and $\tilde{Y}^{t}=D(z^{t}_{con},z^{t}_{sty})$, respectively.

The proposed LFDA framework is trained with the following objective functions.

Inspired by style transfer techniques [18, 20], we adopt the translation loss to separate the content and style features of an input images. Let us consider the case of source-to-target image translation in our framework. Given a source image $I^{s}$ and a target image $I^{t}$, we aim to derive a translated image $I^{s\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}t}=G(z^{s}_{con},z^{t}_{sty})$ which consists of the content of $I^{s}$ and the style of $I^{t}$ (see Fig. 2b). We achieve this translation via objective $\mathcal{L}^{s\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}t}_{trans}$, which consists of two perceptual losses [20] and an adversarial loss:

$$\begin{split}\mathcal{L}^{s\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}t}_{trans}&=\sum_{j\in L}w^{trans}_{con,j}\big{\|}\phi_{j}(I^{s})-\phi_{j}(I^{s\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}t})\big{\|}_{1}\\ &+\sum_{j\in L}w^{trans}_{sty,j}\big{\|}\mu(\phi_{j}(I^{t}))-\mu(\phi_{j}(I^{s\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}t}))\big{\|}_{1}\\ &+\eta\,\big{(}Disc^{s\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}t}(I^{s\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}t})-1\big{)}^{2},\end{split}$$

(2)

where $\eta=0.2$, $w^{trans}_{con}$ and $w^{trans}_{sty}$ are pre-defined weights, $L$ denotes the $\{relu1\_1,relu2\_1,relu3\_1,relu4\_1,relu5\_1\}$ layers of a pre-trained VGG network [39] that measures perceptual loss, $\phi_{j}$ is the $j$-th layer in $L$, and $\mu(\cdot)$ returns the channel-wise mean values of a feature space. This translation loss has also been explored by [5].

To elaborate, the first perceptual loss computes the distance of the high-level content features between $I^{s}$ and $I^{s\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}t}$ such that $I^{s\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}t}$ contains the content of $I^{s}$. Since the content information mostly exists in higher layers of VGG, we set $w^{trans}_{con}$ to $\{0,0,0,1/4,1\}$. The second perceptual loss forces $I^{s\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}t}$ to contain the style of $I^{t}$. To explicitly encode the style information of an image, we employ AdaIN structure [18] that measures the distance of the channel-wise mean values of the style features between $I^{t}$ and $I^{s\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}t}$. Since the style information mostly exists in lower layers of VGG, we set $w^{trans}_{sty}$ to $\{1,1,1,0,0\}$. The third term is a standard least-squares adversarial loss [32], where we assign labels 1 and 0 to untranslated and translated images, respectively. This loss helps to improve the quality of image translation. As for the case of target-to-source translation, it is symmetric to source-to-target translation. We define its objective as $\mathcal{L}^{t\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}s}_{trans}$, which replaces $s$ to $t$, $t$ to $s$ and $s\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}t$ to $t\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}s$ in Eq. (2).

The reconstruction loss is used to guarantee that the combination of the decomposed content and style components forms a nearly complete representation of an input image [5]. Let us consider the case of source image reconstruction. Given a source image $I^{s}$, we aim to derive a reconstruction $I^{s\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}s}=G(z^{s}_{con},z^{s}_{sty})$ (see Fig. 2b). This can be achieved via objective $\mathcal{L}^{s\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}s}_{recon}$, which is also based on the perceptual loss:

$$\mathcal{L}^{s\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}s}_{recon}=\sum_{j\in L}w^{recon}_{j}\big{\|}\phi_{j}(I^{s})-\phi_{j}(I^{s\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}s})\big{\|}_{1},$$

(3)

where $w^{recon}=\{1/32,1/16,1/8,1/4,1\}$. Symmetrically, target image reconstruction is achieved via objective $\mathcal{L}^{t\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}t}_{recon}$, which replaces $s$ to $t$ and $s\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}s$ to $t\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}t$, from Eq. (3).

With the above loss functions, LFDA decomposes the feature space into $\{z^{s}_{con},z^{s}_{sty},z^{t}_{con},z^{t}_{sty}\}$, where each of which contains its supposed information exclusively.

During inference, our goal is to predict a depth map from a given target image. This corresponds to the red path in Fig. 2a. Therefore, only $E_{con}$, $E^{t}_{sty}$ and $D$ are retained after training, where $E^{t}_{sty}$ is the only required sub-network in addition to the MDE primary task network $\{E_{con},D\}$. Compared to recent top-performing approaches which require an entire sophisticated image translation network during inference [36, 44], LFDA allows much lower computational complexity. This is attributed to the proposed decomposition learning that reduces the domain gap more efficiently.

For fair comparison, the architectures of sub-networks $\{E_{con},D\}$, $E^{s}_{sty}$, $E^{t}_{sty}$, $Disc$, $Disc^{t\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}s}$ and $Disc^{s\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}t}$ are implemented identical to the corresponding ones in T²Net [45]. Besides, generator $G$ is implemented as in [5]. The models are trained by Adam optimizer [21] with initial learning rates of $1e^{-4}$ for $\{E_{con},D\}$ and $2e^{-5}$ for the other sub-networks. The learning rates decrease according to the polynomial decay policy. We set $\lambda_{geo}=1$, $\lambda_{sm}=\lambda_{align}=0.01$, $\lambda_{recon}=0.5$, and $\lambda_{trans}=0.05$. The entire framework is trained end-to-end in a single stage. The experiments are implemented by PyTorch [35] and conducted on a single NVIDIA Tesla V100 GPU. We will release our source code after the paper gets accepted.

Different cameras may have distinct intrinsic parameters or viewpoints, making the captured images have different scales, fields of view, etc. Such domain gap could cause sub-optimal adaptation performance.

The style and appearance of synthetic images are usually different from that of real images. This can negatively impact the accuracy on real data.

Adverse weather such as fog and rain produce image artifacts. These artifacts can result in accuracy degradation.

We conduct an ablation study using our model of vKITTI-to-KITTI and evaluate on the KITTI Eigen split. The results are reported in Table V. First, we can see that +Tgt+AL makes an obvious improvement over Src-Only, showing the importance of domain adaptation. Second, +Tgt+Con+2BN refers to the model that makes use of the decomposed content features and deploys two separate BN branches for the source and target domains, respectively. +Tgt+Con+2BN greatly improves the abs-rel metric by 0.017, showing our feature decomposition and separate BNs are effective in learning the domain-invariant content feature. Next, +Tgt+Con+2BN+Sty includes $z^{t}_{sty}$ in the pipeline but still maintains only two separate BNs. Results show that it suffers from severe performance degradation. This proves our argument that content and style features have different distributions, so passing them through the same BN would drop model performance. Finally, LFDA (i.e. +Tgt+Con+3BN+Sty), which deploys the third BN for the target style feature exclusively, resolves this issue successfully. Obviously, LFDA performs the best in most metrics, demonstrating the effectiveness of our method.

To verify the effectiveness of our feature decomposition, Fig. 3 shows the qualitative results of the image reconstruction and translation that are illustrated in Fig. 2b. We can observe that the reconstructed images $I^{s\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}s}$ and $I^{t\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}t}$ are very close to the input images $I^{s}$ and $I^{t}$, respectively. In addition, the translated images $I^{s\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}t}$ and $I^{t\mathrel{\hbox{\rule[-0.2pt]{3.0pt}{0.4pt}}\mkern-4.0mu\hbox{\char 41\relax}}s}$ accurately maintain the content information while generating the style appearance of another domain. Such high-quality results can be achieved only if the decomposition of content and style features is successful. This demonstrates the rationale behind the high performance of the proposed method.

In addition to accuracy, model size and computational cost are also important factors when we evaluate a model. They determine the feasibility of a model for practical applications. In Table VI, we compare LFDA to two existing top-performing approaches in terms of the number of parameters and the number of multiply-accumulate operations (MACs) used at inference time. GASDA [44] include three sub-networks during inference, a target data MDE network, a target-to-source translation network, and a target-to-source MDE network. This design places a heavy computational burden. SharinGAN [36] also needs an image translation network plus a MDE network. In contrast, in LFDA, the only sub-network in addition to the primary MDE network is $E^{t}_{sty}$, which increases minimum complexity. LFDA’s number of MACs is 51% and 20% fewer than GASDA and SharinGAN, respectively, showing that our method can bridge the domain gap much more efficiently.