Masked GAN for Unsupervised Depth and Pose Prediction with Scale Consistency

The framework of our unsupervised network is shown in Fig. 3. The generator takes a short sequential frame snippet that consists of a target image $I_{t}$ and an adjacent source image $I_{s}$; in addition, the output of the generator contains a depth map, a 6-DOF pose, and the Boolean mask $M_{b}$, which corresponds to the unreconstructed region in the synthesized images. Based on the predicted depth and pose, the view reconstruction algorithm that is widely used in previous unsupervised methods is applied to warp the image in the source frame to the target frame, which is shown as the synthesized target image. Because of the inconsistent visual information between the frames caused by motion and view-field changes, synthesized target images produced by view reconstruction are incomplete when compared with the real target image. These unreconstructed areas in the synthesized target images become a distinctive feature between the real and fake images for the discriminator, which will help the discriminator to distinguish them, thereby breaking the balance of the max-min game in adversarial learning. Moreover, because the visual field changes between the adjacent frames are inevitable, these distinctive features (unreconstructed regions) usually persist and cannot be eliminated by continuous training of the generator. In this paper, to eliminate the effects of the unreconstructed regions on the discriminator, we present a Boolean mask to preprocess the real target images and construct the same unreconstructed regions on the real target image as the synthesized target image, as shown in Fig. 3. Therefore, the synthesized and real target images contain the same data distribution of these unreconstructed regions, which will have a positive effect on the training of the discriminator and thus the adversarial training process.

Generator: Our generator consists of three networks for different tasks, as shown in Fig. 3; a DepthNet for monocular depth prediction, a PoseNet for regressing the ego-motion between the target and source image, and a MaskNet for mask estimation. The view reconstruction algorithm is utilized to reconstruct the target image from a source image based on the output of the pose and depth networks. The mask predicted by MaskNet is utilized to eliminate the impacts of incomplete reconstruction regions on reconstruction loss and adversarial loss. For reconstruction loss, the effectiveness of mask has been demonstrated in recent works [2, 13]. For adversarial loss, we tackle this challenge by converting the mask into the Boolean mask to preprocess the target images and synthesize the same unreconstructed regions on target images as that on synthesized images. We train the proposed architecture in an unsupervised manner, and the overall objective loss function is formulated as follows:

where $\alpha$, $\varphi$, $\beta$ and $\gamma$ are the balance factors between the loss terms. $\mathcal{L}_{scale}$ refers to the proposed scale consistency loss, and $\mathcal{L}_{mask}$ is a regularization term to constrain the training of MaskNet, which is inspired by Zhou et al. [2]. $\mathcal{L}_{GAN}$ denotes the adversarial loss. $\mathcal{L}_{basic}$ stands for the basic loss. The basic loss is used to assist the training process of the generator and consists of two parts, the traditional reconstruction loss $\mathcal{L}_{rec}$ and the smoothness loss $\mathcal{L}_{smooth}$ :

where $\alpha_{1}$ and $\alpha_{2}$ are the balance factors.

Reconstruction Loss: With the output of DepthNet and PoseNet, the target images can be reconstructed from the source images $I_{s}$ through the view reconstruction algorithm, which is widely used in previous unsupervised monocular depth estimation frameworks [2, 16, 17]. The principle of this algorithm is based on the following projection function:

where $K$ stands for the camera intrinsics matrix. $\hat{D}_{t}$ denotes the depth estimation of the target image $I_{t}$, and $\hat{T}_{t\to s}$ represents the predicted 6-DOF transformation between the target image $I_{t}$ and the source image $I_{s}$. The pixels of two images $p_{t},p_{s}$ establish the correspondence by a projection function. Then, we synthesize $\hat{I}_{s}$ by warping the image in source frame to target frame. Finally, the reconstruction loss is formulated as follows:

where $\alpha_{3}$ refers to a balance factor, and SSIM [18] is an index that shows the structural similarity between $I_{t}$ and $\hat{I}_{s}$.

Smoothness Loss: To filter out erroneous predictions and promote the representation of the geometric structure, a smoothness loss is designed to constrain the smoothness of the predicted depth map. Inspired by [33], we adopt the second-order differential for improving the smoothness:

where $\nabla^{2}$ denotes the second-order differential operator, and $T$ is the transpose operation.

Traditional mask for reconstruction loss: To eliminate the influence of the visual field changes and the dynamic objects on reconstruction loss, we use a MaskNet to predict these unreconstructed regions. During training, a MaskNet is designed to estimate the different regions between the real target image $I_{t}$ and the synthesized target image $\hat{I}_{s}$, formatted as $min[M\times(I_{t}-\hat{I}_{s})]$, and a regularization term $\mathcal{L}_{mask}$ is used to constrain the MaskNet during training, which is similar to [2]. Our intention is to use the MaskNet to predict the regions that could not be reconstructed on synthesized images. Considering the effect of incomplete reconstruction on the reconstruction loss, the mask $M$ predicted by MaskNet is introduced into $\mathcal{L}^{m}_{rec}$:

Our final basic model with the mask is formulated as follow:

Scale Consistency Loss: In unsupervised monocular frameworks, the depth and pose networks are trained by unlabelled short snippets. However, different snippets have different scale factors (compared with the ground truth) because there is no corresponding scale constraint loss applied. Therefore, previous pose and depth networks [2, 16] cannot provide scale-consistent results between different snippets. Bian et al. [17] consider the scale-consistency of pose networks for generating full trajectories over long monocular videos. Similarly, this paper proposes a novel adaptive loss for better constraint on geometric and scale consistency between snippets, which is formulated as follows:

where $\alpha_{4}$ refers to a balance factor. $D^{t}_{s}(p_{s})$ is computed by the projection algorithm shown in Eq. (9), and it has the same scale information as $\hat{D}_{t}$. $\hat{D}_{t}$ stands for the predicted depth map of the target image. $\hat{D}_{s}$ is the predicted depth map of the source image (the target image of the next frame snippet). $\hat{D}^{s}_{s}$ is reconstructed from $\hat{D}_{s}$ by the warping algorithm, which is similar to the view reconstruction process in [2], and it contains the same scale information as $\hat{D}_{s}$. Then, SSIM loss [18] is adopted for the consistency of $D^{t}_{s}$ and $\hat{D}^{s}_{s}$ in such a way that the scale between the snippets is aligned.

Original GAN: consists of two components, a generator ($G$) and a discriminator ($D$) [32]. The $D$ is designed to distinguish the real data $x$ from synthesized data $G(z)$ by learning the difference of data distribution. At the same time, the major task of the generator is to generate a set of data $G(z)$ with the same distribution as the real data to fool the discriminator. Therefore, a max-min game is played between the generator and discriminator, and the loss function of the original GAN [32] is formulated as:

where $p_{data}(x)$ and $p_{z}(z)$ stand for the data distribution of $x$ and $z$, respectively. Finally, a mapping relationship between two data distributions is established through adversarial learning.

The unsupervised monocular depth estimation cannot feed the discriminator with a real depth map because there is no ground truth. To address this limitation, instead of distinguishing between the real and predicted depth map, the RGB images $\hat{I}_{s}$ synthesized by the view reconstruction are sent to discriminator together with the real target images $I_{t}$ in the monocular framework combined with GANs [8, 9]:

where $D(.)$ and $G(.)$ stand for the discriminator and generator of this paper.

Since the data distribution of the unreconstructed regions is unique to the synthetic images, which will affect the adversarial learning, we propose a Boolean mask to reduce the impact of the unreconstructed regions on the adversarial learning, and the same unreconstructed regions as synthesized images are produced on the target images.

Boolean mask processing: First, we transfer the floating-point mask $M$ predicted by MaskNet to a Boolean type $M_{b}$ by comparing it with a threshold $\theta$:

where $p$ stands for the pixel index on the mask. Then, to eliminate the distinctive feature (unreconstructed regions in synthesized images) between the synthesized and real target images, we use the Boolean mask $M_{b}$ to reweight the target image $I_{t}$ and get similar unreconstructed regions as the synthesized image $\hat{I}_{s}$ on the target image $I_{t}$:

Besides, to prevent this Boolean mask processing step from introducing the new and particular noises into target images, which will also influence the training of discriminator, we do the same Boolean mask processing operation to the synthesized image $\hat{I}_{s}$:

Masked GAN: After preprocessing by the Boolean mask, the masked target image $I^{M_{b}}_{t}$ (real) and synthesized target image $\hat{I}^{M_{b}}_{s}$ (fake) are sent to the discriminator, and based on [32], our final adversarial loss is formulated as follows:

where $I^{M_{b}}_{t}$ and $\hat{I}^{M_{b}}_{s}$ are calculated by Boolean mask processing, Eq.13 and Eq. 14.

Floating-point mask processing: To verify the effectiveness of Boolean mask processing, instead of transferring the mask to the Boolean one, the floating-point mask $M_{f}=M$ predicted by MaskNet is directly used to preprocess the synthesized and real target images ($\hat{I}_{s}$, $I_{t}$ ):

Therefore, based on [32], the adversarial loss combined with floating-point mask processing is formulated as:

where $I^{M_{f}}_{t}$ and $\hat{I}^{M_{f}}_{s}$ are calculated by floating-point mask processing, shown in Eq.(16).

Network architecture: Our unsupervised method mainly consists of two modules, the generator and the discriminator, and the model can be divided into three subnetworks, the MaskNet coupled with the PoseNet, the DepthNet, and the discriminator network. For the DepthNet, we follow previous work [6, 17] and take the DispResNet [16] as our DepthNet. The DepthNet takes a single image as input and outputs a depth map in an end-to-end manner. For the PoseNet and MaskNet, we directly adopt the coupled framework proposed in [2, 13], while the main difference is that we add the skip-connections between the corresponding encoding and decoding layers to improve the performance of the mask prediction. This coupled framework takes a concatenated RGB image snippet as input, which consists of one target image and two or four source images. The PoseNet predicts the 6 DOF poses between the target and source images, and the MaskNet predicts the masks to handle the occlusions and visual field changes between the target and source images. The architecture of discriminator network is designed with reference to the encoder of DispNet [34]. The real target images and synthesized target images are both preprocessed by a Boolean mask processing step before being sent to the discriminator. During training, the discriminator outputs the probability that the input image is real or fake, and the adversarial learning between the generator and discriminator improves the training of the depth and pose networks.

Training detail: The proposed adversarial learning framework is implemented using TensorFlow [35]. For the depth prediction task, we train and test our DepthNet on the KITTI raw dataset [19] by using Eigen’s split [1]. For the pose prediction task, following [26], we train and test our PoseNet on the KITTI odometry dataset [19], where the sequences 00-08 are used for training and sequences 09-10 are used for testing. The setup of our training and testing sets is the same as the ones in previous related studies [2, 16, 17, 26]. We regard a snippet of three (for training PoseNet) or five (for training DepthNet) sequential video frames as a training sample, which consists of one target image (middle frame) and two or four source images. We have done experiments with two different resolutions, $416\times 128$ and $640\times 192$. We train our framework in 50 epochs and obtain 10 randomly sampled batches in one epoch for validation, which is different from previous work in which the framework is trained in 100 epochs [36] or 200 epochs [17] and 1000 randomly sampled batches are obtained in one epoch for validation, and thus, it indicates that our model shows a good convergence rate. Following [17, 36], we also pre-train our networks on Cityscapes [37] and fine-tune it on KITTI [19], each for 50 epochs.

The training of our networks is based on a single RTX 8000 GPU. During training, the image resolution is resized to $416\times 128$ or $640\times 192$, the weights of the generator and discriminator are optimized by ADAM optimizers [38] with $\beta_{1}=0.9$, $\beta_{2}=0.999$, and the learning rate is 0.0002. During training, we adopt $\alpha_{1}=1.0$, $\alpha_{2}=0.5$, $\alpha_{3}=0.85$, $\alpha_{4}=0.2$, $\varphi=0.5$, $\beta=0.3$, $\gamma=0.0001$, $\theta=0.9$. Our parameters are set based on the experiments or related work [16], and we show some experimental results of hyper-parameter optimization in Table I.

Evaluation metrics: For depth evaluation, the commonly used evaluation metrics proposed by Eigen et al. [1] are used in this paper to compare our method with others fairly; these include five evaluation indicators: RMSE, RMSE log, Abs Rel, Sq Rel, Accuracy:

• •

  $\textbf{RMSE}=\sqrt{\frac{1}{|N|}\sum_{i\in N}\parallel d_{i}-d_{i}^{*}\parallel^{2}}$,

• •

  $\textbf{RMSE log}=\sqrt{\frac{1}{|N|}\sum_{i\in N}\parallel\log(d_{i})-\log(d_{i}^{*})\parallel^{2}}$,

• •

  $\textbf{Abs Rel}=\frac{1}{|N|}\sum_{i\in N}\frac{\mid d_{i}-d_{i}^{*}\mid}{d_{i}^{*}}$,

• •

  $\textbf{Sq Rel}=\frac{1}{|N|}\sum_{i\in N}\frac{\parallel d_{i}-d_{i}^{*}\parallel^{2}}{d_{i}^{*}}$,

• •

  Accuracy: $\%$ of $d_{i}$ s.t. $\max(\frac{d_{i}}{d_{i}^{*}},\frac{d_{i}^{*}}{d_{i}})=\delta<thr$,

where $d_{i}$ and $d_{i}^{*}$ denote the predicted depth of pixel $i$ and the corresponding ground truth, and $N$ denotes the total number of pixels that correspond to the ground truth depth value. $thr$ denotes a threshold, which is always set to $1.25^{1}$, $1.25^{2}$, and $1.25^{3}$.

For the trajectory evaluation, the generated full trajectory is evaluated by the standard evaluation metrics provided in the dataset [19], including a translation error metric $t_{err}(\%)$ and a rotation error metric $r_{err}(^{\circ}/100m)$. Compared with the 5-frame pose evaluation that proposed in [2], the evaluation metrics in this paper are more widely used in traditional VO methods and are more meaningful.

Ablation study: In this section, we first conduct a series of ablation experiments to validate the efficacy of our proposed framework, as shown in Table I. The “$\mathcal{L}_{\textrm{basic}}$” denotes that our framework is trained by the basic loss $\mathcal{L}_{\textrm{basic}}$ (Eq. (2)), which consists of the reconstruction loss ($\mathcal{L}_{\textrm{rec}}$, Eq. (4)) and smoothness loss ($\mathcal{L}_{\textrm{smooth}}$, Eq. (5)). Then, scale consistency loss ($\mathcal{L}_{\textrm{scale}}$, Eq. (8)), adversarial learning ($\mathcal{L}_{\textrm{GAN}}$, Eq. (11)), masks for reconstruction loss ($\mathcal{L}^{m}_{\textrm{basic}}$, Eq. (7), $\mathcal{L}_{mask}$), and Boolean mask processing ($\mathcal{L}^{M_{b}}_{\textrm{GAN}}$, Eq. (15)) are introduced sequentially into training. Although both the mask and Boolean mask processing (BMP) in Table I are based on the output of MaskNet, the mask is to reduce the effect of the unreconstructed regions on the reconstruction loss, while BMP is applied to eliminate the impact of the unreconstructed regions on the adversarial loss.

From the results shown in Table I, comparing lines 1 and 2 in the table, the proposed scale consistency loss $\mathcal{L}_{\textrm{scale}}$ is conducive to improvement in the depth estimation. Comparing lines 2 and 3, the introduction of adversarial learning (GAN) does not effectively improve the accuracy of the depth network, $i.e.$, the introduced adversarial learning does not play a key role in the training of DepthNet. In addition, we achieve a better result when the proposed $\mathcal{L}^{M_{b}}_{\textrm{GAN}}$ is used to reduce the impact of the unreconstructed regions on adversarial learning. Therefore, we believe that the unreconstructed regions influence the performance of adversarial learning. Then, the mask predicted by MaskNet is introduced into the framework, which is shown as “$\mathcal{L}^{m}_{\textrm{basic}}$ + $\mathcal{L}_{\textrm{scale}}$ + $\mathcal{L}_{\textrm{GAN}}$ + $\mathcal{L}_{\textrm{mask}}$”. From the evaluation metrics, the depth estimation accuracy of “$\mathcal{L}^{m}_{\textrm{basic}}$ + $\mathcal{L}_{\textrm{scale}}$ + $\mathcal{L}_{\textrm{GAN}}$ + $\mathcal{L}_{\textrm{mask}}$” outperforms that of “$\mathcal{L}_{\textrm{basic}}$ + $\mathcal{L}_{\textrm{scale}}$ + $\mathcal{L}_{\textrm{GAN}}$”, which means that the adopted mask improves the training process of the network. With the qualitative results on the mask in Fig. 4, the unreconstructed regions of the synthesized images are accurately predicted in the masks. Therefore, the reason for accuracy improvement in the depth estimation is that the mask can significantly mitigate the influence of the unreconstructed regions on the reconstruction loss. Afterwards, we adopt the BMP in this paper to mask the real target images (real) and synthesized target images (fake) before sending them to the discriminator, which is shown as “$\mathcal{L}^{m}_{\textrm{basic}}$ + $\mathcal{L}_{\textrm{scale}}$ + $\mathcal{L}^{M_{b}}_{\textrm{GAN}}$ + $\mathcal{L}_{\textrm{mask}}$”, and the best result among the four cases is achieved. In addition, the qualitative results of the BMP shown in Fig. 4 also demonstrate that our proposed BMP can construct similar unreconstructed regions in target images, which helps to reduce the impact of the unreconstructed regions on the adversarial learning and plays a key role in adversarial learning. In summary, this ablation study indicates the effectiveness of our proposed modules.

Comparison of different mask networks: Since the mask predicted by the MaskNet is very important in this paper and plays a key role in the adversarial loss and reconstructing loss, we applied some modifications (skip connection between encoder and decoder) on the mask network proposed in [2] to improve its performance. As shown in Table I, the experimental result shows that these modifications effectively improve the performance of our framework, which also illustrates the importance of the MaskNet to our framework.

Comparison of different mask processing: To further verify the ability of the Boolean mask, we compare the GAN processed by the BMP ($\mathcal{L}^{M_{b}}_{\textrm{GAN}}$) with cases of GANs processed by floating-point mask (FMP) ($\mathcal{L}^{M_{f}}_{\textrm{GAN}}$) and the original GAN without mask processing ($\mathcal{L}_{GAN}$); the results are shown in lines 4, 5 and 6 of Table I. Although the FMP cannot create similar unreconstructed regions on real target images as synthesized target images, it can be seen from Table I that FMP plays a positive role in improving the performance of adversarial learning when compared with the original GAN. According to the examples of the different mask processing results shown in Fig. 5, the effect of the FMP is not as obvious as that of the BMP. In addition, the experimental results in Table I show that both the FMP and BMP are effective in accuracy improvement of the depth estimation, and the BMP works the best.

Comparison with the methods using GANs: We compare our method with the unsupervised monocular methods proposed in [8, 9, 13], which introduce adversarial learning into the training framework. Note that the GANVO [9] generates its depth prediction from a vector, which means that their depth network cannot be used independently and predict the depth in an end-to-end manner. At the same time, the depth network in [13] uses a single image as well as the temporal information for the depth estimation, and as a result, this network takes more information than ours on depth estimation. In addition, because of the LSTM module used in their framework in [13], the accuracy of the depth network depends on the image sequences, and the depth network cannot predict the depth map from only a single image, which limits its practical applications. Although our DepthNet is jointly trained with other networks in an unsupervised manner, it can be used independently during testing and has the ability to accurately generate depth maps from single images, which enables it to perform depth estimation on some independent images such as network images. The quantitative results are shown in Table II, and our method obtain the competitive results.

Comparison with previous work: The qualitative and quantitative results of our DepthNet are evaluated by public metrics and are shown in Fig. 6 and Table III. Comparing the ground truth with our predicted depth maps in Fig. 6, our deep network predicts the depth information of the geometric structures in the scenes, such as trees, streets, cars, and buildings. Note that higher resolutions include detailed geometric details, and thus, we divide the results of the different methods according to the resolution of their input images for fairness. We choose images with different resolutions as input to our DepthNet. As shown in Table III, we obtain competitive performance on end-to-end monocular depth prediction with different resolution, 416$\times$128 and 640$\times$192. Our results also prove that a high-resolution input is conducive to improvement in the accuracy.

Evaluation based on other datasets: Previous methods [16, 17, 6] have proved that depth networks will get better results if the models are pre-trained on the Cityscapes datasets [37] before training on KITTI dataset [19]. Following this approach, we first use the Cityscapes dataset to pre-train our total framework, and then the pre-trained model is further trained by the KITTI training set, which is shown as “CS+K” in Tables III and V.

To further evaluate the performance of our depth network on other data domains, following [36], we directly test our depth model trained by “CS+K” on the Make3D test set [20]. As shown in Table IV, our DepthNet also shows good performance when compare with previous methods, which means that our model has better transferability. Our qualitative results are illustrated in Fig. 6.

For PoseNet, following Zhan et al. [26] and Bian et al. [17], the standard evaluation tools provided by the dataset are used to evaluate the full predicted trajectories, which are different from previous monocular DL-based pose evaluation methods [2, 16, 6, 42]. Table V shows the average rotation and translation errors of the predicted trajectories on the KITTI odometry sequences 09-10. As shown in Table V, comparing the results of “Ours(Without $\mathcal{L}_{\textrm{scale}}$)” and “Ours”, the proposed scale consistency loss $\mathcal{L}_{\textrm{scale}}$ is effective in constraining the scale consistency in the trajectory prediction. Although the method in [13] also considers the scale-inconsistency of pose estimation, the generated trajectories are neither evaluated in the article nor published online, and thus, we cannot make a quantitative comparison with their trajectories. The visual results are shown in Fig. 7, which are drawn by evo tools [43] with automatic scale alignment for the full trajectory. The method [26] trained with stereo image pairs does not have the problem of scale inconsistency because it learns the scale information from stereo image pairs during the training. Monocular methods [2, 16, 44] train their network with monocular sequence and suffer from per-pose scale ambiguity and inconsistency. At the same time, the scale information between the different snippets predicted by PoseNet is inconsistent in such a way that it cannot provide an accurate trajectory. Bian et al. tackle this problem by geometric alignment, and our framework obtains a better result than theirs. As shown in Table. V, our model, when trained on “CS+K”, does not obtain a large improvement similar to others [28, 17] when compared with the model trained on “K”.

In addition, as shown in Table V, because of the strong back-end optimization, ORB-SLAM [15] shows more powerful performance in position and orientation prediction than deep learning-based VO methods. Although there is still a large gap compared with the traditional VO method [15], our PoseNet has the ability to predict the full trajectory over a long monocular video in an end-to-end manner.