Distortion-Tolerant Monocular Depth Estimation On Omnidirectional Images Using Dual-cubemap

Monocular depth estimation, a process of mapping from low to high dimensions, can be very challenging. Early researchers solve the mapping function in a hand-crafted features manner. Nowadays, many recent approaches extract features using deep learning techniques. Eigen $et\ al.$ [10] propose the first CNNs based network. They adopted a coarse-to-fine strategy by designing a multi-scale module to improve predicted results. Inspired by  [11], Laina $et\ al.$ [12] merged pre-trained ResNet-50 [13] and up-sampling blocks into FRCN [12] to estimate the depth. Considering the auxiliary role of other visual tasks, Jiao $et\ al.$ [14] introduced semantic information and designed an attention-driven loss as a bond. Different from treating depth estimation as a regression task, the approach of Fu $et\ al.$ [15] adopted an ordinal regression strategy to transform depth estimation into a classification task by discretizing depth value. However, all the methods mentioned above are designed for NFoV images, and can not be directly applied to our task.

Solving distortion is the key to applying traditional CNNs based approaches. Previous researches attempted to make it by designing various distortion-aware structures. Noticing that the distortion changes from the equator to the poles, Su $et\ al.$ [6] designed convolution kernels of different sizes to deal with the distortion from different regions. However, the relationship between the convolution kernels’ size and the distortion regions is not a simple linear mapping. Zioulis $et\ al.$ [16] proposed OmniDepth to adopt the spherical layer in  [6] as the pre-processing module. In the omnidirectional target detection task, Benjamin $et\ al.$ [5] proposed SphereNet to correct distortion by introducing distortion invariance and spherical sampling position mode. DACF [17] designed a distortion-aware deformable convolution filter to directly perform depth regression on panoramas, getting a more accurate result.

On the other hand, some estimate omnidirectional depth using cubemap. This representation suffers from less distortion but brings discontinuity on boundaries for each face. To overcome this issue, Cheng $et\ al.$ [18] propose cube padding (cp) to utilize the connectivity between faces of the cube for image padding. Nevertheless, they ignore the characteristic of perspective projection. Spherical padding (sp) is presented in  [7] to reduce the boundary discontinuity and combined with an equirectangular map to estimate depth. Unfortunately, the equirectangular format introduces distortion again, which inevitably leads to the instability of their training.

Compared with the equirectangular map, the cubemap contains far less distortion, significantly decreasing depth estimation difficulty. However, when converting to an equirectangular map, estimating each face of the cubemap can easily lead to depth discontinuities and an inability to recover missing pixels accurately, as shown in Fig.2.

To solve this problem, another reference depth map is used to smooth the discontinuous depth. The original cube and the $\phi$ rotated one completely revise the discontinuity because they have complementary boundary discontinuity areas and a consistent context (illustrated in Fig. 1, center). Besides, we observe that when $\phi$ is equal to 45°, the revision effect comes to the best.

We define a rotation operation $\boldsymbol{R}$ representing the corresponding horizontal rotation. Besides, equirectangular-to-cube and cube-to-equirectangular transformations are donated as $\mathfrak{T}$ and $\mathfrak{T}^{-1}$, respectively. Given an equirectangular representation input $I_{equi}\in\mathbb{R}^{W\times H\times 3}$, the whole process can be expressed as follows:

	$\displaystyle D_{equi}$	$\displaystyle=$	$\displaystyle\mathfrak{T}^{-1}(\boldsymbol{f}_{DCDE}(\mathfrak{T}(I_{equi}))),$		(1)
	$\displaystyle D_{equi,\phi}$	$\displaystyle=$	$\displaystyle\boldsymbol{R}^{-1}(\mathfrak{T}^{-1}(\boldsymbol{f}_{DCDE}(\mathfrak{T}(\boldsymbol{R}(I_{equi},\phi))))),$		(2)

where $D_{equi},D_{equi,\phi}$ represents the output of DCDE module.

That is feasible but results in a substantial computational overhead. To optimize the algorithm, we quickly implement the rotation-based process by moving the map blocks (illustrated in Fig. 4) according to the conversion relationship between the sample points and spherical coordinates.

The dual-cubemap depth module is established using two independent and parallel branches, and each branch shares the same architecture. In each branch, we adopt the pre-trained ResNet-50 [13] to extract features and up-projection [7] to generate the depth from features.

However, the discontinuous depth will be caused due to the discontinuous features in each cube face. To eliminate depth discontinuity at the feature-level, we design a boundary-aware block to promote the interaction between the two branches. Specifically, we convert the cube feature maps that output from layer $i$ into equirectangular format to expose the discontinuity and fuse the two branches’ features. The input of layer $i+1$ of the two branches can be respectively expressed as:

	$\displaystyle f_{1}^{i+1}$	$\displaystyle=$	$\displaystyle f_{1}^{i}+\mathfrak{T}(\boldsymbol{R}^{-1}(\mathfrak{T}^{-1}(f_{2}^{i}))\otimes\mathfrak{T}^{-1}(f_{1}^{i})),$		(3)
	$\displaystyle f_{2}^{i+1}$	$\displaystyle=$	$\displaystyle f_{2}^{i}+\mathfrak{T}((\mathfrak{T}^{-1}(f_{2}^{i})\otimes\boldsymbol{R}(\mathfrak{T}^{-1}(f_{1}^{i}))),$		(4)

where subscript 1, 2 represent the first and second branch, respectively, and $\otimes$ is Hadamard product. We get two estimated depth maps in cubemap representation from this module.

The depth estimated from the previous module is merely a coarse result with obvious boundary and depth discontinuity, another BR module is designed to revise the boundary at pixel-level further. We adopt an encoder-decoder architecture to implement this module. The network contains 10 convolution blocks, as shown in Fig. 3(right), and each of which comprises two different convolution layers. A maxpooling or deconvolution is adopted every two layers. Besides, skip connections are applied to connect the low-level and high-level features with the same resolution by a 3×3 convolutional layer to prevent the gradient vanishing problem and information imbalance in training [19].

Depth mutations often appear at the boundary of the object. Therefore, to make the depth estimation at that area more accurate, we introduce gradient loss [20]. Besides, the reverse Huber [21] or Berhu loss [12] is also added in our objective function:

$\displaystyle\beta=\begin{cases}\left|x\right|\qquad\left|x\right|\leq c,\cr\frac{x^{2}+c^{2}}{2c}\qquad\left|x\right|>c.\cr\end{cases}$

(5)

$\displaystyle L_{grad}(y,\hat{y})=\frac{1}{n}\sum_{p}^{n}\left|g_{x}(y_{p},\hat{y_{p}})\right|+\left|g_{y}(y_{p},\hat{y_{p}})\right|$

(6)

And our loss function can be written as:

$\displaystyle L_{\beta 1}$

$\displaystyle=$

$\displaystyle\sum_{i\epsilon p}\beta(\mathfrak{T}^{-1}(D_{1}^{i}),D_{gt}^{i}),$

(7)

where $D_{1}$ is the prediction produced by the first branch; $D_{gt}$ is the ground truth; and P represents all the valid value in the ground truth map.

$\displaystyle L_{\beta 2}$

$\displaystyle=$

$\displaystyle\sum_{i\epsilon p}\beta(\mathfrak{T}^{-1}(D_{2}^{i}),D_{gt}^{i}),$

(8)

where $D_{2}$ is the prediction produced by the second branch.

$\displaystyle L_{\beta f}$

$\displaystyle=$

$\displaystyle\sum_{i\epsilon p}\beta(D_{f}^{i},D_{gt}^{i}),$

(9)

where $D_{f}$ is the final prediction.

$\displaystyle L_{grad}(D_{gt},D_{f})=\frac{1}{n}\sum_{i\in p}^{n}\left|g_{x}(D_{gt}^{i},D_{f}^{i})\right|+\left|g_{y}(D_{gt}^{i},D_{f}^{i})\right|,$

(10)

where $g_{x}$, $g_{y}$ represent horizontal gradient and vertical gradient, respectively; and n is the total number of valid values from the ground truth. The total objective function is:

$\displaystyle L_{total}=\lambda_{\beta 1}L_{\beta 1}+\lambda_{\beta 2}L_{\beta 2}+\lambda_{\beta f}L_{\beta f}+L_{grad},$

(11)

where $\lambda_{\beta 1},\lambda_{\beta 2},\lambda_{\beta f}$ are the weight of each part. While training our model, the weights are set to $\lambda_{\beta 1}=0.1,\lambda_{\beta 2}=0.1,$ and $\lambda_{\beta f}=0.8$.

We conduct our experiments on an Nvidia RTX 2080 MQ GPU. Adam optimizer is chosen, and the batch size is set as 4. The learning rate is $5\times 10^{-4}$ for the first 20 epochs and then adjusted to $1\times 10^{-4}$ for the next 20 epochs. In addition, a model dynamic storage mechanism is set up to pick the best performance model when the training is stable.

We adopt the popular evaluation metrics used by previous works [22, 16] for performance analysis, including MAE, RMSE, RMSE(log),and $\delta_{t}$, $t\in(1.25,1.25^{2},1.25^{3})$.

The experiments are carried out on Matterport3D and Stanford2D3D, two real indoor datasets. The depth values of the two datasets are collected by 3D cameras or sensors, which means there is noise or missing values in certain areas. We follow the official guidance to filter the unavailable images and limit the data range to filter out invalid values. This process can avoid the noise interfering in the experiment (According to the official description, the depth collection values range in 0 m to 10 m). Finally, we get 6319 images for training, 865 for validating, and 1632 for testing from Matterport3D; and 898 for training, 81 for validating, 365 for testing from Stanford2D3D.

We exhibit quantitative comparisons on the two datasets in Table 1, where we mark the 1st best solution in red and the 2nd best solution in blue. The results show that our method outperforms all the other methods in every metric on Matterport3D. On Stanford2D3D, we perform 1st in MAE, $\delta_{1}$, $\delta_{2}$, $\delta_{3}$ and 2nd in the remaining two metrics. Since there is more noise in Stanford2D3D, MAE can better measure the performance of the model than RMSE. Overall, our predictions are on average 4$\%$ more accurate in $\delta_{1}$ than that reported by BiFuse [7]. The comparisons experimental validates the effectiveness of our dual-cubemap approach. The two cubemaps not only focus on the distortion-free regions but also retain the information from the large FoV of the panorama as much as possible, which both contribute to improving estimation accuracy.

Fig. 5 demonstrates the qualitative comparisons of the two datasets. Note that since BiFuse [7] does not release the training code, we can only use their model trained on 360D [16] for qualitative analysis. In general, our results are visually more clear and sharper around boundaries, which can be attributed to the complementary information from the dual-cubemap and gradient loss constraint on edges.

In this section, ablation studies are performed on the following components: 1) $I_{cube1}$: the 0° rotated cubemap branch; 2) $I_{cube2}$: the 45° rotated cubemap branch; 3) BR: boundary revision module; 4) GL: gradient loss. Especially, 1) and 2) are chosen simultaneously to represent the dual-cubemap depth module.

From Table 2, we can observe that one branch with a cubemap representation image as input performs much worse than others because a single cubemap has a smaller FoV than an equirectangular map. Furthermore, the supplement information from the other cubemap has dramatically improved the performance. Since GL is only used to sharpen edges [20] without eliminating the boundary discontinuity, the BR module is proposed. It finally revises the discontinuity of boundaries and outperforms 13$\%$ in terms of MAE.

As shown in Fig. 6, the depth estimated from our dual-cubemap is much better than that from a single cubemap, but there is still visible discontinuity. Moreover, the BR module revises the discontinuity effectively, and gradient loss makes the results more accurate on edges.