Learning a Geometric Representation for Data-Efficient
Depth Estimation via Gradient Field and Contrastive Loss

Recent studies on monocular depth estimation use deep learning algorithms with a large-scale labeled dataset. Due to the properties of the ConvNet, the size of the receptive field decreases as the image passes multiple layers with different sizes of the convolution kernels and strides. Therefore, most depth estimation networks leverage an encoder-decoder structure to increase the size of the output depth map.

The two-scale ConvNets which generate coarse and fine depth maps each to feed additional information by concatenation from coarse to fine networks have been suggested in [18]. Later, their work is extended to three-scale ConvNets for other auxiliary estimation tasks like normal prediction or segmentation[19]. In [5], a deep residual network ResNet[20] is used as the encoder to extract features from the images and has used a novel up-projection algorithm for efficient and faster training. [2] formulates the depth estimation problem as the quantized ordinal regression and [3] applies a U-Net[21] with DenseNet[22] as the encoder to generate a high-quality depth map with complex neural networks.

Self-supervised learning based depth estimation has also been studied by estimating a camera pose or disparity maps [14, 11, 12, 13], but they need paired stereo images[11, 12, 23] or at least sequences of the monocular images[14, 13]. It means that the algorithms above cannot be trained with independently sampled monocular images.

As most images are unlabeled, significant research efforts aim to train neural networks more efficiently with a small number of labeled dataset. Self-supervised learning formulates a pretext task using only unlabeled data so that the network can learn useful visual representation from the image before task-specific supervised learning such as classification or object detection.

In [24], a single image is divided into several non-overlapped patches and the networks are trained to predict their relative positions. Follow-up studies [25, 26] also generate patches but the patches are randomly shuffled and the pretext task is to recover the original images by predicting the permutation of the patches which is similar to solving jigsaw puzzles.

A rotation pretext that rotates the input image is introduced in [7], and the network predicts the degree of applied rotation. The rotation degree is an element of a set that has a finite cardinality of specific numbers, so this poses the rotation pretext task as a classification problem, not a regression.

An exemplar task has been proposed in [8] which decreases the distance between the original RGB image (seed image) and its heavily augmented image and increases the distance between the seed and augmented image from the other RGB images. It allows the network to extract a visual representation which is invariant to a wide range of image transformations.

Recently, a visual representation learning algorithm based contrastive learning has achieved huge success in the image classification task. Contrastive learning [27] is an approach to learn representations by enforcing the attractive force to positive pairs and the repulsive forces to negative pairs. The proposed method in [28] divides the image into several non-overlapping patches and the task is to predict the pixel values of the next unseen patch. They use the target patch as the positive pair and patches from the other location in the same image or patches from the other images as negative pairs for the contrastive learning. [9] and [10] impose multiple image transformations on RGB images and set the positive pair as the augmented image from the same image and the negative pair as the augmented image from the other image. The difference is that [9] uses 2 encoders with the same structure but [10] only uses a single encoder.

We pre-train the encoder of the depth estimation network with the contrastive loss to learn the representation of the image without any supervisory signals. We adopt momentum contrastive learning suggested in MoCo[9], with 2 encoders which are query and key encoders respectively.

The encoder maps the positive and negative pairs of the image to the feature space $\mathcal{Z}$ to extract the latent vector $z$ which contains the compressed information of the input data. Although the latent vector $z$ has a low dimension, it is still vulnerable to overfitting to a specific training data domain, so the $head$ module is adopted which is a 2-layer fully-connected network followed by ReLU non-linear activation function[10]. The head module projects the latent vector $z$ to the feature space $\mathcal{H}$ which has much lower dimensionality with similar representation compared to the original feature space $\mathcal{Z}$. The contrastive loss is then calculated by the similarity between the projected latent vectors $h$ of positive and negative pairs in the low-dimensional feature space.

MoCo[9] leverages a dynamic dictionary as a queue of latent vector $h$ of input data $K=\{h_{k_{0}},h_{k_{1}},h_{k_{2}},...\}$ so that the contrastive learning can be less dependent on the batch size. The encoded key value $h_{k}$ is not discarded after training but stacked in the dictionary to formulate large negative pairs during training. The size of the dynamic dictionary $K$ is fixed, so the most outdated values are removed when the number of the latent vectors in the dictionary exceeds the maximum size. To avoid a rapid change of the key value $h_{k}$ from the same RGB image $\mathcal{I}$ for stable representation learning, MoCo[9] separates two encoders each for the query and key data. The key encoder is not updated by contrastive loss but rather by the momentum update as to keep the key latent vector $h_{k}$ consistent.

As MoCo[9] measures the similarity between the query latent vector $h_{q}$ and the key latent vector $h_{k}$ with a dot product, the contrastive loss function $\mathcal{L}_{q}$ can be formulated as Eq. (1) which is called InfoNCE[29]. The subscript $+$ indicates the positive pair, and $\tau$ is a temperature parameter to control the concentration of the distribution[30].

$$\mathcal{L}_{q}=-\mathrm{log}\frac{\mathrm{exp}(h_{q}\cdot h_{k_{+}}/\tau)}{\Sigma\mathrm{exp}^{K}_{i=0}(h_{q}\cdot h_{k_{i}}/\tau)}$$

(1)

The difference between our method and MoCo[9] is that we choose the query data as the RGB image $\mathcal{I}_{q}$ and the key data as the gradient field $G$ whereas MoCo[9] uses the augmented images by color distortion both as the query and key data. By adopting the gradient field as the key data for momentum contrastive learning, the encoder can learn the geometric representation of the image rather than semantic representation. We distinguish the type of query data from the type of key data, which are RGB image and gradient field respectively. It is because we intend to update the query encoder by the back-propagation[31] only with the error signal from RGB image that is the input of our final objective of the network, i.e., monocular depth estimation. After training with the momentum contrastive loss, the query encoder is used as the initial point of depth estimation network as shown in Fig. 2.

Previous self-supervised learning algorithms are mainly focused on capturing the semantic information of the image, so most of them formulate the unlabeled pretext task by imposing wide range of image transformations like HSV-space color randomization[8], Inception cropping[32] or Gaussian blur[10]. In this paper, we exploit a gradient field of the image as positive and negative pairs of contrastive learning so that the network can extract the visual geometric features.

To generate the gradient field $G$ of the image $\mathcal{I}$, we use Canny edge detector[15] to extract the edge of the image. Unlike the standard Canny algorithm which generates a binary mask to filter the magnitude of the gradient from the Sobel operator by comparing its value with the neighbor pixels, we modify the Canny detector to extract the value of the magnitude of the dominant gradient itself as well as its location so that the pixel of the gradient field has different intensity value according to its edge dominance. The procedure of generating a gradient field can be mathematically expressed as Eq. (2):

$$\begin{gathered}\mathcal{I}\in\mathbb{R}^{h\times w},\>\mathcal{I}_{u},\,\mathcal{I}_{v}\in\mathbb{R}^{h\times w},\\ \|E\|=\sqrt{\mathcal{I}_{u}^{2}+\mathcal{I}_{v}^{2}},\\ G=B_{Canny}\otimes\,\|E\|,\\ \end{gathered}$$

(2)

where

$\displaystyle\mathcal{I}_{u}=\frac{\partial\mathcal{I}}{\partial u},\;\mathcal{I}_{v}=\frac{\partial\mathcal{I}}{\partial v}.$

$\|E\|$ and $B_{Canny}$ denote the magnitude of the gradient from the Sobel operator and the binary result of the Canny algorithm respectively, and the operator $\otimes:(\mathbb{R}^{h\times w},\mathbb{R}^{h\times w})\mapsto\mathbb{R}^{h\times w}$ indicates the pixel-wise multiplication. Lastly, as the range of the magnitude of the gradient field is different from the input RGB image, we both normalize pixel values of the image and gradient field from 0 to 1 for a faster and stabler model learning. We visualize the magnitude of the gradient, binary mask from Canny detector and our proposed gradient field in Fig. 3.

In short, we pre-train the encoder of the depth estimation network with the momentum-based contrastive learning by[9] and the gradient field from the modified Canny algorithm[15] to learn the geometric representation of the images. Our proposed momentum contrastive learning with the gradient field is summarized in Fig. 1.

We both train the encoder in an unsupervised manner and fine-tune the entire depth estimation network with supervisory signals on the NYU Depth v2[17] dataset. The dataset consists of indoor 120K train samples and 654 test samples collected with Microsoft Kinect. We train our method on the 50K subsets of the train sample and evaluate the method on the entire 654 test samples. A resolution of the dataset is 640$\times$480 and we do not resize the input RGB image, but the output depth map is down-sampled to half resolution 320$\times$240 for training speed and computational efficiency. During the evaluation, we trim the test input image with a pre-defined center cropping by [18] for a precise evaluation.

We implement our self-supervised pre-training and monocular depth estimation networks[3, 5] on a public deep learning platform PyTorch[34]. We re-experimented both the self-supervised and depth estimation tasks to eliminate the factors that can affect the performance of the network except for the pre-trained weights of the encoder. For training an encoder with momentum-based contrastive learning, we set the batch size as 64, the size of the dynamic dictionary $K$ as 16384, and the temperature parameter $\tau$ as 0.07. We adopt the SGD optimizer with the learning rate of 0.015, the momentum of 0.9, and set the weight decay as 0.0001. For fine-tuning the depth estimation network, we train depth estimation networks with the same input size 640$\times$480 and use the Adam optimizer with the learning rate of 0.0001. We set the batch size as 4 for training and set the batch size as 1 for evaluation.

We compare the results on monocular depth estimation according to the pre-trained weights of the encoder with two different structures of networks, [3] [5]. The work in [3] adopts DenseNet-161[22] as the encoder for high-quality complex depth estimation and [5] uses ResNet-50[20] as the encoder and its up-projection module as the decoder for a real-time operation. TABLE I shows some meaningful results. Our method seems to show worse performance than supervised ImageNet[33] pre-training due to the difference in the dataset size and the presence of the labels: NYU Depth v2[17] contains 120K images whereas ImageNet contains 14M images with classification labels. However, our method outperforms the existing state-of-the-art self-supervised algorithms on all the quantitative metrics. We improves the accuracy by 5% $\sim$ 30% trained with [3] and by 2% $\sim$ 11% with [5]. Interestingly, Rotation[7] and Exemplar[8] pretext degrade the performance of the depth estimation network compared to random initialization by [16]. It indicates that pre-training the network to capture the semantic information of the image for classification or object detection does not always provide a better initialization for other tasks such as depth estimation. Some qualitative results of our proposed method and prior self-supervised methods trained with [3] are shown in Fig. 4.

In Fig. 5, we visualize the kernels of the first layer of DenseNet-161[22] learned by our proposed self-supervised pre-training and fine-tuning for depth estimation. We observe that the patterns of the kernels which play a critical role in capturing specific information of the image barely change. The absolute value of the kernels are slightly changed by task adaptation, but the general patterns and shapes of the kernels are not affected by the loss signals from the depth estimation. It indicates that how we pre-train the encoder of the entire network determines the dominant representation of the images that the network can capture regardless of task-specific fine-tuning.

We evaluate our model on a subset of the entire dataset with annotated depth labels compared to random initialization by [16]. We intentionally restrict the amount of the labeled data varying from 1% to 10% of NYU Depth v2[17] for fine-tuning the depth estimation networks. Both training and evaluation have been done with [3] where DenseNet-161[22] is used as the encoder. From Table II, our method trained with only 1% and 5% of labels outperforms the random initialization by [16] trained with 3% and 10% of labeled data respectively in all the accuracy metrics. The gain in data-efficiency of the network increases as the number of the labeled data becomes smaller, where the data- efficiency is triple with 1% of labeled data and double with 5% of labeled data. It is because the network is trained by the gradient descent, so the parameter initialization becomes more important as the number of labeled training data becomes smaller.

To demonstrate the domain generalization capability, we evaluate our proposed method on the outdoor dataset even though the depth estimation network is pre-trained and fine-tuned only on the indoor dataset. We use Make3D dataset [35] for evaluation which consists of 534 outdoor images, 400 training data and 134 test data. For evaluation on Make3D, we exploit three commonly used error metrics [36, 37] and central crop[23]. We adopt the depth estimation network from [3] with the encoder of DenseNet-161[22] training on NYU Depth v2[17] and no further fine-tuning on images and labels from Make3D is done.

Table III confirms that the proposed pre-training method on Make3D still shows a better performance than the random[16], rotation[7] and MoCo[9] initialization. As shown in Fig. 6, depth estimation network pre-trained with our method generates much plausible depth output and preserves more details such as edges of the buildings and branches of the trees compared to random initialization and even to the noisy ground truth depth map. It is because our proposed method trains ConvNet to extract geometric features of the image such as vertical or horizontal structural information which are robust regardless of indoor and outdoor environments.