Dusk Till Dawn: Self-supervised Nighttime Stereo Depth Estimation using Visual Foundation Models

Loosely, we classify previous work as being based either on domain adaptation or image enhancement. Adaptation-based approaches seek to overcome the domain shift between day and night using synthesised data to align day/night or clear/inclement weather image encodings [16, 11]. They aim to obtain the same features across different conditions. Synthesised data is created using standard datasets (i.e. [17]), and a Generative Adversarial Network (GAN). [18] uses a Monodepth2 [4] architecture with an adversarially-trained nighttime image encoder. Features are aligned using a GAN to generate day images from night, and a discriminator to enforce similarity. [19] emulates this approach, but trains in the output space as well as the feature space. [20] uses the GAN outputs to train two feature extractors, a day-night invariant feature extractor that forms the backbone for depth prediction, as well as night and day style extractors for training. Outside of our broader categorisation, [21] uses a feature space (rather than image space) contrastive loss to improve domain generalisation beyond that of [4]. Enhancement-based approaches aim to improve performance over day models by taking greater account of scene lighting, for example, by isolating illumination information within the image. [22] uses a learned image enhancement and adapted masking from [4]. [23] estimates the illumination change between the consecutive nighttime images to relax assumptions of photometric loss for better depth estimation.

Concurrent state-of-the-art approaches [16] and [11] also consider weather conditions in their approaches. Both adapt [4] for their all-day unified networks, using GANs to augment for both weather and lighting conditions. [16] uses a semi-augmented warping to minimise GAN-induced inconsistency between consecutive frames, and uses raw inputs for pose estimation to minimise error propagation. Similarly, [11] uses daytime depth estimation to distill depth knowledge to nighttime using image translation networks. [24] uses a complex, partially adversarial architecture, with a learned image enhancement to estimate illumination and uncertainty, which is then masked from the loss.

All approaches mentioned so far are monocular, even if stereo images are used during training. In terms of supervised stereo depth estimation, [25] uses a transformer and 4D correlation feature matrix, including an iterative refinement inspired by [26], to derive depth, flow, and disparity. [27] uses a coarse-to-fine approach with a hierarchical network and adaptive group correlation for getting fine disparities. [28] builds an attention-based cost volume to suppress redundant information and enhance matching-related information. [10] unifies stereo and optical flow approaches based on 2D convolution, avoiding the memory cost of 3D convolution. [29] trains a model using day datasets that have ground truth depth and further uses domain adaptation to work on day to night. Our method does not use any daytime data, domain adaptation, or ground truth depth for training.

In contrast to existing approaches, we use visual foundation models, DINO [13] and DINOv2 [14], as feature extractors for the input images. The models are trained on the ImageNet [30] and Facebook LVD-142M [14] datasets, respectively. In our experiments, we use the pretrained small-models of DINO and DINOv2, with patch sizes of $8$ and $14$, respectively. For the rest of the paper, we refer to DINO ViT-S/8 (patch size 8) as the encoder unless otherwise stated. Our encoder has a conv-layer followed by a series of 12 transformer layers. Given an image $I\in\mathbb{R}^{H\times W\times 3}$ with height $H$ and width $W$, the conv-layer converts the image into overlapping $8\times 8$ patches with stride $4$, resulting in a feature-map $\hat{f}\in\mathbb{R}^{h^{\prime}\times w^{\prime}\times 384}$ where $h^{\prime}=H/4$, $w^{\prime}=W/4$. After processing $\hat{f}$ through the first 6 transformer layers, the estimated output feature-map $\tilde{f}\in\mathbb{R}^{h^{\prime}\times w^{\prime}\times 384}$ is subsampled with stride 2 in the height and width dimensions, giving an $8\times$ downscaled feature map that is further processed by 6 more transformer layers resulting $f\in\mathbb{R}^{h\times w\times 384}$, where $h=H/8$, $w=W/8$.

Selection of the transformer layers is based on [31]. They observe that various encoder layers act similarly to hierarchical CNN layers in capturing local-to-global information as the depth of the network increases. Deeper layers capture semantics, and shallow layers capture local details (including positions). Middle layers tend to carry both. For accurate disparity estimation, we use both deep and middle layer features, $(f,\tilde{f})$, for stereo matching.

The projection head takes feature vectors $f,\tilde{f}$ from the encoder and projects them into lower dimensional space $\mathbb{R}^{D}$ where $D<384$. This is done because (1) when PCA is performed on the $384$ dimensions, we observe the 10 first principal components to explain more than $50\%$ of the total variance, suggesting it is safe to project to lower dimensional space and (2) the computational complexity of the stereo matcher during inference and training will be reduced.

The projection head consists of two conv-layers with kernel size as $1\times 1$, and $\mathrm{ReLU}$ as activation function in the middle. We use $D=128$ as the output description dimension and the same projection head for both left and right features.

The stereo matcher takes the multi-scale left and right features $\{(f_{l},f_{r}),(\tilde{f}_{l},\tilde{f}_{r})\}$ as input and estimates the disparity map $\tilde{d}_{r}\in\mathbb{R}^{h^{\prime}\times w^{\prime}\times 1}$ in three stages. First, features are enhanced with cross-image feature context using a transformer module. Then, a disparity map $d_{g}$ is estimated at $1/8$ scale using the $(f_{l},f_{r})$ features and global matching. Lastly, the disparity is bilinearly interpolated by $2\times$ and corrected for the interpolation artifacts with a residual disparity estimated using $(\tilde{f}_{l},\tilde{f}_{r})$ and local matching. These modules are explained in detail in the following sections.

The disparity output $\tilde{d}_{r}$ is upsampled to give final output disparity $d\in\mathbb{R}^{H\times W\times 1}$. Instead of bilinear interpolation, which can result in blurry borders, we use the learnable convex upsampling method proposed by RAFT [26]. In it, two (3 $\times$ 3) convolutional layers are used to predict a mask and perform softmax over the 9 neighbours of a given pixel.

In order to make the estimated disparity spatially smooth while preserving the edges, the common edge-aware Disparity Smoothness Loss $L_{\rm smooth}$ from [3] is used. Finally, the total loss is $L_{\rm total}=L_{\rm photo}+\beta_{1}L_{\rm reg}+\beta_{2}L_{\rm smooth}$, where $\beta_{1}$ balances how much we spread the features on the unit-sphere and is set as $\beta_{1}=1$, and we choose $\beta_{2}=0.1$.

Throughout our experiments, we train on the RobotCar Dataset [17] and test on both the Robotcar [17] and MS2 [37] datasets. Details of each are given below.

The framework is trained using the Robotcar dataset for 20 epochs with an input image resolution of $192\times 320$. We used a batch size of $8$ and the Adam [38] optimizer with a constant learning rate of $1e-4$.

To the best of our knowledge, there is no self-supervised system that estimates depth from night-time stereo-images. We therefore compare our method with a classical method, Semi-Global Matching (SGM) [34], and 3 state-of-the-art supervised methods: UniMatch-Stereo²²2We used their in-the-wild use stereo-matching with refinement model from GitHub during the evaluation [25], IGEV-Stereo [10], and Sharma et al. [29]. Note that these methods are trained end-to-end only for the purpose of stereo-matching, with large amounts of ground truth data. Also, we found that the disparity estimation of Unimatch-Stereo and IGEV-Stereo drops drastically when tested at the same resolution as ours. Therefore, we use $2\times$ more resolution while reporting their results.

The depth-estimation performance is evaluated qualitatively and quantitatively on both the Robotcar [17] and MS2 [37] datasets. Quantitative evaluation is carried out using the metrics proposed in [39]. For every test image, existing methods compute their metrics as the mean of all valid pixels up to a given depth range ($50\rm m$ in our evaluation), usually only with a sparse set of LiDAR points. Taking the overall mean is sensible when pixels are uniformly distributed over the depth range, however, this is not the case for the RobotCar dataset [17] as shown in Fig 4(a). The same effect is observed for most autonomous driving datasets, including the MS2 dataset [37]. This is due to the fact that the majority of pixels in an image are occupied by points that are very close to the camera. Evaluating performance on this kind of data skews the evaluation, as the performance on nearby points outweighs the performance on the far-away points. To mitigate this, we propose the use of depth bins splitting the given depth range into $M$ bins. Letting $\mathrm{x}$ be a metric, unweighted ($U$) and weighted ($W$) evaluation metrics for a given test image can be written as:

where $Z$ is the total number of valid pixels, $M$ is the total number of bins and $N_{i}$ is the total number of valid pixels in the $i$-th depth-bin. We set $M=10$, i.e., each bin covers $5\rm m$ depth. We reported the numbers from both metrics, and we did not use eigen or garg crops [4] during our evaluation.

We performed various ablation studies on RobotCar dataset to understand the impact of different modules. The results are shown in Table III. Base model uses DINO-V1 as the encoder, with stereo-matching and upsampling, trained using $L_{\rm photo}$ alone. W/ mask had features masked before stereo-matching, also trained with $L_{\rm photo}$ alone. W/ mask + reg used the regularization loss $L_{\rm reg}$ with $L_{\rm photo}$. The improvement in both error and accuracy metrics explains the importance of the proposed masking and regularization loss for accurate depth estimation.

Interestingly, training with a DINOv2 [14] encoder yielded a performance drop despite being pretrained on a larger amount of data than DINO [13] as shown in the last row of Table III. Also, the overexposed areas and lane markings create undesirable edges in the estimated disparity maps (can be observed in Fig. 5, Fig. 6). This can be a limitation of the commonly used edge-aware disparity smoothness loss $L_{\rm smooth}$. We currently leave these issues for future investigation.