Stealing Stable Diffusion Prior for Robust Monocular Depth Estimation

MDE methods can be categorized into two types: supervised and self-supervised. Eigen et al. [6] introduced the CNN-based architecture for depth estimation, laying the foundation for supervised MDE. Over time, supervised MDE methods have evolved into regression methods[5, 16] and classification methods[7, 1, 19]. In contrast, self-supervised MDE does not rely on costly depth ground truth. It generates the supervisory signal using stereo image pairs[8, 11] or adjacent frames in a video[46, 12]. These methods reconstruct the image based on the positional relationship between the cameras and the estimated depth, improving depth estimation accuracy by reducing the difference between the reconstructed image and the target image. However, both supervised and self-supervised methods are susceptible to the effects of darkness and adverse weather conditions, as depicted in Figure 2. Therefore, there is a need for RMDE methods capable of overcoming the poor performance of current approaches in challenging lighting conditions.

In recent years, significant progress has been made in Robust Monocular Depth Estimation (RMDE), with several methods demonstrating promising results. These methods can be broadly categorized into two groups: model-based and data-based approaches.

Model-based methods aim to enhance the model architecture to handle challenging conditions. DeFeat-Net [32] proposes a unified framework for learning robust monocular depth estimation and dense feature representation, specifically targeting improved performance under darkness. RNW [37] employs image enhancement techniques and an adversarial approach to enhance model performance in dark environments. WSGD [34] addresses darkness by estimating flow maps and modeling light changes between adjacent frames. MonoViT [45] utilizes Vision Transformer (ViT) [4] to extract image features, enabling the model to perform well under various weather conditions.

Data-based methods focus on leveraging additional modalities or augmenting the image signal through techniques such as domain adaptation. DEISR [20] utilizes sparse Radar data to enable depth estimation under adverse conditions. R4Dyn [9] demonstrates the benefits of weak supervision using sparse Radar data during training and the advantages of using Radar data as an additional input during inference for RMDE. DET [31] leverages thermal images to achieve depth estimation in darkness and adverse weather conditions. ADFA [33] employs adversarial training to enable the model to adapt to darkness. ADDS [22] utilizes different feature extractors to extract invariant and private features in different domains, enabling the extraction of universal image features and improving depth estimation. ITDFA [44] employs CycleGAN to translate images from daytime to other domains and extracts features from different domains to handle darkness and adverse weather. Robust-Depth [29] introduces bi-directional pseudo-supervision loss and pseudo-supervised pose loss to compensate for the performance degradation caused by the use of translated images. WeatherDepth [36] introduces contrastive learning based on domain adaptation methods. Md4all [10] achieves robust depth estimation by not distinguishing images in standard and challenging conditions, allowing the model to learn from the trained teacher network.

It is worth noting that the robustness of a model encompasses not only its ability to handle challenging conditions but also its generalization to other corruption types such as noise, blur, and digital artifacts. Several works [27, 41, 26, 2, 38] have explored the generalization of MDE models and strategies to improve their performance in various corruption types. In contrast, our work specifically focuses on enhancing the performance of MDE models in real-world weather corruptions and darkness.

DDPM is a generative model, also known as a diffusion model, that achieves image generation by performing a diffusion process in the image space. The impressive generative power of the diffusion model has led to the desire to incorporate control conditions into the generated images. In the field of text-based image generation, Rombach et al. proposed the latent diffusion model (LDM) [28], which performs the diffusion process in the latent space. They also utilize the cross-attention mechanism [35] to introduce conditions for the LDM. Their text-to-image model is now known as Stable Diffusion. To control the spatial structure of an image, Zhang et al. proposed ControlNet [43], which provides an additional control image, such as a depth map, semantic map, canny map, etc., to govern the spatial structure of the generated image. To control the style of generated images, Hu et al. proposed IP-Adapter [40], an effective and lightweight neural network architecture aimed at achieving image prompt capability for pre-trained text-to-image diffusion models. The text-to-image diffusion models can generate images in the style of the images inputted into the IP-Adapter. IP-Adapter achieves this through a decoupled cross-attention mechanism where the cross-attention layers for text features and image features are separate. Our GDT model supports depth maps, text, and prompts for black-and-white or rainy day images as conditions, enabling the generation of images that satisfy the aforementioned conditions.

In supervised monocular depth estimation, DepthNet are trained using sensor data as ground truth. The prediction process can be represented as $D=\mathcal{D}(I)$, where $I$ is the input image, $D$ is the dense depth map of $I$, and $\mathcal{D}$ is the DepthNet. For self-supervised MDE, adjacent frames in videos are used to train DepthNet. In addition to the DepthNet, a PoseNet is required to estimate the ego-motion between consecutive frames. For a frame $I_{t}$ in a video, DepthNet is used to predict the corresponding depth map $D_{t}$, and PoseNet is used to predict the camera ego-motion $P_{t\rightarrow t^{\prime}}$. The process of predicting the camera ego-motion can be expressed as $P_{t\rightarrow t^{\prime}}=\mathcal{P}(I_{t},I_{t^{\prime}})$, where $I_{t^{\prime}}$ represents the adjacent frame of $I_{t}$ sampled from $\{I_{t-1},I_{t+1}\}$, and $\mathcal{P}$ represents the PoseNet. The depth map and camera ego-motion are then used to synthesize the target image $I_{t}$. This process is described in Equation 1, where $K$ represents the camera intrinsics, $Proj(\cdot)$ is the function that outputs the 2D coordinates, and $\langle\cdot\rangle$ is the sampling operator.

Finally, the photometric loss ($pe$) is calculated as defined in Equation 2, where $\alpha$ is set to 0.85. In Monodepth2 [12], the per-pixel photometric loss $\mathcal{L}{p}$ is defined as $\min\limits{t^{\prime}}pe(I_{t},I_{t^{\prime}\rightarrow t})$.

As discussed in Section 1, GAN-based translation models have exhibited certain limitations. In this paper, we present a novel translation model called GDT (Generative Diffusion Model-based Translation). The pipeline of GDT is depicted in Figure 3. The objective of GDT is to generate a training sample $I_{g}$ that closely resembles the day-clear image $I_{d}$ in terms of depth. To achieve this, we leverage the Stable Diffusion prior and introduce several control mechanisms to transform the day-clear image into challenging conditions while preserving specific characteristics.

First, we employ the BLIP2 model [18] to obtain a scene description $T_{cap}$, which aids in preserving the content of the image. Second, we utilize the d2i model of the controlNet [43] to maintain approximate depth consistency. We have observed that the clearer the depth information, the more realistic the generated RGB image becomes. Thus, we employ PatchFusion to enhance the depth estimation obtained from the MiDaS model. Third, we address the challenge of introducing diverse styles into the generated images. We combine the $T_{cap}$ with challenging condition descriptors. However, we encountered an issue where the generated challenging condition images lacked realism. To overcome this, we introduce an image prompt for challenging conditions. In our implementation, we randomly select night images or rain images from the train dataset as image prompts. We utilize the IPAdapter model [40] to incorporate both text prompts and image prompts as inputs for image generation. The text prompt assists in preserving the content, while the image prompt facilitates style transfer.

In comparison to previous GAN-based methods, our GDT approach generates images with greater diversity and does not necessitate training for each specific scene. It can be considered a plug-and-play module. As more powerful generative diffusion models continue to emerge, our GDT method will become even more robust. The generation of the image $I_{g}$ can be expressed as follows:

Here, $SD$ represents the Stable Diffusion prior, $IP$ denotes the IPAdapter model that combines the text prompt $T_{p}$ and image prompt $I_{p}$, $CN$ corresponds to the controlNet model for depth consistency (utilizing $D_{h}$ as input), and $z$ represents the latent noise variable.

In order to steal a stable diffusion prior for robust depth estimation, we have made modifications to the architecture of Monodepth2 [12]. Additionally, we have employed a self-training strategy and introduced additional loss functions. Figure 4 provides a brief illustration of our SSD pipeline.

The nuScenes dataset [3] is divided into three scenarios: day-clear, night, and day-rain. Following the setup proposed in [10], we evaluated the performance of our model in each of these scenarios. The quantitative results are presented in Table 1.

Due to limitations imposed by the model architecture (which requires 3D convolutions) and the number of samples in the dataset, (b)PackNet’s performance shows slight improvement for night and rain scenes but is greatly reduced for day-clear scenes. The weak velocity loss in (b)PackNet contributes to obtaining metric depth. Although (c)RNW employs a unique image enhancement method and adversarial training, it performs worse than our baseline. On the other hand, (e)md4all-AD and (f)md4all-DD are simple and efficient methods that enhance the model’s robustness. Building upon the md4all approach, we have made improvements to the model architecture and translation model. Comparing (d) and (g), our model architecture (DINOv2 encoder with DPT head) demonstrates significant superiority over monoDepth2. Additionally, thanks to the pre-trained weights from the DINOv2 encoder, our depth model can extract more robust features. Comparing (g) and (h), our student model outperforms the teacher model. In the night scenario, absRel is reduced by 7.8%, and in the day-rain scenario, absRel is reduced by 17.9%. This demonstrates that the student model indeed benefits from the GDT-transformed images and effectively incorporates the stable diffusion prior.

The RobotCar dataset [24] is divided into day and night scenarios. (a), (f), and (g) demonstrate the strong generalization capability of our model architecture. Despite not being trained on night data, our model performs well and even outperforms md4all-DD. Furthermore, (g) and (h) illustrate that, despite already achieving high performance, our GDT provides a prior that enables the student model to surpass the teacher model. This leads to further improvements in depth estimation accuracy for both day and night scenarios.

The qualitative results depicted in Figure 5 and Figure 6 showcase the performance of our SSD (Single Shot Depth) model in depth estimation, specifically on the nuScenes [3] and RobotCar [24] datasets. In the nuScenes dataset, our SSD model demonstrates enhanced performance in extracting depth information from regions with low illumination, such as night scenes. Moreover, our approach effectively restores depth information even in rainy scenes. Figure 5 highlights the advantages of our SSD-T (teacher) and SSD-S (student) models over the monodepth2 and md4all-DD methods in night scenes. Our models accurately estimate the depth of cars and pedestrians, whereas the other methods fail to do so. However, in rainy scenes, SSD-T’s performance is compromised due to the superior perceptive ability of the DINOv2 architecture. SSD-T mistakenly perceives virtual images in water as objects, leading to incorrect depth estimations. Nonetheless, SSD-S, which incorporates semantic loss and teacher loss, achieves excellent results. Figure 6 displays the results obtained in both day-clear and night scenes. In the day-clear scene, both SSD-T and SSD-S successfully capture the edge of the wall in their depth maps, whereas monodepth2 and md4all-DD fail to do so. In the night scene, monodepth2 produces unreliable depth maps, whereas our SSD-T and SSD-S accurately estimate the depth of the bicycle on the right side.

We conducted ablation experiments on the nuScenes validation set to validate the effectiveness of our proposed approach. The results are presented in Table 3. In the ablation study, our metrics represent the average performance of the model across all samples in the validation set. In Method 1, we changed the original translation model of md4all to our GDT, without incorporating PatchFusion. The performance of the model improved as GDT can produce detailed images that preserve fine depth information. In Method 2, we introduced PatchFusion, which enhanced the model’s performance by generating high-resolution depth maps. These maps contribute to the preservation of fine depth details and aid GDT in producing more accurate images. During knowledge distillation, the teacher model may produce unreasonable depth estimations. In Method 3, we replaced the distillation loss with the teacher loss, resolving this issue and further improving the model’s performance. Considering the powerful generative capability of SSD, it is necessary to have a more robust backbone to produce reliable image features. DINOv2, known for its ability to generate robust image features, is suitable for our task. In Method 4, after adopting DINOv2 as our encoder, the performance of the depth model improved significantly. In Method 5, we employed semantic loss to align the image features, resulting in a further improvement in model performance.