An Explicit Method for Fast Monocular Depth Recovery in Corridor Environments

In the context of long corridor/hallway scenarios, aside from ground edges, there often exist additional linear features such as door frames and objects, which can introduce interference with the detection of ground edge lines. Consequently, it becomes imperative to establish a Region of Interest (ROI) within the scene. This selection process is based on empirical observations. Due to geometric perspective, ground edge lines typically manifest in the lower-to-middle section of the image, converging from the bottom sides towards the center. Relying on this a priori knowledge, an ROI with a height of H within the image is designated, encompassing rows from the middle to the bottom of the image, as shown in Figure$~{}\ref{fig2}(\textbf{a})$.

For the extraction process, this study employs a line feature detection algorithm based on the Hough transform. The Hough transform is a methodology designed to extract linear features from images. It leverages the duality between points and lines, mapping the discrete pixel points along a straight line in image space to curves in Hough space through parameter equations. Subsequently, the intersection points of multiple curves in Hough space are mapped back to straight line equations in image space, thereby forming the detected straight lines.

Before performing the Hough transform, the image is initially binarized and subjected to Canny edge detection thus expediting the process of Hough transformationas shown in Figure$~{}\ref{fig2}(\textbf{b})$. Subsequently, linear features are extracted through the Hough transform,as shown in Figure$~{}\ref{fig2}(\textbf{c})$, and the extracted features undergo linear fitting to yield the set of linear features, denoted as $S_{l}$.

Where,${\theta}_{k},{pt}_{k}^{0}$are the inclination angle of $l_{k}$,and the the pixel coordinates of the starting point of $l_{k}$, respectively, as illustrated in Figure$~{}\ref{fig2}(\textbf{d})$.

Based on scene priors, a reasonable range for setting the length and inclination angle of the lines is established, which is used to filter the lines within set $S_{l}$. If the image is strict symmetry of axis, the length of the edge lines $L\left(l_{k}\right)$ and the angle ${\theta}_{k}$ should satisfy the following conditions:

According to (2), the left and right line are picked out. And the edge line set is $S_{le}=\{l_{l}\left({\theta}_{l},{pt}_{l}^{0}\right),l_{r}\left({\theta}_{r},{pt}_{r}^{0}\right)\}$, where $l_{l}$ and $l_{r}$ are left and right line, respectively.

The primary function of the virtual camera is to ensure consistency in the imaging process across different scenes, currently predominantly employed within deep learning-based computer vision methodologies. In 2023, BEV-LaneDet [30] introduced the concept of the virtual camera in the context of 3D lane detection tasks on the Bird’s Eye View (BEV) plane. Due to variations in camera intrinsic parameters, installation positions, and camera poses, images captured by the same scene may have different dimensions and scaling ratios. The BEV-LaneDet method employs a deep neural network that requires image parameters to be as consistent as possible with those in the training dataset. This necessitates aligning the camera position, pose, and height above the ground during imaging to avoid substantial disparities between the predictive accuracy of the deployed model and that observed during offline training.

The virtual camera is manually defined, with its intrinsic parameters, installation position, and orientation preconfigured. Prior to training and inference with deep neural networks, images are initially projected onto the imaging space of the virtual camera through perspective transformation. This process ensures that the input images to the model exhibit consistency.

In this study, the concept of the virtual camera is introduced to achieve algorithmic consistency across various scenes. The intrinsic coordinate system for both the real and virtual cameras is established as right-down-forward. The coordinate definitions for the two types of cameras and their respective images are depicted in Figure 3. Within this hypothesis, the width of the long corridor remains constant, and the virtual camera is positioned at the center of the scene, equidistant from the left and right walls. Its optical axis is directed straight ahead along the length of the corridor, while maintaining consistent pitch angles, camera intrinsic parameters, and mounting height as the current real camera.

To achieve the pose transformation from the real camera to the virtual camera, this study begins by projecting ground edge feature points onto the imaging plane of the virtual camera. Subsequently, a geometric error model is constructed, and the maximum likelihood estimation of the pose transformation is computed through an iterative optimization process.

In the assumption of this study, the virtual camera is positioned at the center of the scene, ensuring that the edge lines in the image maintain its symmetry in the virtual camera. To fully utilize this structural feature and ensure algorithm consistency across different input images, the original image is initially transformed into the imaging space of the virtual camera. After obtaining the reference depth plane of the virtual camera, the depth plane information is then reprojected onto the current image. This process ultimately yields the depth distribution of the current image. Therefore, prior to conducting depth estimation, it is necessary to acquire the pose transformation from the current camera to the virtual camera.

Let $P^{W}$ represent a spatial point within the current scene, the pixel coordinate $P^{C}\left(u,v\right)$ of space point $P^{W}\left(x,y,z\right)$ is calculated as

Where, K is the camera internal parameter matrix, $f_{x}$ and $f_{y}$ are the pixel focal lengths in the x and y directions, respectively, corresponding to the physical focal length f of the camera. And the cx and cy represent the pixel offsets of the optical center in the image of the camera. In general, the horizontal and vertical dimensions of the camera sensor have equal pixel sizes, in which case $f_{x}$=$f_{y}$. The rotation matrix R and displacement t are used to represent the pose transformation from the current camera to the virtual camera, and the current frame is projected onto the plane of the virtual camera. The projection result is calculated according to (3)

Where $s^{V}$ is the scale in the transformed virtual camera. In the assumption of the virtual camera, the disparity in pose between the virtual camera and the real camera arises from the yaw angle $\phi$ and the displacement $\tau$ in the x-direction. In which case, R and t are calculated as

As depth has not been recovered yet, the images lack scale information. In order to unify the scales in both cameras, a assumption is accepted of that for every pixel in the original image, its corresponding original spatial point lies in front of the camera at a distance of 1 meter. It means, for any spatial point $P^{W}\left(x,y,z\right)$, $z\equiv 1$. Therefore, the scale factor $s^{V}$ for the virtual camera is calculated as:

Based on equations (2) to (5), the transformation from the current frame image pixel point $pt^{C}(u,v)$ to the virtual camera image pixel point $pt^{V}(u^{V},v^{V})$ can be obtained as follows:

Where,$\xi={1}/\left({f_{t}\cos{\phi}-u\sin{\phi}-c_{x}\cos{\phi}}\right)$,$f_{t}=f_{x}=f_{y}$.Each pixel of $l_{l}\left({\theta}_{l},pt_{l}^{0}\right)$,$l_{r}\left({\theta}_{r},pt_{r}^{0}\right)$along the edge can be transformed into the virtual camera according (7).

Within the assumption, the optical center of the virtual camera oincides with the symmetry axis of the corridor. As a result, the symmetry of the road edge lines is preserved in the virtual camera imaging space. The axis of symmetry is located at position $u=\frac{W}{2}$ on the image plane of virtual camera, where W represents the image width. Based on this structured feature, the geometric residual is calculated. As shown in Figure 4., Set $pt_{k}^{lt}=\left(u_{k}^{l}.v_{k}^{l}\right)$ as a point in $l_{l}$, and its symmetry point about $u=\frac{W}{2}$ is $pt_{k}^{lt}$. Substituting $v=v_{k}^{l}$ into the equation of the right line, $pt_{k}^{lt1}$ can be obtained, and the distance D between two points can be calculated. According to (7), D is a function of both the yaw angle $\phi$ and the displacement $\tau$ in the x-direction. By uniformly selecting N feature points at intervals of $l_{l}$, the sum of D calculated for these N feature points yields symmetry error the left line $E_{L}$. Similarly, the symmetry error of the right line $E_{R}$ can be obtained. Ultimately, this process yields the symmetry geometric error $E_{G}$.

Once the mathematical model for the symmetry-based geometric error $E_{G}$ is established, leveraging the prior features of the scene, a range of variability for the yaw angle $\phi$ and the displacement $\tau$ is defined. Nonlinear optimization is employed to compute the maximum likelihood estimation values $\widehat{\phi}$ and $\widehat{\tau}$ for the yaw angle $\phi$ and displacement $\tau$. Iterative calculations are performed within the range of variability for both parameters to minimize $E_{G}$. The resulting yaw angle and displacement that yield the smallest $E_{G}$ are considered as the maximum likelihood estimation results.

After obtaining the estimated value of the heading Angle and displacement of the heading Angle, the edge points are projected to the virtual camera space through the formula(4)(5), and $S^{V}=\left(pt_{k}^{Vl},pt_{k}^{Vr}\right),k=0,1,2...,n$.

In the virtual camera space, there are now pixel coordinates along two ground edges. Using the ground plane hypothesis, the points along each edge are assumed to be coplanar, and the 3D spatial coordinates of each point meet $\forall P^{W}_{k}\left(x_{k},y_{k},z_{k}\right)\in S^{W},y_{k}=0$. According to the installation position and aperture Angle information of the camera, the 3D spatial coordinates of each edge point corresponding to the virtual camera can be solved by geometric method. The depth plane can then be constructed.

The space coordinate system is selected as the lower left front, the camera height is known as h, the vertical aperture Angle is $\theta_{v}$ . Assume that the pitch Angle of the virtual camera in the current scene is $\theta_{p}$ , and for a pixel point $pt_{0}$ at the bottom of the image, its pixel coordinate is $\left(u_{t},v_{t}\right)$ meet $0\leq u_{t}\leq W$,$v_{t}=H$ , W,H are the image height and image width respectively, and its depth is calculated as

The theoretical method of Inverse Perspective Transform (IPM) is used to calculate the depth of each edge point, as shown in Figure  5. For any point in the set of edge points $S^{V}=\left(pt_{k}^{Vl},pt_{k}^{Vr}\right),k=0,1,2...,n$, its pixel coordinate is $\left(u_{k}^{V},v_{k}^{V}\right)$, and its depth is calculated according to the inverse perspective transform.

The equation is constructed to solve according to the perspective geometry as (12)

Where , $\eta_{k}=f_{y}\Delta v_{k},\Delta v_{k}=H-v_{k}^{V}$is the depth in space at the bottom of the imaging plane 1m away from the optical center of the camera,

Solving (9),$\Delta z$ is calculated as

As shown in (14), the calculation of edge point depth $\Delta z$ is related to camera elevation Angle $\theta_{p}$. In this paper, $\Delta z$ and $\theta_{p}$ are estimated simultaneously by nonlinear optimization. In the corridor, the road surface in the scene is flat and the pitch Angle variation range is small. The iterative optimization algorithm is designed to iteratively calculate the geometric residual of 3D space points within the interval $\left[\lambda_{1},\lambda_{2}\right]$, taking $\alpha$ as the step length, and calculate the optimal $\theta_{p}$,to estimate the result.

Set the initial value of $\theta_{p}$, according to (14) for the edge point pixel set $S^{V}$, in which all pixels recover the depth distribution, according to camera imaging model (3), the 3D spatial coordinates of edge point $pt_{k}^{Vl}$ are calculated as

And the geometry residual is calculated as

The approximate optimal estimate of $\theta_{p}$ can be obtained by iterative optimization

In the image space of the virtual camera, the spatial points in the same depth plane retain the parallel characteristics, and the contour points of the same depth are connected to build a depth plane. The depth plane $\Gamma_{k}^{V}$ is guided by the contour lines. The depth plane $\Gamma_{k}^{V}$ is determined by the left and right edge points $\left\{pt_{k}^{Vl}\left(u_{k}^{Vl},v_{k}^{Vl}\right),pt_{k}^{Vr}\left(u_{k}^{Vr},v_{k}^{Vr}\right)\right\}$, and is uniquely determined, $Dp\left(\Gamma_{k}^{V}\right)$ is the depth of $\Gamma_{k}^{V}$, $Dp\left(\Gamma_{k}^{V}\right)=z_{k}$.

The depth plane $\Gamma_{k}^{V}$, obtained in the virtual camera imaging space, is inversely transformed according to (4-5), and the set of depth planes in the real camera image can be obtained: $S_{\Gamma}^{C}=\left\{\Gamma_{k}^{C}\right\},k=1,2,3,...,L$. In a real camera image, if the pixel $pt_{k}^{C}\left(u_{k}^{C},v_{k}^{C}\right)$ belongs to the depth plane $\Gamma_{k}^{C}$ then equation (18) is a necessary condition.

Where $f\left(u_{k}^{Cl}\right)=\alpha u_{k}^{Cl}-\beta$ is a linear transform of $u_{k}^{Cl}$ , and $\alpha$ , $\beta$ are empirical coefficients.

In the corridor scene, the spatial point distribution is relatively ideal, ignoring the influence of dynamic objects, occlusion, etc., and taking (18) as sufficient and necessary conditions of $pt_{k}^{cl}\in\Gamma_{k}^{C}$ , each pixel in the image is classified.

After pixel classification is completed, it is determined whether it is a ground point according to the coordinates of each pixel. The condition that $pt_{i}^{V}\left(u_{i}^{V},v_{i}^{V}\right)$ is a ground point is

For the pixel point $pt_{k}^{C}\left(u_{k}^{C},v_{k}^{C}\right)\in\Gamma_{k}^{C}$, its depth is calculated by interpolation method (20).

In the formula, $\sigma$ is a nonlinear coefficient. When the depth plane is sufficiently dense, linear interpolation method is adopted. If the point is a ground point, then

Otherwise,