StairNetV3: Depth-aware Stair Modeling using Deep Learning

Different from the approach of only focusing on line extraction in StairNet and StairNetV2, StairNetV3 is dedicated to achieving complete modeling of stair structures. We use a simple neural network to simultaneously conduct line extraction and surface extraction, distinguishing between convex lines/concave lines and riser surfaces/tread surfaces. For stair lines, endpoint information is crucial in the post-processing of network results. We hope that the network can directly learn the stair line endpoints, so we introduce Gaussian kernels.

The input of the network is a full-color RGB image, denoted as $I\in[0,255]^{H\times W\times 3}$, and its height and width are $H$ and $W$, respectively. Our goal is to obtain a heatmap that reflects the position of stair lines, denoted as $O\in[0,1]^{\frac{H}{S_{H}}\times\frac{W}{S_{W}}\times 1}$, where $S_{H}$ and $S_{W}$ are the output strides in the width and height directions, respectively, determining the heatmap size. To express the endpoint information of stair lines, we hope that the response of stair line endpoints in the heatmap is 1, while the response of the rest of the stair lines decreases as the distance from the endpoint increases. Therefore, we apply Gaussian kernels to determine the value of each pixel in the heatmap, as shown in equation 1.

Where $(x,y)$ represents the pixel coordinates in the original image, $Y$ represents the pixel value at the corresponding pixel coordinates $(\biggl{\lfloor}\frac{x}{S_{W}}\biggr{\rfloor},\biggl{\lfloor}\frac{y}{S_{H}}\biggr{\rfloor})$ in the heatmap, $(p_{x},p_{y})$ represents the pixel coordinates of the closer endpoint of stair line $L$ to $(x,y)$ in the original image, and $\sigma$ is the standard deviation and can be adjusted according to demand [34]. According to equation 1, it can be seen that when $(x,y)$ is the midpoint of stair line $L$, $Y$ takes the minimum value. Since the threshold for distinguishing positive and negative samples is usually set to 0.5, we hope that the $Y$ value at the midpoint is greater than 0.5. Therefore, let $\sigma=\frac{1}{2}||L||_{2}=\frac{1}{2}\sqrt{(p_{x1}-p_{x2})^{2}+(p_{y1}-p_{y2})^{2}}$, where $(p_{x1},p_{y1})$ and $(p_{x2},p_{y2})$ represents the left and right endpoints of stair line $L$ in the original image, respectively. So when $x=\frac{p_{x1}+p_{x2}}{2}$ and $y=\frac{p_{y1}+p_{y2}}{2}$, $Y_{min}=e^{-\frac{1}{2}}\approx 0.6$. For the prediction of stair step surfaces, we add a branch to the network for generating semantic segmentation masks. The branch has four upsampling operations to obtain tensors that match the input image size, and the segmentation results have three classifications, including riser, tread, and background. The above process is shown in Figure 2.

Different from the network architecture with two input branches in StairNetV2, StairNetV3 only has an input branch of RGB image, and the branch of depth image is set in the output branches. To achieve effective depth-awareness, the network contains two parallel backbones, which are stacked with several squeeze-and-excitation (SE)-ResNeXt [35, 36] blocks with dilated convolution. Both backbones follow an encoder-decoder structure, where one backbone is used for depth-awareness and the other backbone is used for obtaining stair modeling information. The interaction between the two backbones includes two types. Through the shared input branch, the stair modeling backbone get the implicit supervision from depth information by the way of gradient back propagation. In the decoder of the depth-awareness backbone, a depth-aware block (DAB) is applied to aggregate depth information from different feature layers and directly pass it to the encoder of the stair modeling backbone, and the feature fusion is achieved through a selective module [17] to directly supervise the learning of stair modeling. The network output includes 6 branches. The depth-awareness backbone outputs an estimated depth image with a size of 512 $\times$ 512 $\times$ 1. For the stair modeling backbone, to make the network classification results more robust, we decouple the output branches of the two types of stair lines, so there are a total of 4 branches, including 2 heatmap branches with a size of 64 $\times$ 32 $\times$ 1 and 2 location branches with a size of 64 $\times$ 32 $\times$ 4. The heatmap branch outputs the confidence that the cell contains stair line and the location branch outputs the normalized coordinates of the contained stair lines. In addition, the stair modeling backbone also includes a semantic segmentation branch with a size of 512 $\times$ 512 $\times$ 3 for classifying stair step surfaces. The network architecture is shown in Figure 3.

StairNetV3 has multiple output branches, which can be divided into branches for classification and branches for location. For the classification task, we use the Binary Cross Entropy (BCE) loss function with a sigmoid function. For the regression task, we use the Mean Squared Error (MSE) loss function. For the stair line extraction task, we regard it as a regression problem, and the loss function is described as equation 5:

Where, $M$ and $N$ represents the height and width of the output feature maps, namely, 64 and 32, and $i$ and $j$ represents the position of the cell in the feature maps. $L_{MSE}$ represents MSE loss function, $p_{ij}$ represents the confidence predicted by each cell, and $p_{ij}^{*}$ is the corresponding ground truth. $h_{ij}$ represents the prediction probability of containing stair line and its values are 0 and 1, which represent containing stair line and without stair line, respectively. $t_{ij}$ represents the normalized coordinates of stair line endpoints in each cell, and $t_{ij}^{*}$ is the corresponding ground-truth. For the confidence loss, we give different weights to positive and negative samples to adjust the imbalance between positive samples and negative samples, and $\alpha_{1}$ and $\alpha_{2}$ represents the weight coefficients of positive samples and negative samples, respectively. In our research, $\alpha_{1}$=15, $\alpha_{2}$=5.

For location loss, we inherit the loss function with dynamic weights [17] from StairNetV2, as shown in equation 6.

Where, $x_{ij}$ represents the predicted normalized coordinates in the horizontal direction, and $x_{ij}^{*}$ is the corresponding ground truth. $y_{ij}$ represents the predicted normalized coordinates in the vertical direction, and $y_{ij}^{*}$ is the corresponding ground truth. $\lambda_{1}$ and $\lambda_{2}$ represents the corresponding dynamic weights, respectively. In initial epoch, $\lambda_{1}$=$\lambda_{2}$=10, After each epoch, the dynamic weights are adjusted automatically according to the evaluation indexes. The rest parameters have the same meanings as equation 5.

For the semantic segmentation task of the stair step surfaces, its essence is a classification task. We apply BEC loss and its loss function is shown in equation 7:

Where, $N$ represents the width or height of the output feature maps, namely, 512. $C$ represents the number of classifications, which is 3, $i$ and $j$ represents the position of the pixel in the feature maps, and $k$ represents the classification. $L_{BCE}$ represents the BCE loss function. $m_{ij}$ represents the prediction probability of a pixel belonging to a certain classification, and $m_{ij}^{*}$ is the corresponding ground-truth.

For the depth estimation task, it is essentially a regression task, and we still use the MSE loss, and the loss function is shown in equation 8:

Where $N$, $i$ and $j$ have the same meanings as in equation 7, $L_{MSE}$ represents the MSE loss function, $d_{ij}$ represents the predicted normalized pixel value, and $d_{ij}^{*}$ is its corresponding ground-truth. $\psi$ is a weight coefficient, and in this study, $\psi$=10. Overall, the loss function $L$ of StairNetV3 can be represented as equation 9:

Where, $L_{l}^{b}$ and $L_{l}^{r}$ represent the loss of convex lines and the loss of concave lines respectively.

The data used to evaluate our method all come from the Stair dataset with depth maps [37]. Based on the dataset, we remove the images taken under no light source conditions, and finally retain 2276 RGB-D image pairs as the training set and 556 RGB-D image pairs as the validation set. To evaluate the performance of point cloud reconstruction, we use the Realsense D435i depth camera [38] to collect 154 RGB-D image pairs with a resolution of 640 $\times$ 480 as the test set. During the collection, we record the pose of camera and the linear transformation relationship of depth image generation for point cloud reconstruction. To implement the semantic segmentation of stair step surfaces, we make pixel-level segmentation labels for the training set and the validation set, and the classifications include stair riser surfaces, stair tread surfaces and background. The organized dataset is named as the RGB-D stair dataset [39].

To accurately evaluate our method, we apply several normal evaluation metrics. For stair lines, we use precision, recall, and IOU with a confidence level of 0.5 for evaluation. This is consistent with the evaluation methods of StairNet and StairNetV2. For stair step surfaces, we use pixel accuracy (PA) and mean pixel accuracy (MPA) for evaluation.

We implement StairNetV3 with the PyTorch framework on a workstation that has an i9 13900 CPU and an RTX 3090 GPU. During training, we use Adam optimizer [40] with weight decay of $10^{-6}$ to optimize the network. The learning rate is initialized as 2.5 $\times 10^{-4}$ and halved every 50 epochs. The batch size is set to 4. The input images have the size of 512 $\times$ 512, and the random horizontal flip with a probability of 0.5 is applied for data augmentation.

To verify the role of the DAB and explore its optimal combination with implicit deep supervision, we conduct several ablation experiments. We take a model without DAB as the baseline, and other models place the DAB after the second downsampling, the third downsampling and the fourth downsampling for comparison. All ablation experiments follow the same settings and procedures, and the results are shown in Table 1.

Table 1 shows that the position of the DAB can impact on network performance. When the DAB is placed after the second downsampling, the model performance is even comparable to that without DAB. When the DAB is placed after the third downsampling and fourth downsampling, the model performance is significantly improved. This is mainly because the DAB placed after the third and fourth downsampling can achieve feature fusion at different levels. The DAB contains deep features of the decoder, and the depth features of DAB are closer to the features of original depth image. These features correspond to the shallow features of encoder. Through the feature fusion of deep features and shallow features in the encoder, we realize a method of multi-level feature map fusion. Compared with the fusion of shallow features after the second downsampling, the fusion with deep features after the third downsampling and fourth downsampling can further improve the network performance.

In this section, we test the performance of the models, mainly including the accuracy on the validation set and the inference speed. To meet various application scenes, we provide three versions of StairNetV3, including StairNetV3-large, StairNetV3-medium, and StairNetV3-small, which can be obtained by adjusting the width factor of the network. To prevent the group convolution in the residual block from becoming depthwise separable convolution [41], which may reduce the feature expression ability of the small model. In StairNetV3, the width factor not only affects the number of channels in each layer of the network, but also the number of groups in each group convolution to ensure that the number of filters in each group remains unchanged. We use the precision, recall, and IOU at a confidence of 0.5, as well as PA and MPA as evaluation indicators, the results are shown in Table 2.

For the network inference speed test, we use a desktop platform with an i9 13900 CPU and an RTX 3090 GPU, and a mobile platform with Ryzen7 7735H CPU and RTX4060 laptop GPU for testing. To reflect the inference speed of the network under different workloads, we test the model inference time in two scenes: the batch size is set to 1, and the batch size is set to the maximum value that the GPU can handle. The former corresponds to online image or video stream applications and the latter corresponds to offline image or video stream applications. The detailed test results are shown in Table 3.

As we can see from Tables 2 and  3, as the model becomes lighter, the model accuracy gradually decreases, the speed gradually increases, and the performance fluctuation of the models is stable. When batch size is set to 1, the difference in inference time between the three models is not significant, allowing for flexible selection based on accuracy demands, real-time demands, and computational resource allocation. When batch size is set to the maximum value, there is a significant difference in the inference speed of the models. The large and medium models are suitable for tasks that prioritize accuracy, while the small model is suitable for tasks that prioritize speed. Some visualization results of StairNetV3-large are shown in Figure 7, which intuitively demonstrates the adaptability of StairNetV3 to various stairs and shooting conditions. Particularly, even without depth images as input, StairNetV3 can still show stable performance in scenes with fuzzy visual clues, such as night and descending stairs.

After StairNetV3 obtains the segmentation results of the stair step surfaces, the stair modeling can be realized through point cloud reconstruction. Since the pixels of the image and the points of the point cloud correspond one-to-one, the PA and MPA results in Table 2 can be considered as the segmentation precision of the stair point cloud on the validation set. Because the images in the validation set are padded and scaled and the linear transformation relationships used to generate the depth images are not saved, these images cannot be truly used for point cloud reconstruction. We test point cloud reconstruction in the test set and because the test set does not have ground-truth labels, we only test the running speed, and the results are shown in Table 4. Some visualization results of the stair step surface segmentation are shown in Figure 8.

As can be seen from Table 4, the point cloud reconstruction algorithm has superior real-time performance on both desktop and mobile platforms. This is benefit from our approach of implementing point cloud segmentation by combining image segmentation with point cloud reconstruction, which avoids the slow speed and large data processing drawbacks of direct point cloud segmentation. From Figure 8, it can be intuitively seen that our method can accurately segment point clouds belonging to the stair step surfaces in various scenes.

In this section, we compare StairNetV3 with some previous methods, including traditional image processing methods, methods combining traditional image processing and deep learning, and the deep learning method StairNetV1. We implement traditional image processing method using Gabor filters, Canny edge detection and Hough transform, and then we combine them with YOLOv5 [24] as a combination of deep learning method. The outputs of all the methods in the comparison experiments are adjusted to be consistent with StairNetV3, and all the methods are carried out on the desktop platform described in section 4.4. The test dataset is the RGB-D stair dataset, and the detailed experimental results are shown in Table 5.

It can be seen that deep learning methods have much higher accuracy than traditional methods. Because traditional methods have heavy reliance on manually designed rules and the selection of thresholds, it is difficult to perform well in large dataset containing complex and variable scenes. While, deep learning methods extract features by learning, which can greatly improve adaptability to different scenes compared to traditional methods. Among the deep learning methods, in fact, the feature representation method with Gaussian kernels used by StairNetV3 is not conducive to improving accuracy. This is because the confidences of some positive samples are weakened, especially the positive samples in the middle of the stair lines have a confidence of only 0.6. For the feature representation method of StairNet, the confidences of positive samples are all 1, so there is a strong distinction between the foreground and background. Even under these circumstances, due to the awareness of depth information, StairNetV3 still has better performance than StairNet. Furthermore, StairNetV3-small with less parameters and faster speed still surpasses StairNet 1 $\times$ in accuracy, which proves the effectiveness of our method.

Some visualization results of the comparison experiments are shown in Figure 9. It can be seen that traditional method has a certain degree of false detection and missed detection in different scenes due to their sensitivity to manually designed rules and thresholds. The deep learning method combined with traditional image processing can eliminate false detection to some extent, but the missed detection is still serious. The deep learning methods show great adaptability to various complex scenes, and StairNetV3 has better performance in environments with fuzzy visual cues compared to StairNet, such as night and descending stairs. In addition, when there is ground with similar texture to stairs, StairNetV3 has less false detection compared to StairNet.