Applying Surface Normal Information in Drivable Area and Road Anomaly Detection for Ground Mobile Robots

Ground mobile robots, such as sweeping robots and robotic wheelchairs, are playing significant roles in improving the quality of human life [1, 2, 3]. Visual environment perception and autonomous navigation are two fundamental components for ground mobile robots. The former takes as input sensory data and outputs environmental perception results, with which the latter automatically moves the robot from point A to point B. Among the environment perception tasks for ground mobile robots, the joint detection of drivable areas and road anomalies is a critical component that labels the image as the drivable area or road anomaly at the pixel-level. In this paper, the drivable area refers to a region where ground mobile robots can pass through, while a road anomaly refers to an area with a large difference in height from the surface of the drivable area. Accurate and real-time drivable area and road anomaly detection could avoid accidents for ground mobile robots.

With the great advancement of deep learning technologies, many effective semantic segmentation networks that could be used for the task of drivable area and road anomaly detection have been proposed [4, 5]. Specifically, Chen et al. [4] proposed DeepLabv3+, which combines the spatial pyramid pooling (SPP) module and the encoder-decoder architecture to generate accurate semantic predictions. However, most of the networks simply use RGB images, which suffer from degraded illumination conditions [6]. Recently, some data-fusion convolutional neural networks (CNNs) have been proposed to improve the accuracy of semantic segmentation. Such architectures generally utilize two different types of sensory data to learn informative learning representations. For example, Wang et al. [5] proposed a novel depth-aware CNN to fuse depth images with RGB images, which has improved the performance of semantic segmentation. Thus, the fusion of different modalities of data is a promising research direction that deserves more attention.

In this paper, we first introduce a novel module named Normal Inference Module (NIM), which generates surface normal information from dense depth images with high accuracy and efficiency. The surface normal information serves as a different modality of data, which can be deployed in existing semantic segmentation networks to improve performance, as illustrated in Fig. 1. To validate the effectiveness and robustness of our proposed NIM, we use our previous ground mobile robots perception (GMRP) dataset¹¹1https://github.com/hlwang1124/GMRPD [1] to train twelve state-of-the-art CNNs (eight single-modal CNNs and four data-fusion CNNs) with and also without our proposed NIM embedded. The experimental results demonstrate that our proposed NIM can greatly enhance the performance of the aforementioned CNNs for the task of drivable area and road anomaly detection. Furthermore, our proposed NIM-RTFNet ranks 8th on the KITTI road benchmark²²2www.cvlibs.net/datasets/kitti/eval_road.php [7] and exhibits a real-time inference speed. The contributions of this paper are summarized as follows:

• •

  We develop a novel NIM and show its effectiveness on improving the semantic segmentation performance.

• •

  We conduct extensive studies on the impact of different modalities of data on semantic segmentation networks.

• •

  Our proposed NIM-RTFNet greatly minimizes the trade-off between speed and accuracy on the KITTI road benchmark.

In this section, we briefly overview twelve state-of-the-art semantic segmentation networks, including eight single-modal networks, i.e., fully convolutional network (FCN) [8], SegNet [9], U-Net [10], DeepLabv3+ [4], DenseASPP [11], DUpsampling [12], ESPNet [13] and Gated-SCNN (GSCNN) [14], as well as four data-fusion networks, i.e., FuseNet [15], Depth-aware CNN [5], MFNet [16] and RTFNet [17].

Our proposed NIM, as illustrated in Fig. 2, can generate surface normal information from dense depth images with both high precision and efficiency. The most common way of estimating the surface normal $\mathbf{n}=[n_{x},n_{y},n_{z}]^{\top}$ of a given 3D point $\mathbf{p}^{\text{C}}=[x,y,z]^{\top}$ in the camera coordinate system (CCS) is to fit a local plane: $\mathbf{n}^{\top}\mathbf{p}^{\text{C}}+\beta=0$ to $\mathbf{Q}=[\mathbf{p}^{\text{C}},\mathbf{q}_{1},\dots,\mathbf{q}_{k}]^{\top}$, where $\mathbf{q}_{1},\dots,\mathbf{q}_{k}$ are a collection of $k$ nearest neighboring points of $\mathbf{p}^{\text{C}}$. For a pinhole camera model, $\mathbf{p}^{\text{C}}$ is linked with a pixel ${\mathbf{p}}^{\text{I}}=[u,v]^{\top}$ in the depth image $\mathbf{Z}$ by [18]:

$$z\begin{bmatrix}{\mathbf{p}}^{\text{I}}\\ 1\end{bmatrix}=\begin{bmatrix}f_{x}&0&u_{\text{o}}\\ 0&f_{y}&v_{\text{o}}\\ 0&0&1\end{bmatrix}\mathbf{p}^{\text{C}}.$$

(1)

A depth image can be considered as an undirected graph $\mathcal{G}=(\mathcal{P},\mathcal{E})$ [19], where $\mathcal{P}=\{\mathbf{p}^{\text{I}}_{11},\mathbf{p}^{\text{I}}_{12},\dots,\mathbf{p}^{\text{I}}_{mn}\}$ is a set of nodes (vertices) connected by edges $\mathcal{E}=\{(\mathbf{p}^{\text{I}}_{ij},\mathbf{p}^{\text{I}}_{st})\ |\ \mathbf{p}^{\text{I}}_{ij},\mathbf{p}^{\text{I}}_{st}\in\mathcal{P}\}$. Please note that in this paper, $\mathbf{p}^{\text{I}}_{ij}$ are simply written as $\mathbf{p}^{\text{I}}$. Plugging (1) into the local plane equation obtains [20]:

$$\frac{1}{z}=-\frac{1}{\beta}\bigg{(}n_{x}\frac{u-u_{\text{o}}}{f_{x}}+n_{y}\frac{v-v_{\text{o}}}{f_{y}}+n_{z}\bigg{)}.$$

(2)

Differentiating (2) with respect to $u$ and $v$ leads to

$$\begin{split}\frac{\partial 1/z}{\partial u}=-\frac{1}{\beta f_{x}}n_{x}\approx\frac{1}{{\mathbf{Z}}(\mathbf{p}^{\text{I}}+[1,0]^{\top})}-\frac{1}{{\mathbf{Z}}(\mathbf{p}^{\text{I}}-[1,0]^{\top})}=g_{u},\\ \frac{\partial 1/z}{\partial v}=-\frac{1}{\beta f_{y}}n_{y}\approx\frac{1}{{\mathbf{Z}}(\mathbf{p}^{\text{I}}+[0,1]^{\top})}-\frac{1}{{\mathbf{Z}}(\mathbf{p}^{\text{I}}-[0,1]^{\top})}=g_{v}.\end{split}$$

(3)

Rearranging (3) results in

$$\begin{split}n_{x}\approx-\beta f_{x}g_{u},\ \ \ n_{y}\approx-\beta f_{y}g_{v}.\end{split}$$

(4)

Given a pair of ${\mathbf{q}_{i}}$ and ${\mathbf{p}^{\text{C}}}$, we can work out the corresponding ${n_{z}}_{i}$ as follows:

$${n_{z}}_{i}=\beta\Bigg{(}f_{x}\frac{\partial 1/z}{\partial u}\frac{\Delta{x_{i}}}{\Delta{z_{i}}}+f_{y}\frac{\partial 1/z}{\partial v}\frac{\Delta{y_{i}}}{\Delta{z_{i}}}\Bigg{)},$$

(5)

where ${\mathbf{q}_{i}}-\mathbf{p}^{\text{C}}=[\Delta{x_{i}},\Delta{y_{i}},\Delta{z_{i}}]^{\top}$. Therefore, each neighboring point of $\mathbf{p}^{\text{C}}$ can produce a surface normal candidate as follows [21]:

$$\mathbf{n}_{i}=\begin{bmatrix}&-f_{x}\frac{\partial 1/z}{\partial u}\\ &-f_{y}\frac{\partial 1/z}{\partial v}\\ &f_{x}\frac{\partial 1/z}{\partial u}\frac{\Delta{x_{i}}}{\Delta{z_{i}}}+f_{y}\frac{\partial 1/z}{\partial v}\frac{\Delta{y_{i}}}{\Delta{z_{i}}}\\ \end{bmatrix}.$$

(6)

The optimal surface normal

$$\hat{\mathbf{n}}=\begin{bmatrix}\sin\theta\cos\phi\\ \sin\theta\sin\phi\\ \cos\theta\end{bmatrix}$$

(7)

can, therefore, be determined by finding the position at which the projections of $\bar{\mathbf{n}}_{i}=\frac{{\mathbf{n}}_{i}}{||{\mathbf{n}}_{i}||_{2}}=[\bar{n}_{x_{i}},\bar{n}_{y_{i}},\bar{n}_{z_{i}}]^{\top}$ distribute most intensively [6]. The visual perception module in a ground mobile robot should typically perform in real time, and taking more candidates into consideration usually makes the inference of $\hat{\mathbf{n}}$ more time-consuming. Therefore, we only consider the four neighbors adjacent to $\mathbf{p}^{\text{I}}$ in this paper. $\hat{\mathbf{n}}$ can be estimated by solving [6]

$$\underset{\phi,\theta}{\operatorname*{arg\,min}}\sum_{i=1}^{4}-\hat{\mathbf{n}}\cdot{\bar{\mathbf{{n}}}}_{i},$$

(8)

which has a closed-form solution as follows:

$$\phi=\arctan\bigg{(}\frac{\sum_{i=1}^{4}\bar{n}_{y_{i}}}{\sum_{i=1}^{4}\bar{n}_{x_{i}}}\bigg{)},$$

(9)

$$\begin{split}\theta=\arctan\Bigg{(}\frac{1}{\sum_{i=1}^{4}\bar{n}_{z_{i}}}\bigg{(}\sum_{i=1}^{4}\bar{n}_{x_{i}}\cos\phi+\sum_{i=1}^{4}\bar{n}_{y_{i}}\sin\phi\bigg{)}\Bigg{)}.\end{split}$$

(10)

Substituting (10) and (9) into (7) results in the optimal surface normal inference, as shown in Fig. 2.

In this paper, we proposed a novel module NIM, which can be easily deployed in various CNNs to refine semantic image segmentation. The experimental results demonstrate that our NIM can greatly enhance the performance of CNNs for the joint detection of drivable areas and road anomalies. Furthermore, our NIM-RTFNet ranks 8th on the KITTI road benchmark and exhibits a real-time inference speed. In the future, we plan to propose a more feasible and computationally efficient cost function for our NIM.