Structure Guided Lane Detection

To solve the problem of lane detection, traditional methods are usually based on hand-crafted features by detecting shapes of markings and fitting the spline.  Veit et al. (2008) presents a comprehensive overview of features used to detect road markings. And Wu and Ranganathan (2012) uses Maximally Stable Extremal Regions features and performs the template matching to detect multiple road markings. However, there approaches often fail in unfamiliar conditions.

With the development of deep learning, methods Pizzati and García (2019); Van Gansbeke et al. (2019); Guo et al. (2020) based on deep neural networks achieve progress in lane detection. SCNN Pan et al. (2018) generalizes traditional deep layer-by-layer convolutions to enable message passing between pixels across rows and columns. ENet-SAD Hou et al. (2019) presents a knowledge distillation approach, which allows a model to learn from itself without any additional supervision or labels. PolyLaneNet Tabelini et al. (2020) adopts a polynomial representation for the lane markings, and outputs polynomials via the deep polynomial regression. UltraFast Qin et al. (2020) treats the process of lane detection as a row-based selecting problem using global features. CurveLanes Xu et al. (2020) proposes a lane-sensitive architecture search framework to automatically capture both long-ranged coherent and accurate short-range curve information.

In these methods, different lane representations are adopted and some structural information is considered for performance improvement. However, these methods are usually based on the powerful learning ability of neural networks to learn the fitting or shapes of lanes, and the role of scene-related structural information for lanes has not been paid enough attention to and discussed.

For adapting to different styles of lane annotation, we introduce a new box-line based method for lane representation. Firstly, we calculate the minimum circumscribed rectangle $R$ (“box”) with the height $h$ and width $w$ for the lane instance $L_{lane}$. For this rectangle, center line $L_{center}$ (“line”) perpendicular to the short side is obtained. And the angle between the positive $X$-axis and $L_{center}$ in clockwise direction is $\theta$. In this manner, $L_{center}$ provides the position of the lane instance, and $h$ and $w$ restrict the areas involved. Based on $R$ and $L_{center}$ , lane prediction based on points, masks, markers, grids and other formats can be performed. In this paper, the solution based on key points of lane detection is taken just because of the point-based styles of lane annotation in public datasets (e.g., CULane TuSimple (2017) and Tusimple Pan et al. (2018)).

Inspired by existing methods Li et al. (2019); Chen et al. (2019); Qin et al. (2020), we define key points of the lane instance with equally spaced $y$ coordinates $Y=\{y_{i}\}$ and $y_{i}=\frac{H}{P-1}\cdot i(i=1,2,...,P-1)$, where $P$ means the number of all key points through image height, which is fixed on images with same height $H$ and width $W$. Accordingly, the $x$ coordinates of the lane is expressed as $X=\{x_{i}\}$. For the convenience of expression, the straight line equation of $L_{center}$ is defined as

$$ax+by+c=0,a\neq 0\ or\ b\neq 0$$

(1)

where $a$, $b$ and $c$ can be easily computed by $\theta$ and any point on $L_{center}$. Next, when the $y$ coordinate of the center line is $y_{i}$, we can compute the corresponding $x$ coordinate as

$$x_{i}=L_{center}(y_{i})=\frac{-c-by_{i}}{a},a\neq 0.$$

(2)

Then, we define the offset of $x$ coordinate $\Delta X$ between the lane $L_{lane}$ and center line $L_{center}$ as

	$\displaystyle\Delta X$	$\displaystyle=\{\Delta x_{i}\}=\{x_{i}-\frac{-c-by_{i}}{a}\},$		(3)
	$\displaystyle X$	$\displaystyle=\{\frac{-c-by_{i}}{a}\}+\Delta X.$

Therefore, based on $L_{center}$ and $\Delta X$, we can calculate the lane instance $L_{lane}$. Usually, it is easier to learn $L_{center}$ and $\Delta X$ than the directly fitting key points of $L_{lane}$.

To see Fig. 2, SGNet takes ResNet He et al. (2016) as the feature extractor, which is modified to remove the last global pooling and fully connected layers for the pixel-level prediction task. Feature extractor has five residual modules for encoding, named as $\mathcal{E}_{i}(\pi_{i})$ with parameters $\pi_{i}(i=1,2,...,5)$. To obtain larger feature maps, we convolve $\mathcal{E}_{5}(\pi_{5})$ by a convolutional layer with 256 kernels of $3\times 3$ and then $\times 2$ upsample the features, followed by an element-wise summation with $\mathcal{E}_{4}(\pi_{4})$ to obtain $\mathcal{E}_{4}^{\prime}(\pi_{4}^{\prime})$. Finally, for a $H\times W$ input image, a $\frac{H}{16}\times\frac{W}{16}$ feature map is output by the feature extractor.

In order to learn the lane representation, there are two main ways to learn the center line $L_{center}$ and $x$ offset $\Delta X$. The first way is to learn the determined $L_{center}$ directly with angle, number and position regression, which is usually difficult to achieve precise results because of the inherent difficulty of regression tasks. The second way is based on mature detection tasks, using dense anchors to classify, regress and then obtain proposals representing the lane instance. And the second one has been proved to work well in general object detection tasks, so we choose it as our base model.

To learn the center line $L_{center}$ and $x$ offset $\Delta X$ well, we propose a novel vanishing point guided anchoring mechanism (named as VPG-Anchoring). The vanishing point (VP) provides strong characterization of geometric scene, representing the end of the road and also the “virtual” point where the lanes intersect in the distance. Since VP is the intersection point of lanes, lanes in the scene must pass through VPs, and lines that do not pass through VPs are not lanes in the scene with high probability. Therefore, dense lines radiated from VPs can theoretically cover all lanes in the image, which is equivalent to reducing the generation space of anchors from $\mathbb{R}^{H\times W\times N_{proposal}}$ to $\mathbb{R}^{N_{proposal}}$. $N_{proposal}$ represents the number of anchors generated at one pixel.

As shown in Fig. 2, the features map $\mathcal{E}^{\prime}_{4}(\pi^{\prime}_{4})$ is feed to VPG-Anchoring. In the mechanism, VP is predicted by a simple branch, which is implemented by a multi-scale context-aware atrous spatial pyramid pooling (ASPP) Chen et al. (2018) followed by a convolutional layer with 256 kernels of $3\times 3$ and a softmax activation. The VP prediction branch is denoted as $\phi_{\mathcal{V}}({\pi_{\mathcal{V}}})$ with parameters $\pi_{\mathcal{V}}$.

Usually, VP is not annotated in lane datasets, such as CULane Pan et al. (2018), so we average the intersection points of the center lines of all lane instances and get the approximate VP. In addition, a single point is usually difficult to predict, so we expand the area of VP to a radius of 16 pixels and use segmentation algorithm to predict. To achieve this, we expect the output of $\phi_{\mathcal{V}}({\pi_{\mathcal{V}}})$ to approximate the ground-truth masks of VP (represented as $G_{\mathcal{V}}$) by minimizing the loss

$\displaystyle{\mathcal{L}}_{\mathcal{V}}=BCE(\phi_{\mathcal{V}}({\pi_{\mathcal{V}}}),G_{\mathcal{V}}),$

(4)

where $BCE(\cdot,\cdot)$ represents the pixel-level binary cross-entropy loss function.

In order to ensure that generated anchors are dense enough, we choose a $W_{anchor}\times W_{anchor}$ rectangular area with VP as the center, and take one point every $S_{anchor}$ to generate anchors. For each point, anchors are generated every $A_{anchor}$ angle ($A_{anchor}\in[0,180]$) as shown in Fig. 4.

In this way, anchors are targeted, intensive and not redundant, compared with general full-scale uniform generation and even specially designed methods for lanes Li et al. (2019). Note that anchors run through the whole image, and only the part below VP is shown for convenient display in Figs. 2 and 4.

In order to classify and regress the generated anchors, we extract high-level feature maps based on $\mathcal{E}_{4}(\pi_{4})$ with several convolutional layers. The feature map is named as $\text{F}_{\mathcal{A}}\in\mathbb{R}^{H^{\prime}\times W^{\prime}\times C^{\prime}}$, where $H^{\prime},W^{\prime}$ and $C^{\prime}$ are the height, width and channel of $\text{F}_{\mathcal{A}}$. For each anchor $L_{lane}$, the channel-level features of each point on anchors are extracted from $\text{F}_{\mathcal{A}}$ to obtain lane descriptor $\text{D}_{\mathcal{A}}\in\mathbb{R}^{H^{\prime}\times C^{\prime}}$, which are used to classify the existence $Conf^{L_{lane}}$ and regress $x$ offsets $\Delta X^{L_{lane}}$ including the length $len$ of lanes. To learn these, we expect the output to approximate the ground-truth existence $GConf^{L_{lane}}$ and $x$ offsets $G\Delta X^{L_{lane}}$ by minimizing the loss

	$\displaystyle{\mathcal{L}}_{\mathcal{C}}$	$\displaystyle=\sum_{L_{lane}=0}^{L-1}BCE(Conf^{L_{lane}},GConf^{L_{lane}}),$		(5)
	$\displaystyle{\mathcal{L}}_{\mathcal{R}}$	$\displaystyle=\sum_{L_{lane}=0}^{L-1}SL1(\Delta X^{L_{lane}},G\Delta X^{L_{lane}}),$

where $SL1(\cdot,\cdot)$ means smooth L1 loss and L means the number of proposals. Finally, Line-NMS Li et al. (2019) is used to obtain the finally result with confidence thresholds.

In order to further improve lane perception, we ask for the structural relationship between scenes and lanes, and deeply explore the pixel-level, lane-level and image-level structures.

We compare our approach with state-of-the-arts including DeeplabV2 Chen et al. (2017), SCNN Pan et al. (2018), FD Philion (2019), ENet-SAD Hou et al. (2019) , PointLane Chen et al. (2019), RONELD Chng et al. (2020), PINet Ko et al. (2020), ERFNet-E2E Yoo et al. (2020), IntRA-KD Hou et al. (2020), UltraFast Qin et al. (2020), CurveLanes Xu et al. (2020), Cascaded-CNN Pizzati et al. (2019) and PolyLaneNet Tabelini et al. (2020).

We compare our approach with 10 state-of-the-art methods on CULane dataset, as listed in Tab. 1. Comparing our ResNet34-based method with others, we can see that the proposed method consistently outperforms other methods across total and almost all categories. For the total dataset, our method is noticeably improved from 74.80% to 77.27% compared with the second best method. Also, it is worth noting that our method is significantly better on Crowd (+2.31%), Arrow (+2.17%) and Night (+3.79%) compared with second best methods, respectively. In addition, we also obviously lower FP on Cross by 3.78% relative to the second best one. As for Curve, we are slightly below the best method (ERFNet-E2E), which conducts special treatment for curve points while maybe damaging other categories. Moreover, our method has a faster FPS than almost all results. These observations present the efficiency and robustness of our proposed method and validate that VPG-Anchoring and multi-level structures are useful for the task of lane detection.

Some examples generated by our approach and other state-of-the-art algorithms are shown in Fig. 5. We can see that lanes can be detected with accurate location and precise shape by the proposed method, even in complex situations. These visualizations indicate that the proposed lane representation has a good characterization of lanes, and also show the superiority of the proposed method.

Moreover, we list the comparisons on Tusimple as shown in Tab. 2. It can be seen that our method is competitive in highway scenes without adjustment, which further proves the effectiveness of structural information for lane detection.

To validate the effectiveness of different components of the proposed method, we conduct several experiments on CULane to compare the performance variations of our methods.