BézierFormer: A Unified Architecture for 2D and 3D Lane Detection

In BézierFormer, we represent lane curves as Bézier Curves. Bézier curve of order $N$ uses $N+1$ control points $\left(c_{1},c_{2},...,c_{N+1}\right)$ to represent a curve $S$ and is defined by:

Variable $t$ ranges from 0 to 1, representing the Bézier curve being sampled from the start to the end. Coefficient $b_{n,N}(t)$ is the Bernstein basis polynomial of degree $n$ given by:

We adopt the classic cubic Bézier curve, which uses four control points $\left(c_{1},c_{2},c_{3},c_{4}\right)$ to represent a curve. For 2D lane detection, $c_{1},c_{2},c_{3},c_{4}$ are all 2D vectors representing the $xy$ coordinates of control points. For 3D lane detection, they are 3D vectors representing $xyz$ coordinates. The Bézier curve’s ability to represent lane curves without relying on variables $x$, $y$ or $z$ allows for a unified representation of 2D and 3D lanes of any orientation.

As shown in Figure 2, BézierFormer consists of a feature extractor and Bézier curve decoder. The feature extractor extracts multi-scale features of input images through a backbone network and a transformer encoder containing multi-scale deformable self-attention. The generated features are denoted as $X=\left\{x_{l}\right\}_{l=1}^{L}$, where $L$ represents the number of feature map scales. The decoder consists of $N_{layer}$ homogeneous layers indexed starting from 1. Decoder layer_i receives features $X$, Bézier control point queries and corresponding curve embeddings coming from Decoder layer_i-1 as inputs. The Bézier control point queries model the shapes and positions of lane curves and are denoted as $C_{i-1}=\{\left(c_{1},c_{2},c_{3},c_{4}\right)_{i-1,j}\}_{j=1}^{N_{query}}$. $N_{query}$ is the number of queries, which means the maximum number of lane curves that BézierFormer can detect. Curve embeddings contain lane features and are denoted as $E_{i-1}=\{e_{i-1,j}\}_{j=1}^{N_{query}}$. Decoder layer_i outputs more precise control points $C_{i}$ and better embeddings $E_{i}$ for the next layer. For the first decoder layer, initial embeddings $E_{0}$ are randomly initialized learnable parameters. Initial control points $C_{0}$ could be initialized by $C_{0}=MLP\left(X\right)$. Because $C_{0}$ is related to the input image, this way brings a good generalization ability.

Bézier curve decoder consists of $N_{layer}$ layers with the same structure. It iteratively refines curve embeddings and control point queries. We employ a classification head for category recognition and use Bézier curve sampling defined by Eq.(1) to produce dense lane curve points for detection results.

Taking Decoder layer_i as an example, the inputs are $X$, $C_{i-1}$, and $E_{i-1}$. To leverage the geometric information of lane curves in self-attention and Bézier curve attention, the FV sampling module first sample $N_{ref}$ points for each query. Then, using sine positional encoding and a MLP, we fuse the positions of these $N_{ref}$ points and obtain the query’s positional encoding. The formula is:

After positional encoding, the relationships among lanes are modeled via multi-head self-attention, which enhances $E_{i-1}$ to better curve embeddings $E^{\prime}_{i-1}$ with global context:

$e$ and $e^{\prime}$ are single embeddings in $E_{i-1}$ and $E^{\prime}_{i-1}$. $m$ is the index of the attention head. $W_{m}\in\mathbb{R}^{C_{e}\times C_{v}}$and $W^{\prime}_{m}\in\mathbb{R}^{C_{v}\times C_{e}}$ are learnable parameters, where $C_{e}$ and $C_{v}=C_{e}/M$ are the feature dimensions of the embedding and the key in attention respectively. The attention weights $A_{m,j}\propto\exp\{\frac{e_{j}^{T}U_{m}^{T}V_{m}e_{j}}{\sqrt{C_{v}}}\}$ are normalized to $\sum_{j=1}^{N_{query}}A_{m,j}=1$, and $U_{m},V_{m}\in\mathbb{R}^{C_{v}\times C_{e}}$ are learnable parameters.

Subsequent Bézier curve attention employs the $N_{ref}$ points produced by FV sampling module as reference points. It collects curve features of each query from $X$ and updates $E^{\prime}_{i-1}$ to better embeddings $E_{i}$ for the next layer. A classification head calculates the category vectors $V_{i}$ from $E_{i}$. A simple FFN predicts the offset $\Delta C_{i}$, then the control points are refined as $C_{i}=\Delta C_{i}+C_{i-1}$. To achieve layer-by-layer refinement, $V_{i}$ and $C_{i}$ of each layer need to be supervised.

Bézier curve attention mechanism aims to better capture features of the slender lane curves represented by Bézier curves. We refrain from directly using vanilla deformable attention due to two primary reasons that render it suboptimal for extracting slender lane features. Firstly, deformable attention originates from object detection and generates only one reference point per object, but one reference point is insufficient to describe a slender lane curve comprehensively. Secondly, the reference points of deformable attention are adaptively generated, which causes slow convergence and less accurate lane feature extraction. To overcome these shortcomings, Bézier curve attention mechanism leverages Bézier curve sampling and Bézier curve attention operation to sample and fuse multiple reference points along the curve for accurate and comprehensive lane features.

The Bézier curve attention operation does not involve any interaction among $\{{e^{\prime}}_{i-1,j}\}_{j=1}^{N_{query}}$, so we simplify the explanation of this operation by considering the update of a single $e^{\prime}_{i-1,j}$. Let $\{r_{k}\}_{k=1}^{N_{ref}}$ be the $N_{ref}$ reference points. As shown in the left-top part of Figure 2, the features $\{\{f_{k,l}\}_{k=1}^{N_{ref}}\}_{l=1}^{L}$ of the reference points at different scales of $X$ are sampled, then two MLPs calculate $f_{k,l}$ to obtain $N_{sample}$ attention location offsets $\{o_{k,l,s}\}_{s=1}^{N_{sample}}$ and attention weights $\{w_{k,l,s}\}_{s=1}^{N_{sample}}$ for $r_{k}$ at scale $l$. $\{o_{k,l,s}\}_{s=1}^{N_{sample}}$ are added to $r_{k}$ to get the attention locations $\{p_{k,l,s}\}_{s=1}^{N_{sample}}$. To update $e^{\prime}_{i-1,j}$, the formula is:

For simplicity, $e^{\prime}$ and $e$ represent $e^{\prime}_{i-1,j}$ and $e_{i,j}$, respectively. $m$, $W_{m}$ and $W^{\prime}_{m}$ have the same meanings as they have in Eq.(4). The dynamically generated attention weights $W_{m,k,l,s}$ are normalized to $\sum_{k=1}^{N_{ref}}\sum_{s=1}^{N_{sample}}W_{m,k,l,s}=1$. Bézier curve attention operation extracts lane curve features from FV features and updates curve embeddings $E^{\prime}_{i-1}$ via Eq.(5). Enhanced curve embeddings $E_{i}$ are subsequently used for control points refinement and category recognition.

We present a lane regression loss derived from Chamfer Distance (CD) to better regress Bézier curve and use Focal Loss [20] for classification. [17] shows that regressing the sampled points can achieve better results than directly regressing Bézier control points. However, this method may sometimes be limited to optimizing the distance between local points, while ignoring the overall shape and location fit between two curves. We give an intuitive explanation of this limitation in the Appendix. We propose calculating CD between curves, as it considers each curve a whole entity to get the shortest distance from a point to the curve. As CD is calculated between point sets instead of continuous curves, $N_{dis}$ points $\{p_{i}^{A}\}_{i=1}^{N_{dis}}$ and $\{p_{i}^{B}\}_{i=1}^{N_{dis}}$ on curve $A$ and $B$ are sampled. Inspired by [3], we give the lane curve a width $e$ and normalize CD to IoU, which we call Chamfer IoU, or CIoU for short. It is defined by:

The regression loss between a pair of predicted curve $S_{pred}$ and ground truth curve $S_{gt}$ can be formulated as:

However, the computation of CIoU ignores the order among the sampled points. To ensure the correct order of the sampled points, we add two simple geometric constraints.

$len()$ means calculating the curve length. $\hat{p}_{1},\hat{p}_{N_{dis}}$ represent the normalized endpoints of the curve. Then, we get regression loss as $L_{reg}=L_{loc}+L_{len}+L_{endpoint}$ and total loss is the sum of the losses of all decoder layers. The formula is:

To match predictions with ground truth, we adopt SimOTA [21] to dynamically assign $topk$ predictions to each ground truth with the matching cost $cost_{i,j}=L_{reg_{i,j}}+L_{cls_{i,j}}$. Specifically, we set $topk=1$ for an NMS-free pipeline.

In this section, we conduct all the studies with ResNet18.

We implement BézierFormer based on MMDetection [26]. For reproducibility, we give the detailed experimental settings, including hyperparameters of the training and testing process, data augmentation strategies, and network settings of BézierFormer in Table VI.

For a more intuitive understanding of the advantage of Chamfer IoU-based lane regression loss, Figure 4 visualizes different losses. Figure 4(c) shows that the distances between two curves’ endpoints are relatively larger than other locations, contributing to a more effective learning process compared with Figure 4(a) and Figure 4(b).

For a fair comparison with polynomial and row-based lane representation, we employ the same feature extractor and design delicated decoders for them, similar with Bézier curve decoder as shown in Figure 5.

For polynomial representation, we refer to LSTR [16] and use $\left(k,f,m,n,b^{{}^{\prime}},b^{{}^{\prime\prime}},\alpha,\beta\right)$ to formulate a curve. LSTR uses vertical $y$ as variable to describe a curve, $k,f,m,n,b^{{}^{\prime}},b^{{}^{\prime\prime}}$ is the polynomial parameters, $\alpha,\beta$ represents the $y$ coordinates of endpoints. The formula is:

To sample the reference points for Poly Curve Attention, which has the same equation with Bézier Curve Attention, we selects five equally-spaced $y$ coordinates:

Thus, we can get the reference points of Poly Curve Attention according to Eq.(11). Finally, the reference points are:

For row-based representation, we refer to Laneformer [27]. Laneformer formulates a lane curve as $\left(x_{1},x_{2},...,x_{72},y_{start},y_{end}\right)$, where $\left(x_{1},x_{2},...,x_{72}\right)$ are the $x$ coordinates for the 72 equally-spaced $y$ coordinates, and $y_{start},y_{end}$ denote for the start $y$ coordinate and end $y$ coordinate of the curve. Row-based Curve Attention also shares the same equation with Bézier Curve Attention, but has a different reference points sampling method. To get the five reference points, we compute five equally-spaced indexes of $x$ according to $y_{start}$ and $y_{end}$, as follows.

Then we can get the five reference points:

CULane [12] is also a famous large-scale 2D lane detection dataset including nine challenging scenarios on highways and urban roads. CULane’s challenge lies in various difficult visual scenes. Although BézierFormer is not designed with this challenge in mind, it still slightly outperforms methods which are proposed to address this challenge, such as CLRNet [3], on CULane.

To have an intuitive understanding of the performance of BézierFormer, we give the qualitative results on CULane [12], CurveLanes [1] and OpenLane [6]. As shown in Figure 6, BézierFormer is robust in challenging scenarios like Highlight, Night, Crowd, Arrow, Curve, Shadow, and No Line. Figure 7 shows the results on CurveLanes and indicates that BézierFormer performs well in complex topologies like merged lanes, forked lanes, curves lanes, dense lanes, and nearly horizontal lanes. Figure 8 draws the results on OpenLane and proves the effectiveness of BézierFormer’s 3D lane detection in different scenes.