CEMFormer: Learning to Predict Driver Intentions from In-Cabin and External Cameras via Spatial-Temporal Transformers

Given multi-view streaming camera inputs and traffic-related context information, our objective is to predict a future event based on observations up to the present time. These future events fall into one of several predefined categories. We denote the input space as $\mathcal{X}$, the output space as $\mathcal{Y}$, and the context space as $\mathcal{C}$. During training, a set of $N$ training samples $\left\{(\mathbf{x}_{1},\mathbf{x}_{2},...,\mathbf{x}_{T_{j}})j,y_{j},\mathbf{c}_{j}\right\}_{j=1}^{N}$ is provided, where $\mathbf{x}_{t}\in\mathcal{X}$ represents the observation at time $t$. $y\in\mathcal{Y}$ is the ground-truth label of the event that occurs at the end of the video at time $T_{j}$. $\mathbf{c}\in\mathcal{C}$ is the vector containing auxiliary context information, which is utilized during training. During inference, however, the algorithm processes each incoming video frame in an online manner. Specifically, the online prediction system receives $\mathbf{x}_{t}$ at each time step, with the goal of predicting the event $y$ that will occur at time $T$ given only past and current observations $(\mathbf{x}_{1},\mathbf{x}_{2},...,\mathbf{x}_{t})$, where $t<T$.

We propose a novel transformer-based framework for online driver intention anticipation, designed to effectively aggregate spatio-temporal features from in-cabin and external cameras as well as episodic memory representations using attention mechanisms. As illustrated in Fig. 1, CEMFormer comprises $L$ encoder layers, each adopting the conventional structure from vision transformers [10] with some tailored designs.

Firstly, distinct patch projection layers are applied to each input view. Suppose we have the observation $\mathbf{x}_{t}$ at time $t$, which contains $M$ views:

$$\mathbf{x}_{t}=\left\{\mathbf{x}_{t,m}\in\mathbb{R}^{C\times H_{m}\times W_{m}}\right\}_{m=1}^{M},$$

(1)

where $H_{m}$ and $W_{m}$ represent the height and width of the input image from view $m$, and $C$ denotes the number of channels. For simplicity, we omit $t$ in this paragraph. The image $\mathbf{x}_{m}$ is then divided into a grid of $N_{m}$ patches, each with a size of $P^{2}\cdot C$, where $(P,P)$ is the patch size and $N_{m}=H_{m}W_{m}/P^{2}$ is the number of patches. The patches are subsequently flattened and linearly projected to $D$-dimensional tokens:

$\displaystyle\mathbf{x}_{m}^{\textit{vit}}$

$\displaystyle=[\mathbf{x}_{m}^{1}E_{m},...,\mathbf{x}_{m}^{N_{m}}E_{m}]+E_{m}^{\textit{pos}}$

$\displaystyle m=1,2,...,M,$

(2)

where $E_{m}\in\mathbb{R}^{(P^{2}\cdot C)\times D}$ is the projection matrix, and $E_{m}^{\textit{pos}}\in\mathbb{R}^{N_{m}\times D}$ represents the position embedding. Inputs from multiple views are then concatenated into a combined sequence $\mathbf{z}^{0}$:

$\displaystyle\mathbf{z}^{0}$

$\displaystyle=[\mathbf{x}_{1}^{\textit{vit}};\mathbf{x}_{2}^{\textit{vit}};\cdots;\mathbf{x}_{M}^{\textit{vit}}],$

(3)

which is subsequently fed into the spatial-temporal encoder.

Episodic memory pertains to the capacity to recall latent representations from the past time series [11]. CEMFormer incorporates this concept to process camera inputs online, allowing it to retain and reference prior context while facilitating information flow between iterations. Consequently, CEMFormer can make predictions based on both previous and current frames, resulting in enhanced robustness and accuracy.

Specifically, $K$ episodic memory embeddings $E_{t}^{\text{mem}}\in\mathbb{R}^{K\times D}$ are prepended to the input sequence:

$$\bar{\mathbf{z}}_{t}^{0}=[E_{t}^{\text{mem}};\mathbf{z}_{t}^{0}],$$

(4)

The spatial-temporal encoder then aggregates the features from in-cabin and external cameras as well as episodic memory representations by stacking $L$ transformer blocks:

$$\mathbf{z}_{t}^{L}=\operatorname{SpatialTemporalEncoder}\left(\bar{\mathbf{z}_{t}}^{0}\right)$$

(5)

In the last layer, to ensure information flow between frames, the current episodic memory representations are passed to the input sequence of the next moment:

$$E_{t+1}^{\text{mem}}=\mathbf{z}_{t,1:K}^{L},\bar{\mathbf{z}}_{t+1}^{0}=[E_{t+1}^{\text{mem}};\mathbf{z}_{t+1}^{0}].$$

(6)

The episodic memory representations at the output of the spatial-temporal encoder are fed into a prediction head to generate the final outputs. Directly optimizing the standard cross-entropy loss $\ell^{ce}$ can lead to incorrect predictions of some scenarios into categories that conflict with the traffic context. To address this issue, we propose a new context-consistency (CC) loss, which applies a penalty for making such wrong predictions. Specifically, let $\mathcal{S}=\left\{(r,\mathcal{A})\mid r\in\mathcal{Y},\mathcal{A}\subset\mathcal{C}\right\}$ be the set of contradicting scenarios, which is a subset of false positive cases. The CC loss can be defined as:

$${\ell^{cc}}=-{\sum\limits_{(r,\mathcal{A})\in\mathcal{S}}}{\mathds{1}_{[\mathbf{c}\in\mathcal{A}]}\log(1-p_{r})},$$

(7)

where $\mathbf{c}$ is the current traffic context, $\mathds{1}_{[\mathbf{c}\in\mathcal{A}]}$ is a binary indicator, and $p_{r}$ is the predicted probability of event $r$ given by the model.

Following [12], we take advantage of the exponentially growing loss to encourage the model to predict early while ensuring that it does not over-fit the training data when there is insufficient context for anticipation. Combining the cross-entropy loss, the context-consistent loss, and the exponentially growing loss, we refer to the unified loss function as the joint Context-Consistent cross entropy loss:

$$\mathcal{L}_{\text{joint}}={\sum\limits_{i=1}^{N}}{\sum\limits_{t=1}^{T}}{e^{-(T-t)}(\ell_{i}^{cc}+\ell_{i}^{ce})}.$$

(8)

Since the derivatives with respect to all parameters can be computed, we can effectively train the proposed CEMFormer using an off-the-shelf optimizer to minimize the loss function with back-propagation through time (BPTT) as in [13].

We assess the performance of our proposed method for maneuver anticipation using the publicly available Brain4Cars dataset [14]. This dataset comprises 594 video clips, showcasing both in-cabin and forward-facing views of a vehicle¹¹1Though it was reported that the dataset includes 700 videos, a portion of them are missing in the public release.. The dataset encompasses five driver maneuver categories, which defines the output space for our experiments $\mathcal{Y}=${go straight, left lane change, left turn, right lane change, right turn}. Lane changes and turns are annotated with the maneuver’s start time, corresponding to when the wheel touches the lane marking or when the vehicle begins to yaw at the intersection, respectively [12].

Based on the available traffic context information in the dataset, we define the traffic context vector $\mathbf{c}\in\mathcal{C}\subseteq\mathbb{R}^{3}$, as a three-dimensional binary vector. Each dimension represents whether the ego vehicle is in the left-most lane, in the right-most lane, and near an intersection, respectively. Additionally, the set $\mathcal{S}$ of contradicting scenarios in Eq. 7 is formally defined as $\mathcal{S}=\{$(left lane change, $(1,\cdot,\cdot)$), (right lane change, $(\cdot,1,\cdot)$), (left turn, $(0,1,\cdot)$), (right turn, $(1,0,\cdot)$), (left turn, $(\cdot,\cdot,0)$), (right turn, $(\cdot,\cdot,0)$)$\}$, and is visualized in  Fig. 2.

To ensure the reliability of the results, we employ a 5-fold cross-validation in all our experiments, which is consistent with previous studies. The final evaluation metrics include the average accuracy and F1 score, along with their standard deviations.

For our experiments, we initialize our model using the DINO ViT-B/16 [15] pre-trained weighs, which was trained on ImageNet [16]. We adopt AdamW [17] as the optimizer with a weight decay of 0.05 and apply a cosine learning rate scheduler [18] with the base learning rate set to $5\times 10^{-5}$. Both the in-cabin and external vehicle camera streams have a resolution of $224\times 224$. With a patch size of $16\times 16$, this results in a total of 392 patches. We empirically set the number of memory tokens $K=4$ (Ablation study is in Sec. III-D). For data augmentation, we divide each video into $T$ segments of equal duration and randomly sample one frame from each segment, where $T$ is the video length in seconds, drawing inspiration from [19]. As a frame-level data augmentation strategy, we also employ simple random crop with random horizontal flip, introduced in [20]. Owing to the limited size of the Brain4Cars dataset, we only fine-tune the multi-head self-attention layers in the spatial-temporal encoder, as recommended in [21]. Our model is trained for 200 epochs on a single NVIDIA RTX 3090 Ti GPU with a batch size of 10.

Tab. I presents the comparison results on the Brain4Cars test set. We compare CEMFormer to two other widely used end-to-end methods [6, 7], as they have outperformed traditional machine learning approaches in driver intention prediction tasks²²2For a fair comparison, we do not consider results from [12] as part of its training data is not accessible.. CEMFormer achieves the highest accuracy of 85.37% and an F1 score of 87.09% with the multi-view inputs. Furthermore, we observe that CEMFormer surpasses the other two methods in terms of the number of parameters. These results indicate that the proper use of an episodic memory-guided architecture allows the model to learn complex spatial-temporal relationships from both in-cabin and external driving views. This is in contrast to previous work such as [6], which claimed that outside views were not helpful for driver intention prediction tasks. Additionally, our model significantly reduces the number of parameters compared to previous methods while maintaining a compact size, even when incorporating additional views. This demonstrates the scalability of the proposed model.

To offer a thorough understanding of the CEMFormer model’s performance during online inference, we present detailed visualizations that demonstrate how the individual transformer modules contribute to identifying crucial regions within multi-view streams. The well-trained CEMFormer model performs remarkably in most scenarios, as illustrated in Fig. 3 using one episode as an example.

We visualize attention maps for both in-cabin and external views in the last layer of the spatial-temporal encoder. Attention scores are employed to generate attention maps, which are then superimposed onto the original images. These attention scores are reshaped according to their original spatial positions to produce the attention maps, with red representing the highest level of focus on a specific region. Images displayed in the same row show results from the same model. The following observations can be made based on the visualization results: