Deep Attention Driven Reinforcement Learning (DAD-RL) for Autonomous Decision-Making in Dynamic Environment

The ego-vehicle can obtain historical information of its surrounding vehicles $H_{t}$, segmented Bird-Eye-View (BEV) context $C_{t}$, and an elaborate ego’s odometric state history $E_{t}$ as shown in Fig. 2. The whole observation space tensor is represented as $\mathcal{O}_{t}=\left[H_{t};E_{t};\mathcal{C}_{t}\right]$. The subscript $t$ attributes to the tensor’s value at time $t$. The BEV context is defined as $\mathcal{C}_{t_{o}}=\{D_{t};W_{t}\}$, where $D_{t}$ is drivable area map and $W_{t}$ waypoint map of size $128\times 128$. The historical surrounding vehicle data is defined as ${H}_{t}=\left[H^{1}_{t},...,H^{n}_{t}\right]$ for $n$ surrounding vehicles, with $H^{i_{sv}}_{t}=\left[\left(X^{i_{sv}}_{t-5k.\delta t},\phi^{i_{sv}}_{t-5k.\delta t},v^{i_{sv}}_{t-5k.\delta t},l^{i_{sv}}_{t-5k.\delta t}\right)\right]^{4}_{k=0}$ of vehicle $i$’s current and past timesteps spaced four simulation steps $\delta t$ apart. Here, for vehicle $i$, $X^{i_{sv}}_{t}=(x^{i_{sv}}_{t},y^{i_{sv}}_{t})$, $\phi^{i_{sv}}_{t}$, $v^{i_{sv}}_{t}$, and $l^{i_{sv}}_{t}$ are historical data of the relative position to the ego, the heading, speed of the vehicle and the lane it is in respectively. Observation $E_{t}=\left[\left(E^{(1)}_{t};E^{(2)}_{t}\right)\right]$ consists of various historical odometric values split into two vectors based on utility in the DAD-RL framework. The vector $E^{(1)}_{t}$ contains the same quantities as in $H^{i_{sv}}_{t}$ and $E^{(2)}_{t}=\left[\left(\omega_{t-5k.\delta t},\psi_{t-5k.\delta t},v^{ego}_{t-5k.\delta t},a^{ego}_{t-5k.\delta t},j_{t-5k.\delta t}\right)\right]^{4}_{k=0}$, a tuple having steering, yaw rate, linear velocity, linear acceleration, and linear jerk of the ego at time $t$ respectively. The following section explains how the spatiotemporal deep attention encoder processes this observation space. The encoder has two parts: a) Spatio-Temporal Attention Encoder (STAE) and b) Context Encoder (CE).

Spatio-Temporal Attention Encoder (STAE): This encoder $\rho_{\eta}$ takes in the historical kinematic states of the ego and surrounding vehicles $E_{t}$ and $H_{t}$ at timestep $t$ as input. To encode the temporal kinematic relationships (past 2.1 seconds or 21 $\delta t$ simulation steps) of a dynamic traffic scenario, the states of the surrounding vehicles $H^{i_{sv}}_{t}$ and the ego vehicle $E^{(1)}_{t}$ is passed through a shared Long Short-Term Memory (LSTM) network. Let’s denote the aforementioned vectors of each vehicle $H^{i_{sv}}_{t}$ and ego $E^{(1)}_{t}$ as $I_{t}$. Eq 1 represents the intermediate temporal encoding $p_{t}$.

	$\displaystyle I^{i}_{t}=E^{(1)}_{t}\mbox{or}\ H^{i_{sv}}_{t}$		(1)
	$\displaystyle p^{i}_{t}=LSTM(I^{i}_{t})$

The temporal encoding for ego $p^{ego}_{t}$ and $p^{i_{sv}}_{t}$ for all $I_{t}$ can be found from the hidden states of the LSTM. The ego vehicle’s attention has been modeled based on the temporal kinematic encodings. This has been obtained using an attention mechanism between each temporal encoding. Let $W_{q}$, $W_{k}$, and $W_{v}$ are the query, key, and value weight matrices, respectively. To employ the attention mechanism, $p^{ego}_{t}$ is considered as the query $Q$ and $p^{i_{sv}}_{t}$ for all $n$ surrounding vehicles as both the key $K$ and the value $V$ vectors since it is important for the ego encoding $p^{ego}_{t}$ to pay attention to each surrounding vehicle encoding $p^{i_{sv}}_{t}$. The following Equ.2 shows the employed attention mechanism. Let query, key and value be $Q=p^{ego}_{t};K=V=p^{1:n_{sv}}_{t}$ and $\alpha_{t}$ be attention.

$\displaystyle\alpha_{t}(Q,K,V)=\sigma\left(\frac{Q.W_{q}[K.W_{k}]^{T}+M}{\sqrt{d_{k}}}\right)V$

(2)

The softmax or the $\sigma(Q,K)$ term outputs the attention weights for each vehicle to every other vehicle present. However, since the interest is on the attention weights associated with the ego on other vehicles, only the first row of the matrix in the $\sigma(Q,K)$ is taken. After the attention block, the final spatio-temporal attention $Z_{t}$ is derived as given in the Equ.3, where $\it{Norm}$, $\oplus$, and $\it{Linear}$ represent layer normalization, concatenation method, and fully connected layer, respectively. $E_{t[k=0]}$ is ego state vector at current time instant. $M$ is a mask that hides absent vehicles from the attention calculation. This mask helps in cases where the number of surrounding vehicles in the sensor range is less than the maximum number of vehicles (n).

	$\displaystyle m_{t}=Norm\ (\alpha_{t}+p^{ego}_{t})$		(3)
	$\displaystyle{Z_{t}}=Linear\ (E_{t[k=0]}\oplus m_{t})$

Context Encoder (CE): To process the BEV context maps $\mathcal{C}_{t}$, which contain drivable area map $D_{t}$ and waypoint map $W_{t}$, which are images, a Convolutional Neural Network (CNN) is used. To obtain a concise vector, CNN layers are employed in a manner that decreases the image size after each successive layer. After the CNN layers, the intermediate output is flattened and passed through a fully connected layer, giving the context encoding vector $c_{t}$. After obtaining spatio-temporal attention encoding $Z_{t}$ of the surrounding vehicles and odometric states of the ego and the context encoding $c_{t}$, the vectors $Z_{t}$ and $c_{t}$ are concatenated to obtain a final state encoding $s_{t}$ as shown in Equ.4.

$$s_{t}=Z_{t}\oplus c_{t}$$

(4)

The ego vehicle has a stochastic policy network $\pi_{\theta}$ with $\theta$ as its parameters mapping state $s_{t}$ to actions. As in any RL task, the objective is to learn $\pi_{\theta}$ along with our STAE ($\rho_{\eta}$) that can maximize the cumulative reward obtained by the ego-vehicle. Soft Actor-Critic (SAC), a state-of-the-art RL algorithm, is used for training policy networks and STAE.

Reward Structure: The ego-vehicle is trained using a dense reward structure for safety and comfort (evaluated by ‘Humanness error’ metrics). The reward structure (Equation 5) is a linear combination of rewards and penalties:

	$\displaystyle R=\lambda_{1}.r_{crash}+\lambda_{2}.r_{road}+\lambda_{3}.r_{v^{ego}}+\lambda_{4}.r_{goal}$		(5)
	$\displaystyle+\lambda_{5}.r_{prog}+\lambda_{6}.r_{oroute}+\lambda_{7}.r_{ww}+\lambda_{8}.r_{slow}$

The positive reward terms are $r_{goal}$ and $r_{prog}$. $r_{goal}=1$ if the agent reaches the goal; otherwise, $r_{goal}=0$; $r_{prog}$represents distance travelled towards goal. To encourage the ego vehicle’s momentum towards reaching the goal under speed limits, the reward is defined as $r_{v^{ego}}=v^{ego}/V_{max}$ when $v^{ego}<V_{max}$ and a penalty defined as $r_{v^{ego}}=-\mbox{abs}(v^{ego}-V_{max})/V_{max}$ when the ego vehicle is overspeeding. The penalty terms are defined as $r_{crash}=r_{road}=r_{route}=r_{ww}=r_{slow}=-1$, representing penalty for crashing, going offroad, off route and wrong way, and for not moving for ten consecutive seconds. The $\lambda_{i}$ coefficients are used to scale up/down and adjust the weights of each term in the reward structure.

The action space is mid-level control, combining continuous and discrete actions. Mathematically, the action space is defined as $\mathcal{A}_{t}=\left[V_{t}^{target},\Lambda_{t}\right]$, where $V_{t}^{target}\in(0,V_{max})$ represents the target speed of the vehicle at time $t$ and $\Lambda_{t}$ represents lane commands such as ’switch to left/right lane’ or ’keep lane.’ A classical controller present in the SMARTS simulator executes these middle-level commands. The SAC algorithm’s policy network $\pi_{\theta}$ is designed for continuous action spaces. To accommodate discrete lane change commands, $\pi_{\theta}$ which outputs parameters of a Gaussian Distribution from which lane actions are sampled in the range $(-1,1)$, is divided into partitions a) $(-1,-1/3)$, b) $(-1/3,1/3)$ and c) $(1/3,1)$, where a) and c) are mapped to ’switch to left/right lane’ respectively and b) to ’keep lane’.

The DAD-RL framework strongly emphasizes utilizing interaction encoding techniques to comprehend intricate and realistic traffic settings. SMARTS has been chosen as the simulation platform to assess the effectiveness of the DAD-RL framework in handling such complex scenarios. Within SMARTS, several demanding scenarios were constructed aimed at both training and testing the DAD-RL approach. These scenarios are designed to encompass diverse interactive and stochastic traffic dynamics. The following scenarios capture various aspects of realistic driving behaviors.

Left Turn-T: This is an urban T-junction with heavy traffic and no traffic signals. The goal of the ego-vehicle is to take an unprotected left turn.

Roundabout: In this urban roundabout scenario, the ego transitions from a 2-lane bi-directional road to a 2-lane unidirectional roundabout with four exits—three versions: Roundabout-A, B, and C, with increasing difficulties in that order. The ego vehicle is supposed to cover a quarter, half, and three-quarters in Roundabout-A, B, and C, respectively. The commute distance sets the difficulty level.

Double-Merge: In this scenario, an autonomous vehicle (’ego’) starts from a single-lane road, navigates a two-lane one-way road with two entrances/exits, and exits on the opposite side. The ’ego’ must effectively perform lane changes amidst traffic flows from all entrances to exits, honing its navigation and lane-changing skills.

The scenarios above incorporate heavy traffic flows randomly selected for each simulation episode. The agent engages in multiple consecutive scenarios to assess its overall performance during evaluation.

This subsection explains how the DAD-RL framework is trained. The SMARTS simulator is used as an RL-Gym environment for training. Each simulation step is $\delta t=0.1s$ apart. The ego-vehicle obtains information about surrounding vehicles, itself, and the context at every step. The ego-vehicle processes the observation, takes action, and proceeds to the next simulation step. A route is initialized for the vehicle to follow at the beginning of each episode. Each step consists of a tuple $(\mathcal{O}_{t},a_{t},\mathcal{O}_{(t+1)},r_{t},d_{t})$ with $d_{t}$ being done/terminal signal. A series of such steps and tuples make one episode. The DAD-RL framework is trained on five scenarios mentioned in the previous subsection. A vectorized environment runs several simulations parallelly to speed up the training process. As several steps/episodes progress, the ego-vehicle stores the experience tuple $(\mathcal{O}_{0:B},a_{0:B},\mathcal{O}_{0:B},r_{0:B},d_{0:B})$ in a buffer B. After gaining sufficient experience, a batch is sampled from this buffer B to estimate the objective function, and the loss is back-propagated from the end of the policy network $\pi_{\theta}$ till the beginning of STAE $\rho_{\eta}$ based on loss functions of SAC. After this network training process, the ego-vehicle collects more experience, repeating the process. The training was done on a machine with at least 32 GB RAM, an Nvidia 2080Ti GPU, and Ubuntu 20.04.

To thoroughly compare the proposed framework’s performance, the DAD-RL framework was evaluated against several baseline methods, and SOTA [11]. PPO: Proximal Policy Optimization stands out as a SOTA policy gradient approach with promising performance on robotic decision-making tasks. SAC: Soft Actor-Critic is another SOTA RL algorithm, an off-policy approach capitalizing on entropy maximization to assist agent training. DrQ: Data-regularized Q-learning employs a CNN-based augmentation to SAC, facilitating direct learning of a policy function from pixels. RDM: Rule-based Driver Model mimicking the driving behavior of neighboring vehicles within SMARTS simulations, offering a benchmark for evaluating the rule-based performance. DT: Decision Transformer is an innovative approach that reformulates the RL problem as a conditional sequence modeling task and employs a transformer to generate future actions from past states, actions, and rewards. SRT: Scene-Rep Transformer (SRT) is a transformer-based framework recently proposed in [11]. SRT employs two transformer architectures - a Multi-Stage Transformer (MST) for encoding the multi-modal scene input and another Sequential Latent Transformer (SRT) to instill predictive information into the latent state vector.

For all the mentioned baselines, the implementation provided by [11] is considered for the performance comparison, and the results are directly sourced from the same.

In evaluating the proposed framework, several performance metrics were utilized to measure the experiments’ effectiveness. These metrics provide insights into the agent’s behavior and performance. Here is an explanation of the performance metrics adopted for the evaluation: Succ.% (Success Rate): This metric represents the proportion of episodes where the agent successfully reaches the goal out of the total evaluation episodes. A higher success rate indicates better performance in goal achievement. Coll.% (Collision Rate): The collision rate is the proportion of episodes that resulted in a collision between the agent and surrounding vehicles. A lower collision rate signifies better navigation and avoidance capabilities. Stag.% (Stagnation Rate): Stagnation rate reflects the proportion of episodes ending prematurely due to exceeding the maximum time steps allowed. A lower stagnation rate indicates efficient decision-making and goal pursuit. Humanness Error: Humanness Error is calculated based on comfort factors such as jerk and angular acceleration. It assesses how closely the agent’s movements resemble human-like driving behavior. Overall Score: Overall Score depicts a comprehensive measure of driving performance. As implemented in SMARTS, it combines multiple factors such as progress, rule violations, and comfort. Humanness Error and Overall Score are both provided by SMARTS. To assess the performance of the DAD-RL framework based on the defined metrics, we follow an evaluation strategy similar to [11]. The evaluation strategy involves playing 50 episodes in simulation to test the agent’s performance across varied traffic patterns, enabling a comprehensive analysis of the proposed framework’s effectiveness. The initialization for the testing environment is randomly selected.

Table I shows the results of three evaluation metrics, namely success rate, collision rate, and stagnation rate for all the baselines mentioned in the previous section and the DAD-RL framework. It is evident from the results that the DAD-RL framework exhibits significant performance improvements compared to the baseline methods in most of the scenarios. DAD-RL shows exceptional performance on the left turn scenario, with a $0\%$ collision rate. DAD-RL significantly improves the success and collision rates on all roundabout scenarios compared to the SRT. The trend of success rates in the three roundabout scenarios reflects their difficulties. As we go from R-A to R-B to R-C, the route length increases, which means the ego will face more traffic interactions, increasing the difficulty. In the Double Merge scenario, DAD-RL faces tough competition from MST and SRT, yet it still surpasses other baselines like DT and DrQ in terms of overall performance. Unlike the other four scenarios, Double Merge requires the agent to assess the perfect time to merge into a lane with ongoing traffic to avoid collisions. This could explain the discrepancies in results for Double Merge, and it can be solved with a spatiotemporal prediction module similar to SRT. On average, across all five scenarios, DAD-RL increases the success rate by 29.6% compared to SAC and 2.4% compared to SRT. These quantitative findings emphasize the superior capabilities of DAD-RL in completing driving tasks and mitigating collision incidents. The architectural difference between DAD-RL and SRT is noteworthy, with the latter leveraging a full Transformers model while the former opts for a simpler approach using a compact spatio-temporal encoder with a single attention layer. This streamlined model design enhances efficiency by concentrating on crucial learning components. The lightweight nature of the DAD-RL framework not only delivers enhanced average performance and simplifies architectural complexity, making it a practical and effective choice for driving tasks.

Multiple experiments on three distinct scenarios were conducted to understand the individual aspects of the spatio-temporal encoder and the context encoder. The results of these experiments are depicted in Fig. 3. Both the Roundabout and Double Merge scenarios exhibit similar trends among the metrics. Compared to SAC, which only utilizes a context encoder, using only the proposed spatio-temporal encoder (Context) improves the overall score while also reducing humanness errors. A combination of the spatio-temporal attention encoder and the context encoder (DAD-RL) achieves the highest Overall Score and the lowest humanness error. In the Left Turn-T scenario case, the trends are predominantly consistent with the other scenarios except for one metric. SAC obtains a lower humanness error than both variants of DAD-RL. Upon visualizing SAC’s performance in simulation, it was noticed that the agent shows conservative behavior, making it difficult for the agent to maneuver through the traffic at the T-junction, leading to a lower Overall score. This conservative driving style is the reason for the lower humanness error of SAC. DAD-RL shows a slight compromise regarding humanness error to improve the overall score greatly.