Enhancing High-Speed Cruising Performance of Autonomous Vehicles through Integrated Deep Reinforcement Learning Framework

Let us consider a highway scenario where an AV seeks to optimize its high-speed cruising performance. We assume that all surrounding vehicles around this AV are HDVs, which is typical in the early deployment stages of AVs with very low penetration rates. The driving behavior of these surrounding HDVs is characterized by the Intelligent Driver Model (IDM) for car following and the MOBIL model for lane changing, both of which are widely adopted in transportation research [24]. The vehicle dynamics of AVs are modeled using a bicycle model (see Fig. 1) with configuration parameters obtained from a high-fidelity vehicle model in the commercial software Carsim, as detailed in Table I.

This study aims to devise a controller for the longitudinal (i.e., lane-keeping) and lateral (i.e., lane-changing) movements of the AV in high-speed cruising scenarios. Hence, the AV’s control input is defined by $v=\left[a_{x},\delta_{f}\right]^{T}$, where $a_{x}$ and $\delta_{f}$ are the longitudinal acceleration and front-wheel steering input, respectively. The AV’s state is defined by $\zeta=\left[V_{x},V_{y},\varphi,\gamma,X,Y\right]^{T}$, where $V_{x},~{}V_{y},~{}\varphi,~{}\gamma,~{}X$, and $Y$ represent the longitudinal velocity, lateral velocity, yaw angle, yaw rate, global longitudinal position, and global lateral position, respectively.

With these notations, the vehicle’s position under the global coordinate system can be modeled as

When the vehicle operates with a small steering input, the dynamics of the longitudinal velocity, lateral velocity, and yaw angle can be represented as follows [35].

where $M$ represents the vehicle mass, $J_{z}$ represents the inertia yaw moment of the vehicle, $F_{fy}$ and $F_{fx}$ represent the lateral and longitudinal forces of the front tire, respectively, $F_{ry}$ and $F_{rx}$ represent the lateral and longitudinal forces of the rear tire, respectively, and $L_{f}$ and $L_{r}$ represent the distances from the center of gravity (CG) to the front axle and rear axle, respectively. As shown in Fig. 1, the relation between these variables can be described by

where $K_{f}$ and $K_{r}$ represent the cornering stiffness of front and rear tires, respectively, $\alpha_{f}$ and $\alpha_{r}$ represent the slip angles of front and rear tires, respectively. $\beta$ is the vehicle sideslip angle and calculated by $V_{y}/V_{x}$.

By combining Eq.(1)-Eq.(3), the vehicle dynamics can be written as follows.

where

This section presents an integrated framework (see Fig. 2) to systematically combine the decision-making, path-planning, and motion-control modules. The decision-making module chooses between lane-keeping and lane-changing maneuvers, using a DRL agent that takes into account high-speed cruising performance and collision avoidance. If lane keeping is chosen, the DRL agent generates an acceleration/deceleration control signal. If lane changing is chosen, the path-planning module generates an efficient and human-like path that effectively trade off vehicle safety, driving comfort, and travel efficiency, with the corresponding weights learned from experienced drivers using an IRL algorithm. Finally, the motion-control module calculates the control signals with a model predictive controller that tracks the planned path while ensuring the physical feasibility by fully considering the vehicle dynamics constraints and actuator saturation.

Note that, different from existing works that combine only the decision-making module with the path-planning module, we further incorporate the motion-control module into the integrated framework. Such an integration involves two aspects. First, the DRL agent in the behavioral decision-making module generates not only the behavioral decisions but also the control weights that will be used in the motion-control module. Such a treatment systematically aligns the control objectives of the motion-control module with the other two modules. Second, the motion-control module is incorporated into the DRL agent’s training process, thereby ensuring that the trained DRL agent can generate physically feasible actions.

The decision-making process can be described as a Markov Decision Process (MDP), denoted by a tuple $\mathcal{M}=(\mathcal{S},\mathcal{A},P,R)$, where $\mathcal{S}$, $\mathcal{A}$, $P$, and $R$ are the state space, action space, state transition probability, and reward function, respectively. The details are presented below.

where Eq.(6) encourages the AV to drive faster, with $V_{ev,i}$, $V_{\min}$, and $V_{\max}$ indicating the ego AV velocity at decision step $i$, minimum velocity, and maximum velocity, respectively. The constraint $V_{ev,i}\in\left[V_{\min},V_{\max}\right]$ is ensured in the lower motion-control module via the MPC controller. Eq. (7) employs an artificial potential field (APF) function to penalize the ego AV from being too close to surrounding HDVs to enhance its collision-avoidance ability in high-speed cruising scenarios, where $Y_{ev,i}$ and $X_{ev,i}$ represent the global lateral and longitudinal positions of ego AV at decision step $i$, while $Y_{m,i}$ and $X_{m,i}$ indicate the global lateral and longitudinal positions of HDV $m$ at decision step $i$. The scale factors $\sigma_{1}$ and $\sigma_{2}$ normalize the longitudinal and lateral spacing between vehicles. Eq.(8) is a Gaussian-type reward with scale factor $\sigma_{3}$ implemented to discourage excessive lane-changing maneuvers to reduce the impact on congestion, where $Y_{\text{mid }}$ is the lateral coordinate of the lane centerline, and $P_{\text{lane }}$ normalizes the reward term to be between 0 and 1. During the training process, the scale factors are tuned to minimize collisions. After experiments, $\sigma_{1}$, $\sigma_{2},\sigma_{3}$, and $P_{\text{lane }}$ are set by -3, -0.5, -0.1, and 2.5, respectively.

The bootstrapped DQN is leveraged as the RL agent in this paper to enhance training efficiency, which provides exponential learning speed by approximating the population distribution using the distribution of bootstrapped samples $\Omega(\tilde{E})$, where the data set $\tilde{E}$ is obtained by sampling uniformly with replacement from the real data set $E$.

Fig. 3 illustrates the network structure of the bootstrapped DQN that consists of a shared core network and $k$ independent branches, each representing a head network. Each head network combined with the core network can be seen as a $Q$-value network, denoted by $Q^{k}$. Note that the training data for each head network only includes the bootstrapped sample with distribution $\Omega(\tilde{E})$. This allows $K$ independent $Q$-networks to collaborate and produce improved $Q$-value estimation results.

The bootstrapped DQN is implemented by generating a bootstrap mask to acquire a subsample each time after an action is selected. The bootstrap mask at decision step $i$, represented by a binary vector $d_{i}=\{d_{i}^{k}\}_{k=1}^{K}\in\{0,1\}^{K}$ with $K$ denoting the number of head networks, indicates which neural network to update with the current transition. Considering the bootstrap mask $d$, the state transition at decision step $i$ can be augmented by $\left(s_{i},u_{i},r_{i+1},s_{i+1},d_{i}\right)$ and stored in the experience buffer. Consequently, the gradient of the $k$-th $Q$-value network can be represented as follows.

where

where $\theta_{i}^{k}$ represents the $k$-th neural network parameters at decision step $i$, $y_{i}^{Q^{k}}$ represents the approximate target value at decision step $i$, and $r_{i}$ and $\gamma$ refer to the interactive reward with the environment and the discount factor, respectively. $\max_{u}Q\left(s_{i+1},u;\theta^{-}\right)$ represents the action corresponding to the maximum action value of the target $Q$ network at the state $s_{i+1}$. During the training process, similar to the traditional Q-network [26], we introduce $\theta^{-}$ to denote the target network parameter, which satisfies the fixed $\theta^{-}=\theta_{i}^{k}$. It means the target network parameters $\theta^{-}$ are updated at a frequency of $\tau$ and remain fixed in between updates. Furthermore, gradient normalization is used to mitigate the impact of back-propagation from each $Q$-value network onto the shared core network. The normalized process is represented as follows.

When an RL agent starts the training process and explores potential rewards from the environment, it first follows the basic DQN method. Note that a $Q$-value network is randomly chosen at each episode. After completing the training, the optimal policy can output a set of actions. Considering the diversity of the $Q$-value network in the Bootstrapped DQN structure, a voting mechanism is introduced to select the optimal action, whereby the action with the highest number of votes across $K$ different $Q$-value networks will be executed. The pseudo-code for the bootstrapped DQN is shown in Algorithm 1.

To simplify the path-planning process, we employed a widely used polynomial expression [36] to generate candidate paths. The representation of the candidate paths is as follows.

where $Y_{\text{ref }}$ is the reference lateral position. $\epsilon_{0},\epsilon_{1},\epsilon_{2},\epsilon_{3}$, $\epsilon_{4}$, and $\epsilon_{5}$ are the coefficients of the polynomial. In the path-planning process, the constraints, which include both the initial and terminal points of the lane-changing behavior, are designed as follows to determine these coefficients.

where $\left(X_{\text{ref },0},Y_{\text{ref },0}\right)$ and $\left(X_{\text{ref },t},Y_{\text{ref },t}\right)$ represent the initial and terminal position of the lane-changing maneuver along the reference path, respectively.

Then by substituting Eq.(13) into Eq.(12), the candidate paths can be further represented as follows.

where $F$ and $P$ represent the lateral and longitudinal lane-changing distances, respectively. In this work, to facilitate system design, the longitudinal velocity of the AV is treated as constant during the lane-changing maneuver [22], which can be achieved through a Proportional-Integral algorithm in the motion-control module. Let $t_{c}$ denote the lane-changing duration, and hence the longitudinal distance of the lane-changing maneuver can be expressed as $P=V_{x}t_{c}$. Note that the mathematical expression for the bounds of $t_{c}$ is obtained by the constraint (15):

Eq.(15) is derived to ensure the collision avoidance and stability of the AV, where $d_{avoi}$ represents the collision-avoidance distance, and $d_{c}$ and $d_{a}$ are the available space for the lane-changing maneuver at the current and adjacent lanes, respectively. $s_{saf}$ indicates the minimum safety distance between vehicles, $a_{x,\max}$ is the maximum allowed deceleration rate, and $\mu$ is the road adhesion coefficient. $\beta_{\text{ref}}$ represents the vehicle sideslip angle at the reference path, which can serve as an indicator to characterize the AV’s stability in vehicle dynamics [21]. Specifically, the reference value of $\beta_{\text{ref}}$ can be calculated as follows.

where

A set of candidate paths as shown in Fig. 4 are generated from Eq.(14) with a time interval of 0.1 s between $t_{c,\text{ min }}$ and $t_{c,\text{ max }}$, where Let $t_{c,\min}$ and $t_{c,\max}$ denote the lower and upper bounds for $t_{c}$, respectively, calculated from Eq.(15).

To make the behavior of AVs (characterized by their trajectories) understandable by HDVs, this work captures the preference of human drivers over multiple control objectives by introducing an IRL approach to learn skilled drivers’ reward function for planning lane-changing paths. After finishing training, an optimal reference path would be selected to achieve a human-like tradeoff between these control objectives.

Specifically, AV’s safety is quantified by considering the stability of the vehicle dynamics and the potential collision risk. Driving comfort is indicated by the change rate of the yaw angle, while travel efficiency is represented by the lane-changing duration $t_{c}$. A vector of control objectives for a considered candidate path is constructed as follows:

where $H_{sta},~{}H_{col},~{}H_{com}$ and $H_{tra}$ represent the control objectives for vehicle stability, collision risk, comfort, and travel efficiency, respectively, with the specific forms given as follows:

where we discretize the lane-changing duration $[t_{c,\min},t_{c,\max}]$ into $\eta$ time instants with equal intervals of 0.1 s, indexed by $l$, and $\dot{\varphi}_{\text{l}}$ represents the change rate of the yaw angle at the time instant $l$.

The reward function of IRL is then constructed by a linear combination of the feature vector.

where $\varpi=\left[\varpi_{sta},\varpi_{col},\varpi_{com},\varpi_{tra}\right]$.

During the training process, we utilize the driving behavior of skilled drivers as expert experience to optimize the feature weight coefficients $\varpi$ in Eq.(23). Note that we adopt a maximum entropy method [37] for training the IRL model. The objective of such method is to maximize the likelihood of the driver’s trajectory $\lambda_{q}\in\Theta,(q=1,2,\ldots,Q)$.

where $Q$ indicates the total driver’s trajectories collected in the tests. $\tilde{\lambda}_{j}$ is the candidate path generated by the polynomial expression Eq.(14). $S$ and $H_{\lambda}$ represent the number of candidate paths and feature vector of the driver trajectory, respectively. To guarantee the effectiveness of the training data, the initial state when generating the trajectory $\tilde{\lambda}_{j}$ is the same as that of driver trajectory $\lambda$.

The objective function of the IRL framework can be written as follows.

The gradient of $\Omega(\varpi)$ is computed as:

Eq.(26) could reflect the difference between the driver trajectory and candidate paths. It is employed to iterate and update the feature weight vector $\varpi$ using the gradient ascent method Eq.(27). The IRL algorithm flow is presented in Algorithm 2. After completing the training, the feature weight vector is determined. Hence, the AV’s path-planning ability can emulate a skilled driver to effectively balance different control objectives in high-speed cruising scenarios. In the IRL framework, the learning rate $\rho$ and episodes $E$ can impact the training effectiveness. By comparing the human-like driving results with different settings, $\rho=0.08$ and $E=150$.

The lane-keeping motion-control module aims to track the reference acceleration/deceleration signal $a_{x,\text{ ref }}$ calculated in the behavioral decision module, while ensuring high-speed cruising of the ego AV. This is achieved via a MPC-based controller with an embedded quadratic programming (QP) problem described as follows. Note that the system state-space equation adopts the vehicle longitudinal dynamics model and can be derived from Eq.(4) [40].

where the objective function Eq.(28a) minimizes the total costs over a prediction horizon $N_{p}$, including (i) the deviation from the maximum speed (the first term) and (ii) the tracking error between the actual acceleration rate and the reference acceleration rate $a_{x,ref}$ given by the decision-making module (the second term), weighted by constants $P_{1}$ and $R_{1}$. The specific values of $P_{1}$ and $R_{1}$ are determined by the DRL agent in the behavioral decision-making module. Constraints (28b) and (28c) impose physical bounds on the acceleration and velocity at each decision step. Constraint (28d) ensures that the spacing between the ego AV and the preceding HDV is lower bounded by the safe spacing. Notice that here we assume that the following HDV will not actively collide with the AV.

The lane-changing motion-control module devises an MPC-based controller to track the reference path generated by the path-planning module, while ensuring the AV’s lateral stability [38]. The embedded optimization problem of the MPC can be represented as follows. The system state-space equation employs the vehicle lateral dynamics model [22].

where $\bar{\zeta}=[\varphi,Y]^{T}$, and $\bar{\zeta}_{ref}=\left[\bar{\varphi}_{\text{ref }},Y_{\text{ref }}\right]^{T}$. The objective function Eq.(29a) includes (i) the deviation from the reference trajectories (the first term) and (ii) the cost of the control inputs (the second term), weighted by positive-definite matrices $P_{2}\in\mathbb{R}^{2\times 2}$ and $R_{2}\in\mathbb{R}$. The specific values of $P_{2}$ and $R_{2}$ are determined by the behavioral decision-making module, where $P_{2}=\operatorname{diag}\left(P_{21},P_{22}\right)$. $\bar{\varphi}_{\text{ref }}=0$ is set to guarantee the AV’s heading angle is zero after completing the lane-changing maneuver. Constraint (29b) imposes physical bounds on the front-wheel steering input. Constraints (29c), (29d), and (29e) ensure the AV’s lateral handling stability [38].

The integrated framework requires the training of the feature weight vector $\varpi$ and the behavioral decision-making policy, which is performed in a sequential manner. First, using collected data of skilled human drivers, the IRL method is trained based on Algorithm 2 to obtain the feature weight vector $\varpi$. Then, Algorithm 1 is employed to train an AV navigating in a high-speed cruising scenario by integrating the behavioral decision-making, path-planning, and motion-control modules. After tuning in the training progress, the hyperparameters of the bootstrapped DQN are given in Table III. Note that during the training process, the initial positions and velocities of HDVs are randomly set. Meanwhile, the Long Short-Term Memory network is employed to obtain the driving behavior of the leading HDVs from the historical data.

Furthermore, to simplify the system design, we set a benchmark for the training of control weights in the motion-control module. For lane-keeping control (Eq.(28a)), to prioritize the optimization objective of tracking the reference acceleration signal, $R_{1}=4P_{1}$ is set as a benchmark. As for the lane-changing control (Eq.(29a)), considering the priority of path-tracking performance, we set $P_{22}=10P_{21}$. After completing the training, the RL agent in the behavioral decision-making module would output the specific values of control weights.

In the path-planning module, we introduce the IRL method to learn skilled drivers’ reward function for planning lane-changing paths. The driving simulator, as depicted in Fig. 5, is employed to collect driver behavior data. More details regarding the driving simulator can be found in [39]. We invite skilled drivers to participate in the tests, who will get familiar with the testing equipment in advance to ensure the accuracy of the test data.

Through these experiments, we construct a dataset comprising 50 drivers’ lane-changing trajectories under different test conditions. This dataset is utilized to update the IRL weight vector $\varpi$. After finishing the training, we employ a set of 10 driver trajectories to evaluate the learning performance. Here, the feature vector $H$ in Eq.(18) is calculated to provide a comparison of human-like path-planning performance. The specific indices from IRL’s learning results and driver’s trajectories are presented in Table IV. The average learning errors for different performance indices, including vehicle safety, driving comfort, and travel efficiency, in ten tests are $6.92\%,3.36\%,6.14\%$, and $0.83\%$, respectively. These results demonstrate the effectiveness of the proposed IRL method in acquiring preferences of human drivers, thereby balancing different control objectives during lane-changing maneuvers.

After completing the training for these two frameworks, ten test cases with different initial positions and velocities of HDVs are conducted to verify the control performance. As shown in Figs. 6 and 7, it is clear that the proposed integrated framework outperforms the sequential framework in both the average AV velocity and reward. Overall, the total average velocity and reward with the integrated framework for ten test cases can be improved by $2.12\%$ and $10.25\%$, respectively. Although the average velocity of the AV with the sequential framework is slightly higher than that of the integrated framework in test cases 3 and 8, the integrated framework performs significantly better in ensuring a high reward. This is because the sequential framework cannot make optimal decisions from a global perspective due to the lack of coordination between these fundamental modules. The results demonstrate that the integrated framework can enhance high-speed cruising performance while considering collision avoidance, the interpretability of AVs’ trajectories, and the feasibility of AV motion.

We specifically highlight the value of integrating the motion-control module by dynamically setting up control weights, which has not been considered in the existing literature. To this end, we compare our integrated framework with a semi-integrated framework that only incorporates the path-planning module, while the control weights in the motion-control module are assumed to be pre-defined constants. For both the integrated and semi-integrated frameworks, we simulate a continuous overtaking maneuver scenario, which provides a comprehensive test case to better assess the longitudinal and lateral driving performance of AVs. Specifically, the continuous overtaking maneuver scenario under the integrated framework is shown in Fig. 8.

The value of integrating the motion-control module is illustrated by evaluating the performance indicators summarized in Tables V and VI, respectively, for the longitudinal and lateral control of AVs. The comparison between the integrated framework (i.e., with dynamic weights) and the semi-integrated framework (i.e., with constant weights) is shown in Figs. 9 and 10, whereby it is clear that the ego AV exhibits better high-speed cruising performance and stability. This can be attributed to the DRL agent’s ability to dynamically adjust control weights, thereby ensuring optimal control performance in dynamic driving conditions. From Fig. 10, although the semi-integrated framework with constant control weights consumes slightly less control effort, it performs worse in ensuring path-tracking and stability.

In addition, we assess the stability performance of the ego AV using the $\beta{\rm{-}}\gamma$ phase plane [35]. As depicted in Fig. 11, when introducing the integrated framework with dynamic control weights, the ego AV runs within a smaller region, indicating a more stable state. As the third and fourth lane-changing maneuvers are executed at higher speeds, the stability state of the ego AV deteriorates when using constant control weights in the motion-control module. In contrast, the proposed integrated framework reliably ensures the AV’s stability.

We perform a sensitivity analysis on the number of head networks in the bootstrapped DQN. As shown in Figs. 14 and 15, the bootstrapped DQN results in a higher average reward and a lower collision rate when employing more head networks. This can be attributed to the enhancement of the exploration capability with the increasing number of head networks. However, the marginal benefits of having more head networks diminish when the number of head networks reaches 6. This is because an adequate number of head networks could guarantee the precision of the uncertain Q-value estimation. Since more head networks consume more computing resources, we select the number of head networks as 6 in this work.