Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priors

Toward defining or getting the skill that can be exploited in reinforcement learning for autonomous driving and robotics, two main approaches exist: (1) manually designing task-specific or object-centric motion skills [46, 5, 4, 48, 14, 31], such as cutting in a target lane before a specific car, or grasping a specific object. However, when AVs run on actual roads, a dense-traffic setting, AVs’ motion usually should be synthesized considering relationships between multiple surrounding vehicles, which is usually too complicated to design manually. With limited flexibility and expressiveness, such manually-designed skills can hardly cover diverse driving and interaction situations. (2) another approach is to extract or learn skills from offline motion datasets, such as clustering or segmenting motion skills [15, 33, 30], or distill offline motions into low-dimension latent space [29, 24, 25], so that the issues and labor of manual design can be bypassed. However, such datasets can be expensive and labor-intensive to collect if they are not already available. It would also be difficult for such learned skills to transfer to new tasks or environments as they are usually task-specific and conditioned on the environment. Moreover, it is also not guaranteed that all necessary skills are covered in the datasets as they are usually unbalanced in distribution and sub-optimal. To tackle the limitations above, one recent work [50] combines the two approaches by first building a task-agnostic and ego-centric motion skill library in a pure ego vehicle motion perspective and then encoding the motion skills into a low-dimension latent skill space. However, the skill library construction still requires considerable effort, and the latent space encoding further makes the decision-making less interpretable. Thus, in this work, we propose to directly learn over the skill parameter space instead of the latent space, to spare the efforts in skill library construction, enable more interpretability in the decision-making process, and also provide an parameterized interface to further leverage priors.

Although RL has been shown to be effective in several problems, learning efficiency issues [11] limit its applications. Inspired by human decision-making processes where prior knowledge is quite helpful when we learn new tasks, many works use expert prior knowledge to avoid learning from scratch with exhaustive interactions. There exist three approaches to leveraging prior knowledge: (1) utilize expert demonstration for a warm-up pretraining or policy (actor) initialization before RL [20, 10, 19, 27, 32]; (2) train an expert policy based on expert demonstrations, which is then used to guide RL process [29, 34, 21]; (3) maintain an expert data buffer, which is mixed with interaction buffer during RL for more fruitful experiences [20, 10, 27, 32, 42, 26, 22]. The first method is effective in policy-based RL methods but only initializing the actor was reported to be not helpful in the actor-critic framework [29]. This is because the actor and critic are interacting with each other during the learning process, and the actor’s objective is to maximize the Q-value output by the critic without pretrianing. Thus, the actor could quickly lose the prior knowledge learned from the expert after several updates. Empirically, we found performance drops can happen at early training iterations when adopting the first method (see Section IV-C2). The second and third methods use expert priors as guidance during RL training but might suppress the performance when the expert performance is suboptimal [34]. To overcome the limitations of the three methods, we propose a ‘double initialization’ technique with both the actor and critic initialized simultaneously and demonstrate this approach can achieve strong performance even when a suboptimal expert is employed.

Inspired by the sampling-based motion planning [18, 8, 45], we first introduce the parameterized motion skill. Defined in a pure ego-motion perspective, such motion skills are task-agnostic and ego-centric and can cover diverse motions needed in dense-traffic settings, which can be generalizable to different scenarios. Technically, the beginning boundary of the motion skill is determined by the vehicle’s current state, and one motion skill further necessitates four parameters of the end boundary i.e., the lateral position $y_{e}$, heading angle $\phi_{e}$, speed $v_{e}$, and acceleration $a_{e}$ at the end of the motion skill. RL agents can learn and explore these parameters to generate diverse motion skills. Given the skill parameters, the generation of one motion skill consists of three steps, as shown in Fig. 3: (a) path generation, generating future path on the road, (b) speed profile generation, specifying the variation of speed in the skill time window, and (c) trajectory (motion skill) generation, projecting the integral of the speed profile onto the path to generate motion trajectory. All details are introduced below.

Path Generation. The path is generated by connecting the origin and an endpoint on the road by cubic splines, in the ego vehicle’s coordinate system. The endpoint is characterized by three parameters: longitudinal position $x_{e}$, lateral position $y_{e}$, and heading angle $\phi_{e}$. To ensure feasible speed-to-path projection, the path length should be longer than the integral of the speed profile. We thus fix the longitudinal position of the endpoint as the longest distance the ego vehicle can reach within the skill time window $T$. Therefore, the path generation consists of two free parameters the RL agent needs to learn: the lateral position $y_{e}$ and heading angle $\phi_{e}$ of the endpoint, which can cover diverse lateral driving intentions and maneuvers, such as lane keeping, overtaking, and cutting-in.

Speed Profile Generation. The speed profile is represented by a cubic polynomial in the time horizon $[0,T]$, parameterized by the speed $v$ and acceleration $a$ at the beginning and end of the time horizon. When generating the speed profile, we specify the speed and acceleration at the beginning time by the vehicle’s current state. Thus, the speed profile generation phase provides two free parameters that the RL agent needs to learn: the speed $v_{e}$ and acceleration $a_{e}$ at the end time, which can cover diverse temporal intentions, such as accelerating, decelerating, and emergent stop.

Parameterized Motion Skill Generation. Given the path and speed profile, the motion skill is generated by projecting the integral of the speed profile onto the path. Each motion skill is a sequence of vehicle states $\mathbf{X}=[\mathbf{x_{1}},\mathbf{x_{2}},...,\mathbf{x_{T}}]$, with each state as a tuple $\mathbf{x_{t}}=\{x_{t},y_{t},\phi_{t},v_{t},a_{t}\}$.

Note that each skill generation is conditioned on the vehicle’s current state, which is the final state of the last executed skill, which ensures smoothness between skill segments. The dynamic constraint (acceleration, curvature) is also enforced by restricting the planning parameters in reasonable ranges.

Another way to accelerate RL is by leveraging prior knowledge from expert demonstrations. However, most expert demonstrations $\mathcal{D}_{u}=\{(s_{i},u_{i})\}$ are in control space and do not contain the skill and reward information, making it not readily usable. We propose an inverse parameter recovery procedure to annotate expert data with skill parameters and construct the expert demonstration in skill space $\mathcal{D}_{\boldsymbol{\theta}}=\{(s_{i},\boldsymbol{\theta}_{i})\}$.

Consider the motion skill generation (see Section III-A) as a forward process, where one motion skill $\mathbf{X}$ is generated given skill parameters $\boldsymbol{\theta}$. Then the skill parameter recovery can be regarded as an inverse procedure, where $\boldsymbol{\theta}$ are determined given one demonstrated motion skill $\mathbf{X}_{d}$. Practically, the motion skills are retrieved by sequentially splitting the demonstration trajectory into segments, each with a length of $T$. Formally, this recovery process can be formulated as

where $f_{s}$ denotes the motion skill generation process in Section III-A. Specifically, we use the sequential quadratic programming (SQP)  [16] as the optimization method. For each motion skill, we run the optimization process multiple times with different skill parameter initializations to mitigate issues of local optima and achieve better recovery accuracy.

With the expert demonstration in skill space $\mathcal{D}_{\boldsymbol{\theta}}=\{(s_{i},\boldsymbol{\theta}_{i})\}$, we can leverage the skill and expert prior simultaneously in RL. In this paper, we modify the maximum-entropy RL to do so. Specifically, we adopt the Soft Actor-Critic framework (SAC [9]), which consists of an actor $\pi(\boldsymbol{\theta}|s)$ for policy improvement and a critic $Q(s,\boldsymbol{\theta})$ for policy evaluation. This section will introduce how the two networks can be pretrained to capture expert priors.

Actor Pretraining To leverage the expert demonstration in skill space $\mathcal{D}_{\boldsymbol{\theta}}$ as priors, we first pretrain an actor $\pi(\boldsymbol{\theta}|s)$ to capture the skill priors conditional on the current state. The training of this model aims at maximizing the log probability of the recovered expert skill parameter in the distribution output by the actor:

where for each data $(s,\boldsymbol{\theta})$ in the expert demonstration $\mathcal{D}_{\boldsymbol{\theta}}$, the pretrained actor $\pi(\boldsymbol{\theta}|s)$ takes the current state $s$ as inputs and outputs the Gaussian distribution of the skill parameters $\boldsymbol{\theta}$. $\mathcal{H}(\boldsymbol{\theta})$ denotes the entropy regularization term and $\beta$ denotes entropy weight. The pretrained actor can provide prior knowledge on which skills are more promising to explore conditioning in the current situation.

Critic Pretraining Since the actor and critic interact with each other during RL training, only pretraining the actor does not fully leverage the prior knowledge. However, pretraining the critic $Q(s,\boldsymbol{\theta})$ is not always available since there is no reward information in the expert demonstration in either control space $\mathcal{D}_{u}$ or skill space $\mathcal{D}_{\boldsymbol{\theta}}$. Fortunately, we already have the pretrained actor who has learned expert priors. Thus we propose to roll out the pretrained actor in the environment to collect an expert demonstration with both skill and reward information $\mathcal{D}_{\boldsymbol{\theta}}^{{}^{\prime}}=\{(s_{i},\boldsymbol{\theta}_{i},s_{i}^{\prime},r)\}$, which is then used to pretrain the critic $Q(s,\boldsymbol{\theta})$. The pretaining of the critic follows typical policy evaluation in SAC, as in Line 38 of Algorithm 1. As in Sec IV-C2, we will discuss in detail how the simple but effective double initialization method can outperform other methods to incorporate expert priors.

To simultaneously leverage parameterized motion skill and expert priors for higher learning efficiency, we modify the maximum-entropy RL, whose objective function is to encourage reward maximization and exploration in skill space:

where $\sum_{t=1}^{T}\gamma^{t}r_{t}$ denotes the accumulated discounted reward return from the environment after one motion skill of length $T$ is executed, $\mathcal{H}\big{(}\pi(\boldsymbol{\theta}|s)\big{)}$ denotes the entropy term, and $\alpha$ denotes the temperature parameter. Instead of learning a policy over raw control actions $\pi(u|s)$ at a single time step, we learn a policy that outputs skill parameters, $\pi(\boldsymbol{\theta}|s)$, which is then used to generate a motion skill by the procedure defined in Section III-A. Each motion skill is tracked for $T$ time steps before the next skill is generated, which follows a typical semi-MDP process [41, 1] with temporal abstraction and accelerated reward signaling [24, 25, 21]. The time horizon, $T$, is fixed and consistent with the motion skill length in Section III-A and Section III-B. While some works investigated variable-length policy conditional on the task and environment  [30, 13, 37], we empirically found a fixed-length policy can achieve strong performance and leave the extension to dynamic-length policies to future work. On top of parameterized motion skills, we adopt the double initialization method to further leverage priors, where we initialize the actor and the critic with the pretrained weight in Section III-B.

We follow the proposed three-stage strategy to evaluate the performance of baselines and our method. As illustrated in Fig. 5, the reward metric in the first evaluation stage indicates that the performance of PPO, SAC, and SPiRL is significantly lower than that of the other three methods. Zhou et al. [50] reported that PPO and SAC had poor performance in MetaDrive tasks even under dense reward settings, so it is not surprising that the performance is poor under the more difficult sparse reward conditions used in this work. Though SPiRL was shown to be effective in manipulation tasks [4, 29], it performs poorly in our driving setting, likely due to the fact that the driving task requires very diverse skills while SPiRL can only learn skills from limited expert demonstrations: since the expert demonstrations of driving tasks are usually unbalanced to mostly consist of data where the vehicle is driving forward, it is hard to ensure that all essential skills are learned or covered in SPiRL agent. We empirically found that the agents learned with SPiRL mostly drive straight forward and can barely exhibit maneuvers, leading to a high collision rate.

The constant-SAC, which simply repeats the same action in the skill window $T$, obtains a slightly lower reward in the three scenarios compared to TaEcRL and ASAP-RL. However, when we move to the second/third evaluation stage and focus on the detailed driving-context metrics, the Constant-SAC performed much worse than TaEcRL and ASAP-RL: the constant-SAC agent drives very conservatively and slowly to avoid collisions. Though a low collision rate and a decent road completion ratio are achieved, the agent usually cannot arrive at the destination within the given time, which leads to a close-to-zero success rate and passed cars per episode. Thus it is not a good driving strategy and shows the importance of proper skill design.

Finally, we compare the results between TaEcRL and ASAP-RL. ASAP-RL achieves better performance in the first 10k iterations in terms of all metrics due to the usage of the expert prior. In addition, ASAP-RL has more passed cars per episode than TaEcRL by a large margin, with better or similar success rates, road completion ratio, and collision rates. This is hypothetically because 1) our ASAP-RL can generate very diverse skills by combining different skill parameters, while TaEcRL relies on offline datasets and can only use limited skills. For example, we found agents learned by TaEcRL usually drive in an erratic way, leading to a low ability to overtake. 2) ASAP-RL can leverage expert priors to accelerate convergence and bias the agent toward better optima and higher performance, due to our proposed skill parameter recovery and double-initialization method. In contrast, TaEcRL is limited by its own algorithm design and cannot utilize expert data. These results suggest ASAP-RL could be a useful tool to enable autonomous driving in interactive, diverse dense-traffic scenarios.

Visualizations of our ASAP-RL agent running on the three scenarios can be found in Fig. 8. The agent drives in dense traffic efficiently and safely with diverse maneuvers such as lane changing, cutting in, braking, etc.