Personalized Federated Learning of Driver Prediction Models for Autonomous Driving

Even in the setting we showed above, general FL algorithms such as FedAVG can potentially perform well. Namely, we can obtain a better set of parameters $\hat{\theta}_{k}$ than randomly initialized without sharing $\mathbf{x}_{k}$ in a cloud environment $\mathcal{C}$. However, we hypothesize that we can improve $\hat{\theta}_{k}$ by syncing parameters from a global model as in standard FL, but also specializing the model for robot $\mathcal{R}_{k}$ with the local data $\mathbf{x}_{k}$. Therefore, we propose a variant of FL algorithm with an additional training step for personalization, which is shown in Alg. 1. There are two major differences from general FL. First, we add a training step executed on each robot $\mathcal{R}_{k}$ for personalization in Alg. 1 line 7. Unlike some prior personalized FL works such as [8], we assume $\mathcal{R}_{k}$ can execute many more training steps (e.g. not one step of SGD but a few epochs) because today’s robot systems can do so with modern low-power deep learning accelerators. Our main innovation is to carefully limit parameter updates to avoid overfitting by personalized training. This is inspired by standard methods in fine-tuning DNNs using transfer learning, which freeze some layers’ parameters. Also, retraining all the parameters on a local dataset, which can be of limited size and highly biased, often causes overfitting. We found that it is effective to slow down the update of parameters that have common features across many robots’ data during personalized training. To do so, our key insight is to estimate the variance of different parameters across robots during FL, which is codified in Algorithm lines 7, 13, and 14.

Specifically, we now discuss how to properly adjust the extent of parameter updates. In short, we propose to apply an adaptive learning rate to FL for robotics systems. Generally, it is hard to accurately determine which parameters learn common dynamics and shared trends across robots, especially when training a model such as a DNN with a huge number of parameters. However, we can expect such common parameters tend to eventually converge to a similar value on any dataset. Therefore, we propose a method to estimate how much parameters have in common from the variation of parameters learned by each robot in a fleet. The estimated degree of similarity finally affects the learning rate of each parameter (lines 13-14). As a result, higher learning rates are assigned to parameters with higher variation in personalized training, which might indicate they need specialization per robot. Specifically, we introduce a variation vector $\vec{\sigma}$ whose elements represent the degree of variability of each parameter. Then, the learning rate of each parameter is determined based on a ratio of the variability. Namely, the learning rate of the $i$-th parameter of parameter vector $\theta$ is denoted by $L\sigma_{i}/\max\vec{\sigma}$, where $L$ is a constant representing the maximum learning rate. In practical systems, we can cluster parameters (e.g. parameters of a layer of a DNN) and calculate learning rates for each cluster for computational scalability.

In this section, we introduce a very simple task where a point mass must be controlled by a linear-quadratic regulator (LQR) controller in a 1-dimensional space, as illustrated in Fig. 3. The goal is to learn parameters of an LQR controller that moves the point mass from an arbitrary initial state to the origin. Importantly, we have $K=3$ robots with identical dynamics, but different LQR cost functions. Our goal is to estimate both the dynamics model and LQR control policy parameters using local datasets $\mathbf{x}_{k}$ for robot $k$ and personalized federated learning to share knowledge amongst robots $k$. We now introduce the system dynamics and LQR controller for each robot $k$.

Each robot $k$’s system state at discrete time $t$ is denoted by $x^{k}_{t}=[y^{k}_{t},v^{k}_{t}]$, which is a vector of its velocity $v^{k}_{t}$ and position $y^{k}_{t}$. The dynamics of this simple linear system are given by $x^{k}_{t+1}=Ax^{k}_{t}+Bu^{k}_{t}$, where $A\in\mathbb{R}^{2\times 2}$ is the dynamics matrix, $B\in\mathbb{R}^{2\times 1}$ is the control matrix, and $u^{k}_{t}\in\mathbb{R}$ is the control input. Assuming a unit-mass system, the dynamics of our linear system are given by:

Next, we define a cost function to identify the optimal control input. The standard LQR cost is defined as $\sum_{t=0}^{\infty}((x^{k}_{t})^{\mathrm{T}}Q^{k}x^{k}_{t}+(u^{k}_{t})^{\mathrm{T}}R^{k}u^{t}_{k})$, where $Q^{k}$ and $R^{k}$ are weight matrices ( $Q^{k}\in\mathbb{R}^{2\times 2},R^{k}\in\mathbb{R}^{1\times 1}$ ). It is well-known that the optimal control input $u^{*,k}_{t}=\mathcal{K}^{\mathrm{LQR}}x^{k}_{t}$, where $\mathcal{K}^{\mathrm{LQR}}$ is a feedback matrix that arises from the solution to the discrete time Riccati Equation.

In our toy experiment, we assume each robot $\mathcal{R}_{k}$ knows the parametric form of the linear dynamics equations and linear feedback policy, but does not know the specific parameters $A$, $B$, $Q^{k}$ and $R^{k}$. Instead, each robot must learn these parameters from measured rollouts (e.g., trajectory data) of an expert controlling the system. We note that it is sufficient to learn dynamics matrices $A$ and $B$ to predict the next system state from the current state and applied control, as well as the LQR feedback matrix $\mathcal{K}^{\mathrm{LQR},k}$ to general controls from the current system state.

Crucially, since each robot has the same dynamics, our experiments should show that $A$ and $B$ quickly converge to the same global value using knowledge transfer. However, since the cost functions differ, we should see each robot converge to its unique LQR feedback matrix, illustrating the benefits of personalization.

Fig. 4 provides further evidence for the benefits of our APFL method by showing how close the learned parameters $\hat{A}^{k}$, $\hat{B}^{k}$, and $\hat{\mathcal{K}}^{\mathrm{LQR},k}$ differ from the ground-truth parameters. Clearly, our method of APFL (purple) shows strong benefits of not only personalization, but also our scheme of adaptively adjusting the learning rate of parameters based on their variation across robots. Specifically, we achieve both low errors for dynamics and control matrices (x-axis) and the estimated LQR feedback matrix (y-axis). Next we evaluate the benefits of APFL on two challenging AV tasks featuring rich human-robot interaction in our user studies.

The second case study is shown on the left side of Fig. 5, where a human-driven vehicle and AV must safely interchange lanes in a short distance, inspired by [24]. The AV is equipped with a trajectory forecasting model that predicts the future motion of the human-driven car, conditioned on past interaction history and a candidate robot future control decision. A key feature of our work is we performed a study with 7 real human drivers, who exhibited a mixture of aggressive and cautious driving styles when deciding whether to overtake or yield to the AV. To quickly collect data with diverse human volunteers, we used the CARLO 2D driving simulator [3], which is a light-weight version of CARLA without photo-realistic rendered scenes. Crucially, our FL framework ensures that the human-robot interaction datasets, which often show risky driving styles from human subjects, are kept private and never shared with a central server.

When the robot controller decides a series of control inputs $[(c^{rth}_{t+1},c^{rst}_{t+1}),(c^{rth}_{t+2},c^{rst}_{t+2}),..,(c^{rth}_{t+\tau},c^{rst}_{t+\tau})]$ at time $t$, it computes the sum of the expected cost function in the time horizon $H>\tau$ for all possible robot control candidates. The controller chooses a series of control inputs that minimizes the sum of the expected cost, which naturally depends on anticipated future human controls. We set the control interval $\tau$ and planning horizon $H$ to $3$ and $20$ respectively, which performed well with an acceptable compute budget.

Once there is enough longitudinal distance to swap lanes, the controller switches to the second phase, and just works as a lane change controller. Since we observed humans’ driving styles almost do not differ during the lane change (and largely govern when to safely initiate it), the lane change controller is a static LQR controller. This static LQR controller drives the robot to a target lateral position $p^{rx}$ representing the x-coordinate of a target lane with longitudinal velocity $v^{ry}$. In the following quantitative evaluation, we incorporate human trajectory prediction models trained with the benchmarks in Table 1 into the MPC-based AV controller. Then, we compare these controllers in terms of how safe and efficiently the cars are able to swap lanes given the different human’s driving styles.

For the training set, seven humans play the lane swapping scenario running on the CARLO simulator with 50 initial states above. Since the prediction model with CVAE shown in Fig. 6 is not trained at this time, the robot controller uses a very simple model based on only the current state. Specifically, the predicted human control $\hat{C}^{h}_{i}$ for future time $i$ is always the current human control $C^{h}_{t}$ for any $i\in\{t+1,t+2,..,t+\tau\}$.

Once we gather initial data for how humans respond to a baseline AV without a sophisticated model of human behavior, we can train a better driven prediction model. This prediction model is in turn used in the AV’s MPC and resulting rollouts against human drivers allows us to generate an evaluation set. Notably, this evaluation set is used to fine-tune and re-train the human trajectory forecasting model.

Similar to the experiment IV-A, each of the 7 robot cars observes only one type of data corresponding to one human driver. During training, both the default learning rate and $L$ are 0.001, and the loss function is the evidence lower bound objective (ELBO) [13, 21]. The number of epochs is 30 for each training step in Alg. 1. We used 40 sessions for training and 10 challenging held-out sessions for testing. We trained the model with four algorithms shown in Tab. I. Similar to the experiment in Sec. IV-A, the federated learning algorithms use FedAVG to aggregate trained parameters. In our method (APFL), we used layer-wise learning rate instead of parameter-wise to improve compute efficiency, which essentially groups parameters in one layer.

Additionally, evaluation results show an AV (with our learned prediction models in-the-loop) can safely and efficiently swap lanes with actual human drivers in test episodes in the CARLO simulator. Fig. 8 shows distributions of key quantitative metrics over the test episodes, such as the elapsed time until the AV starts a lane change, distance between cars when lane changes commence, and the mean cost for each session. Clearly, an AV using our APFL model can start to change lanes quickly with a larger safe distance to the human car.

We now consider a more sophisticated AV driving scenario in the photo-realistic, standard CARLA [6] simulator. As shown on the right of Fig. 5, our scenario is similar to the lane-swap scenario of the previous section. However, there is additional complexity since there is a gray human-driven car that starts at random distances from the AV with random velocities, which requires the AV to reason about whether there is enough safety margin to overtake the red car.

Unlike the lane swapping case, we use a programmed synthetic controller for human cars to test on uniformly diverse driving styles, including risky behaviors. This synthetic human controller is also a two-phase controller, but control inputs are decided according to a target velocity in the first phase. The target velocity is chosen as high $v_{\mathrm{high}}$ or low $v_{\mathrm{low}}$, where the high velocity $v_{\mathrm{high}}$ is only chosen if the human car has a safe relative difference from the other cars, modulated by its risk tolerance parameter $\gamma$:

In the above equation, $\gamma$ is a parameter representing the degree of preference for overtaking robot cars, which we henceforth called the risk-tolerance parameter. Further, in the above equation, $D_{\mathrm{rel}}$ indicates the relative gap between cars based on the positions and velocities denoted by $(p^{hy}_{t}-p^{ry}_{t})+(v^{hy}_{t}-v^{ry}_{t})$, and $p^{gy}_{t}$ and $D_{\mathrm{safe}}$ are a longitudinal position of the gray car and a safety distance constant, respectively.

The target velocity for the human-driven vehicle is set depending on the relative velocity to the robot car driving alongside it. Once the target velocity is determined, the synthetic human-driver controller computes control inputs with a PID controller in CARLA. We tested 5 kinds of driving styles whose risk-sensitivity ($\gamma$) parameters were in the range $[-1.0,-0.5,0,0.5,1.0]$. These driving styles include a mixture of cautious and aggressive styles to ensure a diverse dataset. For instance, a human car with $\gamma=1.0$ sometimes tries to pass the robot car even when the current relative position to the robot car is minimal.

We collected a diverse dataset in the same manner as in the previous scenario. Specifically, we generated 54 initial states where the relative position between cars and velocities are uniformly distributed. We trained trajectory forecasting models according to Table I with the same hyper-parameters and LSTM encoder-decoder model as the previous scenario. Finally, we integrated the trained trajectory forecasting models into the AV’s controller and evaluated it in 9 challenging scenarios where the robot cars need to change their behavior depending on humans’ driving styles.