Mobility-Aware Cooperative Caching in Vehicular Edge Computing Based on Asynchronous Federated and Deep Reinforcement Learning

Qiong Wu, , Yu Zhao, Qiang Fan, Pingyi Fan, ,
Jiangzhou Wang, , Cui Zhang This work was supported in part by the National Natural Science Foundation of China under Grant No. 61701197, in part by the open research fund of State Key Laboratory of Integrated Services Networks under Grant No. ISN23-11, in part by the 111 Project under Grant No. B12018. (Corresponding author: Qiong Wu) Qiong Wu and Yu Zhao are with School of Internet of Things Engineering, Jiangnan University, Wuxi 214122, China, and also with the State Key Laboratory of Integrated Services Networks (Xidian University), Xi’an 710071, China (Email: [email protected], [email protected]). Qiang Fan is with Qualcomm, San Jose CA 95110 USA (Email: [email protected]). Pingyi Fan is with the Department of Electronic Engineering, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China (Email: [email protected]). Jiangzhou Wang is with the School of Engineering, University of Kent, CT2 7NT Canterbury, U.K. (Email: [email protected]). Cui Zhang is with Banma Network Technology Co., Ltd., Shanghai 200000, China (Email: [email protected]).

Abstract

The vehicular edge computing (VEC) can cache contents in different RSUs at the network edge to support the real-time vehicular applications. In VEC, owing to the high-mobility characteristics of vehicles, it is necessary to cache the user data in advance and learn the most popular and interesting contents for vehicular users. Since user data usually contains privacy information, users are reluctant to share their data with others. To solve this problem, traditional federated learning (FL) needs to update the global model synchronously through aggregating all users’ local models to protect users’ privacy. However, vehicles may frequently drive out of the coverage area of the VEC before they achieve their local model trainings and thus the local models cannot be uploaded as expected, which would reduce the accuracy of the global model. In addition, the caching capacity of the local RSU is limited and the popular contents are diverse, thus the size of the predicted popular contents usually exceeds the cache capacity of the local RSU. Hence, the VEC should cache the predicted popular contents in different RSUs while considering the content transmission delay. In this paper, we consider the mobility of vehicles and propose a cooperative Caching scheme in the VEC based on Asynchronous Federated and deep Reinforcement learning (CAFR). We first consider the mobility of vehicles and propose an asynchronous FL algorithm to obtain an accurate global model, and then propose an algorithm to predict the popular contents based on the global model. In addition, we consider the mobility of vehicles and propose a deep reinforcement learning algorithm to obtain the optimal cooperative caching location for the predicted popular contents in order to optimize the content transmission delay. Extensive experimental results have demonstrated that the CAFR scheme outperforms other baseline caching schemes.

Index Terms:

cooperative caching, VEC, asynchronous federated learning, deep reinforcement learning

I Introduction

With the development of the internet of vehicles (IoV) and cloud computing, caching technology facilitates various real-time vehicular applications for vehicular users (VUs), such as automatic navigation, pattern recognition and multimedia entertainment [1] [2]. For the standard caching technology, the cloud caches various contents like data, video and web pages. In this scheme, vehicles transmit the required contents to a macro base station (MBS) connected to a cloud server, and could fetch the contents from the MBS, which would cause high content transmission delay from the MBS to vehicles due to the communication congestion caused by frequently requested contents from vehicles [3]. The content transmission delay can be effectively reduced by the emergence of vehicular edge computing (VEC), which caches contents in the road side unit (RSU) deployed at the edge of vehicular networks (VNs) [4]. Thus, vehicles can fetch contents directly from the local RSU, to reduce the content transmission delay. In the VEC, since the caching capacity of the local RSU is limited, if some vehicles cannot fetch their required contents, a neighboring RSU who has the required contents could forward them to the local RSU. The worst case is that vehicles need to fetch contents from the MBS due to both local and neighboring RSUs not having cached the requested contents.

In the VEC, it is critical to design a caching scheme to cache the popular contents. The traditional caching schemes cache contents based on the previously requested contents [5]. However, owing to the high-mobility characteristics of vehicles in VEC, the previously requested contents from vehicles may become outdated quickly, thus the traditional caching schemes may not satisfy all the VUs’ requirement. Therefore, it is necessary to predict the most popular contents in the VEC and cache them in the suitable RSUs in advance. Machine learning (ML) as a new tool, can extract hidden features by training user data to efficiently predict popular contents[6]. However, the user data usually contains privacy information and users are reluctant to share their data directly with others, which make it difficult to collect and train users’ data. Federated learning (FL) can protect the privacy of users by sharing their local models instead of data[7]. In traditional FL, the global model is periodically updated by aggregating all vehicles’ local models[8] -[10]. However, vehicles may frequently drive out of the coverage area of the VEC before they update their local models and thus the local models cannot be uploaded in the same area, which would reduce the accuracy of the global model as well as the probability of getting the predicted popular contents. Hence, it motivates us to consider the mobility of vehicles and propose an asynchronous FL to predict accurate popular contents in VEC.

Generally, the predicted popular contents should be cached in their local RSU of vehicles to guarantee a low content transmission delay. However, the caching capacity of each local RSU is limited and the popular contents may be diverse, thus the size of the predicted popular contents usually exceeds the cache capacity of the local RSU. Hence, the VEC has to determine where the predicted popular contents are cached and updated. The content transmission delay is an important metric for vehicles to provide real-time vehicular application. The different popular contents cached in the local and neighboring RSUs would impact the way vehicles fetch contents, and thus affect the content transmission delay. In addition, the content transmission delay of each vehicle is impacted by its channel condition, which is affected by vehicle mobility. Therefore, it is necessary to consider the mobility of vehicles to design a cooperative caching scheme, in which the predicted popular contents can be cached among RSUs to optimize the content transmission delay. In contrast to some conventional decision algorithms, deep reinforcement learning (DRL) is a favorable tool to construct the decision-making framework and optimize the cooperative caching for the contents in complex vehicular environment [11]. Therefore, we shall employ DRL to determine the optimal cooperative caching to reduce the content transmission delay of vehicles.

In this paper, we consider the vehicle mobility and propose a cooperative Caching scheme in VEC based on Asynchronous Federated and deep Reinforcement learning (CAFR). The main contributions of this paper are summarized as follows.

1)

By considering the mobility characteristics of vehicles including the positions and velocities, we propose an asynchronous FL algorithm to improve the accuracy of the global model.
2)

We propose an algorithm to predict the popular contents based on the global model, where each vehicle adopts the autoencoder (AE) to predict its interested contents based on the global model, while the local RSU collects the interested contents of all vehicles within the coverage area to catch the popular contents.
3)

We elaborately design a DRL framework (dueling deep Q-network (DQN)) to illustrate the cooperative caching problem, where the state, action and reward function have been defined. Then the local RSU can determine optimal cooperative caching to minimize the content transmission delay based on the dueling DQN algorithm.

The rest of the paper is organized as follows. Section II reviews the related works on content caching in VNs. Section III briefly describes the system model. Section IV proposes a mobility-aware cooperative caching in the VEC based on asynchronous federated and deep reinforcement learning method. We present some simulation results in Section V, and then conclude them in Section VI.

II Related Work

In this section, we first review the existing works related to the content caching in vehicular networks (VNs), and then survey the current state of art of the cooperative content caching schemes in VEC.

In [12], Dai et al. proposed a distributed content caching framework with empowering blockchain to achieve security and protect privacy, and considered the mobility of vehicles to design an intelligent content caching scheme based on DRL framework. In [13], Yu et al. proposed a mobility-aware proactive edge caching scheme in VNs that allows multiple vehicles with private data to collaboratively train a global model for predicting content popularity, in order to meet the requirements for computationally intensive and latency-sensitive vehicular applications. In [14], Zhao et al. optimized the edge caching and computation management for service caching, and adopted Lyapunov optimization to deal with the dynamic and unpredictable challenges in VNs. In [15], Jiang et al. constructed a two-tier secure access control structure for providing content caching in VNs with the assistance of edge devices, and proposed the group signature-based scheme for the purpose of anonymous authentication. In [16], Tang et al. proposed a new optimization method to reduce the average response time of caching in VNs, and then adopted Lyapunov optimization technology to constrain the long-term energy consumption to guarantee the stability of response time. In [17], Dai et al. proposed a VN with digital twin to cache contents for adaptive network management and policy arrangement, and designed an offloading scheme based on the DRL framework to minimize the total offloading delay. However, the above content caching schemes in VNs did not take into account the cooperative caching in the VEC environment.

There are some works considering cooperative content caching schemes in VEC. In [18], Qiao et al. proposed a cooperative edge caching scheme in VEC and constructed the double time-scale markov decision process to minimize the content access cost, and employed the deep deterministic policy gradient (DDPG) method to solve the long-term mixed-integer linear programming problems. In [19], Chen et al. proposed a cooperative edge caching scheme in VEC which considered the location-based contents and the popular contents, while designing an optimal scheme for cooperative content placement based on an ant colony algorithm to minimize the total transmission delay and cost. In [20], Yao et al. designed a cooperative edge caching scheme with consistent hash and mobility prediction in VEC to predict the path of each vehicle, and also proposed a cache replacement policy based on the content popularity to decide the priorities of collaborative contents. In [21], Wang et al. proposed a cooperative edge caching scheme in VEC based on the long short-term memory (LSTM) networks, which caches the predicted contents in RSUs or other vehicles and thus reduces the content transmission delay. In [22], Gupta et al. proposed a cooperative caching scheme that jointly considers cache location, content popularity and predicted rating of contents to make caching decision based on the non-negative matrix factorization, where it employs a legitimate user authorization to ensure the secure delivery of cached contents. In [23], Yao et al. proposed a cooperative caching scheme based on the mobility prediction and drivers’ social similarities in VEC, where the regularity of vehicles’ movement behaviors are predicted based on the hidden markov model to improve the caching performance. In [24], Wu et al. proposed a hybrid service provisioning framework and cooperative caching scheme in VEC to solve the profit allocation problem among the content providers (CPs), and proposed an optimization model to improve the caching performance in managing the caching resources. In [25], Yao et al. proposed a cooperative caching scheme based on mobility prediction, where the popular contents may be cached in the mobile vehicles within the coverage area of hot spot. They also designed a cache replacement scheme according to the content popularity to solve the limited caching capacity problem for each edge cache device. In [26], Zhang et al. proposed a cooperative edge caching architecture that focuses on the mobility-aware caching, where the vehicles cache the contents with base stations collaboratively. They also introduced a vehicle-aided edge caching scheme to improve the capability of edge caching. In [27], Liu et al. designed a cooperative caching scheme that allows vehicles to search the unrequested contents. This scheme facilitates the content sharing among vehicles and improves the service performance. In [28], Wang et al. proposed a VEC caching scheme to reduce the total transmission delay. This scheme extends the capability of the data center from the core network to the edge nodes by cooperatively caching popular contents in different CPs. It minimizes the VUs’ average delay according to an iterative ascending price method. In [29], Liu et al. proposed a real-time caching scheme in which edge devices cooperate to improve the caching resource utilization. In addition, they adopted the DRL framework to optimize the problem of searching requests and utility models to guarantee the search efficiency. In [30], Ko et al. proposed an adaptive scheduling scheme consisting of the centralized scheduling mechanism, ad hoc scheduling mechanism and cluster management mechanism to exploit the ad hoc data sharing among different RSUs. In [31], Cui et al. proposed a privacy-preserving data downloading method in VEC, where the RSUs can find popular contents by analyzing encrypted requests of nearby vehicles to improve the downloading efficiency of the network. In [32], Luo et al. designed a communication, computation and cooperative caching framework, where computing-enabled RSUs provide computation and bandwidth resource to the VUs to minimize the data processing cost in VEC.

As mentioned above, no other works has considered the vehicle mobility and privacy of VUs simultaneously to design cooperative caching schemes in VEC, which motivates us to propose a mobility-aware cooperative caching in VEC based on the asynchronous FL and DRL.

Refer to caption — Figure 1: VEC scenario

III System Model

III-A System Scenario

As shown in Fig. 1, we consider a three-tier VEC in an urban scenario that consists of a local RSU, a neighboring RSU, a MBS attached with a cloud and some vehicles moving in the coverage area of the local RSU. The top tier is the MBS deployed at the center of the VEC, while middle tier is the RSUs deployed in the coverage area of the MBS. They are placed on one side of the road. The bottom tier is the vehicles driving within the coverage area of the RSUs.

Each vehicle stores a large amount of VUs’ historical data, i.e., local data. Each data is a vector reflecting different information of a VU, including the VU’s personal information such as identity (ID) number, gender, age and postcode, the contents that the VU may request, as well as the VU’s ratings for the contents where a larger rating for a content indicates that the VU is more interested in the content. Particularly, the rating for a content may be $0$ , which means that it is not popular or is not requested by VUs. Each vehicle randomly chooses a part of the local data to form a training set while the rest is used as a testing set. The time duration of vehicles within the coverage area of the MBS is divided into rounds. For each round, each vehicle randomly selects contents from its testing set as the requested contents, and sends the request information to the local RSU to fetch the contents at the beginning of each round. We consider the MBS has abundant storage capacity and caches all available contents, while the limited storage capacity of each RSU can only accommodate part of contents. Therefore, the vehicle fetches each of the requested content from the local RSU, neighboring RSU or MBS in different conditions. Specifically,

III-A1 Local RSU

If a requested content is cached in the local RSU, the local RSU sends back the requested content to the vehicle. In this case the vehicle fetches the content from the local RSU.

III-A2 neighboring RSU

If a requested content is not cached in the local RSU, the local RSU transfers the request to the neighboring RSU, and the neighboring RSU sends the content to the local RSU if it caches the requested content. Afterward, the local RSU sends back the content to the vehicle. In this case the vehicle fetches the content from the neighboring RSU.

III-A3 MBS

If a content is neither cached in the local RSU nor the neighboring RSU, the vehicle sends the request to the MBS that directly sends back the requested content to the vehicle. In this case, the VU fetches the content from the MBS.

III-B Mobility Model of Vehicles

The model assumes that all vehicles drive in the same direction and vehicles arrive at a local RSU, following a Poisson distribution with the arrival rate $\lambda_{v}$ . Once a vehicle enters the coverage of the local RSU, it sends request information to the local RSU. Each vehicle keeps the same mobility characteristics including position and velocity within a round and may change its mobility characteristics at the beginning of each round. The velocity of different vehicles follows an independent identically distribution. The velocity of each vehicle is generated by a truncated Gaussian distribution, which is flexible and consistent with the real dynamic vehicular environment. For round $r$ , the number of vehicles driving in the coverage area of the local RSU is $N^{r}$ . The set of $N^{r}$ vehicles are denoted as $\mathbb{V}^{r}=\left\{V_{1}^{r},V_{2}^{r},\ldots,V_{i}^{r},\ldots,V_{N^{r}}^{r}\right\}$ , where $V_{i}^{r}$ is vehicle $i$ driving in the local RSU $(1\leq i\leq N^{r})$ . Let $\left\{U_{1}^{r},U_{2}^{r},\ldots,U_{i}^{r},\ldots,U_{N^{r}}^{r}\right\}$ be the velocities of all vehicles driving in the local RSU, where $U_{i}^{r}$ is velocity of $V_{i}^{r}$ . According to [33], the probability density function of $U_{i}^{r}$ is expressed as

f({U_{i}^{r}})=\left\{\begin{aligned} \frac{{{e^{-\frac{1}{{2{\sigma^{2}}}}{{({U_{i}^{r}}-\mu)}^{2}}}}}}{{\sqrt{2\pi{\sigma^{2}}}(erf(\frac{{{U_{\max}}-\mu}}{{\sigma\sqrt{2}}})-erf(\frac{{{U_{\min}}-\mu}}{{\sigma\sqrt{2}}}))}},\\ {U_{min}}\leq{U_{i}^{r}}\leq{U_{max}},\\ 0\qquad\qquad\qquad\qquad\quad otherwise.\end{aligned}\right.

(1)

where $U_{\max}$ and $U_{\min}$ are the maximum and minimum velocity threshold of each vehicle, respectively, and $erf\left(\frac{U_{i}^{r}-\mu}{\sigma\sqrt{2}}\right)$ is the Gauss error function of $U_{i}^{r}$ under the mean $\mu$ and variance $\sigma^{2}$ .

III-C Communication Model

The communications between the local RSU and neighboring RSU adopt the wired link. Each vehicle keeps the same communication model during a round and changes its communication model for different rounds. When the round is $r$ , the channel gain of $V_{i}^{r}$ is modeled as [34]

	$\displaystyle h_{i}^{r}(dis(x,V_{i}^{r}))=\alpha_{i}^{r}(dis(x,V_{i}^{r}))g_{i}^{r}(dis(x,V_{i}^{r})),$		(2)
	$\displaystyle x=S,M,$		(2)

where $x=S$ means the local RSU and $x=M$ means the MBS, $dis(x,V_{i}^{r})$ is the distance between the local RSU $/$ MBS and $V_{i}^{r}$ , $\alpha_{i}^{r}(dis(x,V_{i}^{r}))$ is the path loss between the local RSU $/$ MBS and $V_{i}^{r}$ , and $g_{i}^{r}(dis(x,V_{i}^{r}))$ is the shadowing channel fading between the local RSU $/$ MBS and $V_{i}^{r}$ , which follows a Log-normal distribution.

Each RSU communicates with the vehicles in its coverage area through vehicle to RSU (V2R) link, while the MBS communicates with vehicles through vehicle to base station (V2B) link. Since the distances between the local RSU $/$ MBS and $V_{i}^{r}$ are different in different rounds, V2R $/$ V2B link suffers from different channel impairments, and thus transmit with different transmission rates in different rounds. The transmission rates under V2R and V2B link are calculated as follows.

According to the Shannon theorem, the transmission rate between the local RSU and $V_{i}^{r}$ is calculated as [35]

R_{R,i}^{r}=B\log_{2}\left(1+\frac{p_{B}h_{i}^{r}(dis(S,V_{i}^{r}))}{\sigma_{c}^{2}}\right),

(3)

where $B$ is the available bandwidth, $p_{B}$ is the transmit power level used by the local RSU and $\sigma_{c}^{2}$ is the noise power.

Similarly, the transmission rate between the MBS and $V_{i}^{r}$ is calculated as

R_{B,i}^{r}=B\log_{2}\left(1+\frac{p_{M}h_{i}^{r}(dis(M,V_{i}^{r}))}{\sigma_{c}^{2}}\right),

(4)

where $p_{M}$ is the transmit power level used by MBS.

IV Cooperative Caching Scheme

In this section, we propose a cooperative cache scheme to optimize the content transmission delay in each round $r$ . We first propose an asynchronous FL algorithm to protect VU’s information and obtain an accurate model. Then we propose an algorithm to predict the popular contents based on the obtained model. Finally, we present a DRL based algorithm to determine the optimal cooperative caching according to the predicted popular contents. Next, we will introduce the asynchronous FL algorithm, the popular content prediction algorithm and the DRL-based algorithm, respectively.

IV-A Asynchronous Federated Learning

As shown in Fig. 2, the asynchronous FL algorithm consists of 5 steps as follows.

IV-A1 Select Vehicles

The main goal of this step is to select the vehicles whose staying time in the local RSU is long enough to ensure they can participate in the asynchronous FL and complete the training process.

Each vehicle first sends its mobility characteristics including its velocity and position (i.e., the distance to the local RSU and distance it has traversed within the coverage of the local RSU), then the local RSU selects vehicles according to the staying time that is calculated based on the vehicle’s mobility characteristics. The staying time of $V_{i}^{r}$ in the local RSU is calculated as

T_{r,i}^{staying}=\left(L_{s}-P_{i}^{r}\right)/U_{i}^{r},

(5)

where $L_{s}$ is the coverage range of the local RSU, $P_{i}^{r}$ is the distance that $V_{i}^{r}$ has traversed within the coverage of the local RSU.

The staying time of $V_{i}^{r}$ should be larger than the sum of the average training time $T_{training}$ and inference time $T_{inference}$ to guarantee that $V_{i}^{r}$ can complete the training process. Therefore, if $T_{r,i}^{staying}>T_{training}+T_{inference}$ , the local RSU selects $V_{i}^{r}$ to participate in asynchronous FL training. Otherwise, $V_{i}^{r}$ is ignored.

IV-A2 Download Model

In this step, the local RSU will generate the global model $\omega^{r}$ . For the first round, the local RSU initializes a global model based on the AE, which can extract the hidden features used for popular content prediction. In each round, the local RSU updates the global model and transfers the global model $\omega^{r}$ to all the selected vehicles in the end.

IV-A3 Local Training

In this step, each vehicle in the local RSU sets the downloaded global model $\omega^{r}$ as the initial local model and updates the local model iteratively through training. Afterward, the updated local model will be the feedback to the local RSU. For each iteration $k$ , $V_{i}^{r}$ randomly samples some training data $n_{i,k}^{r}$ from the training set. Then, it uses $n_{i,k}^{r}$ to train the local model based on the AE that consists of an encoder and a decoder. Let $W_{i,k}^{r,e}$ and $b_{i,k}^{r,e}$ be the weight matrix and bias vector of the encoder for iteration $k$ , respectively, $W_{i,k}^{r,d}$ and $b_{i,k}^{r,d}$ be the weight matrix and bias vector of the decoder for iteration $k$ , respectively. Thus the local model of $V_{i,j}^{r}$ for iteration $k$ is expressed as $\omega_{i,k}^{r}=\{W_{i,k}^{r,e},b_{i,k}^{r,e},W_{i,k}^{r,d},b_{i,k}^{r,d}\}$ . For each training data $x$ in $n_{i,k}^{r}$ , the encoder first maps the original training data $x$ to a hidden layer to obtain the hidden feature of $x$ , i.e., $z(x)=f\left(W_{i,k}^{r,e}x+b_{i,k}^{r,e}\right)$ . Then the decoder calculates the reconstructed input $\hat{x}$ , i.e., $\hat{x}=g\left(W_{i,k}^{r,d}z(x)+b_{i,k}^{r,d}\right)$ , where $f{(\cdot)}$ and $g{(\cdot)}$ are the nonlinear and logical activation function [36]. Afterward, the loss function of data $x$ under the local model $\omega_{i,k}^{r}$ is calculated as

l\left(\omega_{i,k}^{r};x\right)=(x-\hat{x})^{2},

(6)

where $\omega^{r}_{i,1}=\omega^{r}$ .

After the loss functions of all the data in $n_{i,k}^{r}$ are calculated, the local loss function for iteration $k$ is calculated as

f(\omega_{i,k}^{r})=\frac{1}{\left|n_{i,k}^{r}\right|}\sum_{x\in n_{i,k}^{r}}l\left(\omega_{i,k}^{r};x\right),

(7)

where $\left|n_{i,k}^{r}\right|$ is the number of data in $n_{i,k}^{r}$ .

Then the regularized local loss function is calculated to reduce the deviation between the local model $\omega_{i,k}^{r}$ and global model $\omega^{r}$ to improve the algorithm convergence, i.e.,

g\left(\omega_{i,k}^{r}\right)=f\left(\omega_{i,k}^{r}\right)+\frac{\rho}{2}\left\|\omega^{r}-\omega_{i,k}^{r}\right\|^{2},

(8)

where $\rho$ is the regularization parameter.

Let $\nabla g(\omega_{i,k}^{r})$ be the gradient of $g\left(\omega_{i,k}^{r}\right)$ , which is referred to as the local gradient. In the previous round, some vehicles may upload the updated local model unsuccessfully due to the delayed training time, and thus adversely affect the convergence of global model [37][38][39]. Here, these vehicles are called stragglers and the local gradient of a straggler in the previous round is referred to as the delayed local gradient. To solve this problem, the delayed local gradient will be aggregated into the local gradient of the current round $r$ . Thus, the aggregated local gradient can be calculated as

\nabla\zeta_{i,k}^{r}=\nabla g(\omega_{i,k}^{r})+\beta\nabla g_{i}^{d},

(9)

where $\beta$ is the decay coefficient and $\nabla g_{i}^{d}$ is the delayed local gradient. Note that $\nabla g_{i}^{d}=0$ if $V_{i}^{r}$ uploads successfully in the previous round.

Then the local model for the next iteration is updated as

\omega^{r}_{i,k+1}=\omega^{r}-\eta_{l}^{r}\nabla\zeta_{i,k}^{r},

(10)

where $\eta_{l}^{r}$ is the local learning rate in round $r$ , which is calculated as

\eta_{l}^{r}=\eta_{l}\max\{1,\log(r)\},

(11)

where $\eta_{l}$ is the initial value of local learning rate.

Then iteration $k$ is finished and $V_{i}^{r}$ randomly samples some training data again to start the next iteration. When the number of iterations reaches the threshold $e$ , $V_{i}^{r}$ completes the local training and upload the updated local model $\omega_{i}^{r}$ to the local RSU.

IV-A4 Upload Model

Each vehicle uploads its updated local model to the local RSU after it completes local training.

IV-A5 Asynchronous aggregation

If the local model of $V_{i}^{r}$ , i.e., $\omega^{r}_{i}$ , is the first model received by the local RSU, the upload is successful and the local RSU updates the global model. Otherwise, the local RSU drops $\omega^{r}_{i}$ and thus the upload is not successful.

When the upload is successful, the local RSU updates the global model $\omega^{r}$ by weighted averaging as follows:

1 Set global model

\omega^{r}

;

2 for each round $r$ from $1$ to $R^{max}$ do

3 for each vehicle $V^{r}_{i}\in\mathbb{V}^{r}$ in parallel do

T_{r,i}^{staying}=\left(L_{s}-P_{i}^{r}\right)/U_{i}^{r}

;

5 if $T_{r,i}^{staying}>T_{training}+T_{inference}$ then

V^{r}_{i}

is selected to participate in asynchronous FL training;

8 for each selected vehicle $V^{r}_{i}$ do

\omega^{r}_{i}\leftarrow\textbf{Vehicle Updates}(\omega^{r},i)

;

10 Upload the local model

\omega^{r}_{i}

to the local RSU;

12 Receive the updated model

\omega^{r}_{i}

;

13 Calculate the weight of the asynchronous aggregation

\chi_{i}

based on Eq. (14);

14 Update the global model based on Eq. (12);

15 return

w^{r+1}

16Vehicle Update(

w,i

17 Input:

w^{r}

18 Calculate the local learning rate

\eta_{l}^{r}

based on Eq. (11);

19 for each local epoch k from $1$ to $e$ do

20 Randomly samples some data

n_{i,k}^{r}

from the training set;

21 for each data $x\in n_{i,k}^{r}$ do

22 Calculate the loss function of data

x

based on Eq. (6);

24 Calculate the local loss function for interation

k

based on Eq. (7);

25 Calculate the regularized local loss function

g\left(\omega_{i,k}^{r}\right)

based on Eq. (8);

26 Aggregate local gradient

\nabla\zeta_{i,k}^{r}

based on Eq. (9);

27 Update the local model

\omega^{r}_{i,k}

based on Eq. (10);

29Set

\omega^{r}_{i}=\omega^{r}_{i,e}

;

30 return

\omega^{r}_{i}

Algorithm 1 The Asynchronous Federated Learning Algorithm

\omega^{r}=\omega^{r-1}+\frac{d_{i}^{r}}{d^{r}}\chi_{i}\omega^{r}_{i},

(12)

where $d_{i}^{r}$ is the size of local data in $V_{i}^{r}$ , $d^{r}$ is the total local data size of the selected vehicles and $\chi_{i}$ is the weight of the asynchronous aggregation for $V_{i}^{r}$ . The weight of the asynchronous aggregation $\chi_{i}$ is calculated by considering the traversed distance of $V_{i}^{r}$ in the coverage area of the local RSU and the content transmission delay from local RSU to $V_{i}^{r}$ to improve the accuracy of the global model and reduce the content transmission delay. Specifically, if the traversed distance of $V_{i}^{r}$ is large, it may have long available time to participate in the training, thus its local model should occupy large weight for aggregation to improve the accuracy of the global model. In addition, the content transmission delay from local RSU to $V_{i}^{r}$ is important because $V_{i}^{r}$ would finally download the content from the local RSU when the content is either cached in the local or neighboring RSU. Thus, if the content transmission delay from local RSU to $V_{i}^{r}$ is small, its local model should also occupy large weight for aggregation to reduce the content transmission delay. The weight of the asynchronous aggregation $\chi_{i}$ is calculated as

\chi_{i}=\mu_{1}{(L_{s}-P_{i}^{r})}+\mu_{2}\frac{s}{R_{R,i}^{r}},

(13)

where $\mu_{1}$ and $\mu_{2}$ are coefficients of the position weight and transmission weight, respectively (i.e., $\mu_{1}+\mu_{2}=1$ ), $s$ is the size of each content. Thus, the content transmission delay from local RSU to $V_{i}^{r}$ is affected by the transmission rate between the local RSU and $V_{i}^{r}$ , i.e., $R_{R,i}^{r}$ . We can further calculate $\chi_{i}$ based on the normalized $L_{s}-P_{i}^{r}$ and $R_{R,i}^{r}$ , i.e.,

\chi_{i}=\mu_{1}\frac{(L_{s}-P_{i}^{r})}{L_{s}}+\mu_{2}\frac{R_{R,i}^{r}}{\max_{k\in N^{r}}\left(R_{R,k}^{r}\right)}.

(14)

Since the local RSU knows $dis(S,V_{i}^{r})$ and $P_{i}^{r}$ for each vehicle $i$ at the beginning of the asynchronous FL, the local RSU can calculate $R_{R,i}^{r}$ according to Eqs. (LABEL:eq2) and (3), and further calculate $\chi_{i}$ according to Eq. (13).

Up to now, the asynchronous FL in round $r$ is finished and the updated global model $\omega^{r}$ is obtained. The process of the asynchronous FL algorithm is shown in Algorithm 1 for ease of understanding, where $R^{max}$ is the maximum number of rounds, $e$ is the maximum number of local epochs. Then, the local RSU sends the obtained model to each vehicle to predict popular contents.

IV-B Popular Content Prediction

In this subsection, we propose an algorithm to predict the popular contents. As shown in Fig. 3, the popular content prediction algorithm consists of the 4 steps as follows.

IV-B1 Data Preprocessing

The VU’s rating for a content is $0$ when VU is uninterested in the content or has not requested a content. Thus, it is difficult to differentiate if a content is an interesting one for the VU when its rating is $0$ . Marking all contents with rating $0$ as uninterested contents is a bias prediction. Therefore, we adopt the obtained model to reconstruct the rating for each content in the first step, which is described as follows.

Each vehicle abstracts a rating matrix from the data in the testing set, where the first dimension of the matrix is VUs’ ID and the second dimension is VU’s ratings for all contents. Denote the rating matrix of $V_{i}^{r}$ as $\bm{R}_{i}^{r}$ . Then, the AE with the obtained model is adopted to reconstruct $\bm{R}_{i}^{r}$ . The rating matrix $\bm{R}_{i}^{r}$ is used as the input data for the AE that outputs the reconstructed rating matrix $\hat{\bm{R}}_{i}^{r}$ . Since $\hat{\bm{R}}_{i}^{r}$ is reconstructed based on the obtained model which reflects the hidden features of data, $\hat{\bm{R}}_{i}^{r}$ can be used to approximate the rating matrix $\bm{R}_{i}^{r}$ . Then, similar to the rating matrix, each vehicle also abstracts a personal information matrix from the data of the testing set, where the first dimension of the matrix is VUs’ ID and the second dimension is VU’s personal information.

IV-B2 Cosine Similarity

$V_{i}^{r}$ counts the number of the nonzero ratings for each VU in $\bm{R}_{i}^{r}$ and marks the VUs with the $1$ $/$ $m$ largest numbers as active VUs. Then, each vehicle combines $\hat{\bm{R}}_{i}^{r}$ and the personal information matrix (denoted as $\bm{H}_{i}^{r}$ ) to calculate the similarity between each active VU and other VUs. The similarity between an active VU $a$ and $b$ is calculated according to cosine similarity [40]

	$\displaystyle\operatorname{sim}_{a,b}^{r,i}=\cos\left(\bm{H}_{i}^{r}(a,:),\bm{H}_{i}^{r}(b,:)\right)$		(15)
	$\displaystyle=\frac{\bm{H}_{i}^{r}(a,:)\cdot\bm{H}_{i}^{r}(b,:)^{T}}{\left\\|\bm{H}_{i}^{r}(a,:)\right\\|_{2}\times\left\\|\bm{H}_{i}^{r}(b,:)\right\\|_{2}}$		(15)

where $\bm{H}_{i}^{r}(a,:)$ and $\bm{H}_{i}^{r}(b,:)$ are the vectors corresponding to the active VU $a$ and $b$ in the combined matrixes, respectively, $\left\|\bm{H}_{i}^{r}(a,:)\right\|_{2}$ and $\left\|\bm{H}_{i}^{r}(b,:)\right\|_{2}$ are the 2-norm of $\bm{H}_{i}^{r}(a,:)$ and $\bm{H}_{i}^{r}(b,:)$ , respectively. Then for each active VU $a$ , $V_{i}^{r}$ selects the VUs with the $K$ largest similarities as the $K$ neighboring VUs of VU $a$ . The ratings of the $K$ neighboring VUs also reflect the preferences of VU $a$ to a certain extent.

IV-B3 Interested Contents

After determining the neighboring VUs of active VUs, in $\bm{R}_{i}^{r}$ , the vectors of neighboring VUs for each active VU are abstracted to construct a matrix $\bm{H}_{K}$ , where the first dimension of $\bm{H}_{K}$ is the IDs of the neighboring VUs for active VUs, while the second dimension of $\bm{H}_{K}$ is the ratings of the contents from neighboring VUs. In $\bm{H}_{K}$ , a content with a VU’s nonzero rating is regarded as the VU’s interested content. Then the number of interested contents is counted for each VU, where the counted number of a content is referred to as the content popularity of the content. $V_{i}^{r}$ selects the contents with the $F_{c}$ largest content popularity as the predicted interested contents.

IV-B4 Popular Contents

After vehicle in the local RSU uploads their predicted interested contents, the local RSU collects and compares the predicted interested contents uploaded from all vehicles to select the contents with the $F_{c}$ largest content popularity as the popular contents. The proposed popular content prediction algorithm is illustrated in Algorithm 2, where $\mathbb{C}^{r}$ is the set of the popular contents and $\mathbb{C}_{i}^{r}$ is the set of interested contents of $V^{r}_{i}$ .

1 Input:

\omega^{r}

2 for each vehicle $V^{r}_{i}\in\mathbb{V}^{r}$ do

3 Construct the rating matrix

\bm{R}_{i}^{r}

and personal information matrix;

\hat{\bm{R}}_{i}^{r}\leftarrow AE(\omega^{r},\bm{R}_{i}^{r})

;

5 Combine

\hat{\bm{R}}_{i}^{r}

and information matrix as

\bm{H}_{i}^{r}

;

\mathbb{C}_{i}^{r}\leftarrow\textbf{Vehicle Predicts}(\bm{H}_{i}^{r},i)

;

7 Uploads

\mathbb{C}_{i}^{r}

to the local RSU;

9Compare received contents and select the

F_{c}

most interested contents into

\mathbb{C}^{r}

10 return

\mathbb{C}^{r}

11 Vehicle Predicts

(\bm{H}_{i}^{r},i)

12 Input:

\bm{H}_{i}^{r},i\in{1,2,...,N^{r}}

13 Calculate the similarity between

V_{i}^{r}

and other vehicles based on Eq. (15);

14 Select the first

K

vehicles with the largest similarity as neighboring vehicles of

V_{i}^{r}

;

15 Construct reconstructed rating matrixes of

K

neighboring vehicles as

\bm{H}_{K}

;

16 Select the

F_{c}

most interested contents as

\mathbb{C}_{i}^{r}

;

17 return

\mathbb{C}_{i}^{r}

Algorithm 2 The Popular Content Prediction Algorithm

The cache capacity of the each RSU $c$ , i.e., the largest number of contents that each RSU can accommodate, is usually smaller than $F_{c}$ . Next, we will propose a cooperative caching to determine where the predicted popular contents can be cached.

IV-C Cooperative Caching Based on DRL

We consider the computation capability of each RSU is powerful and the cooperative caching can be determined within a short time. The main goal is to find an optimal cooperative caching based on DRL to minimize the content transmission delay. Next, we will formulate the DRL framework and then introduce the DRL algorithm.

IV-C1 DRL Framework

The DRL framework includes state, action and reward. The training process is divided into slots. For the current slot $t$ , the local RSU observes the current state $s(t)$ and decides the current action $a(t)$ based on $s(t)$ according to a policy $\pi$ , which is used to generate the action based on the state at each slot. Then the local RSU can obtain the current reward $r(t)$ and observes the next state $s(t+1)$ that is transited from the current state $s(t)$ . We will design $s(t)$ , $a(t)$ and $r(t)$ , respectively, for this DRL framework.

State

We consider the contents cached by the local RSU as the current state $s(t)$ . In order to focus on the contents with high popularity, the contents of the state space $s(t)$ are sorted in descending order based on the predicted content popularity of the $F_{c}$ popular contents, thus the current state can be expressed as $s(t)=\left(s_{1},s_{2},\ldots,s_{c}\right)$ , where $s_{i}$ is the $i$ th most popular content.

Action

Action $a(t)$ represents whether the contents cached in the local RSU need to be relocated or not. In the $F_{c}$ predicted popular contents, the contents that are not cached in the local RSU form a set $\mathbb{N}$ . If $a(t)=1$ , the local RSU randomly selects $n(n<c)$ contents from $\mathbb{N}$ and exchanges them with the $n$ lowest popular contents cached in the local RSU, and then sorts the contents in a descending order based on their content popularity to get $s(t+1)$ . Neighboring RSU also randomly samples $c$ contents from $F_{c}$ popular contents that do not belong to $s(t+1)$ as the cached contents of the neighboring RSU within the next slot $t+1$ . We denote the contents cached by the neighboring RSU as $s_{n}(t+1)$ . If $a(t)=0$ , the contents cached in the local RSU will not be relocated and the neighboring RSU also determines its cached contents, similar to the case when $a(t)=1$ .

Reward

The reward function $r(t)$ is designed to minimize the total content transmission delay to fetch the contents requested by vehicles. Note that the local RSU has recorded all the contents requested by the vehicles. The content transmission delays to fetch a requested content $f$ are different when the content is cached in different places.

If content $f$ is cached in the local RSU, i.e., $f\in s(t)$ , the local RSU transmits content $f$ to $V_{i}^{r}$ , thus the content transmission delay is calculated as

d_{R,i,f}^{r}=\frac{s}{R_{R,i}^{r}},

(16)

where $R_{R,i}^{r}$ is the transmission rate between the local RSU and $V_{i}^{r}$ , which has been calculated by Eq. (3).

If content $f$ is cached in the neighboring RSU, i.e., $f\in s_{n}(t)$ , the neighboring RSU sends the content to the local RSU that forwards the content to $V_{i}^{r}$ , thus the transmission delay is calculated as

\bar{d}_{R,i,f}^{r}=\frac{s}{R_{R,i}^{r}}+\frac{s}{R_{R-R}},

(17)

where $R_{R-R}$ is the transmission rate between the local RSU and neighboring RSU, which is a constant transmission rate in the wired link.

If content $f$ is neither cached in the local RSU nor in the neighboring RSU, i.e., $f\notin s(t)\text{ and }f\notin s_{n}(t)$ , the MBS transmits content $f$ to $V_{i}^{r}$ , thus the content transmission delay is expressed as

d_{B,i,f}^{r}=\frac{s}{R_{B,i}^{r}},

(18)

where $R_{B,i}^{r}$ is the transmission rate between the MBS and $V_{i}^{r}$ , which is calculated according to Eq. (4).

In order to clearly distinguish the content transmission delays under different conditions, we set the reward that $V_{i}^{r}$ fetches content $f$ at slot $t$ as

r_{i,f}^{r}(t)=\begin{cases}e^{-\lambda_{1}d_{R,i,f}^{r}}&f\in s(t)\\ e^{-\left(\lambda_{1}d_{R,i,f}^{r}+\lambda_{2}\bar{d}_{R,i,f}^{r}\right)}&f\in s_{n}(t)\\ e^{-\lambda_{3}d_{M,i,f}^{r}}&f\notin s(t)\text{ and }f\notin s_{n}(t)\end{cases},

(19)

where $\lambda_{1}+\lambda_{2}+\lambda_{3}=1$ and $\lambda_{1}<\lambda_{2}\ll\lambda_{3}$ .

Thus the reward function $r(t)$ is calculated as

r(t)=\sum_{i=1}^{N^{r}}\sum_{f=1}^{F_{i}^{r}}r_{i,f}^{r}(t),

(20)

where $F_{i}^{r}$ is the number of requested contents from $V_{i}^{r}$ .

IV-C2 DRL Algorithm

As mentioned above, the next state will change when the action is $1$ . The dueling DQN algorithm is a particular algorithm which works for the cases where the partial actions have no relevant effects on subsequent states [41]. Specifically, the dueling DQN decomposes the Q-value into two functions $V$ and $A$ . Function $V$ is the state value function that is unrelated to the action, while $A$ is the action advantage function that is related to the action. Therefore, we adopt the dueling DQN algorithm to solve this problem.

1 Initialize replay buffer

\mathcal{D}

, the parameters of the prediction network

\theta

, the parameters of the target network

\theta^{\prime}

;

2 Input: requested contents from all vehicles in the local RSU for round

r

3 for episode from $1$ to $T_{s}$ do

4 Local RSU randomly caches

c

contents from

F_{c}

V Simulation and Analytical Results

TABLE I: Values of the parameters in the experiments.

Parameters of System Model
Parameter	Value	Parameter	Value
$B$	$540$ kHz	$K$	$10$
$m$	$3$	$p_{B}$	$30$ dBm
$p_{M}$	$43$ dBm	$R_{R,R}$	$15$ Mbps
$s$	$100$ bytes	$T_{training}$	$2$ s
$T_{inference}$	$0.5$ s	$U_{\max}$	$60$ km/h
$U_{\min}$	$50$ km/h	$\mu$	$55$ km/h
$\sigma$	$2.5$ km/h	$\sigma_{c}^{2}$	$-114$ dBm
Parameters of Asynchronous FL
Parameter	Value	Parameter	Value
$L_{s}$	$1000$ m	$\beta$	$0.001$
$\eta_{l}$	$0.01$	$\mu_{1}$	$0.5$
$\mu_{2}$	$0.5$	$\rho$	$0.0001$
Parameters of DRL
Parameter	Value	Parameter	Value
$I$	$32$	$\gamma_{D}$	$0.99$
$\eta_{\theta}$	$0.01$	$\lambda_{1}$	$0.0001$
$\lambda_{2}$	$0.4$	$\lambda_{3}$	$0.5999$

We have evaluated the performance of the proposed CAFR scheme in this section.

V-A Settings and Dataset

We simulate a VEC environment on the urban road as shown in Fig. 1 and the simulation tool is Python $3.8$ . The communications between vehicle and RSU/MBS employ the 3rd Generation Partnership Project (3GPP) cellular V2X (C-V2X) architecture, where the parameters are set according to the 3GPP standard [34]. The simulation parameters are listed in Table I. A real-world dataset from the MovieLens website, i.e., MovieLens 1M, is used in the experiments. MovieLens 1M contains $1,000,209$ rating values for $3,883$ movies from $6,040$ anonymous VUs with movie ratings ranging from $0$ to $1$ , where each VU rates at least $20$ movies [42]. MovieLens lM also provides personal information about VUs including ID number, gender, age and postcode. We randomly divide MovieLens lM data set to each vehicle as its local data. Each vehicle randomly chooses $99.8\%$ data from its local data as its training set and $0.2\%$ data as its testing set. For each round, each vehicle randomly samples a part of the movies from testing set as its requested contents.

V-B Performance Evaluation

We use cache hit ratio and the content transmission delay as performance metrics to evaluate the CAFR scheme. The cache hit rate is defined as the probability of fetching requested contents from the local RSU [43]. If a requested content is cached in the local RSU, it can be fetched directly from the local RSU, which is referred to as a cache hit, otherwise, it is referred to as a cache miss. Thus, the cache hit rate is calculated as

\text{ cache hit radio }=\frac{\text{cache hits }}{\text{cache hits }+\text{cache misses }}\times 100\%.

(28)

The content transmission delay indicates the average delay for all vehicles to fetch contents, which is calculated as

\text{content transmission delay}=\frac{D^{\text{total}}}{\text{the number of vehicles }},

(29)

where $D^{\text{total}}$ is the delay for all vehicles to fetch contents, and it is calculated by aggregating the content transmission delay for every vehicle to fetch contents.

We compare the CAFR scheme with other baseline schemes such as:

•

Random: Randomly selecting $c$ contents from the all contents to cache in the local and neighboring RSU.
•

c- $\epsilon$ -greedy: Selecting the contents with $c$ largest numbers of requests based on probability $1-\epsilon$ and selecting $c$ contents randomly based on probability $\epsilon$ to cache in the local RSU. In our simulation, $\epsilon=0.1$ .
•

Thompson sampling: For each round, the contents cached in the local RSU is updated based on the number of cache hits and cache misses in the previous round [9], and $c$ contents with the highest value are selected to cache in the local RSU.
•

FedAVG: Federated averaging (FedAVG) is a typical synchronous FL scheme where the local RSU needs to wait for the local model updates to update its global model according to weighted average method:

$\omega^{r}=\sum_{i=1}^{N^{r}}\frac{d^{r}_{i}}{d^{r}}\omega^{r}_{i}.$ (30)
•

CAFR without DRL: Compared with the CAFR scheme, this scheme does not adopt the DRL algorithm to optimize caching scheme. Specifically, after predicting the popular contents, $c$ contents are randomly selected from the predicted popular contents to cache in the local RSU and neighboring RSU, respectively.

Now, we will evaluate the performance of the CAFR scheme through simulation experiments. In the following performance evaluation, each result is the average value of five experiments.

Fig. 5 shows the cache hit ratio of different schemes under different cache capacities of each RSU, where the result of CAFR is obtained when the vehicle density is $15$ vehicles/km (i.e., the number of vehicles is 15 per kilometer), and the results of other schemes are independent with the vehicle density. It can be seen that the cache hit ratio of all schemes increases with a larger capacity. This is because that the local RSU caches more contents with a larger capacity, thus the requested contents of vehicles are more likely to be fetched from the local RSU. Moreover, it is seen that the random scheme provides the worst cache hit ratio, because the scheme just selects contents randomly without considering the content popularity. In addition, CAFR and c- $\epsilon$ -greedy outperform the random scheme and the thompson sampling. This is because that random and thompson sampling schemes do not predict the caching contents through learning, whereas CAFR and c- $\epsilon$ -greedy decide the caching contents by observing the historical requested contents. Furthermore, CAFR outperforms c- $\epsilon$ -greedy. This is because that CAFR captures useful hidden features from the data to predict the accurate popular contents.

Fig. 6 shows the content transmission delay of different schemes under different cache capacities of each RSU, where the vehicle density is $15$ vehicles/km. It is seen that the content transmission delays of all schemes decrease as the cache capacity increases. This is because that each RSU caches more contents as the cache capacity increases, and each vehicle fetches contents from local RSU and neighboring RSU with a higher possibility, thus reducing the content transmission delay. Moreover, the content transmission delay of CAFR is smaller than other schemes. This is because that the cache hit rate of CAFR is better than those of schemes, and more vehicles can fetch contents from local RSU directly, thus reducing the content transmission delay.

Fig. 7 shows the cache hit ratio and the content transmission delay of the CAFR scheme under different vehicle densities when the cache capacity of each RSU is $100$ . As shown in this figure, the cache hit rate increases as the vehicle density increases. This is because when more vehicles enter the coverage area of the RSU, the global model of the local RSU is trained based on more data, and thus can predict accurately. In addition, the content transmission delay decreases as the vehicle density increases. This is because the cache hit rate increases when the vehicle density increases, which enables more vehicles to fetch contents directly from local RSU.

Fig. 8 compares the cache hit rate of the CAFR scheme and the FedAVG scheme under different rounds when the vehicle density is $15$ vehicles/km and the cache capacity of each RSU is $100$ contents. It can be seen that the cache hit radio of CAFR fluctuates between $22.5\%$ and $24\%$ within $30$ rounds, while the cache hit rate of FedAVG scheme fluctuates between $22\%$ and $23.5\%$ within $30$ rounds. This indicates that the CAFR scheme is slightly better than the FedAVG scheme. This is because the CAFR scheme has considered the vehicles’ mobility characteristics including the positions and velocities to select vehicles and aggregate the local model, thus improving the accuracy of the global model.

Fig. 9 shows the training time of CAFR and FedAVG schemes for each round when the vehicle density is $15$ vehicles/km and the cache capacity of each RSU is $100$ contents. It can be seen that the training time of CAFR scheme for each round is within $1$ s and $2$ s, while the training time of FedAVG scheme for each round is within $22$ s and $24$ s. This indicates that CAFR scheme has a much smaller training time than the FedAVG scheme. This is because the FedAVG scheme needs to aggregate all vehicles’ local models for the global model updating in each round, while the CAFR scheme aggregates as soon as a vehicle’s local model is received for each round.

Fig. 10 shows the cache hit rate and content transmission delay of each episode in the DRL of the CAFR scheme when the vehicle density is $15$ vehicles/km and the cache capacity of RSU is $100$ . As the episode increases, the cache hit rate gradually increases and the content transmission delay decreases gradually in the first ten episodes. This is because the local RSU and neighboring RSU gradually cache appropriate popular contents in the first ten episodes. In addition, it is seen that the cache hit rate and content transmission delay converge at around episode $10$ . This is because the local RSU is able to learn the policy to perform optimal cooperative caching at around $10$ episodes.

Fig. 11 compares the cache hit ratio of the CAFR scheme with CAFR scheme without DRL under different cache capacities of each RSU when the vehicle density is $15$ vehicles/km. As shown in Fig. 11, the cache hit ratio of CAFR outperforms the CAFR without DRL. This is because DRL can determine the optimal cooperative caching according to the predicted popular contents, and thus more suitable popular contents can be cached in the local RSU.

Fig. 12 compares the content transmission delay of the CAFR scheme with CAFR scheme without DRL under different cache capacities of each RSU when the vehicle density is $15$ vehicles/km. As shown in Fig. 12, the content transmission delay of CAFR is less than that of CAFR without DRL. This is because the cache hit ratio of CAFR outperforms the CAFR without DRL and more vehicles can fetch contents from local RSU directly.

VI Conclusions

In this paper, we considered the vehicle mobility and proposed a cooperative caching scheme CAFR to reduce the content transmission delay and improve the cache hit radio. We first proposed an asynchronous FL algorithm to obtain an accurate global model, and then proposed an algorithm to predict the popular contents based on the global model. Afterwards, we proposed a cooperative caching scheme to minimize the content transmission delay based on the dueling DQN algorithm. Simulation results have demonstrated that the CAFR scheme outperforms other baseline caching schemes. According to the theoretical analysis and simulation results, the conclusions can be summarized as follows:

•

CAFR scheme can learn from the local data of vehicles to capture useful hidden features and predict the accurate popular contents.
•

CAFR greatly reduces the training time for each round by aggregating the local model of a single vehicle in each round. In addition, CAFR considers vehicles’ mobility characteristics including the positions and velocities to select vehicles and aggregate the local model, which can improve the accuracy of the training model.
•

The DRL in the CAFR scheme determines the optimal cooperative caching policy according to the predicted popular contents, and thus more suitable popular contents are cached in the local RSU and neighboring RSU to reduce the content transmission delay.

References

[1] L. Liu, C. Chen, Q. Pei, S. Maharjan and Y. Zhang, "Vehicular Edge Computing and Networking: A Survey," Mob. Networks Appl., vol. 26, pp. 1145-1168, 2021.
[2] Q. Wu, Y. Zhao and Q. Fan, "Time-Dependent Performance Modeling for Platooning Communications at Intersection," IEEE IoT-J, doi: 10.1109/JIOT.2022.3161028.
[3] Y. Dai, D. Xu, S. Maharjan, G. Qiao and Y. Zhang, "Artificial Intelligence Empowered Edge Computing and Caching for Internet of Vehicles," IEEE Wireless Commun. Mag., vol. 26, no. 3, pp. 12-18, Jun. 2019.
[4] M. A. Javed and S. Zeadally, "AI-Empowered Content Caching in Vehicular Edge Computing: Opportunities and Challenges," IEEE Network, vol. 35, no. 3, pp. 109-115, May/June 2021.
[5] A. Narayanan, S. Verma, E. Ramadan, P. Babaie, and Z.-L. Zhang, "DeepCache: A deep learning based framework for content caching," Proc. Workshop Netw. Meets AI ML NetAI, 2018, pp. 48–53.
[6] M. Yan, C. A. Chan, W. Li, L. Lei, A. F. Gygax and C. -L. I, "Assessing the Energy Consumption of Proactive Mobile Edge Caching in Wireless Networks," IEEE Access, vol. 7, pp. 104394-104404, 2019, doi: 10.1109/ACCESS.2019.2931449.
[7] M. Chen, Z. Yang, W. Saad, C. Yin, H. V. Poor and S. Cui, "A Joint Learning and Communications Framework for Federated Learning Over Wireless Networks," IEEE Trans. Wireless Commun., vol. 20, no. 1, pp. 269-283, Jan. 2021, doi: 10.1109/TWC.2020.3024629.
[8] X. Wang, C. Wang, X. Li, V. C. M. Leung and T. Taleb, "Federated Deep Reinforcement Learning for Internet of Things With Decentralized Cooperative Edge Caching," IEEE IoT-J, vol. 7, no. 10, pp. 9441-9455, Oct. 2020, doi: 10.1109/JIOT.2020.2986803.
[9] L. Cui, X. Su, Z. Ming, Z. Chen, S. Yang, Y. Zhou and W. Xiao, "CREAT: Blockchain-assisted Compression Algorithm of Federated Learning for Content Caching in Edge Computing," IEEE J-IoT, doi: 10.1109/JIOT.2020.3014370.
[10] R. Cheng, Y. Sun, Y. Liu, L. Xia, D. Feng and M. Imran, "Blockchain-empowered Federated Learning Approach for An Intelligent and Reliable D2D Caching Scheme," IEEE J-IoT, doi: 10.1109/JIOT.2021.3103107.
[11] H. Zhu, Q. Wu, X. -J. Wu, Q. Fan, P. Fan and J. Wang, "Decentralized Power Allocation for MIMO-NOMA Vehicular Edge Computing Based on Deep Reinforcement Learning," IEEE J-IoT, doi: 10.1109/JIOT.2021.3138434.
[12] Y. Dai, D. Xu, K. Zhang, S. Maharjan and Y. Zhang, "Deep Reinforcement Learning and Permissioned Blockchain for Content Caching in Vehicular Edge Computing and Networks," IEEE Trans. Veh. Technol., vol. 69, no. 4, pp. 4312-4324, April 2020, doi: 10.1109/TVT.2020.2973705.
[13] Z. Yu, J. Hu, G. Min, Z. Zhao, W. Miao and M. S. Hossain, "Mobility-Aware Proactive Edge Caching for Connected Vehicles Using Federated Learning," IEEE Trans. Intell. Transp. Syst., vol. 22, no. 8, pp. 5341-5351, Aug. 2021, doi: 10.1109/TITS.2020.3017474.
[14] J. Zhao, X. Sun, Q. Li and X. Ma, "Edge Caching and Computation Management for Real-Time Internet of Vehicles: An Online and Distributed Approach," IEEE Trans. Intell. Transp. Syst., vol. 22, no. 4, pp. 2183-2197, April 2021, doi: 10.1109/TITS.2020.3012966.
[15] S. Jiang, J. Liu, L. Huang, H. Wu and Y. Zhou, "Vehicular Edge Computing Meets Cache: An Access Control Scheme for Content Delivery," ICC 2020 - 2020 IEEE International Conference on Communications (ICC), 2020, pp. 1-6, doi: 10.1109/ICC40277.2020.9148755.
[16] C. Tang, C. Zhu, H. Wu, Q. Li and J. J. P. C. Rodrigues, "Toward Response Time Minimization Considering Energy Consumption in Caching-Assisted Vehicular Edge Computing," IEEE J-IoT, vol. 9, no. 7, pp. 5051-5064, 1 April1, 2022, doi: 10.1109/JIOT.2021.3108902.
[17] Y. Dai and Y. Zhang, "Adaptive Digital Twin for Vehicular Edge Computing and Networks," Journal of Communications and Information Networks (JCIN), vol. 7, no. 1, pp. 48-59, March 2022.
[18] G. Qiao, S. Leng, S. Maharjan, Y. Zhang and N. Ansari, "Deep Reinforcement Learning for Cooperative Content Caching in Vehicular Edge Computing and Networks," IEEE J-IoT , vol. 7, no. 1, pp. 247-257, Jan. 2020, doi: 10.1109/JIOT.2019.2945640.
[19] J. Chen, H. Wu, P. Yang, F. Lyu and X. Shen, "Cooperative Edge Caching With Location-Based and Popular Contents for Vehicular Networks," IEEE Trans. Veh. Technol., vol. 69, no. 9, pp. 10291-10305, Sept. 2020, doi: 10.1109/TVT.2020.3004720.
[20] L. Yao, X. Xu, J. Deng, G. Wu and Z. Li, "A Cooperative Caching Scheme for VCCN With Mobility Prediction and Consistent Hashing," IEEE Trans. Intell. Transp. Syst., doi: 10.1109/TITS.2022.3171071.
[21] R. Wang, Z. Kan, Y. Cui, D. Wu and Y. Zhen, "Cooperative Caching Strategy With Content Request Prediction in Internet of Vehicles," IEEE J-IoT, vol. 8, no. 11, pp. 8964-8975, 1 June1, 2021, doi: 10.1109/JIOT.2021.3056084.
[22] D. Gupta, S. Rani, S. H. Ahmed, S. Garg, M. Jalil Piran and M. Alrashoud, "ICN-Based Enhanced Cooperative Caching for Multimedia Streaming in Resource Constrained Vehicular Environment," IEEE Trans. Intell. Transp. Syst., vol. 22, no. 7, pp. 4588-4600, July 2021, doi: 10.1109/TITS.2020.3043593.
[23] L. Yao, Y. Wang, X. Wang and G. WU, "Cooperative Caching in Vehicular Content Centric Network Based on Social Attributes and Mobility," IEEE Trans. Mob. Comput., vol. 20, no. 2, pp. 391-402, 1 Feb. 2021, doi: 10.1109/TMC.2019.2944829.
[24] R. Wu, G. Tang, T. Chen, D. Guo, L. Luo and W. Kang, "A Profit-Aware Coalition Game for Cooperative Content Caching at the Network Edge," IEEE J-IoT, vol. 9, no. 2, pp. 1361-1373, 15 Jan.15, 2022, doi: 10.1109/JIOT.2021.3087719.
[25] L. Yao, A. Chen, J. Deng, J. Wang and G. Wu, "A Cooperative Caching Scheme Based on Mobility Prediction in Vehicular Content Centric Networks," IEEE Trans. Veh. Technol., vol. 67, no. 6, pp. 5435-5444, June 2018, doi: 10.1109/TVT.2017.2784562.
[26] K. Zhang, S. Leng, Y. He, S. Maharjan and Y. Zhang, "Cooperative Content Caching in 5G Networks with Mobile Edge Computing," IEEE Wirel. Commun., vol. 25, no. 3, pp. 80-87, JUNE 2018, doi: 10.1109/MWC.2018.1700303.
[27] K. Liu, J. K. -Y. Ng, J. Wang, V. C. S. Lee, W. Wu and S. H. Son, "Network-Coding-Assisted Data Dissemination via Cooperative Vehicle-to-Vehicle/-Infrastructure Communications," IEEE Trans. Intell. Transp. Syst., vol. 17, no. 6, pp. 1509-1520, June 2016, doi: 10.1109/TITS.2015.2495269.
[28] S. Wang, Z. Zhang, R. Yu and Y. Zhang, "Low-latency caching with auction game in vehicular edge computing," 2017 IEEE/CIC International Conference on Communications in China (ICCC), 2017, pp. 1-6, doi: 10.1109/ICCChina.2017.8330526.
[29] M. Liu, D. Li, H. Wu, F. Lyu and X. S. Shen, "Real-Time Search-Driven Caching for Sensing Data in Vehicular Networks," IEEE J-IoT, doi: 10.1109/JIOT.2021.3134964.
[30] B. Ko, K. Liu, S. H. Son and K. -J. Park, "RSU-Assisted Adaptive Scheduling for Vehicle-to-Vehicle Data Sharing in Bidirectional Road Scenarios," IEEE Trans. Intell. Transp. Syst., vol. 22, no. 2, pp. 977-989, Feb. 2021, doi: 10.1109/TITS.2019.2961705.
[31] J. Cui, L. Wei, H. Zhong, J. Zhang, Y. Xu and L. Liu, "Edge Computing in VANETs-An Efficient and Privacy-Preserving Cooperative Downloading Scheme," IEEE J. Sel. Areas Commun., vol. 38, no. 6, pp. 1191-1204, June 2020, doi: 10.1109/JSAC.2020.2986617.
[32] Q. Luo, C. Li, T. H. Luan and W. Shi, "Collaborative Data Scheduling for Vehicular Edge Computing via Deep Reinforcement Learning," IEEE J-IoT, vol. 7, no. 10, pp. 9637-9650, Oct. 2020, doi: 10.1109/JIOT.2020.2983660.
[33] Y. AlNagar, S. Hosny, and A. A. El-Sherif, "Towards mobility-aware proactive caching for vehicular ad hoc networks," Proc. IEEE Wireless Commun. Netw. Conf. Workshop (WCNCW), Apr. 2019, pp. 1–6.
[34] "Study on LTE-based V2X Services," 3rd Generation Partnership Project (3GPP), Technical Specification (TS) 36.885, June 2016, version 14.0.0.
[35] J. Chen, H. Wu, P. Yang, F. Lyu and X. Shen, "Cooperative Edge Caching With Location-Based and Popular Contents for Vehicular Networks," IEEE Trans. Veh. Technol., vol. 69, no. 9, pp. 10291-10305, Sept. 2020, doi: 10.1109/TVT.2020.3004720.
[36] A. Ng, "Sparse autoencoder," CS294A Lecture notes, vol. 72, no. 2011, pp. 1–19, 2011.
[37] Y. Chen, Y. Ning, M. Slawski and H. Rangwala, "Asynchronous online federated learning for edge devices with non-IID data," 2020 IEEE International Conference on Big Data (Big Data), pp. 15-24, 2020.
[38] C. Xie, S. Koyejo, and I. Gupta, "Asynchronous federated optimization," arXiv preprint arXiv:1903.03934, 2019.
[39] H. -S. Lee and J. -W. Lee, "Adaptive Transmission Scheduling in Wireless Networks for Asynchronous Federated Learning," IEEE J. Sel. Areas Commun., vol. 39, no. 12, pp. 3673-3687, Dec. 2021, doi: 10.1109/JSAC.2021.3118353.
[40] Z. Yu, J. Hu, G. Min, H. Lu, Z. Zhao, H. Wang and N. Georgalas, "Federated Learning Based Proactive Content Caching in Edge Computing," 2018 IEEE Global Communications Conference (GLOBECOM), 2018, pp. 1-6, doi: 10.1109/GLOCOM.2018.8647616.
[41] Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot and N. Freitas, "Dueling Network Architectures for Deep Reinforcement Learning," ArXiv, abs/1511.065811 (2016): n. pag.
[42] F. Harper and J. Konstan, "The movielens datasets: History and context," ACM Trans. Interact. Intell. Syst., vol. 5, no. 4, p. 19, 2016.
[43] S. Müller, O. Atan, M. van der Schaar and A. Klein, "Context-Aware Proactive Content Caching With Service Differentiation in Wireless Networks," IEEE Trans. Wireless Commun., vol. 16, no. 2, pp. 1024-1036, Feb. 2017.