Joint Optimization of Spectrum and Energy Efficiency Considering the C-V2X Security: A Deep Reinforcement Learning Approach

The subchannels multiplexing allocation matrix is

The transmission rate of nth V2I link can be expressed as

where ${P_{V2I}}$ and ${P_{V2V}}$ mean the transmission power of nth V2I link and mth V2V link, respectively. ${a_{mn}}$ means the subchannels multiplexing allocation element. ${h_{n,b}}$ and ${h_{m,b}}$ denote the power gains of the channel from nth V2I link to the BS and from mth V2V link using the same subchannel with nth V2I link to the BS, respectively. ${\sigma^{2}}$ is the noise power. And the transmission rate of mth V2V link can be expressed as

where ${h_{m}}$ and ${h_{m,j}}$ are the power gain of the channel corresponding to mth V2V link and the interference power gain of jth V2V link. ${I_{m}}$ means the interference to mth V2V link. Other elements are the same as mentioned above.

For the eavesdropper, the transmission rate of eavesdropping on a V2V link is

where ${h_{m,e}}$, ${h_{n,e}}$ and ${h_{j,e}}$ are the power gain of channel eavesdropping on other V2V links, the interference power gain of nth V2I link and jth V2V link, respectively, and $I_{eve}$ is the interference to the eavesdropper. The concept of secrecy rate is the non-negative difference between transmission rate and eavesdropping rate. The secrecy rate of V2V links are

where y = ${[x]^{+}}$ means

In a dense and resource-limited traffic environment like intersections, there is a marked drop in the SE and EE of V2V and V2I links. And there may be users called eavesdroppers, who illegally monitor vehicular information and keep records of it to get their privacy. This article optimizes the vehicular platooning network resource utilization including both SE and EE under the premise of ensuring the V2V secrecy rates.

Multi-objective optimization problems with different dimensions need to figure out the Pareto optimality. To optimize the problem efficiently and comprehensively, a compositive efficiency is defined as the ratio of SE and the total power consumption, which integrates the meaning of SE and EE [9]. The compositive efficiency including SE and EE of V2V links is

where ${P_{C}}$ is the circuit power, ${B_{n}}$ means the bandwidth, and the compositive efficiency of V2I links is

Overall, the objective optimization function can be summarized as

subject to:

(9) is the objective optimization function to maximize the SE and EE of V2V and V2I links. (10a) demonstrates mth V2V link can most reuse one V2I link. (10b) means the constraint of the V2V secrecy rate. (10c) and (10d) limit the V2I and V2V transmission power. (10e) is the subchannels multiplexing allocation element limited on 0 or 1.

In the Part B of Section III, the optimization function (9) is a combinatorial optimization problem with discrete integers. Obviously, the optimization problem is a NP-hard problem, so it is difficult to obtain the optimal solution. Based on these assumptions and models, the SEED is proposed to optimize the resource utilization of V2V and V2I links, which is a security-aware approach to enhancing the spectrum efficiency and energy efficiency based on deep reinforcement learning.

Deep reinforcement learning (DRL) is a combination of deep learning and reinforcement learning. DRL sees learning as a heuristic evaluation process. The main idea of DRL is that an agent may learn by interacting with its environment using the experience gathered. It should be able to optimize some objectives given in the form of cumulative rewards [10]. Compared to traditional Q learning, deep Q network is an improved algorithm which solves the problem of state and action space explosion. Thus it is used to solve our optimization problem heuristically [11]. The parameters of DQN are set below in this article’s scenario.

• •

  Agent: v $\in\{{v_{1}},{v_{2}},...,{v_{m}},...\}$, where v is a V2V link existing in the network.

• •

  State Space: s $\in\{H_{m},H_{n,b},H_{m,e},H_{n,e},I_{i},N_{i}\}$, where s is the state observed from environment by the agent v. ${H_{m}}$ are the subchannel power gains of V2V links. ${H_{n,b}}$ are the subchannel power gains from V2I UEs to BS. ${H_{m,e}}$ and ${H_{n,e}}$ mean are power gains to eavesdroppers. ${I_{i}}$ are the interferences to the agents. ${N_{i}}$ are the subchannels selected by local observations.

• •

  Action Space: a $\in\{\{{N_{1}},{P_{V2{V^{(1)}}}}\},\{{N_{2}},{P_{V2{V^{(2)}}}}\},...,\\ \{{N_{K}},{P_{V2{V^{(K)}}}}\}\}$, where ${N_{k}}$ is the subchannel that the agent selects, ${P_{V2{V^{(K)}}}}$ is the V2V transmission power level. Vehicles platooning’s communication is different from that of free vehicles, because platoon members mainly transmit and receive intra-platoon messages (e.g. intra-platoon CAM) to maintain the internal safety of the platoon. Therefore, it isn’t necessary for them to use the transmission power, which closes to the maximum power in the standard protocol of 3GPP. Thus, the V2V transmission power level of platoon members is lower than that of platoon leaders and free vehicles.

• •

  Reward Function:

  $$\emph{r}=\left\{\begin{array}[]{l}{\lambda_{\alpha}}{\zeta_{V2V}}+{\lambda_{\beta}}{\zeta_{V2I}},{\rm{}}if{\rm{}}{R_{m,\sec urity}}\geq{R_{T}}\\ {\rm{-}}1,{\rm{}}otherwise\end{array}\right.$$

  (11)

where ${\lambda_{\alpha}}$ and ${\lambda_{\beta}}$ are weighted coefficients representing its importance in the vehicular network, and ${\lambda_{\alpha}}$ + ${\lambda_{\beta}}$ = 1. A negative reward is introduced in order to make DQN select better policies. Obviously, the threshold of V2V secrecy rate is the boundary of positive and negative rewards, and guarantees that the agent can get more positive rewards if it have a certain number of secrecy rates. This reward function is shared by each agent. During the interaction between each agent and the environment, the corresponding actions are continuously updated to obtain the maximum cumulative reward.

The structure of DQN in the SEED is illustrated in figure 2. The framework of DQN consists of agents and the vehicular environment. In a subframe, an agent v observes a state $s_{t}$ from the environment, and the MainNet can get the state-action function $Q(s,a;\theta)$, which is used to evaluate the selection of the action. According to the action the agent selects, it can get a new state ${s_{t+1}}$. The function $Q({s}^{\prime},{a}^{\prime};{\theta}^{\prime})$ means the output of the TargetNet, and the target Q is $r+\gamma\max Q({s}^{\prime},{a}^{\prime},{\theta}^{\prime})$. So the loss function in the deep neural network is

where r is the value of reward when taking action a. $\gamma$ is the discount factor, the closer it is to 1, the more it will value future rewards, and the closer it is to 0, the less sensitive it is to future rewards. The asynchronization between parameters ${\theta}^{\prime}$ and $\theta$ sloves the problem of instability when using nonlinear network to evaluate value function.

Before the simulation analysis, the basis of parameter selection should be introduced. The frequency operates in 2 GHz to realize spectrum sharing between V2V and V2I links detailed in 3GPP TR 36.785. One V2V link multiplexes at most one V2I link. We follow the urban scenario in 3GPP TR 37.885. It is 1299 m long and 750 m wide. The vehicles are placed in the scenario as platooning, and each platooning has five vehicles including one platooning leader and four platooning members. Vehicle speed is set at 60 km/h.

Experimental environment relies on Python. The DQN model is built in TensorFlow 1.9.0 and the GPU model used is GeForce GTX 1080 Ti. The number of hidden layers is 3. Optimizer we select is RMSProp, learning rate is set as 0.01. Discount factor is 0.5. The size of mini batch is 2000. More details are marked in Table I.

In order to evaluate the performance of the proposed SEED, we compare it with the DQN algorithm without power allocation (DQN-wopa) [12] and the baseline algorithm that uses random allocation.

As shown in figure 3, with the increase of the vehicles, the network efficiency including SE and EE in both V2V and V2I links is decreasing. The reason is that the power consumption and the interference of the network will be ascending when the number of vehicles increases. From this figure, we can see that the SEED proposed alleviates this phenomenon to a greater extent. The numerical value of network efficiency is 1.192 when the number of vehicles is 20, and the numerical values of DQN-wopa algorithm and the baseline algorithm are 1.040 and 0.626, respectively. When the number of vehicles is 100, the environment of the system becomes severe, the network efficiency of V2V and V2I links is 0.492 using the SEED proposed, which is better than 0.383 and 0.167 using the other two schemes. The remaining values in the SEED are 0.917, 0.716 and 0.597. We can see that the down curve of using SEED is relatively flat, indicating that for the high-density vehicles environment, it can better slow down the decrease in efficiency.

Figure 4 demonstrates the mean V2V secrecy capacity versus the number of vehicles. The SEED and the DQN-wopa algorithm both have higher performance on protecting vehicles’ privacy than the baseline algorithm, because the V2V secrecy rate is considered as the constraint of the optimization problem. Because the SEED can choose different power for vehicles adaptively, it will interfere as much as possible with the eavesdropper. While the performance of random algorithm is bad because it does not care about the security of C-V2X. The values of mean V2V secrecy rates are 1.068, 0.778, 0.663, 0.521 and 0.443 in the SEED.

Figure 5 and figure 6 show that the network efficiency of V2V links and the network efficiency of V2I links. As shown in formula (9), the optimization problem in this paper is a combinatorial one. ${\lambda_{\alpha}}$ and ${\lambda_{\beta}}$ are set as 0.9 and 0.1, respectively. These weighted coefficients mean that the network efficiency of V2V links is more important than that of V2I links, the reason is that the usage frequency of V2V is more frequent than that of V2I.

From figure 5 we can see that the network efficiency of V2V links in the SEED far superior to the other two schemes. In the SEED, when the number of vehicles is from 20 to 100, the numerical values are 0.879, 0.644, 0.554, 0.409 and 0.372, respectively. The reason is that the network efficiency of V2V links accounts for a large proportion in the optimization function (9).

As shown in figure 6, the network efficiency of V2I links in the approach proposed is almost as good as DQN-wopa algorithm. When the number of vehicles is from 20 to 100, the numerical values are 4.004, 3.373, 2.213, 2.276 and 1.593 in our approach, respectively. The performance of the SEED on the V2I efficiency is not the most excellent in these three schemes every time. The reason is that the weighted coefficient of V2I is small, to maximize the efficiency of the whole network including V2V and V2I links, the SEED makes its choice.

The loss function of our DQN model in 20 vehicles is shown in figure 7. It can be seen that when training about 150 steps, the loss value drops from 1.288 to about 0.107. After long time steps, the loss value falls to about 0.047. It seems that the model has already converged.