iRDRC: An Intelligent Real-time Dual-functional Radar-Communication System for Automotive Vehicles

In the communication mode, the AV uses the V2I capability to transmit the data, e.g., of current road traffic or live on-board video streaming, to the Base Stations (BSs) distributed along the road. Assume that the AV uses a single channel for the data transmission and has a data queue for storing incoming data packets, e.g., from its sensor devices. Let $D$ be the capacity of the data queue. In the radar mode, the AV performs an automotive millimeter-wave radar to detect unexpected events. As shown in Fig. 1, the radar mode can be used to detect unexpected events, e.g., a car coming from another road obscured by a truck. In particular, we define an unexpected event as an event that can possibility cause collisions with the AV. We consider that the occurrence of an unexpected event is influenced by four main factors: the road condition, weather condition, speed of the AV, and nearby moving object [5, 6]. Note that the values of these factors can be obtained by the AV’s sensing system, e.g., road friction sensor, weather station instrument, speedometer, and cameras.

Let $r\in\{0,1\}$, $w\in\{0,1\}$, $v\in\{0,1\}$, and $m\in\{0,1\}$ be the road state, weather state, speed state, and moving object state, respectively. In particular, $r=1$, $w=1$, $v=1$, and $m=1$ represent unfavorable conditions, e.g., slippery road, rainy weather, high speed of the AV, and a moving object nearby, respectively. In contrast, $r=0,w=0,v=0$ and $m=0$ express favorable conditions, e.g., straight road, good weather, low speed and without a moving object nearby, respectively. Let $p_{j}^{i}$ denote the probability to occur an unexpected event at the current condition $j$ (where $j\in\{0,1\}$ corresponds to favorable or unfavorable conditions, respectively) of factor $i$, $i\in\{r,w,v,m\}$. For example, $p_{1}^{r}$ expresses the probability of an unexpected event to occur given the slippery road condition, i.e., $r=1$. Note that the generalization of the states beyond 0 and 1 is straightforward. For example, the speed of the AV can be divided into multiple levels, e.g., low, medium, and high.

To model the dynamic of environment, the probabilities $p_{j}^{v}$, $p_{j}^{w}$ are taken from the real-world data in [5, 6], and other probabilities are assumed to be pre-defined. Then, we can determine the probability of an unexpected event to occur given factor states $(r,w,v,m)$ using the Bayes’ theorem. For this, let $\oplus$ denote the occurrence of an unexpected event, and $\ominus$ denote that no unexpected event occurs. Let $\tau_{i}$ be the probability that factor $i$ is at state $0$, where $i\in\{r,w,v,m\}$. Thus, the probability that factor $i$ at state $1$ is $1-\tau_{i}$. By using the Bayes’ theorem, the probability of an unexpected event to occur given factor states $(r,w,v,m)$ is determined by:

In general, when the probability of an unexpected event to occur, $P(\oplus)$, is high, the environment is more dynamic and uncertain. We introduce a metric, i.e., the miss detection probability, to evaluate the performance of the proposed system. The miss detection probability is defined by the ratio of the number of unexpected events that the AV cannot detect to the total number of unexpected events on the road. A high miss detection probability results in a high risk of accident for the AV. We also introduce the second metric to evaluate the performance of the proposed system that is the data throughput. The data throughput is defined as the average number of packets per time unit that is successfully transmitted from the AV to the BSs. Note that, we assume that the accuracy of the autonomous radar system is perfect, i.e., there is no miss detection or false alarm, when the AV uses the radar mode. However, the system model can be straightforwardly extended by considering the miss detection and false alarm caused by sensing accuracy of the radar. In this case, the proposed DRL scheme still can work well as it can learn these parameters through real-time interactions with the environment.

Intuitively, to minimize the miss detection probability, the AV can use the radar mode more frequently to detect unexpected events, but this reduces the data throughput. Conversely, to increase the throughput, the AV can use the communication mode more frequently, but this may increase the miss detection probability. Consider this tradeoff with the uncertainty of environment, the AV’s decision making problem can be modeled as an MDP. We then develop a DRL algorithm to quickly obtain the optimal policy for the AV without requiring completed information about environment. The details about the DRL scheme that enables the AV to quickly find the optimal policy will be discussed in Section V-B.

At each time step, the AV decides to use either the communication mode or the radar mode. Let $\mathcal{A}$ denote the action space of the AV, $\mathcal{A}=\big{\{}a;a\in\left\{0,1\right\}\big{\}}$, where $a=0$ means that the AV chooses the communication mode, and $a=1$ means that the AV chooses the radar mode. The state of the AV is the combination of (i) the state of the data queue, (ii) the state of the channel that the AV uses for its data communication, (iii) the state of the road, (iv) the weather state, (v) the speed state of the AV, and (vi) the nearby moving object state. Thus, the state space of the AV can be defined as

where $d$ represents the state of the data queue, i.e., the number of packets in the data queue, $c$ refers to the state of the communication channel that the AV uses to transmit data to the BSs. $c=0$ if the channel is good, i.e., low interference, and $c=1$ if the channel is bad, i.e., high interference. $r,w,v$, and $m$ are defined in Section II-A. The state of the system at time step $t$ is defined as $s_{t}=(d,c,r,w,v,m)\in\mathcal{S}$.

At each time step $t$, the AV chooses an action $a_{t}\in\mathcal{A}$ at state $s_{t}\in\mathcal{S}$ and receives an immediate reward $r_{t}$. The reward is designed to encourage the AV to increase the data throughput and at the same time decrease its miss detection probability. For this, we define the reward function as follows.

When the AV selects the communication mode and if the channel state is good, the AV successfully transmits $\nu_{1}$ packets and receives a reward $r_{1}$. Otherwise, when the AV selects the communication mode and if the channel is bad, the AV successfully transmits $\nu_{2}$ packets and receives a reward $r_{2}$. Moreover, when the AV selects the communication mode and an unexpected event occurs, the AV receives a penalty of $-r_{3}$. When the AV selects the radar mode and if the AV does not detect any unexpected event, the AV receives no reward. Otherwise, when the AV selects the radar mode and if the AV detects an unexpected event, the AV receives a reward that is proportional to the number of unfavorable conditions in $\{r,w,m,v\}$, i.e., the number of values $1$ in $\{r,w,m,v\}$. This means that the AV receives a high reward if the probability of an unexpected event to occur is high, e.g., the AV is under very unfavorable conditions, and if the unexpected event is detected. This definition is to encourage the AV to use the radar mode when the environment conditions are unfavorable. In summary, the immediate reward can be defined as follows:

where $b$ is the number of values of $1$ in the set $\{r,w,m,v\}$. Note that the probability of an unexpected event to occur given $\{r,w,m,v\}$, $P(\oplus)$, is defined in (1).

In this paper, we aim to find the optimal policy for the AV, denoted by $\pi^{*}$, to maximize its long-term discounted cumulative reward, i.e., discounted return, as defined by

where $G(\pi)$ is the expected discounted return under the policy $\pi$, $r_{t+1}(\pi)$ is the immediate reward under policy $\pi$ at time step $t+1$, $T$ is the time horizon, and $\gamma\in(0,1)$, is the discount factor. The optimal policy $\pi^{*}$ will allow the AV to make optimal decisions at any state $s_{t}$, i.e., $a^{*}_{t}=\pi^{*}(s_{t})$.

To find the optimal policy for the AV, standard Q-learning [8] can be adopted by estimating Q-values of all state-action pairs, i.e., $Q(s,a)$. The Q-values are iteratively updated in a Q-table, and thus the Q-learning suffers the large state space problem. Therefore, we propose to use the DRL with DQN to quickly find the optimal policy.

For the comparison purpose, the capacity of the data queue is set to $D=10$ packets, and the arrival packets follow a Poisson distribution with an average arrival rate of $\lambda_{d}=1$ packet/time step. If the channel state is good, i.e., $c=0$, the AV can transmit $\nu_{1}=4$ packets, if the channel state is bad, i.e., $c=1$, the AV can transmit $\nu_{2}=2$ packets. We assume that the probability that the channel is at the bad state is $p_{c}=0.1$, and the probability that the channel is at the good state is $1-p_{c}$. For the reward values, to minimize the miss detection probability, the value of $r_{3}$ should be much higher than other values, i.e., $r_{1}$, $r_{2}$, and $r_{4}$. In particular, we set the values $(r_{1},r_{2},r_{3},r_{4})$ to be $(2,1,50,5)$. The values of $p_{0}^{v}$ and $p_{1}^{v}$ are taken from [5] in which if the AV’s speed exceeds $60$ km/h, the AV’s speed is high and otherwise the AV’s speed is low. Specifically, the values $p_{0}^{v}$ and $p_{1}^{v}$ are set to be $0.005$ and $0.1$, respectively. Rain can be considered to be a common unfavorable weather state, and thus the values of $p_{1}^{w}$ and $p_{0}^{w}$ can be taken from [6] in which $p_{1}^{w}=0.046$ and $p_{0}^{w}=0.005$. The parameters of the DQN scheme are set as follows. The neural network used in DQN is a Multilayer Perceptron with $1$ input layer, $2$ hidden layers and $1$ output layer. The input layer contains $6$ units which correspond to the number of dimensions of the state space. The output layer contains $2$ units corresponding to the number of dimensions of the action space of the AV. The DQN and the environment for the AV are implemented by using Keras library and OpenAI Gym environment, respectively. To evaluate the DQN scheme, we introduce the Q-learning [8] and Round-robin scheme, i.e., the AV switches back and forth between the radar mode and the communication mode, as baseline schemes.

We first compare the total rewards obtained by the schemes. As shown in Fig. 2(a), the total rewards obtained by the DQN and Q-learning are much higher than that of the Round-robin. Furthermore, the DQN and Q-learning converge to the same reward. However, the convergence speed of the DQN is much faster than that of the Q-learning. In particular, the DQN requires $170$ episodes to approach the optimal value, while the Q-learning scheme requires $280$ episodes. The reason is that the DQN updates multiple Q-values in a mini-batch at each training iteration [7], while Q-learning performs only one Q-values update at each training iteration [8]. As a result, the convergence rate of the Q-learning is usually much lower than that of the DQN, especially for the large state/action spaces [7].

Next, we evaluate the DQN scheme by varying the environmental factors. Without loss of generality, we evaluate the proposed scheme when the probability to occur an unexpected event given the high speed of the AV, $p_{1}^{v}$, varies from $0.1$ to $1$. As shown in Fig. 2(b), as $p_{1}^{v}$ increases, the average reward obtained by the Round-robin scheme decreases, while those obtained by the DQN and Q-learning schemes increase. The reason can be explained as follows. With the Round-robin scheme, the radar mode is chosen according to a fixed policy, meaning that the radar mode may not be frequently used even if the occurrence probability of an unexpected event is high. Thus, the AV may receive high penalties that results in a decrease of the average reward. With the DQN and Q-learning schemes, the AV uses the radar mode more frequently as $p_{1}^{v}$ increases to minimize the penalties. As a result, the DQN and Q-learning schemes can achieve higher average rewards compared with that of the Round-robin scheme.

Following the optimal policy, the DQN and Q-learning can significantly outperform the Round-robin in terms of throughput (see Fig. 2(c)) and miss detection probability (see Fig. 2(d)). As shown in Fig. 2(d), the miss detection probabilities obtained by the DQN and Q-learning decrease as $p_{1}^{v}$ increases. The reason is that the optimal policies obtained by the DQN and Q-learning enable the AV to select the radar mode more frequently as unexpected events are likely to occur. Thus, the AV can detect more unexpected events and reduce the miss detection probability. Note that our simulation results presented in this section are especially useful to design key parameters for real AV systems to ensure the safety for the users. In particular, given the current simulation setting $(r_{1},r_{2},r_{3},r_{4})=(2,1,50,5)$, the AV can achieve a miss detection probability ranging from $0.15$ to $0.3$. We can further reduce the miss detection probability of the AV to meet its requirement by increasing the reward when the AV selects the radar mode, e.g., increasing $r_{4}$ from $5$ to $50$ or $100$.