Vehicle Tracking in Wireless Sensor Networks via Deep Reinforcement Learning

With the advances in the fabrication technologies that integrate the sensing and the wireless communication, a large-scale wireless sensor networks (WSNs) is formed in the desired fields by deploying dense tiny sensor nodes. Vehicle tracking in WSNs has several prominent merits: firstly, the sensing unit is close to the vehicle, thus the sensed data will be of a qualitatively good geometric fidelity via vehicle-to-infrastructure (V2I) technology [1]; secondly, the information about the target is simultaneously generated by multiple sensors and thus contains redundancy [2]. However, there still exist some unsolved problems as far as vehicle tracking in WSNs concerned. The first issue is how to guarantee the vehicle to be tracked by sensor nodes. Second is how to maximize the tracking accuracy with the limited resources of WSNs, such as the energy restriction of each node [3].

There are two strategies based on the data processing mechanism, namely centralized strategy and decentralized strategy. Specifically, in centralized strategy, a sensor is artificially selected as a cluster head, and the tracking estimation is performed at this node with all received data [4].

However, the tasks at the cluster head may be overloaded in this strategy. In each iteration of the decentralized strategy, each cluster uses the data from their neighbours to refine its local estimate [5]. Therefore, although the head is closer to the data source than the fixed head in centralized strategy, it has higher energy efficiency. Meanwhile, the works in [6] point out that deep reinforcement learning (DRL) is a problem-solving tool and suitable for decentralized systems in WSNs.

The existing decentralized tracking strategies in WSNs are based on the prediction position to activate a fixed number of nodes, which will result in unnecessary energy consumption. Thus, we propose a dynamic activation area adjustment scheme based on DRL, to save energy consumption while ensuring tracking accuracy. Based on this, we first formulate the problem as a Markov decision process (MDP). Then, we construct the optimization problem as maximizing average rewards of MDP, where the reward consists of both tracking accuracy and energy consumption. Furthermore, we propose a pair of schemes based on deep Q network (DQN) and on deep determined policy gradient (DDPG), to maximize average rewards of MDP. In simulation, our proposed DQN and DDPG based algorithms outperform the conventional Q-learning based method in terms of tracking accuracy and energy consumption.

Fig. 1 illustrates an vehicle tracking scenario in a WSNs, which consists of a single vehicle and multiple sensor nodes. The WSNs starts tracking and broadcasting when the object vehicle appears in its monitoring area. Consider that the time slots are denoted by $t=1,2,...,T$, with unequally duration $\tau(t)$ seconds. The vehicle derives into the monitoring area of the WSNs at time slot $t=1$, and leaves the WSNs or the WSNs loss vehicle at $t=T$. Let the vehicle’s vector denote as $\emph{{x}}(t)=[x_{t},v_{t}^{x},y_{t},v_{t}^{y}]^{\text{T}}$, where $\emph{{l}}(t)=[x_{t},y_{t}]^{\text{T}}$ is the 2-D position of the vehicle, and $[v_{t}^{x},v_{t}^{y}]^{\text{T}}$ is the velocity of the vehicle at time slot $t$ on the plane. As shown in Fig. 1, the connection between the sensor nodes and the vehicle can be established by V2I within a certain range, thus the radius $r_{\text{s}}$ of sensing area $n_{\text{s}}(t)$ is fixed. Moreover, the WSNs activates the sensors in activation area $n_{\text{a}}(t)$ with radius $r_{\text{a}}(t)$ in advance, for tracking with dynamic energy consumption. Additionally, let the sensors in set of intersection area $S_{\text{int}}(t)=n_{\text{s}}(t)\cap n_{\text{a}}(t)$ denote as $i=1,2,\ldots,|m(t)|$, where $m(t)$ is the collection of sensors in $S_{\text{int}}(t)$, and $|m(t)|=\text{card}[m(t)]$ represents the amount of elements in set $m(t)$.

At time slot $t-1$, WSNs enables an activation area $n_{\text{a}}(t)$ in advance, which is an area based on the predicted vector $\hat{\emph{{x}}}^{\prime}(t)$ and $r_{\text{a}}(t)$. When the vehicle moves from $t-1$ to $t$, sensors in $m(t)$ track the vehicle cooperatively, and then only one sensor node is selected for transmitting data to the sensors in $n_{\text{a}}(t+1)$.

This paper solves the problem of improving tracking accuracy and energy saving when tracking vehicle in WSNs. Explicitly, the number of the sensor nodes for tracking contributes to the tracking accuracy [5]. Nevertheless, with the increase of the sensor nodes, the energy consumption for communication and data processing becomes higher. To efficiently track the vehicle, we propose a dynamic activation area adjustment scheme to balance the trade-off between tracking accuracy and energy consumption. In this section, we model this scheme as an MDP based on the nature of the dynamic decision-making problem.

• •

  State Space: Let $\mathcal{S}=\{s(t)|t=1,2...,T\}$ denote the state space of the WSNs, where $s(t)=\{\hat{\emph{{x}}}^{\prime}(t),\hat{\emph{{x}}}(t),r_{\text{a}}(t)\}$ includes the estimated vector $\hat{\emph{{x}}}^{\prime}(t)$ of the vehicle in time slot $t$, the estimated vector of vehicle $\hat{\emph{{x}}}(t)$ based on $\hat{\emph{{x}}}^{\prime}(t)$ and the measurement ${\emph{{z}}_{i}}(t)$, and the radius of the activation area $r_{\text{a}}(t)$. The elements in $\mathcal{S}$ except $r_{\text{a}}(t)$ are described in the Kalman filtering model.

• •

  Action Space: Let $\mathcal{A}$ denote the action space of the WSNs, where $r_{\text{a}}(t)\in\mathcal{A}$ is the radius of $n_{\text{a}}(t)$.

• •

  Reward: Assuming that $n_{\text{a}}(t)$ and $n_{\text{s}}(t)$ are circles with radius of $r_{\text{a}}(t)$ and $r_{\text{s}}$, the prediction error in time slot $t$ is denoted as $L=||\hat{\emph{{l}}}^{\prime}(t)-\hat{\emph{{l}}}(t)||_{2}$, where the predicted position is $\hat{\emph{{l}}}^{\prime}(t)=\emph{{H}}\hat{\emph{{x}}}^{\prime}(t)$. In the process of the tracking, we consider the reward consisting of both the tracking accuracy and the energy consumption in each time slot:

  1. 1.

     Tracking Accuracy: For quantifying the tracking accuracy to measure the tracking effect of WSNs, classical measurement (such as estimation error covariance) represents the error between $\hat{\emph{{l}}}^{\prime}(t)$ and $\hat{\emph{{l}}}(t)$, but we would like to measure the error between $n_{\text{s}}(t)$ and $n_{\text{a}}(t)$, which depends on $\hat{\emph{{l}}}^{\prime}(t)$ and $r_{\text{a}}(t)$. By adopting $S_{\text{int}}(t)=n_{\text{s}}(t)\cap n_{\text{a}}(t)$ as the metric, we can can measure the effect of $r_{\text{a}}(t)$ on energy consumption and track accuracy. As such, $\hat{\emph{{l}}}^{\prime}(t)=\hat{\emph{{l}}}(t)$ satisfies when $L=0$ and $\text{max}[r_{\text{a}}(t)]=r_{\text{s}}$, thus as $S_{\text{int}}(t)$ increases, the predicted position becomes more accurate. We define the $S_{\text{int}}(t)=r_{\text{a}}^{2}(t)[\theta_{1}(t)-\text{sin}\theta_{1}(t)\text{cos}\theta_{1}(t)]+r_{\text{s}}^{2}[\theta_{2}(t)-\text{sin}\theta_{2}(t)\text{cos}\theta_{2}(t)]$. Herein, $\theta_{1}(t)$ and $\theta_{2}(t)$ are the angles of intersection area, which are obtained by $\theta_{1}(t)=\text{cos}^{-1}\frac{r_{\text{a}}^{2}(t)+L^{2}-r_{\text{s}}^{2}}{2r_{\text{a}}(t)L}$ and $\theta_{2}(t)=\text{cos}^{-1}\frac{r_{\text{s}}^{2}+L^{2}-r_{\text{a}}^{2}(t)}{2r_{\text{s}}L}$, respectively.

  2. 2.

     Energy Consumption: In duration of $\tau(t)$, the energy consumption generated by data transmission is

     $$\begin{split}e_{\text{com}}(t)=&P_{\text{r}}\sum\limits_{i=1}^{|m(t)|}\tau_{1}^{i}(t)+P_{\text{r}}\tau_{2}^{\prime}(t)+P_{\text{t}}\sum\limits_{j=1}^{|m(t)|-1}\tau_{2}^{j}(t)\\ &+P_{\text{t}}\tau_{3}^{\prime}(t)+P_{\text{r}}\sum\limits_{k=1}^{|m_{\text{a}}(t+1)|}\tau_{3}^{k}(t),\end{split}$$

     (4)

     where $P_{\text{t}}$ and $P_{\text{r}}$ denotes the power of nodes for transmitting and receiving data, respectively. In addition, the energy consumption of nodes for waking up in $n_{\text{a}}(t)$ and working in $m(t)$ is $e_{\text{w}}(t)=\left[P_{\text{w}}|m(t)|+P_{\text{idle}}(|m_{\text{a}}(t)|-|m(t)|)\right]\tau(t)$, where $|m_{\text{a}}(t)|$ denote the amount of sensors of $n_{\text{a}}(t)$, and the power of the nodes in working mode is denoted as $P_{\text{w}}$ and the power of the nodes in idle mode is denoted as $P_{\text{idle}}$[5]. Thus, the total energy consumption is $e(t)=e_{\text{com}}(t)+e_{\text{w}}(t)$.

  Therefore, as the goal of this system is to improve the tracking accuracy and to minimize the energy consumption, we can define reward $R(t)$ by combining normalized $S_{\text{int}}(t)$ and $e(t)$ as $R(t)=\frac{S_{\text{int}}(t)}{\pi r^{2}_{\text{s}}}-\frac{e(t)}{e_{\text{max}}}$, where $e_{\text{max}}$ denotes the maximum energy consumption in each time duration.

It is noteworthy that, as the number of the training episodes increases, the total rewards in a single episode may exist a certain degree of disturbance, thus the increase of the total rewards in a single episode does not represent an accurate improvement of the system performance. Therefore, our goal is to find an optimal policy $\pi^{*}$ with the maximum average total reward, given by:

In this section, to solve the problem (5), we adopt the policy iteration based DDPG method, and the value iteration based DQN method. Additionally, in DQN method, we use the action selection strategy of ’softmax’ to compare with the ’greedy’ strategy.

Based on the previous research in [7], we consider the field of WSNs is $360\times 600\text{m}^{2}$. Besides, the motion model is

where $\emph{{x}}(t)$ is the vector of the vehicle, $\tau=\tau(t)$ is the length of each time interval, and $w(t)$ is the vector noise. The initial vector is $\emph{{x}}(0)=[0\ 7.84\ 0\ 7.84]^{\text{T}}$, which means the initial speed is 40 kmph. The vector noise accounting for the unpredictable modeling error is characterized by $\emph{{Q}}=\mathbb{E}[\emph{{w}}(t)\emph{{w}}^{\text{T}}(t)]=\begin{bmatrix}0.03&0\\ 0&0.03\end{bmatrix}$.

In the tracking of a single vehicle, there are many sensors that can be used in the scenario, such as cameras [5] and logical combination of ranging sensors [2]. By adopting the strategy of uniform distribution of sensor nodes in [5], we set 375 nodes in the sensing range with the density of 1 sensor/24m, and the linear measurement model of each sensor is $\emph{{z}}_{i}(t)=\emph{{H}}\emph{{x}}(t)+\begin{pmatrix}v_{x}(t)\\ v_{y}(t)\end{pmatrix}_{i}$, where $\emph{{H}}=\begin{bmatrix}1&0&0&0\\ 0&0&1&0\end{bmatrix}$. Notably, the motion model can also be a non-linear model, which can be estimated by extended Kalman filter, particle filter or other methods. The parameters of the measurement noise covariance are $\emph{{R}}_{i}=\mathbb{E}\left[\begin{pmatrix}v_{x}(t)\\ v_{y}(t)\end{pmatrix}_{i}\ \begin{pmatrix}v_{x}(t)\\ v_{y}(t)\end{pmatrix}_{i}^{\text{T}}\right]=\begin{bmatrix}400&0\\ 0&400\end{bmatrix}$. We randomly initialize $\emph{{x}}(0)$, and the parameters used in this model are shown in Table 1.

Fig. 2 plots the trends of the average tracking accuracy $\frac{1}{T}\sum\limits_{t=1}^{T}{S_{\text{int}}(t)}$ and the average energy consumption $\frac{1}{T}\sum\limits_{t=1}^{T}{e(t)}$ with different $r_{\text{s}}$, where each value of $e(t)$ and $S_{\text{int}}(t)$ is the final convergence value obtained by training 1800 episodes. Clearly, increasing sensing area $n_{\text{s}}(t)$ achieves higher $S_{\text{int}}(t)$ and $e(t)$.

Fig. 3 shows the average accumulated rewards $\frac{1}{T}\sum\limits_{t=1}^{T}{R(t)}$ of the proposed algorithms. Firstly, we can observe that although $e(t)$ and $S_{\text{int}}(t)$ increase with $r_{\text{s}}$, their gaps in terms of the average accumulated rewards enlarge gradually. Secondly, as episodes go by, the average accumulated rewards based on DQN is much higher than other methods. The reasons include two aspects: first the QL based method leads to the lowest value because of the Q-table has a limited capacity; second the value iteration of DQN is based on the unbiased estimation of data, while the DDPG is based on the biased estimation.

This paper focuses on improving the tracking accuracy and energy saving of target vehicle in WSNs. Thus, we propose a decentralized vehicle tracking strategy, which dynamically adjusts the activation area to improve tracking accuracy and energy saving. Then, we define the problem as a MDP and use DRL method to solve it. Finally, the simulation results show that the DQN-based method has a better performance than other DRL-based methods. In future, we can explore multi-agent DRL algorithms in cooperative tracking scenarios.