Predictive Wireless Based Status Update for Communication-Agnostic Sampling

With the emergence and development of the Internet-of-Things (IoT), more and more applications require timely information be communicated through wireless networks, e.g., collaborative autonomous driving, wherein only the latest information can reflect the accurate status of the physical world. In this context, leveraging the wireless interface for real-time status updating has gained much traction. Status update is a process that the source node transfers information flow to the target node (with possibly feedback), which lets the target node remotely obtain the real-time status generated at the source node.

Age of Information (AoI) [2] serves as a performance metric for characterizing the freshness of information in a status update system. Most existing works for time-sensitive status update in wireless networks focus on AoI optimizations. Recent research advances suggest that many well-known design principles (e.g., for providing high throughput and low delay) that lead to the success of traditional packet-based networks, are inappropriate for AoI optimizations. Many researches focus on queuing theory. The pioneer work in [3, 2] analyzes the tradeoff between sampling frequency and communication delay (e.g., network scheduling delay and queuing delay). It is shown that the source node can transmit more timely data packets by increasing the sampling frequency of status information, thus reducing the error of the status information between the target node and the source node. However, the increase of the sampling packets would cause longer network delay, resulting in network congestion and also the new information of each sampling packet is decreased. Many later studies have extended this result to e.g., scenarios with packet management [4], multi-class queuing systems [5], gamma distribution for the service time [6], controlling the status packet arrival process instead of assuming it is random [7], and energy harvesting sources [8]. In addition, Ref. [9] derives the stationary distribution of AoI under various queuing disciplines, and Ref. [10] models the spatially correlated statuses as a random field. In conclusion, for the status-unaware schemes of minimizing AoI, it is necessary to control the sampling frequency to avoid network congestion.

As the communication medium, there are many researches on wireless networks, especially scheduling problems, for AoI optimizations. A centralized scheduling strategy using the Whittle’s index is used in [11, 12]. Ref. [13, 14, 15, 16, 17, 18] study decentralized scheduling strategies in wireless multi-access networks. Considering that each node may have different channel conditions, service rates and packet arrival rates, Ref. [16, 17, 18] optimize the access probability of the retreat window size based on the existing MAC-layer protocol, e.g., CSMA or ALOHA. The access probability is associated with the index in Whittle’s index method in [14, 15]. When the number of nodes is large, the round-robin scheduling strategy is proved to be optimal in [13]. And [19, 20] show that a stationary policy actually achieves order-optimal performance in general network topology.

On the other hand, there are many schemes based on status-aware update [21, 22]. An update approach based on status-error thresholds is proposed to optimize sampling from Wiener and Ornstein-Uhlenbeck processes in [23, 24]. Ref. [25] proposes a method to capture the change of status by using effective AoI instead of simple AoI. In this respect, the value of information [26, 27] is designed to reflect on the impact of AoI on a specific application. Ref. [28] proposes a mean-field approach to calculate the access probabilities with random-walk status transitions to optimize the wireless network. According to the content of some status packets, Ref. [29] proposes to assign priorities to them. Most of the mentioned works are based on a status model, e.g., Markov source model, to obtain theoretical tractability, and have not considered status predictions.

In summary, existing works either try to design the sampler based on communication constraints modeled by queues, or develop status-aware scheduler without considering status predictions. Is it possible to build a status update system wherein the sampler can sample the information source without any frequency constraints, while the communication interface can automatically adapt to both network conditions and status trends, such that the destination node observes frequent and fresh updates while the network occupancy is still low? In this paper, a novel wireless communication protocol named parallel communications based on communication-agnostic sampling is proposed. Specifically, there are two types of packets transmitted at the source node, one is conventional Over-The-Air (OTA) packets and the other is model prediction packets. Only when statuses are not predicable by the built-in online status predictor, they are transmitted OTA; otherwise, the status updates are produced by model predictions. A novel Status-aware Multi-Agent Reinforcement learning neTworking solution (SMART) based on the Whittle’s index methodology is proposed, that can be extended to any status vectors (coped with DNN) without being limited to a specific network. We use the Software-Defined-Radio (SDR) platform to test the framework proposed in this paper. We simulate a typical autonomous driving scenario on an integrated hardware-software evaluation platform. In the platform, the vehicle control and motion dynamics are simulated by Simulation of Urban MObility (SUMO), and the communications among vehicles are realized by Software-Defined-Radio (SDR). The results show that by adopting parallel communications, the channel occupancy is significantly reduced, while guaranteeing low status recovery error. Then we applied the framework to two realistic scenarios, one is a multi-density platooning scenario implemented on SUMO and the other is a UAV formation flight control scenario implemented by MATLAB. The results show that compared with current packet-based delay-optimized scheme and the AoI-optimized status-unaware scheme, the parallel communication framework has better performance in terms of wireless network control performance, e.g., safe inter-vehicle distance.

The rest of the paper is organized as follows. In Section II, a single-link example is shown to illustrate the core idea of parallel communications; the SDR verification follows immediately; the migration to the hardware-software evaluation platform follows finally. Section III describes the networking solution for parallel communications, i.e., SMART. Section IV provides a case study using SUMO, applying the proposed framework to multi dense platooning. Section V provides another case study using MATLAB to apply the framework to the UAV formation flight control. Finally, in Section VI, conclusions are drawn with future directions discussed.

First, we would like to illustrate intuitively, plain and simple, on the central idea of predictive wireless, which is in analogous to human communications. Suppose that Alice and Bob live in different cities and Bob sends a letter to Alice everyday about whether it is rainy in his city on that day. This is a typical status update setting where a fixed update interval (one day) is adopted. However, further suppose that Bob’s city rarely rains, and then he would soon discover that he could make the status update much more efficiently (in the sense of sending less reports), by defining and communicating a model with Alice prescribing that whenever there is no letter, there is no rain; otherwise, Bob would report an model-unexpected status indicating rainy. Obviously, the model in this case is a time-series forecasting model predicting no rain every time, and the status is a binary scalar value.

Such a simple, but useful idea can be generalized to improve the communication efficiency in status update-based wireless systems. In essence, such systems can benefit much from a status prediction functionality. The proposed system architecture diagram is shown in Fig. 1. We term this architecture as parallel communications, since the status information is communicated by both model predictions and OTA packets. The parallel communication operation is transparent to the upper layers or modules, i.e., communication-agnostic sensing module. Several key design aspects are illustrated in details as follows. We assume a time-slotted system and the time index is denoted by $t$ (the duration of a time slot is $1$ millisecond in the SDR implementation). At each time slot, a status (or a set of statuses) is sampled by the sensing module and fed into the Parallel Transmit Function Block (PTxFB).

In this section, we describe in general a networking scheme for efficient communications among ad hoc wireless devices running the parallel communication protocol. One of the critical issues of this networking scheme that distinguishes from conventional ad hoc network problems is that how to properly make transmit decisions when the decisions depend on locally-observable status information, i.e., status-aware. The scheme is inspired by the Whittle’s index methodology [33] and is explained as follows.

Denote $\bm{x}_{n}(t)\triangleq[\bm{s}_{n}(t),\,\hat{\bm{s}}_{n}(t)]$ as the Markov state for $n$-th source node in the system at time $t$, wherein $\bm{s}_{n}$ denotes the local status information, and $\hat{\bm{s}}_{n}(t)$ denotes the status recovery at the $n$-th destination (estimated at source nodes based on aligned status). Note that for non-Markov status evolution, a collection of finite length history of statuses can be approximately defined as Markov states which is common in practice. The overall system space is hence $\{\bm{x}_{n}(t)|n=1,...,N\}$ wherein $N$ is the total number of source nodes. Denote $\bm{u}(t)=\{u_{n}(t)|n=1,...,N\}$ as the transmit decision of sources at time $t$; $u(t)=1$ denotes transmit and silent otherwise. The Markov Decision Process (MDP) can be defined as

where $\mathcal{E}(\cdot)$ denotes a predefined error function. MDP-1 is essentially a Partially Observable MDP (POMDP) as seen by each source-destination pair, since each source can only observe its local current status. MDP-1 suffers from the curse of dimensionality when using e.g., value iterations. In addition, MDP-1 is a POMDP which does not have a general solution method—multi-agent reinforcement learning techniques may be leveraged but with convergence issues [34].

Towards this end, we adopt the concept of Whittle’s index [33] to decouple MDP-1. The idea of this approach is that instead of considering all nodes simultaneously, the problem is decoupled and reduced to transmit decisions for one source node. To avoid the trivial, selfish and useless solution of always transmitting, each transmission is associated with an auxiliary cost of $m_{n}$. In other words, the decoupled MDP problem for source node-$n$ is formulated as

wherein the top and bottom terms in the minimization operator represent the cost-to-go from state $\bm{x}_{n}$ onwards with the action of silent and transmit, respectively. The expected cost functions are denoted by $\mathcal{E}^{(0)}_{\bm{x}_{n}}$ and $\mathcal{E}^{(1)}_{\bm{x}_{n}}$ respectively for both actions (assuming the error function is decomposable, e.g., $\ell^{1}$, $\ell^{2}$ norms); the transition matrices $\mathcal{P}^{(0)}_{\bm{x}_{n}}$ and $\mathcal{P}^{(1)}_{\bm{x}_{n}}$ are denoted likewise. The relative cost-to-go function of state $\bm{x}_{n}$ and the average cost are denoted by $f(\bm{x}_{n})$ and $\hat{J}^{*}$ respectively. It has been shown that the solution solving (18) for each source node and selecting the node with the largest index ($\max\{m_{n}|n=1,...,N\}$) to transmit leads to near-optimal performance in many problems [11, 12, 14]. Inspired by these results, we design the following scheme, as shown in Algorithm 1, which is termed as SMART. Due to the fact that the status evolution is unknown, as well as that no supervisor is available in real world, we adopt a deep reinforcement learning-based approach. The status in general is comprehended using a Deep Neural Network (DNN), specifically a Deep Deterministic Policy Gradient (DDPG) approach [35], while in practice if the status is straightforward, simpler approaches can be adopted.

In Algorithm 1, we use $m_{\mathsf{max}}$ and $m_{\mathsf{min}}$ to denote the maximum and minimum auxiliary costs, respectively. In practice, $m_{\mathsf{max}}$ and $m_{\mathsf{min}}$ are hyper-parameters that depend on network conditions to provide adaptation capabilities inside the interval $[m_{\mathsf{min}},m_{\mathsf{max}}]$. SMART works roughly as follows. In the single-agent training phase, a set of possible auxiliary cost $m$ is trained for each source node as if only itself is updating with the additional auxiliary cost. A mapping from each $m$ to the DDPG parameters is stored in the node’s database. Afterwards, in the auxiliary cost adaptation phase, the SNs²²2In practice, it is found that representative (not all) destination nodes elected as SN are sufficient. observe the network conditions (i.e., collisions, channel idle and status values) for a period of time $\mathsf{evaInt}$ in Step 1 and thereby adjust the auxiliary cost values. When the cost increases due to congestion, the auxiliary cost should be increased to discourage source nodes from transmitting; in other cases, the auxiliary cost should be decreased, or stays the same after convergence. The corresponding source nodes switch their DDPG parameters based on their respective SNs’ feedback. SMART is also compatible with different MAC protocols. For example, nodes would transmit randomly based on the backoff window size in CSMA and C-V2X protocols. Eventually, the network converges to a situation wherein appropriate auxiliary costs are attained such that only nodes with urgent transmission needs (e.g., large status prediction error) are transmitting and the network is properly loaded. Note that the auxiliary costs can be different among nodes since each source node follows its corresponding SN’s feedback. In the following section, we will observe in details how a system of platooning vehicles behave under SMART and parallel communications.

In this section, we demonstrate through SUMO a Wireless Networked Control (WNC) application, namely multiple dense platooning in steady state using C-V2X communication protocol and the proposed framework (see Fig. 7). The application is selected for two reasons. First, one of the most promising 5G vertical applications is C-V2X-enabled autonomous driving and platooning is perhaps the most attractive fully autonomous driving technology; secondly, platooning represents a WNC system wherein timely (milliseconds level) and precise status update and control are essential.

Scenario and Platoon Control: Several platoons, each led by a leader vehicle, travel close to each other (within the communication range of C-V2X which is typically $700$ meters [36]) on the highway in steady state, i.e., after platoons are formed. We consider only the longitudinal drive control, that is the vehicles travel on a straight line. In each platoon, the leader vehicle is driven by human and the others are controlled by the leader vehicle based on control algorithms such as (11) and wireless signals. Naturally, for the leader vehicles to make precise control, the status information it collects from its follower vehicles, including distances from the front vehicle and instantaneous velocities, should be as timely as possible. The ultimate goal of the system is to save fuel while ensuring safety, and hence the vehicles are designed to follow its corresponding front vehicles as close as possible while not crashing into them. In the simulations, we set a predefined desired distance between two vehicles and let the control algorithm to maintain this desired distance—the minimum safe distance is therefore the maximum distance reduction from the desired distance during the whole trajectory. The simulation scenario is a two-way, four-lane highway with a length of $1500$ meters. The length of vehicles is $5$ meters. The leader vehicle enters with a speed of $10$ m/s, then accelerates to $22.2$ m/s in $5$ seconds; from the $15$-th second, the leader decelerates to $9.7$ m/s in $5$ seconds, and then accelerates to $22.2$ m/s in $15$ seconds (depicted in the bottom-left figure in Fig. 8). The acceleration of vehicles is restricted to $[-2.94,4]$ m/s² to avoid very rapid and abrupt changes in speed. The simulation time step is $1$ ms. The actuation delay and sensing delay are ignored to focus on information timeliness, and hence the performance evaluation can be viewed as an optimistic bound.

Network Protocol: The underlying MAC- and PHY-layer protocols that convey the status and control information are based on C-V2X Mode 4 [31]. Specifically, as shown in Fig. 7, an SPS decentralized time/frequency resource allocation scheme is adopted wherein a vehicle UE with packet (Cooperative Awareness Message, CAM) to send could choose uniformly randomly from a pool of time/frequency resources. The pool is composed of several subchannels in the frequency domain and a number of subframes with length denoted by Resource Reservation Interval (RRI). Thereby, as the selection is fully autonomous, collisions may happen especially when the system is heavily loaded. In addition to collisions, Half-Duplex (HD) error also occurs in C-V2X Mode 4, which represents the error caused by receiving a packet while transmitting on the same subframe. In the simulations, we set the RRI to be $10$ ms and the number of subchannels to be $2$, in consistency with current standards [37].

Parallel Communications Implementation: The majority of implementation details are the same as described in Section II and III. In particular, the status model is estimated based on a sliding-window LMS method (10). We choose the cost function $\mathcal{E}(\cdot)$ in SMART as the standard deviation from the desired distance between successive vehicles. The model calibration time interval is $500$ ms.

Simulation Results: In Fig. 8, the proposed parallel communication framework is tested in comparison with the status-unaware scheme using the optimal status update interval. Status-unaware schemes represent those without knowledge of the status content. Therefore, the best they can achieve is to optimize over the update interval; we obtain the optimum by simulating over a set of status update intervals from $10$ ms to $150$ ms, with step size of $10$ ms, and the optimum is shown to be $40$ ms (see Fig. 9). Note that optimizing the average AoI among all vehicles is equivalent to optimizing over update intervals since the statuses are generate-at-will [11]. Intuitively, the tradeoff is that a smaller update interval leads to worse congestion, but lower update waiting delay; vice versa. Comparing among the first column in Fig. 8, it is observed that the parallel communication scheme with correction packets outperforms the others significantly, showing that by using model predictions, the network load is reduced such that the status attained and control is more timely and hence the vehicle distance variance is much smaller. Observing the middle column (Obv. 1), it is shown that without correction packets (which are transmitted periodically and carry status information) the parallel communication scheme suffers from the issue we discussed in Section II-E. That is, after a status packet loss, the source and destination are unaligned as can be observed in the bottom-middle figure; hence, the control is affected and the vehicle distance is kept to be larger than desired until the next unexpected status packet transmission. The transmissions of correction packets certainly entail overhead. From Obv. 2, it is shown that the optimal status packet update interval for status-unaware schemes is $40$ ms, and the correction packet transmission interval (same with model calibration packets) is $500$ ms. Therefore, the additional overhead of correction packets is well worth it since it is relatively small and useful. A key observation in Fig. 9, which shows that the optimal status update interval of status-unaware schemes is about $40$ ms, is that in status update, sometimes it is better to be timely but unreliable, than ultra-reliable but sacrificing timeliness. This is different with the current ultra-Reliable Low-Latency Communications (uRLLC) principle. Specifically, as shown in the figure, the status becomes stale when waiting to be updated in the out-of-date regime with large status update intervals; on the other hand, the transmission reliability is high in this regime since the network load is low and hence collisions rarely happen.

The effectiveness of dynamic auxiliary cost adaptation in SMART is shown in Fig. 10. The reference design is the status-unaware scheme with the optimal update interval. SMART is tested with different initial auxiliary cost values, as shown in the $x$-axis; note that for fixed auxiliary cost schemes, the initial value never changes afterwards. It is found that the reinforcement learning-based scheme can adapt to different network conditions, as represented by different numbers of vehicles, and obtain the best performance. However, a fixed auxiliary cost scheme without considering the networking aspect is insufficient to provide robust performance.

In this section, we demonstrate through MATLAB another Wireless Networked Control (WNC) application, namely UAV formation flight control using C-V2X communication protocol. The application is selected for two reasons. First, unmanned aerial vehicles (UAVs) are getting a large attention to be widely used in various areas due to their high potential and versatile usability; secondly, for some applications that need high reliability and responsibility, they may need some special requirements of collision avoidance, which means a low-latency and precise status updating wireless network is required.

Scenario and UAV Formation Flight Control: One formation of 10 UAVs, led by a leader UAV, travel close to each other in steady state. We consider only the full-direction flight control, that is the UAVs travel freely in three dimensions. Each UAV can sense its absolute three-dimension coordinates. In the formation, similar to the previous case, the leader UAV is driven by a preset algorithm and the others are controlled by the leader UAV based on control algorithms. The status information, in this case, includes coordinates and instantaneous velocities. The ultimate goal of the system is to precisely control the UAV formation flight, and hence the UAVs are designed to travel to the target coordinates as the leader UAV requires as close as possible. In the simulations, we set each UAV a desired distance away from its leader in each dimension. The simulation scenario is a space where all UAVs can travel freely. The leader UAV enters with a speed of $0$ m/s on all three axes, then accelerates to $0.49$ m/s on x-axis in $0.5$ second, $1.715$ m/s on y-axis in $0.5$ seconds and $0.49$ m/s on z-axis in $0.5$ seconds; from the $0.5$-th second, the leader decelerates to $0.245$ m/s on x-axis in $0.5$ second, $1.0682$ m/s on y-axis in $0.5$ second, $0$ m/s on z-axis in $0.5$ second (the velocity of x-axis is depicted in the bottom-left figure in Fig. 11). The acceleration of UAVs is restricted to $[-4,4]$ m/s² to avoid very rapid and abrupt changes in speed. The simulation time step is $1$ ms.

Network Protocol: The protocol is similar to the previous application’s. In this application, the resource re-selection period is set small so each UAV re-selects the radio resource frequently, which could cause severer radio collision risk.

Parallel Communications Implementation: The majority of implementation details are the same as described in Section IV. In the previous application, the status includes distance between vehicles. However, in this application, it is the absolute coordinates rather than the distance. As there is three axes, two more models are added to estimate the status of the other two dimensions. In particular, the model calibration time interval is $1000$ ms in this application.

Simulation Results: In Fig. 11, the proposed parallel communication framework is tested in comparison with the status-unaware scheme. The figure shows the result from $5$ s to $20$s when the model has converged. Since the status includes the absolute coordinates, the figure shows the coordinates rather than distance. Comparing among the first row in Fig. 11, it is observed that the parallel communication scheme, by using status predictions, outperforms the status-unaware scheme when the radio resource is insufficient comparing to the number of nodes. The distance between each two UAV can not keep constant in the status-unaware scheme while it can in the proposed scheme. Observing the second row, the performance on x-axis is shown. In the middle figure, the model calibration packets is transmitted per 1000ms and there are a few status packets transmitted after the model has converged. The proposed scheme reduces the radio resource cost clearly comparing to the status-unaware scheme. Since the model has converged and there are less radio resource collisions, the model predicted status is close to the act status by observing the right figure. The prediction error from $6000$ ms to $6030$ ms is about $0.005$ m which is smaller than the status transmission trigger threshold. Observing the third row, the performance on y-axis is shown. The performance is similar to the x-axis’s. The prediction error from $6000$ ms to $6030$ ms is about $0.003$ m. The estimation on y-axis is working well after the model is updated. The performance on z-axis is shown in the last row. The prediction error from $6000$ ms to $6030$ ms is about $0.002$ m. However, similar to the x-axis’s, the proposed scheme is working well. The performance of this application can show the generalization of the proposed parallel communication scheme on status communication problems.

In this paper, we proposed a parallel communication scheme whereby status information is communicated by both OTA packets and aligned predictions by calibrated status models. By this scheme, the wireless communication devices can adjust packet transmission frequencies based on network conditions and status predictions, such that the sampler can be communication-agnostic. The system is implemented on an SDR platform, and link-level experimental results show that the network load can be significantly reduced while maintaining low status recovery error by leveraging model predictions, namely revealing much while saying less. We also test the model on an integrated hardware-software evaluation platform for collaborative autonomous driving that we proposed. The results show that by adopting parallel communications, the channel occupancy is significantly reduced by online model predictions, while keeping the same status estimation error. In addition, SMART is proposed for networking such predictive wireless devices in an ad hoc WNC system based on a Whittle’s index-inspired reinforcement learning framework. To test the proposed approach in practice, we simulate two scenarios with promising future applications. First, we simulate on SUMO a multi dense platooning application. Then we use MATLAB to simulate a flight formation control scenario of UAV. It is shown by the vehicle and flight control performance in the two scenarios that the parallel communication scheme significantly outperforms both AoI-optimized status-unaware schemes and conventional uRLLC schemes, due to the fact that the proposed scheme can reduce the network load, thus improving transmission reliability with less packet collisions.

Future research topics include investigating more robust system identification, model estimation and time-series forecasting mechanisms for status model predictions in parallel communications. More generally, the interplay between sensing, communication, computation and control is worth studying to enable more efficient wireless applications.