AoI and Energy Consumption Oriented Dynamic Status Updating in Caching Enabled IoT Networks

We consider an IoT network consisting of $N$ users, one ECN, and one sensor, which serves users with time sensitive information. The user set is denoted by $\mathcal{N}=\{1,2,\cdots,N\}$. We assume a time-slotted system as illustrated in Fig. 1, where each slot is divided into two phases, i.e., the DDP and SUP, whose time durations are fixed and denoted by $D_{d}$ and $D_{u}$, respectively. In each slot $t$, users may request the ECN for the cached data packet, and the corresponding query profile can be expressed as $\mathbf{r}(t)=\left(r_{1}(t),r_{2}(t),\cdots,r_{N}(t)\right)$, where $r_{n}(t)\in\{0,1\},\forall n\in\mathcal{N}$, with $r_{n}(t)=1$ if user $n$ requests the data, and $r_{n}(t)=0$ otherwise. In this paper, we consider the ECN obtains the data query profile $\mathbf{r}(t)$ at the beginning of each time slot $t$, and it has no prior knowledge of users’ request patterns which is random.

As shown in Fig. 1, the ECN would first transmit the required data packet to the corresponding users during DDP. Then, the ECN has to decide whether or not ask the sensor update its status, and, if any, receives the status update during the SUP. Let $A(t)\in\{0,1\}$ denote the ECN’s update decision made in each time slot $t$, i.e., $A(t)=1$ if the sensor is asked to sense the underlying environment and update the status, and ${A}(t)=0$ otherwise. We consider all data packets requested by users can be successfully transmitted from the ECN. However, when each update is transmitted from the sensor to ECN, there is a potential update failure due to the limited transmission power of the sensor, which will be presented in detail in the following subsection.

During SUP in time slot $t$, if the sensor is asked to update its status, then it has to sense the underlying environment and transmit the information packet to the ECN. The energy consumed by the sensor for each environmental sensing is assumed to be constant and denoted by $E_{s}$. Besides, we assume the time spent on the environment sensing is negligible and hence, the transmission time of each update packet is $D_{u}$. The spectral bandwidth available for the data update is $B$ and the transmit power of the sensor is fixed as $p$. Accordingly, the overall energy consumption of one status update is $E=E_{s}+pD_{u}$. Let $g(t)$ denote the channel power gain from the sensor to ECN in time slot $t$, and $\delta^{2}$ the AWGN power density at the ECN. It is assumed that the channel between the sensor and ECN follows the quasi-static Rayleigh fading with mean $\bar{g}$, i.e., its power gain remains the same over one SUP and changes independently across time slots.

The size of each update packet is considered to be constant and denoted by $F$. To complete the data transmission in one SUP, the required transmission rate is set to $R=\frac{F}{D_{u}}$. In this light, if the sensor is asked to transmit its update packet, a transmission failure can occur when the required SNR threshold cannot be reached. Particularly, according to Shannon’s formula, to meet the transmission rate requirement $R$, the SNR threshold should be set as

Furthermore, the update failure probability $P_{f}$ for one update transmission, if there is any, can be expressed as [14, 15]

where $\gamma(t)$ denotes the SNR with the corresponding transmission, and $\bar{\gamma}=\frac{p\bar{g}}{B\delta^{2}}$ the average SNR.

For ease of expression, we use $Z(t)\in\{0,1\}$ to denote whether the asked update is successfully transmitted to the ECN, i.e., $Z(t)=1$ if the update is successfully delivered and $Z(t)=0$ otherwise. Since the sensor will only generate and transmit the update packet when it is asked by the ECN, we further have $Z(t)\leq A(t)$ with each time slot $t$.

In this paper, we consider both the AoI at the users and the ECN just before the decision moment in each time slot, i.e., the beginning of the status updating phase. Before formally depicting the evolution of AoI at each user, we first give the dynamics of AoI for the stored data packet from the ECN’s perspective. Particularly, we denote the AoI at the ECN in slot $t$ by the $\Delta_{0}(t)$, whose evolution can be expressed as

where $Z(t-1)=1$ means in the previous slot $t-1$ an update packet was successfully delivered to the ECN.

Similarly, let $\Delta_{n}(t)$ denote the AoI at each user $n$ with slot $t$. Then, its dynamics can be expressed as

where the product $r_{n}(t)Z(t-1)=1$ means the packet required by user $n$ was just successfully updated to the ECN in the previous slot $t-1$, and (a) holds because of Eq. (3). Without loss of generality, we initialize $\Delta_{0}(1)=\Delta_{n}(1)=D_{u}+D_{d},\forall n\in\mathcal{N}$. Based on the above analyses, we note that, even obtaining data from the same ECN, different AoIs may be experienced by different users, which would be ignored if we just consider the dynamics of the AoI at the ECN.

To strike a balance between the average AoI experienced by users and energy consumed by sensors, we define the overall cost associated with the update decision made by the ECM in each slot $t$ as

where $\omega_{n}$ denotes the weight allocated to user $n$, i.e., $\omega_{n}\in(0,1),\forall n\in\mathcal{N}$ and $\sum\nolimits_{n=1}^{N}\omega_{n}=1$, $E$ represents the energy consumed to complete one status update, and $\beta_{1}$ as well as $\beta_{2}$ are parameters used to nondimensionalize the equation and meanwhile, to make a trade-off between the experienced AoI and consumed energy in the network.

In this work, we aim at designing a dynamic status updating strategy to minimize the expectation of a long term accumulative cost, i.e.,

where $\mathbf{A}$ denotes a sequence of update decisions made by the ECN from slot $1$ to $T$, and $\lambda$ the discount rate introduced to determine the importance of the present cost and meanwhile to guarantee the long term accumulative cost finite. The formulated dynamic optimization problem will be solved in the following section.

The concerned dynamic status updating can be formulated into an MDP consisting of a tuple $\mathcal{M}=\Gamma\left(\mathbb{S},\mathbb{A},U\left({\cdot,\cdot}\right)\right)$, which is depicted as follows:

• •

  State space $\mathbb{S}$: At each time slot $t$, the state $\mathcal{S}(t)$ is defined to be the combination of the AoIs at the ECN and users just before the data updating phase, i.e., $\mathcal{S}(t)=\left(S_{0}(t),S_{1}(t),S_{2}(t),\cdots,S_{N}(t)\right)$, where $S_{m}(t)=\Delta_{m}(t),\forall m\in\{0\}\cup\mathcal{N}$. We set the maximum AoI at the ECN and users to be $\Delta_{\max}=M_{\max}D$ where $D=D_{u}+D_{d}$ and $M_{\max}$ is an integer. In this light, the state space $\mathcal{S}$ is finite and can be expressed as

  $\displaystyle\mathbb{S}=\{{{\mathbb{S}_{0}}}\times{{\mathbb{S}_{1}}}\times{{\mathbb{S}_{2}}}\times\cdots\times{{\mathbb{S}_{N}}}\}$

  (8)

  with $\mathbb{S}_{m}=\{D,2D,\cdots,M_{\max}D\},{\forall m\in\{0\}\cup\mathcal{N}}$.

• •

  Action space $\mathbb{A}$: At each time slot $t$, the ECN could ask the sensor to sense the underlying environment and update its generated data packet. Therefore, the action space can be expressed as $\mathbb{A}=\{0,1\}$, where $A=0$ ($A\in\mathbb{A}$) means the ECN would not ask the sensor to conduct its status update.

• •

  Reward function $U\left({\cdot,\cdot}\right)$: The reward function is a mapping from a state-action pair $\mathcal{S}\times A$ to a real number, which is used to quantify the obtained reward by choosing a action $A\in\mathbb{A}$ at a state $\mathcal{S}\in\mathbb{S}$. In this work, at each time slot $t$, when given a state $\mathcal{S}(t)$ and choosing action $\mathcal{A}(t)$, we define the reward function as

  		$\displaystyle U\left({\mathcal{S}(t),A(t)}\right)$
  		$\displaystyle=C-\left(\beta_{1}\sum\nolimits_{n=1}^{N}{\omega_{n}\left(\Delta_{n}(t)+D_{u}\right)}+\beta_{2}A(t)E\right)$
  		$\displaystyle\mathop{=}\limits^{(a)}\mathop{C}-\beta_{1}D_{u}-L(t)=C_{1}-L(t)$		(9)

  where $C$ is a constant used to ensure the reward positive, and (a) follows Eq. (5). Here, $D_{u}$ is introduced to evaluate the effect of the chosen action after it is conducted, i.e., after the SUP is over.

For an MDP, the goal is generally to find a deterministic stationary policy $\pi:\mathbb{S}\to\mathbb{A}$ to maximize the long term expected discounted return. Particularly, a policy is said to be deterministic and stationary if: 1. Given the state, only one certain action is chosen; 2. The policy is irrelevant to time. In this work, we aim at deriving a deterministic stationary policy $\pi^{*}$ that maximizes the long term expected discounted reward with the initial state $\mathcal{S}(0)$, i.e.,

where (a) holds when all the elements in $\mathcal{S}(0)$ are set to $D$. Comparing Eq. (6) with Eq. (III-A), we note that $\pi^{*}$ can also be used to derive a solution of the original Problem P.

As shown in Eq. (III-A), in each time slot $t$, the immediately obtained reward $U\left({\mathcal{S}(t),A(t)}\right)$ does affect the cumulative reward in the long run. Therefore, to find the optimal strategy $\pi^{*}$, it is essential to accurately estimate the long-term effect of each decision, which is nontrivial because of the causality. In this work, we resort to RL, and design a model-free algorithm, named as EAU, to solve it, which will be presented in detail in the following subsection.

For each deterministic stationary policy $\pi$, we define the action-value function as shown in Eq. (11), with $(\mathcal{S},A)$ denoting the initial action-state pair, and the corresponding Bellman optimality equation can be expressed in Eq. (12) [16], where ${\mathsf{P}\left({{\mathcal{S}}^{\prime}\left|{{\mathcal{S}},{A}}\right.}\right)}$ denotes the transition probability from one state $\mathcal{S}$ to $\mathcal{S}^{\prime}$ by conducting an action $A$. Essentially, if that probability is known, the Bellman optimality equations are actually a system of equations with $\left|{\mathbb{S}\times\mathbb{A}}\right|$ unknowns, where $\left|{\cdot}\right|$ represents the cardinality of a set. In that case, for a finite MDP, we could derive the unique solution by utilizing model based RL algorithms, e.g., dynamic programming[16].

However, in this work, the transition probability cannot be regarded as a prior knowledge due to the unknown of the users’ request patterns and mean channel gain from the sensor to ECN. To deal with the corresponding challenge, we develop a model-free algorithm by resorting to expected Sarsa, one kind of temporal-difference (TD) RL algorithms. We note that, compared with traditional TD algorithms, e.g., classic Sarsa and Q-learning, expected Sarsa may achieve better performance with small additional computational cost for many problems [16]. Our developed EAU algorithm is shown in Algorithm 1, where $\hat{\mathbf{Q}}(t)$ is a row vector with $\left|{\mathbb{S}\times\mathbb{A}}\right|$ elements, denoting the estimate of the action-value function obtained after the $t$-th iteration. In this algorithm, the loop is repeated until the number of iterations reaches the maximum value $T_{\max}$.

At the beginning of EAU, the estimate of the action-value vector $\hat{\mathbf{Q}}(0)$ is initialized as a vector with all elements being 0, and all elements of the initial state $\mathcal{S}(0)$ are set to $D$. We note the constant $C_{1}$ in Eq. (• ‣ III-A) could be chosen arbitrarily, and, as an instance, we set it as

with which the immediately obtained reward in each iteration $t$ is guaranteed to be positive.

When the initialization is completed, the algorithm goes into a loop. At each iteration $t$, we will first choose an action $A(t)$ from the action space $\mathbb{A}$ based on the current state $\mathcal{S}(t)$ by looking up the action-value vector $\hat{\mathbf{Q}}(t)$. To balance the exploration and exploitation, we adopt the $\varepsilon$-greedy policy here, i.e., choosing an action from the space $\mathbb{A}$ with the following probability distribution

where $\hat{Q}\left({\mathcal{S}(t),A}\right)$ denotes the estimated state-action value for the pair $\left({\mathcal{S}(t),A}\right)$ with the vector $\hat{\mathbf{Q}}(t)$, and $\varepsilon$ belongs to $\left(0,1\right)$. After that, we would conduct the action $A(t)$, obtain a reward $U\left({\mathcal{S}(t),A(t)}\right)$, as shown in Eq. (• ‣ III-A), and then observe a new state $\mathcal{S}(t+1)$.

Learning from the new experiences (i.e., $\mathcal{S}(t)$, $A(t)$ and $\mathcal{S}(t+1)$), we would update the estimate of the action-value vector from $\hat{\mathbf{Q}}(t)$ to $\hat{\mathbf{Q}}(t+1)$ by just updating the element associated with the pair ($\mathcal{S}(t)$, $A(t)$). Particularly, if we denote the element with ($\mathcal{S}(t)$, $\mathcal{A}(t)$) in $\hat{\mathbf{Q}}(t)$ and that in $\hat{\mathbf{Q}}(t+1)$ by $\hat{Q}\left({\mathcal{S}(t),\mathcal{A}(t)}\right)$ and $\hat{Q^{\prime}}\left({\mathcal{S}(t),\mathcal{A}(t)}\right)$, respectively, we have

where $\alpha\in(0,1]$ is the learning step-size and $\Xi(t)$ denotes the obtained TD error. Here, $\Xi(t)$ can be as

where $\lambda$ denotes the discount rate and the expectation

represents the expected action-value we will have, if we start from the state $\mathcal{S}(t+1)$, still evaluate the effect of actions according to the action-value vector $\hat{\mathbf{Q}}(t)$, and choose the action with the $\varepsilon$-greedy policy as shown in Eq. (14).

When the algorithm is terminated, we will obtain an estimated action-value function mapping from each state-action pair to a positive number, i.e., $\hat{\mathbf{Q}}(T_{\max})$. With $\hat{\mathbf{Q}}(T_{\max})$ and the initial state $\mathcal{S}=\{D,D,\cdots,D\}$, we can obtain an approximate solution of Problem P by keeping looking up $\hat{\mathbf{Q}}(T_{\max})$ and choosing the action bringing the maximum action-value in each decision time.