Meta-Reinforcement Learning for Timely and Energy-efficient Data Collection in Solar-powered UAV-assisted IoT Networks

The energy consumption of a rotary-wing UAV is chiefly determined by its propulsion and communication necessities. The following is an expression for the UAV’s propulsion power during time slot $t$ [6]:

where $n_{r}$ represents the number of rotors, $x_{T}$, $x_{s}$, and $x_{f}$ indicate the thrust coefficient based on disc area, rotor solidity, and the incremental correction factor of induced power, respectively, $\chi$ is the local blade section drag coefficient, $\rho$ is air density, $A$ denotes the disc area for each rotor, $d_{0}$ denotes the fuselage drag ratio for each rotor, and $T_{\textrm{h}}(t)$ indicates the thrust of each rotor. To better elaborate, we only take into account the acceleration that is parallel to the velocity [6]. Hence, each rotor’s thrust can be expressed as

where $a_{\textrm{c}}(t)=(v_{s}(t+1)-v_{s}(t))/\tau_{0}$, $S_{FA}$ denotes the fuselage equivalent flat plate area, $M$ and $g$ are the UAV’s weight and the gravity acceleration, respectively.

The energy consumption of the UAV $e_{\textrm{c}}(t)$ in a time slot is discussed in the following four scenarios: If the UAV is in a restoring mode at time slot $t$, i.e., $m(t)=0$, and the energy level is less than $E_{\textrm{th}}^{2}$, i.e., $E(t)<E_{\textrm{th}}^{2}$, the energy consumption is zero; if the UAV is in a working mode at time slot $t$, i.e., $m(t)=1$, the energy level is greater than $E_{\textrm{th}}^{1}$, i.e., $E(t)\geq E_{\textrm{th}}^{1}$, and it schedules an SN while offloading data packets, i.e., $b(t)\neq 0$ and $q(t)=1$, the energy consumption is $\tau_{0}P_{\textrm{c}}^{\textrm{p}}(t)+(\tau_{0}-\tau_{c})P_{\textrm{c}}^{\textrm{t}}$; if the UAV is in a working mode at time slot $t$, i.e., $m(t)=1$, the energy level is greater than $E_{\textrm{th}}^{1}$, i.e., $E(t)\geq E_{\textrm{th}}^{1}$, and it only offloads data packets without scheduling an SN, i.e., $b(t)=0$ and $q(t)=1$, the energy consumption is $\tau_{0}\left(P_{\textrm{c}}^{\textrm{p}}(t)+P_{\textrm{c}}^{\textrm{t}}\right)$; in other cases, the UAV uses its propulsion energy just for flying. Hence, the UAV’s energy consumption $e_{\textrm{c}}(t)$ in time slot $t$ is summarized in (11) on the next page.

Providing a sustainable energy supply for the UAV with limited battery capacity is a crucial challenge. Assuming that the UAV can harvest energy from sunlight, solar energy is not reliable and depends on various factors, such as weather and altitude. The process of solar energy harvesting by the UAV is portrayed as an independent Bernoulli process with parameter $\lambda_{1}$ [50]. This signifies that the probability of solar energy arriving at the UAV at each time slot is $\lambda_{1}$. The solar energy reaching the UAV during time slot $t$ can be expressed as [8]

where $h(t)$ is the UAV’s altitude in time slot $t$, $\eta_{1}$ denotes the efficiency of energy conversion, $S_{u}$ indicates the area of the solar panel that effectively receives light, $G_{1}$ represents the average solar radiation on the ground, $\varphi_{1}$ and $\varphi_{2}$ are the maximum atmospheric transmittance value and the atmosphere’s extinction coefficient, respectively, and $h_{1}$ denotes the earth’s scale height. Therefore, the energy harvested by the UAV during time slot $t$ is given by

To sum up, the dynamics of the UAV’s battery level can be represented as

The state in time slot $t$ is denoted as $\boldsymbol{s}(t)=(\boldsymbol{\Psi}(t),v_{s}(t),\phi(t-1),\boldsymbol{Y}(t),\boldsymbol{U}(t),\boldsymbol{\delta}(t),E(t))$, where

• •

  $\boldsymbol{\Psi}(t)$ denotes the set of relative positions between the UAV’s ground projection at time slot $t$ and all the nodes, i.e., $\boldsymbol{\Psi}(t)=(\boldsymbol{\psi}_{0}(t),\boldsymbol{\psi}_{1}(t),\ldots,\boldsymbol{\psi}_{N}(t))$, where $\boldsymbol{\psi}_{n}(t)=\boldsymbol{C}_{u}(t)-\boldsymbol{C}_{n}=(x_{u}(t)-x_{n},y_{u}(t)-y_{n}),n\in\mathcal{N}^{+}$.

• •

  $v_{s}(t)$ denotes the UAV’s speed at the start of time slot $t$.

• •

  $\phi(t-1)$ denotes the UAV’s direction of velocity at time slot $t-1$.

• •

  $\boldsymbol{Y}(t)$ is the set of lifetime of the update data packet in time slot $t$ for all SNs, i.e., $\boldsymbol{Y}(t)=(Y_{1}(t),Y_{2}(t),\ldots,Y_{N}(t))$.

• •

  $\boldsymbol{U}(t)$ is the set of lifetime of the update data packet at the UAV in time slot $t$ for all SNs, i.e., $\boldsymbol{U}(t)=(U_{1}(t),U_{2}(t),\ldots,U_{N}(t))$.

• •

  $\boldsymbol{\delta}(t)$ is the set of AoI values in time slot $t$ for all SNs, i.e., $\boldsymbol{\delta}(t)=(\delta_{1}(t),\delta_{2}(t),\ldots,\delta_{N}(t))$.

• •

  $E(t)$ denotes the UAV’s energy level in time slot $t$.

The UAV’s action in time slot $t$ is characterized by its speed $v_{s}(t+1)$ at the start of time slot $t+1$ and direction $\phi(t)$, the scheduling of SNs $b(t)$, and the offloading decision $q(t)$, i.e., $\boldsymbol{a}(t)=(v_{s}(t+1),\phi(t),b(t),q(t))$, where $v_{s}(t+1)$ and $\phi(t)$ are continuous, while $b(t)$ and $q(t)$ are discrete.

We detail the transition of each element in $\boldsymbol{s}(t)$. The update of the relative position of the UAV’s projection on the ground to the node $n$, $\boldsymbol{\psi}_{n}(t)$ $n\in\mathcal{N}^{+}$, relies on the UAV’s location, speed, and direction of the velocity, which can be expressed as

The lifetime update of the update data packet at each SN depends on the arrival status of the packet. In particular, if a new packet is received at SN $n$, its lifetime at the SN is initialized to zero; otherwise, the lifetime at the SN is increased by one. The update of the lifetime of the update data packet at the SN is given in Eq. (6).

The lifetime update of the update data packet for each SN at the UAV depends on the scheduling status $z_{n}(t)$ from SN $n\in\mathcal{N}$ to the UAV. Specifically, if the update status of SN $n$ is successfully transmitted to the UAV, the lifetime of SN $n$ at the UAV is set to its lifetime at SN $n$ plus one; otherwise, the lifetime at the UAV is increased by one. The update of the lifetime of the update data packet at the UAV is given in Eq. (7).

The AoI update for each SN depends on the offloading status $o(t)$ from the UAV to the DC. Due to the channel’s stochastic characteristics, the scheduled SN’s AoI may not decrease. Specifically, if the UAV successfully offloads the packet of SN $n$ in its buffer to the DC within time slot $t$, the AoI of SN $n$ is updated to the lifetime of SN $n$ at the UAV; otherwise, the AoI of SN $n$ is incremented by one. The AoI update is presented in Eq. (8).

The update of the UAV’s energy level is dependent on the energy consumption and energy arrival status, which is represented in Eq. (14).

The UAV’s reward in time slot $t$ is defined to minimize the weighted sum of the average AoI of SNs and the energy consumption of the UAV, and it is expressed as $r(t)=-\frac{1}{T}(\omega_{1}\sum_{n=1}^{N}\delta_{n}(t)+\omega_{2}e_{\textrm{c}}(t))$.

The goal of the MDP is to discover an optimal policy that, beginning with the initial state $s(1)$, maximizes the expected total reward during $T$ time slots. In particular, the optimal policy is the solution of the MDP, i.e.,

where the expectation is relative to the distribution of the sequence of states and actions following the policy $\pi$ and the transition probabilities.

In this subsection, we first introduce the architecture of the CADRL-based algorithm shown in Fig. 2, which can manage both discrete and continuous actions. This architecture is based on the actor-critic architecture but contains two parallel actor networks. These two actor networks are utilized to perform discrete-action selection and continuous-action selection, respectively. In particular, the two actor networks share the first few layers with parameters $\boldsymbol{\theta}^{\textrm{s}}$ to extract the features from the input state $\boldsymbol{s}$. Then, the single-stream neural network is partitioned into two streams, forming the discrete actor network’s output layer with parameters $\boldsymbol{\theta}^{\textrm{d}}$ and the continuous actor network’s output layer with parameters $\boldsymbol{\theta}^{\textrm{c}}$, respectively. A stochastic policy $\pi(\boldsymbol{s};\boldsymbol{\theta}^{\textrm{s}},\boldsymbol{\theta}^{\textrm{d}})$ is learned by the discrete actor network for selecting discrete actions $\boldsymbol{a}^{\textrm{d}}$, and the continuous actor network learns a stochastic policy $\pi(\boldsymbol{s};\boldsymbol{\theta}^{\textrm{s}},\boldsymbol{\theta}^{\textrm{c}})$ for determining continuous actions $\boldsymbol{a}^{\textrm{c}}$.

To generate the stochastic policy $\pi(\boldsymbol{s};\boldsymbol{\theta}^{\textrm{s}},\boldsymbol{\theta}^{\textrm{d}})$, the discrete actor network outputs a vector of values $f=(f_{\boldsymbol{a}_{1}^{\textrm{d}}},f_{\boldsymbol{a}_{2}^{\textrm{d}}},\ldots,f_{\boldsymbol{a}_{A_{\textrm{d}}}^{\textrm{d}}})$ for the $A_{\textrm{d}}$ discrete actions, where $A_{\textrm{d}}=2(N+1)$ is the dimension of the discrete action space. Then, a discrete action $\boldsymbol{a}^{\textrm{d}}$ is randomly sampled from the distribution represented by $\textrm{softmax}(f)$. The continuous actor network outputs both the mean and variance of the Gaussian distribution to form the stochastic policy $\pi(\boldsymbol{s};\boldsymbol{\theta}^{\textrm{s}},\boldsymbol{\theta}^{\textrm{c}})$ for continuous actions $\boldsymbol{a}^{\textrm{c}}$. Both the discrete and continuous actor networks are updated using the trust region policy optimization (TRPO) method in this study. The stochastic policy $\pi(\boldsymbol{s};\boldsymbol{\theta}^{\textrm{s}},\boldsymbol{\theta}^{\textrm{d}})$ and the stochastic policy $\pi(\boldsymbol{s};\boldsymbol{\theta}^{\textrm{s}},\boldsymbol{\theta}^{\textrm{c}})$ are updated by minimizing their loss functions, respectively. The loss function of the discrete policy $\pi(\boldsymbol{s};\boldsymbol{\theta}^{\textrm{s}},\boldsymbol{\theta}^{\textrm{d}})$ is given by [51]

where $\mathbb{E}[\cdot]$ denotes the expectation for a limited batch of experiences, $(\boldsymbol{\theta}_{\textrm{old}}^{\textrm{s}},\boldsymbol{\theta}_{\textrm{old}}^{\textrm{d}})$ is the parameters before the update, $\textrm{KL}[\pi_{1},\pi_{2}]$ represents the KL divergence between two policies $\pi_{1}$ and $\pi_{2}$, $\epsilon$ denotes the maximum KL divergence constraint. Similarly, the loss function of the continuous policy $\pi(\boldsymbol{s};\boldsymbol{\theta}^{\textrm{s}},\boldsymbol{\theta}^{\textrm{c}})$ is given by [51]

In the CADRL architecture, there is one critic network with parameters $\boldsymbol{\vartheta}$ that is utilized to estimate the state-value function $V(\boldsymbol{s};\boldsymbol{\vartheta})$. The critic network undergoes updates by minimizing the loss function, which is defined as

where $V^{\textrm{target}}(t)=\hat{A}(t)+V(s(t))$, and $\hat{A}(t)$ is the generalized advantage estimation (GAE) that is expressed by [52]

where $\lambda$ is the GAE parameter, $\gamma$ represents the discount factor, and $\hat{A}(0)=0$.

Algorithm 1 outlines the process of the CADRL-based solar-powered UAV-assisted data collection algorithm. The process begins with the initialization of the maximum KL divergence $\epsilon$, replay buffer $D$, as well as the parameters of the actor network $(\boldsymbol{\theta}^{\textrm{s}},\boldsymbol{\theta}^{\textrm{d}},\boldsymbol{\theta}^{\textrm{c}})$ and critic network $\boldsymbol{\vartheta}$ (Line 1). In time slot $t$, the UAV selects the discrete action component $\boldsymbol{a}^{\textrm{d}}(t)\sim\pi(\boldsymbol{s}(t);\boldsymbol{\theta}^{\textrm{s}},\boldsymbol{\theta}^{\textrm{d}})$ and the continuous action component $\boldsymbol{a}^{\textrm{c}}(t)\sim\pi(\boldsymbol{s}(t);\boldsymbol{\theta}^{\textrm{s}},\boldsymbol{\theta}^{\textrm{c}})$ according to the current state $\boldsymbol{s}(t)$, conducts the action $\boldsymbol{a}(t)=(\boldsymbol{a}^{\textrm{d}}(t),\boldsymbol{a}^{\textrm{c}}(t))$, receives a reward $r(t)$, and transitions a new state $\boldsymbol{s}(t+1)$. Then, the UAV places the experience ($\boldsymbol{s}(t),\boldsymbol{a}(t),r(t)$,$\boldsymbol{s}(t+1)$) in the replay buffer (Lines 5~8). To update the actor and critic networks, a mini-batch of experiences is sampled from the replay buffer. The advantage function is computed as (21). The loss functions of the discrete and continuous actor networks are calculated according to (18) and (19), respectively, and are minimized to update the two actor networks. The critic network undergoes updates through the minimization of its loss function (20) (Lines 9~12).

The CADRL-based algorithm is applicable to a specific task where the number and positions of SNs are fixed. When the task changes, the policy learned by the CADRL-based algorithm is not applicable to the new task. The use of a meta-learning-based DRL method brings the advantage of quick adaptation to new tasks, drawing on previous knowledge from prior experiences, and minimising the need for extensive training data [53]. In parallel, model-agnostic meta-learning (MAML) is a gradient-based meta-RL approach that concentrates on parameter optimization of the meta-policy during meta-training, serving as a solid starting point for unforeseen tasks [54]. Hence, we utilize MAML to enhance the generalization of the DRL method across different tasks and propose a meta-learning-based compound-action deep reinforcement learning (MLCADRL) algorithm.

An agent in MAML endeavors to acquire a meta-policy with parameters $\boldsymbol{\theta}=(\boldsymbol{\theta}^{\textrm{s}},\boldsymbol{\theta}^{\textrm{d}},\boldsymbol{\theta}^{\textrm{c}})$ across a multitude of tasks from a given task distribution $p(M)$. Each task $M_{i}\sim p(M)$ can be described as an MDP model defined by $(\mathcal{S}_{i},\mathcal{A}_{i},r_{i},\mathcal{P}_{i})$, where $\mathcal{S}_{i}$, $\mathcal{A}_{i}$, $r_{i}$, and $\mathcal{P}_{i}$ denote the state space, action space, reward function, and state transition function of task $M_{i}$, respectively. As depicted in Fig. 3, the training process for the MLCADRL-based algorithm consists of two alternating learning phases: the inner-loop task learner and the outer-loop meta learner. The meta parameters $\boldsymbol{\theta}$ in both loops are optimized by using gradient descent. In the inner-loop, the task learner is initialized with the meta parameters $\boldsymbol{\theta}$. It computes the updated parameters $\boldsymbol{\theta}_{i}$ of the task learner for each training task $M_{i}$ using the training data $\mathcal{D}_{i}^{\textrm{tr}}$, which is collected by the meta-policy $\boldsymbol{\theta}$, which is expressed as

where $\alpha_{1}$ is the task learner’s learning rate and $\mathcal{L}$ denotes the loss function of an RL learning method. Then, the task learner utilizes validation data $\mathcal{D}_{i}^{\textrm{vd}}$ sampled from trajectories collected with updated parameters $\boldsymbol{\theta}_{i}$ for task $M_{i}$ to estimate the loss function, expressed as

In the outer-loop, as the policy is updated for each training task, the meta learner accumulates the loss $\mathcal{L}_{M_{i}}(\boldsymbol{\theta}_{i},\mathcal{D}_{i}^{\textrm{vd}})$ and carries out a meta-gradient update on the meta parameters $\boldsymbol{\theta}$ as

where $\alpha_{2}$ is the learning rate of the meta learner. These processes iterate for $L_{\textrm{meta}}$ times. Upon the completion of meta-training, the meta-policy can be employed as the initial policy for new tasks, facilitating rapid adaptation to new tasks.

The solar-powered UAV-assisted data collection algorithm, based on MLCADRL, is detailed in Algorithm 2. First, the task distribution $p(M)$, the number of meta-iterations $L_{\textrm{meta}}$, the number of tasks of each meta-iteration $I$, the number of trajectories sampled for each task $K$, and the parameters of the meta-policy $\pi(\boldsymbol{\theta}^{\textrm{s}},\boldsymbol{\theta}^{\textrm{d}},\boldsymbol{\theta}^{\textrm{c}})$ are initialized (Line 1). In the meta-training phase, the inner-loop task learner and outer-loop meta learner are performed alternately. In particular, in the inner-loop task learner phase, the task-specific policy is updated by using the vanilla policy gradient algorithm (REINFORCE) [55]. We sample $I$ tasks from the task distribution (Line 3). The loss functions of the discrete and continuous actor networks for each task $M_{i}$ are given by

where $\boldsymbol{s}_{i,k}(t)$, $\boldsymbol{a}_{i,k}^{\textrm{c}}(t)$, and $\boldsymbol{a}_{i,k}^{\textrm{d}}(t)$ respectively denote the state, continuous action, and discrete action of task $M_{i}$ at time slot $t$ along trajectory $k$. We generate $K$ trajectories $\mathcal{D}_{i}^{\textrm{tr}}$ by executing the meta-policy on task $M_{i}$ to calculate the gradients of the loss functions in (25) and (26). The task-specific policy parameters $(\boldsymbol{\theta}_{i}^{\textrm{s}},\boldsymbol{\theta}_{i}^{\textrm{d}},\boldsymbol{\theta}_{i}^{\textrm{c}})$ are obtained by performing one or more gradient updates in (22) (Lines 5~7). In the outer-loop meta-learner phase, the loss functions of the discrete and continuous actor networks are given by

To calculate the gradients of the loss functions in (27) and (28), the meta-learner summarizes the trajectories $\mathcal{D}_{i}^{\textrm{vd}}$ sampled using the policy $\pi(\boldsymbol{\theta}_{i}^{\textrm{s}},\boldsymbol{\theta}_{i}^{\textrm{d}},\boldsymbol{\theta}_{i}^{\textrm{c}})$ for each task. The meta-policy is updated by minimizing these loss functions using the TRPO method (Lines 8 and 10).