Multi-Agent Graph Reinforcement Learning based On-Demand Wireless Energy Transfer in Multi-UAV-aided IoT Network

Each UAV’s energy consumption is mainly caused by propulsion and WET. According to [11], in each slot $t$, UAV-$u$’s propulsion power consumption is determined by its velocity $V_{u}[t]=\frac{1}{\vartheta}\left\|q_{u}[t+1]-q_{u}[t]\right\|$ as follows:

where the constants $P_{a}$, $P_{b}$, $V_{tip}$, $e_{0}$, $e_{1}$, $f_{0}$ and $\rho$ are the UAV’s mechanical-related parameters. Hence, the propulsion energy consumption of UAV-$u$ in slot $t$ is obtained as $P_{pro}(V_{u}[t])\vartheta$. Denote $P_{u}$ as each UAV-$u$’s transmit power for WET. The energy consumption of UAV-$u$’s WET in slot $t$ is thus $C_{u}[t]P_{u}\vartheta$. Denote $B_{u}[t]$ as the battery level of UAV-$u$ at the beginning of slot $t$, which is updated as

We assume that UAV-$u$ is fully charged in the initial with $B_{u}[0]=B_{u}^{max}$, where $B_{u}^{max}$ is UAV-$u$’s battery capacity.

Each IoT device is installed with a rectenna and an energy harvester. According to [12], the energy harvester transforms the received RF power $p\geq 0$ at the rectenna into the direct-circuit (DC) power $\mathcal{F}(p)$ as follows:

where $P_{sen}$ and $P_{sat}$ with $0<P_{sen}<P_{sat}$ are the sensitivity power and the saturation power at the energy harvester, respectively, and $f(\cdot)$ is a non-linear power transform function that can be easily obtained through the curve fitting technique [12]. From (4), no DC power is harvested if $p$ is below $P_{sen}$ and the harvested DC power keeps unchanged if $p\geq P_{sat}$. The value of $P_{sen}$ is usually high (with, e.g., -10 dBm [13]) in practice. Hence, the transmission distance from the UAV to the IoT device needs to be sufficiently short to assure effective WET with non-zero harvested energy. For each device-$i$, since its harvested energy from multiple UAVs in each slot can be accumulated, based on (1) and (4), the harvested energy at device-$i$ in slot $t$ is obtained as

From (5), device-$i$ can harvest more energy if more UAVs are located nearby and transmit energy to it jointly. Denote $B_{i}[t]$ as the battery level of device-$i$ at the beginning of slot $t$, which is updated as

where $B_{i}^{max}$ is the battery capacity of device-$i$ and the initial battery energy $B_{i}[0]\geq 0$ is given. Denote $B^{thr}$ as the required battery energy level at each IoT device, where $B_{i}^{max}\geq B^{thr}>B_{i}[0]>0$ holds in general. We say an IoT device is energy satisfied if its accumulated battery energy reaches the required $B^{thr}$ before or at the last slot $T$. Hence, $\mathbb{I}_{l}[t]=\{i|B_{i}[t]+E_{i}^{har}[t]<B^{thr}\}$ is the set of energy-unsatisfied IoT devices at the end of slot $t$. After the UAVs’ WET for $T$ slots, the set of energy-unsatisfied IoT devices is obtained as $\mathbb{I}_{l}[T]$.

The HoE is defined to measure the time-varying energy demand of each IoT device. Denote $H_{i}[t]$ as the HoE of device-$i$ at the beginning of slot $t$. We define the time variation of $H_{i}[t]$ as

where $E^{exp}\!\triangleq\!\frac{B^{thr}}{T}$ denotes the average amount of energy that device-$i$ expects to harvest in each slot for reaching $B^{thr}$ after $T$ slots. If device-$i$’s harvested energy $E_{i}^{har}[t-1]$ in slot ($t-1$) reaches the expected $E^{exp}$, but the resultant battery energy $B_{i}[t]$ at the beginning of slot $t$ is still lower than the required $B^{thr}$, $H_{i}[t]$ is reduced by $1$ at the beginning of slot $t$, where the minimum allowable HoE when $B_{i}[t]<B^{thr}$ is set to $1$. If $E_{i}^{har}[t-1]$ is lower than the expected $E^{exp}$ and the resultant $B_{i}[t]$ is also lower than the required $B^{thr}$, $H_{i}[t]$ is increased by $1$ at the beginning of slot $t$. Moreover, if device-$i$’s required energy is satisfied with $B_{i}[t]\geq B^{thr}$ at the beginning of slot $t$, $H_{i}[t]$ becomes $0$. The overall HoE of all the energy-unsatisfied IoT devices over $T$ slots is then obtained as

It is easy to find that $H_{total}$ is $0$, if $\mathbb{I}_{l}[T]$ is empty.

By optimally determining all the UAVs’ WET decisions $\boldsymbol{C}=\{C_{u}[t]\}$ and trajectories $\boldsymbol{Q}=\{q_{u}[t]\}$ over $T$ slots, we minimize the overall HoE $H_{total}$ in (8) under the UAVs’ practical mobility and energy constraints as follows:

The constraint in (9) ensures that the velocity of UAV-$u$ does not exceed its maximal allowable velocity $V_{u}^{max}$. The constraint in (10) gives each UAV’s binary WET decision. The constraint in (11) ensures that each UAV’s remained energy at the end of slot $T$ is no less than the minimum required energy $B_{u}^{min}$ for a safe return after the WET task. The constraint in (12) guarantees a safe distance between any two UAVs in each slot to avoid collisions. The constraint in (13) confines each UAV’s horizontal moving space within an area of length $L_{max}$ and width $W_{max}$.

Problem (P1) is a mixed-integer programming problem. It is also noticed that with the goal to minimize the overall HoE, the multiple UAVs need to be efficiently organized, by either jointly transmitting energy to the same set of IoT devices that are closely located, or separately serving different sets of IoT devices that are distantly located. Hence, all the UAVs’ trajectories and WET decisions are naturally coupled with each other over time. Moreover, as constrained by (11), each UAV must use the limited battery energy wisely for reducing the IoT devices’ HoE. Therefore, problem (P1) is generally difficult to solve efficiently by using the traditional optimization methods.

According to problem (P1), by letting each UAV act as an agent, we model the MDP for each of the $U$ agents[17]. For each agent, define the MDP as a set of states $\mathbb{S}$, a set of actions $\mathbb{A}$, and a set of rewards $\mathbb{R}$. The state set $\mathbb{S}$ embraces all the possible environment configurations at each UAV, including the UAV’s own location, the HoE of all the IoT devices, the battery levels of all the IoT devices ¹¹1It is assumed that all the IoT devices share their HoE and battery levels with the UAVs via a common channel., and the UAV’s own battery level. The action set $\mathbb{A}$ provides the action space of each UAV’s decision on its trajectory and WET. For any given state $s_{u}[t]\in\mathbb{S}$ for UAV-$u$ at the beginning of slot $t$, UAV-$u$ applies the policy $\pi_{u}:s_{u}[t]\to a_{u}[t]$ to select the action $a_{u}[t]\in\mathbb{A}$, and then gets the corresponding reward $r_{u}[t]\in\mathbb{R}$ at the end of slot $t$.

Specifically, the state in slot $t$ is defined as $s_{u}[t]=\left\{\!x_{u}[t]\!,\!y_{u}[t]\!,\!H_{1}[t]\!,...,\!H_{I}[t]\!,\!B_{1}[t]\!,...,\!B_{I}[t]\!,\!B_{u}[t]\!\right\}$, which contains in total of $M=2I+3$ elements. Denoting $\varphi_{u}[t]$ as UAV-$u$’s horizontal rotation angle in slot $t$, UAV-$u$’s horizontal location $\left(x_{u}[t],y_{u}[t]\right)$ in problem (P1) is determined if $\varphi_{u}[t]$ and $V_{u}[t]$ are obtained. Thus UAV-$u$’s MDP action is defined as $a_{u}[t]=\left\{V_{u}[t],\varphi_{u}[t],C_{u}[t]\right\}$, where $C_{u}[t]=0$ if the policy network output is negative or $C_{u}[t]=1$, otherwise. The reward function is proposed as

where $r_{u,0}[t],r_{u,1}[t]$ are the reward and penalty that UAV-$u$ receives in slot $t$, respectively, and $\xi_{0},\xi_{1}\in(0,1)$ are the corresponding weights. Specifically, letting $w_{i}^{u}\triangleq\frac{\mathcal{F}(P_{u}C_{u}[t]G_{i}^{u}[t])\vartheta}{E_{i}^{har}[t]}$ denote the ratio of the DC energy that device-$i$ harvestes from UAV-$u$ to that from all the UAVs, we use $N_{u}[t]=\frac{w_{u}[t]}{\sum_{u\in\mathbb{U}}w_{u}[t]}$ with $w_{u}[t]=\sum_{i\in\mathbb{I}}w_{i}^{u}[t]$ to represent UAV-$u$’s effective WET weight among all the UAVs in slot $t$. The IoT devices can harvest higher amounts of energy from the UAV with a higher $N_{u}[t]$, and vice versa. We then propose to use the following $r_{u,0}[t]$:

From (15), while the UAVs prefer to perform WET more frequently to reduce the IoT devices’ HoE, they also need to use their battery energy carefully to assure the constraint in (11). Hence, the reward in (15) contains two items. The first item is UAV-$u$’s reward for charging IoT devices, where the numerator is the product of UAV-$u$’s weight $N_{u}[t]$ and all the IoT devices harvested energy biased by their HoE, and the denominator is the product of the HoE summation over all the energy-unsatisfied IoT devices and the set size $|\mathbb{I}_{l}[t]|$, which plus $1$ to prevent the denominator from being $0$. It is easy to find that, the more energy the high-HoE IoT devices can harvest, the higher value the first item achieves. The second item is UAV-$u$’s battery energy gap between $B_{u}[t+1]$ and $B_{u}^{min}$ in constraint (11) after taking action $a_{u}[t]$ in slot $t$, where a balance parameter $\xi_{2}$ is multiplied. The penalty in (14) is designed as

where $PEN_{u}^{0}=1$ (or $PEN_{u}^{1}=1$) if the constraint in (12) (or (13)) is not satisfied, or $PEN_{u}^{0}=0$ (or $PEN_{u}^{1}=0$), otherwise.

By receiving each UAV’s state information, the central controller obtains the global information. Let $\boldsymbol{o}[t]=\{s_{1}[t],...,s_{U}[t]\}$, $\boldsymbol{a}[t]=\{a_{1}[t],...,a_{U}[t]\}$ and $\boldsymbol{r}[t]=\{r_{1}[t],...,r_{U}[t]\}$ denote the global observations, actions, and rewards, respectively. To explore the potential connections among the UAVs to improve the overall WET performance as well as to avoid collisions, the central controller uses a graph to represent all the UAVs, by treating each UAV as a node in the graph. According to [14], we use the following similarity matrix among the UAVs to represent the strength of their connections,

where the element $z_{uu^{{}^{\prime}}}[t]=\exp\left(-\frac{\left\|q_{u}[t]-q_{u^{{}^{\prime}}}[t]\right\|^{2}}{2\varrho^{2}}\right)$, $\forall u,u^{{}^{\prime}}\in\mathbb{U},u\neq u^{{}^{\prime}}$, is the Gaussian distance between UAV-$u$ and UAV-$u^{{}^{\prime}}$, $\varrho^{2}$ is a constant, and the element $z_{uu}[t]=\sum_{u^{{}^{\prime}}\in\mathbb{U},u^{{}^{\prime}}\neq u}z_{uu^{{}^{\prime}}}[t]$ on the diagonal is used as the degree of UAV-$u$. Specifically, from [15], to obtain the global feature matrix $\tilde{\boldsymbol{o}}$, the central controller first generates the attention matrix $\boldsymbol{W}_{att}$ and value matrix $\boldsymbol{W}_{v}^{{}^{\prime}}$ as follows:

where the symbol $\times$ denotes the matrix multiplication, $(\cdot)^{T}$ is the matrix transposition. For any matrix $\boldsymbol{Z}\in\mathbf{R}^{U\times U}$, the $softmax(\cdot)$ function transforms the element $z_{uu^{{}^{\prime}}}$ into $\frac{\exp\left(z_{uu^{{}^{\prime}}}\right)}{\sum_{u^{{}^{\prime}}\in\mathbb{U}}\exp\left(z_{uu^{{}^{\prime}}}\right)}$, $\forall u,u^{{}^{\prime}}\in\mathbb{U}$. Then, the global feature matrix $\tilde{\boldsymbol{o}}$ is obtained as $\tilde{\boldsymbol{o}}=\boldsymbol{W}_{att}\times\boldsymbol{W}_{v}^{{}^{\prime}}+\boldsymbol{W}_{v}^{{}^{\prime}}$, which is used as the new observation matrix for the central controller.

As shown in Fig. 2, the MAGRL framework includes two parts, where one is the local training at each of the UAV, and the other is the global training at the central controller. In training stage, a tuple $\left(\boldsymbol{o}[t],\boldsymbol{a}[t],\boldsymbol{r}[t],\boldsymbol{o}[t\!+\!1],\boldsymbol{Z}[t],\boldsymbol{Z}[t\!+\!1]\right)$ is stored in the experience replay buffer $\mathcal{D}$, and all the neural networks for both local and global training apply the stochastic gradient descent (SGD) algorithm to update their parameters.

Local Training: For each UAV’s local training, we apply the SAC algorithm proposed in [18] to enhance the exploration of the environment for all agents. As shown in the left side of Fig. 2, for any UAV-$u$, there are five neural networks employed for its local training, which are the policy network $Actor_{L}$, the local Q-networks $Q_{L0}~{}Critic$ and $Q_{L1}~{}Critic$, and the local V-networks $V_{L0}~{}Critic$ and $V_{L1}~{}Critic$, with the corresponding network parameters denoted by $\theta^{\pi_{u}}$, $\eta_{0,u}$, $\eta_{1,u}$, $\phi_{0,u}$ and $\phi_{1,u}$, respectively. The policy network is trained for the policy function $\pi_{u}(\cdot)$ that maps UAV-$u$’s state $s_{u}[t]$ to its action $a_{u}[t]$, the two local Q-networks are trained for the local state action functions $Q_{L0}(\cdot)$ and $Q_{L1}(\cdot)$, and the two local V-networks are trained for the local state functions $V_{L0}(\cdot)$ and $V_{L1}(\cdot)$. The information entropy $\mathcal{H}(\cdot)$ is used to enhance the agent’s exploration of the environment [18]. The goal of the local training is to obtain the optimal policy $\pi_{u}^{*}=\arg\max_{\pi_{u}}\sum_{t\in\mathbb{T}}\mathbb{E}\left[r_{u}[t]+\alpha_{u}\mathcal{H}\left(\pi_{u}(\cdot|s_{u}[t])\right)\right]$, where the temperature coefficient $\alpha_{u}$ is the weight of the information entropy, and $\mathbb{E}[\cdot]$ is the expectation operation over all the possible actions. The performance of UAV-$u$ when taking action $a_{u}[t]$ in state $s_{u}[t]$ is evaluated by the local Q-networks $Q_{L0}~{}Critic$ and $Q_{L1}~{}Critic$, with $Q_{Lj}(s_{u}[t],a_{u}[t])=r_{u}[t]+\gamma\mathbb{E}[V_{L1}(s_{u}[t\!+\!1])]$, $j\in\{0,1\}$. The performance of UAV-$u$ in state $s_{u}[t]$ is evaluated by the local V-network $V_{L1}~{}Critic$, with $V_{L1}(s_{u}[t])=\mathbb{E}_{a_{u}^{{}^{\prime}}\sim\pi_{u}}[Q_{min}(s_{u}[t],a_{u}[t])-\alpha_{u}\log(\pi_{u}(a_{u}^{{}^{\prime}}|s_{u}[t]))]$, where $a_{u}^{{}^{\prime}}\sim\pi_{u}$ denotes the action taken from policy $\pi_{u}$, and $Q_{min}(\cdot)=\min(Q_{L0}(\cdot),Q_{L1}(\cdot))$.

The parameters of all the five neural networks are updated based on the corresponding loss functions. Specifically, after receiving the $k$-th local experience $(s_{u,k},a_{u,k},r_{u,k},s_{u,k}^{{}^{\prime}})$ from the mini-batch $\mathcal{D}_{k}$ of central controller for the local training, UAV-$u$ uses the loss function

to update $\phi_{0,u}$ for the local V-network $V_{L0}~{}Critic$ in negative the direction of the gradient $\widehat{\nabla}_{\phi_{0,u}}J_{V_{L0}}(\phi_{0,u})$, where $Q_{exp}\!\triangleq\!\mathbb{E}_{a_{u}^{{}^{\prime}}\sim\!\pi_{u}}\left[\!Q_{min}(s_{u,k},a_{u}^{{}^{\prime}};\!\eta_{j,u})\!-\!\alpha_{u}\mathcal{H}_{k}\!\right]$ is the expected entropy-added local minimum Q-value and $\mathcal{H}_{k}\!\triangleq\!\log\left(\!\pi_{u}(a_{u}^{{}^{\prime}}|s_{u,k};\!\theta^{\pi_{u}})\!\right)$ is the entropy. For the parameter $\phi_{1,u}$ of $V_{L1}~{}Critic$ network, we perform a soft update via $\phi_{1,u}\leftarrow\tau\phi_{1,u}+(1-\tau)\phi_{0,u}$, $\tau\in[0,1)$. For the update of the parameters $\eta_{0,u}$ and $\eta_{1,u}$ for $Q_{L0}~{}Critic$ and $Q_{L1}~{}Critic$ networks, respectively, they are also computed in the negative direction of the corresponding loss function’s gradient with

where $j\in\{0,1\}$ and by using $Q_{G}(\cdot)$ to denote the global Q-value used to guide the training of the two local Q-networks, we define $y_{Lj}\!\triangleq\!\epsilon\!\left(\!r_{u,k}\!+\!\gamma\mathbb{E}\!\left[\!V_{L1}(s_{u,k}^{{}^{\prime}};\!\phi_{1,u})\!\right]\!\right)\!+\!(1\!-\epsilon)\mathbb{E}\left[Q_{G}(\cdot)\right]$ with $\epsilon\in(0,1]$. Similarly, according to the loss function for the policy network

the network parameter $\theta^{\pi_{u}}$ is updated in the negative direction of the gradient $\widehat{\nabla}_{\theta^{\pi_{u}}}J_{\pi_{u}}(\theta^{\pi_{u}})$, where the information entropy $\mathcal{H}_{k}^{{}^{\prime}}\triangleq\log(\pi_{u}\left(f_{\theta^{\pi_{u}}}(s_{u,k};\varepsilon)|s_{u,k}\right))$ is calculated from the noise-added action $f_{\theta^{\pi_{u}}}(s_{u,k};\varepsilon)$, and $\varepsilon$ is the noise sampled from a fixed distribution $\mathcal{N}$. Based on [18], adding noise to the action prevents the network from overfitting and ensures the stable network training. We use the loss function

to update the temperature coefficient $\alpha_{u}$ in the negative direction of the gradient $\widehat{\nabla}_{\alpha_{u}}J_{\alpha_{u}}(\alpha_{u})$ with $\tilde{\mathcal{H}}\triangleq|s_{u}[t]|$.

Global Training: The global training is designed for the central controller which consists of three neural networks, which are the global Q-network $Q_{G}~{}Critic$ and the two global V-networks $V_{G0}~{}Critic$ and $V_{G1}~{}Critic$, with the corresponding network parameters denoted as $\eta_{G}$, $\phi_{G0}$ and $\phi_{G1}$, respectively. The global Q-network is trained for the global state action function $Q_{G}(\cdot)$ and the two global V-networks are trained for the global state functions $V_{G0}(\cdot)$ and $V_{G1}(\cdot)$. As shown in the right side of Fig. 2, each neural network contains the self-attention block, which extracts the global feature matrix $\tilde{\boldsymbol{o}}$ to obtain the connections among UAVs as specified in Section III-B. The goal of the central training is to find the optimal global Q-value $Q_{G}^{*}(\boldsymbol{o}[t],\boldsymbol{a}[t])=\boldsymbol{r}+\gamma\mathbb{E}[V_{G1}(\boldsymbol{s}[t])]$.

The parameters of the three networks are also updated based on the corresponding loss functions. Specifically, after receiving the $k$-th global experience ($\boldsymbol{o}_{k},\boldsymbol{a}_{k},\boldsymbol{r}_{k},\boldsymbol{o}^{{}^{\prime}}_{k},\boldsymbol{Z}_{k},\boldsymbol{Z}_{k}^{{}^{\prime}}$) from the mini-batch $\mathcal{D}_{k}$, the central controller uses the loss function

to update $\phi_{G0}$ for the global V-network $V_{G0}~{}Critic$ in the negative direction of the gradient $\widehat{\nabla}_{\phi_{G0}}J_{V_{G0}}(\phi_{G0})$. For the parameter $\phi_{G1}$ of $V_{G1~{}Critic}$ network, we perform a soft update via $\phi_{G1}\leftarrow\tau\phi_{G1}+(1-\tau)\phi_{G0}$, $\tau\in[0,1)$. For the update of the parameter $\eta_{G}$ for $Q_{G}~{}Critic$ network, it is also computed in the negative direction of the corresponding loss function’s gradient, where the loss function is given as

Based on the above framework, we propose the MAGRL-based algorithm to solve problem (P1). The MAGRL-based algorithm is specified in Algorithm 1.