Constrained Deep Reinforcement Learning for Energy Sustainable Multi-UAV based Random Access IoT Networks with NOMA

While internet connectivity plays an increasing role in people’s everyday life in densely populated areas, some rural areas and nature fields such as farms, deserts, oceans, and polar regions, typically lack expansive internet coverage. This is because network providers tend to deploy telecommunication infrastructure in areas where providing wireless service is economically profitable. Nevertheless, farmers, environmental agencies, research organizations, defense agencies, and utility companies among many others, have increasing demands for internet connectivity in such under-served areas, to support massive Internet of Things (IoT) based applications ranging from tracking animal health, agricultural growth, and marine life, to surveillance sensors for defense applications and nuclear waste site management, just to name a few. Provisioning wireless internet access for a massive number of IoT devices in under-served areas at cost effective rates is undoubtedly of great interest for governments, businesses, and end customers.

To support emerging IoT based services, new coverage and distributed channel access solutions should be conceived. Unmanned Aerial Vehicle (UAV) based wireless relays have been proposed to facilitate fast and flexible deployment of communication infrastructure due to their high mobility [1, 2, 3, 4, 5, 6, 7, 8, 9]. UAVs equipped with wireless transceivers can fly to a designated area and provision affordable wireless internet connectivity to a massive number of IoT devices, by relaying data to network servers through satellite back-haul links. Moreover, UAVs can dynamically adjust their location in real-time to counter environmental changes and improve system performance.

With the number of IoT devices projected to reach $3.3$ billion by $2021$ [10], distributed Medium Access Control (MAC) protocols based wireless technologies such as Wi-Fi, Zigbee, and Aloha-based LoRaWAN, are expected to play an important role in provisioning massive IoT access over the unlicensed band in the fifth-generation ($5$G) wireless network era [11, 12, 13]. Non Orthogonal Multiple Access (NOMA), which can improve the spectral efficiency by exploiting Successive Interference Cancellation (SIC) to enable non-orthogonal data transmissions, is yet another promising solution to enable massive machine type communication (mMTC) in $5$G networks and beyond. Recent works propose to apply power-domain NOMA in slotted-Aloha systems to support mMTC of IoT devices [14, 15, 16]. NOMA enabled Aloha-type wireless networks is therefore of significant importance to support massive channel access in UAV-based IoT Networks.

In this work, we consider solar-powered multi-UAV based wireless IoT networks, where UAVs act as wireless Base Stations (BS) for a massive number of IoT devices. IoT devices contend for access to the shared wireless channel using an adaptive $p$-persistent slotted Aloha MAC protocol to send data to the UAVs, which relay the received data to the internet backbone through wireless satellite back-haul links. UAVs on the other hand, are equipped with solar cells to replenish the on-board battery, and exploit power-domain SIC to decode multiple users’ transmissions, thus improving the transmission efficiency. To enable an energy-sustainable and capacity-optimal massive IoT network, we jointly study dynamic multi-UAV altitude control and NOMA-based multi-cell wireless channel access of IoT devices. The objective of the stochastic control problem is to maximize the total network throughput of a massive number of IoT devices, characterized by random uplink channel access and varying wireless channel conditions, and coupled with multiple constraints to ensure energy sustainability of solar-powered UAVs. To the best of our knowledge, our work is the first work to study energy sustainability of a multi-UAV based wireless communication system in support of a massive number of IoT devices with NOMA and random channel access.

The main contributions of our work can be summarized as follows. First, we formulate the joint problem of multi-UAV altitude control and adaptive random channel access of massive IoT devices to attain the maximum capacity under energy sustainability constraints of UAVs over a prespecified operating horizon as a Constrained Markov Decision Process (CMDP). Second, to learn an optimal control policy for the wireless communication system, we design an online model-free Constrained Deep Reinforcement Learning (CDRL) algorithm based on Lagrangian primal-dual policy optimization to solve the CMDP. A cooperative policy is learned among UAVs which ensures energy sustainability of UAVs over an operating horizon, while maximizing the total network capacity with NOMA under the probabilistic mutual interference of IoT devices. Third, we compare the performance of our proposed algorithm to two baseline solutions: 1) unconstrained Deep RL (DRL) approach without energy sustainability constraints, and 2) DRL approach with reward shaping to account for energy costs. Our extensive simulations demonstrate that our proposed algorithm yields feasible policies with higher network capacity, and outperforms baseline solutions which do not guarantee feasible policies. Specifically, the policy learned by our proposed CDRL algorithm achieves a temporal average network capacity that is $82.4\%$ higher than that of a feasible DRL with reward shaping, and $6.47\%$ lower than that of the energy-constraint-free system. Last but not least, we demonstrate that the learned policy, which has been efficiently trained on a small network size, can effectively manage networks with a massive number of IoT devices and varying initial network states.

The remainder of this paper is organized as follows. A literature survey of related research work and a background of unconstrained and constrained MDPs is given in Section II. The system model is described in Section III. The problem formulation and the proposed CDRL algorithm is presented in Section IV, followed by the simulation setup and performance evaluation results in Section V. Finally our concluding remarks and future work are given in Section VI.

Consider a multi-UAV based IoT network consisting of $M$ UAVs and $N$ IoT devices, where the UAVs collect data from a massive deployment of IoT devices, as shown in Figure 1(a). Let $\mathcal{M}=\{1,\cdots,M\}$ be the set of UAVs, and $\mathcal{N}=\{1,\cdots,N\}$ be the set of IoT devices. UAVs are connected via wireless back-haul links to a central controller, which controls the altitude of each UAV and manages the access parameters of wireless IoT devices. IoT devices are independently and uniformly distributed (i.u.d.) across a deployment area $\mathbb{A}$. Let the locations of IoT devices be $\{\hat{x}^{i},\hat{y}^{i}\}_{i=1}^{N}$. Each IoT device is served by the closest UAV. Denote the subset of IoT devices which are associated with UAV $m$ by $\mathcal{N}_{m}\subset\mathcal{\mathcal{N}}$, $|\mathcal{N}_{m}|\leq N$, $\bigcup_{m=1}^{M}\mathcal{N}_{m}=\mathcal{N}$, $\mathcal{N}_{i}\cap\mathcal{N}_{j}=\phi,\forall i\neq j\in\mathcal{M}$. Time is slotted into fixed-length discrete time units indexed by $n$. For instance, the $n$-th time slot is $[t_{n},t_{n+1})$, where $t_{n+1}-t_{n}=\Delta t,~{}\forall n$. Each time slot $n$ is further divided into $L$ communication sub-slots of length $\frac{\Delta t}{L}$ each, as shown in 1(b). Denote the $l$-th communication sub-slot in slot $n$ by $t_{n}^{l}$, $l=\{0,\cdots,L-1\}$. During these communication sub-slots, IoT devices contend for channel access based on an adaptive $p$-persistent slotted Aloha MAC protocol. In this protocol, an IoT device waits until the beginning of a communication sub-slot before attempting to access the channel with probability $p$, which is adapted every time slot by the central controller based on network dynamics. IoT devices transmit uplink data to their associated UAV with a fixed transmission power of $P_{TX}$ watts, and are traffic-saturated, i.e., there is always a data packet ready for transmission.

Denote the location of UAV $m$ during time slot $n$ by $r_{m}(t_{n})=(x^{m},y^{m},z^{m}_{n})$. In our system model, $(x^{m},y^{m}),\forall m\in\mathcal{M}$, is first determined based on the IoT device distribution on the ground using the popular Lloyd’s K-means clustering algorithm with runtime $\mathcal{O}(2NM)$ [48]¹¹1Lloyd’s K-means clustering algorithm is an iterative algorithm to determine a set of $K$ centroids given a large set of IoT device locations $\{\hat{x}^{i},\hat{y}^{i}\}_{i=1}^{N}$, so as to minimize the within-cluster variance (sum of squared distances to cluster centroid), $\text{min}_{\{(x^{m},y^{m})\}}\sum_{m=1}^{M}\sum_{i\in\mathcal{N}_{m}}||(\hat{x}^{i},\hat{y}^{i})-(x^{m},y^{m})||^{2}$. The impacts of adopting other schemes for determining $(x^{m},y^{m}),\forall m\in\mathcal{M}$ on network performance will also be investigated.. Let $u^{i,m}_{n}=\{0,1\}$ indicate whether IoT device $i$ associates with UAV $m$ during time slot $n$. If UAV $m$ located at $r_{m}(t_{n})$ during the $n$-th time slot is the closest to IoT device $i$, $u^{i,m}_{n}=1$, and $u^{i,l}_{n}=0,\forall l\neq m$. The power of the signal transmitted by a wireless IoT device $i$ to a UAV $m$ is subject to independent Rayleigh channel fading, $h_{i,m}(t_{n})$²²2The statistical channel state information, $h_{i,m}(t_{n})$, is assumed to be quasi-static and is fixed during a time slot $n$., and a distance-dependent free-space path-loss $d_{i,m}^{-\alpha}(t_{n})$, where $\alpha$ is the path-loss exponent, and $d_{i,m}^{-\alpha}(t_{n})$ is the propagation distance between IoT device $i$ and UAV $m$, $d_{i,m}(t_{n})=\sqrt{(x^{m}-\hat{x}^{i})^{2}+(y^{m}-\hat{y}^{i})^{2}+(z^{m}_{n})^{2}}$. The received power at UAV $m$ from IoT device $i$ in a communication sub-slot $t_{n}^{l}$ is,

where $c_{0}=\frac{\lambda^{\alpha}}{(4\pi)^{\alpha}}$ is a constant which depends on the wavelength of the transmitted signal, $\lambda$, and $d_{0}$ is a reference distance. $\mathcal{\hat{I}}_{i}(t_{n}^{l})$ is a Bernoulli random variable with parameter $p(t_{n})\in(0,1]$ which indicates whether IoT device $i$ transmits during communication sub-slot $t_{n}^{l}$, i.e., $\mathcal{\hat{I}}_{i}(t_{n}^{l})=1$ with probability $p(t_{n})$. The UAV $m$ first decodes the signal with the highest signal power under the interference from all other IoT devices involved in the NOMA transmissions. Without loss of generality, IoT devices $\mathcal{I}_{m}(t_{n}^{l})=\{i|\mathcal{\hat{I}}_{i}(t_{n}^{l})=1\}$ are sorted in the descending order of their received signal strength at UAV $m$, such that $i=1$ is the IoT device with the highest received signal to interference plus noise (SNIR) at UAV $m$, and $i=2$ is the IoT device with the second highest received SNIR at UAV $m$ ³³3We consider the two highest received signals to trade-off NOMA gain and SIC decoding complexity for uplink transmissions.. The highest received SNIR at UAV $m$ in a communication sub-slot $t_{n}^{l}$ is therefore,

where $n_{0}$ is the noise floor power. Similarly, the second highest received SNIR at UAV $m$ in a communication sub-slot $t_{n}^{l}$ is,

UAV $m$ can decode the signal with $\text{SNIR}_{1,m}(t_{n}^{l})$ if

1. 1.

   user 1 is associated with UAV $m$ during communication sub-slot $n$, $u_{n}^{1,m}=1$, and,

2. 2.

   $\text{SNIR}_{1,m}(t_{n}^{l})$ is larger than the SNIR threshold, i.e., $U(\text{SNIR}_{1,m}(t_{n}^{l}))=\text{SNIR}_{1,m}(t_{n}^{l})$,

where $U(.)$ is a thresholding function to maintain a minimum quality of service,

In addition, UAV $m$ can decode the signal with $\text{SNIR}_{2,m}(t_{n}^{l})$ if

1. 1.

   $\text{SNIR}_{1,m}(t_{n}^{l})$ is successfully decoded,

2. 2.

   user 2 is associated with UAV $m$ during communication sub-slot $n$, $u_{n}^{2,m}=1$, and,

3. 3.

   $\text{SNIR}_{2,m}(t_{n}^{l})$ is larger than the SNIR threshold, i.e., $U(\text{SNIR}_{2,m}(t_{n}^{l}))=\text{SNIR}_{2,m}(t_{n}^{l})$

The sum rates of the received data at UAV $m$ in communication sub-slot $t_{n}^{l}$ is,

where $\mathcal{W}$ is the transmission bandwidth, and $e_{n}^{1,m}=1$ if $U(\text{SNIR}_{1,m}(t_{n}^{l}))=\text{SNIR}_{1,m}(t_{n}^{l})$ and $0$ otherwise. The total network capacity in any given system slot $t_{n}$,

UAVs are equipped with solar panels, which harvest solar energy to replenish the on-board battery. The attenuation of solar light passing through a cloud can be modeled based on [23],

where $\beta_{c}\geq 0$ denotes the absorption coefficient of the cloud, and $d^{cloud}$ is the distance that the solar light travels through the cloud. Following [23] and the references therein, the solar energy harvested by UAV $m$ during time slot $n$ can be modeled as,

where $\psi$ is a constant representing the energy harvesting efficiency, $\tilde{S}$ is the area of solar panels, and $\tilde{G}$ denotes the average solar radiation intensity on earth. $z_{high}$ and $z_{low}$ are the altitudes of upper and lower boundaries of the cloud. During time-slot $n$, UAV $m$ can cruise upwards or downwards from $r_{m}(t_{n})$ to $r_{m}(t_{n+1})$. The energy consumed by UAV $m$ during time slot $n$ [23] is,

where, $V_{z}=\sqrt{\frac{W}{2\rho A}}$, $W$ is the weight of the UAV, $\rho$ is air density, and $A$ is the total area of UAV rotor disks. $P_{\text{static}}$ is the power consumed for maintaining the operation of UAV, and $P_{\text{antenna}}$ is the power consumed by the receiving antenna. It is worth to mention that cruising upwards consumes more power than cruising downward and hovering.

Denote the battery energy storage of UAV $m$ at the beginning of slot $n$ by $B_{m}(t_{n})$. The battery energy in the next slot is given by,

where $\mathbb{B}(t_{n}),\forall n$, are independent zero-mean gaussian random variables with variance $\sigma^{2}_{B}$ which characterizes the randomness in the battery evolution process, and $\left[~{}\right]^{+}$ denotes the positive part.

In this work, we investigate discrete-time UAV altitude control and random channel access management for a multi-UAV based IoT network with NOMA. In order to maximize the total network capacity under stochastic mutual interference of IoT devices while ensuring the energy sustainability of UAVs over the operating horizon $H$, the central controller decides on the altitude of each UAV $m,~{}\forall m\in\mathcal{M}$, at the beginning of each slot $n$, $z_{n}^{m}$, as well as the channel access probability $p(t_{n})$ of IoT devices considering the potential access gain provisioned by NOMA. The channel access probability will be broadcast to IoT devices through beacons at the beginning of each time slot, and IoT devices then adapt their random channel access parameter accordingly.

The problem of maximizing the total network capacity while ensuring energy sustainability of each UAV is a constrained stochastic optimization problem over the operating horizon due to the stochastic channel model and random channel access in the multi-cell IoT network. Offline solutions are generally impractical because it is hard to mathematically track probabilistic mutual interference caused by the uplink transmissions of IoT devices with random access, stochastic channel conditions, and SIC decoding at the UAVs. Hence, we first formulate this problem as a CMDP, and design an online Constrained Deep Reinforcement Learning (CDRL) algorithm, to find an energy sustainable capacity-optimal policy to control the altitude of each UAV and the channel access probability of IoT devices.

We have developed a simulator in Python for the solar-powered multi-UAV based Wireless IoT network with NOMA described in section III, and implemented the proposed CDRL algorithm based on OpenAI’s implementation of PPO  [51]. We trained the proposed CDRL agent in a multi-cell wireless IoT network of $M=2$ solar-powered UAVs and $N=200$ IoT devices. IoT devices were deployed independently and uniformly within a grid of $[0,0]\times[1500,500]$m. The two UAVs were initially deployed at $(250,250,750)$m and $(750,250,1250)$m with $50\%$ initial battery energy, i.e., $111$ Wh. Although we have investigated the impacts of different initial UAV deployments on network performance, note that the $x$ and $y$ coordinates of the two UAVs, i.e., $(250,250)$m and $(750,250)$m, are minimizers of the sum of squared planar distances between IoT devices and the closest UAV, as determined by Lloyd’s K-means clustering algorithm for the uniform random deployment of IoT devices on the ground. UAVs were allowed to cruise vertically between $500$m and $1500$m. As a baseline for comparison, we have compared the performance of the proposed CDRL algorithm with unconstrained PPO agent without energy sustainability constraints, and unconstrained PPO agents that accounts for energy costs via fixed reward shaping (RLWS), where the reward signal was $\mathcal{\hat{R}}(s_{n},a_{n},\bm{\mathbf{\eta}})=\mathcal{R}(s_{n},a_{n})-\eta_{1}\mathcal{C}_{1}(s_{n},a_{n})-\eta_{2}\mathcal{C}_{2}(s_{n},a_{n})$, and $\eta_{1}=10,\eta_{2}=10$ or $\eta_{1}=0,\eta_{2}=10$. The main simulation parameters for the experiments are outlined in Table I.

The output energy of the solar panels as a function of the altitude according to (14) is shown in Figure 3. It can be seen that above the cloud cover at $1300$m, the output solar energy reaches the highest, and it attenuates exponentially through the cloud until $700$m is reached. Below $700$m, the output solar energy is zero. UAVs cruising above $1300$m harvest the most solar energy, while UAVs cruising below $700$m drain their battery energy the fastest. Note that hovering at low altitudes reduces distance-dependant path-loss and improves the wireless communication rates, however, it is not energy sustainable for UAVs.

The learning curves of the trained CDRL agent are shown in Figure 4. It can be seen in Figure 4(a) that the CDRL agent becomes more experienced as training progresses, collecting higher expected total rewards. In addition, the CDRL agent becomes more experienced in satisfying energy constraints, learning policies whose expected costs fall bellow $-B_{min}/B_{max}=-0.1$, which means that the learned policy results in energy increase in the battery of each UAV by at least $B_{min}$ at the end of the flight horizon. On the other hand, the convergence of the Lagrangian multipliers to non-negative values during the training of the proposed CDRL algorithm is shown in Figure 4(b). It can be observed that the two cost constraints are penalized differently, which is primarily due to the different initial conditions and different terminal states. The Lagrangian multiplier of the energy constraint corresponding to UAV 1 is larger than that of UAV 2, which means that the energy constraint of UAV 1 is satisfied farther from the constraint boundary compared to that of UAV 2. Based on this observation, it is therefore expected that UAV 1 will end up its flight with more harvested energy in its battery compared to UAV 2.

The learned policy by our proposed CDRL algorithm is shown in Figure 5. It can be seen from Figures 5(a) and 5(b) that the CDRL agent learns a policy in which the two UAVs take turns in cruising upwards to recharge their on-board batteries, and in serving IoT devices deployed on the ground. UAV $2$ first climbs up to recharge its battery, while UAV $1$ descends down to improve communication performance for IoT devices on the ground. Since IoT devices are associated with the closest UAV, in this case, all IoT devices are associated with UAV $1$. When UAV $2$’s battery is fully charged, it descends down gradually to switch roles with UAV $1$: UAV $2$ becomes the BS with which all IoT devices are associated, while UAV $1$ climbs up to recharge its battery. Such a policy ensures that the battery energy of the two UAVs is not drained throughout the operating horizon as can be seen from Figure 5(c). Note that the terminal energy in UAV’s 1 battery is higher than that of UAV 2, which is expected based on the observation that the Lagrange multiplier corresponding to UAV’s 1 energy constraint is larger than that of UAV 2. In Figure 5(d), the random channel access probability based on the learned CDRL policy is shown. It can be observed that when either of the two UAVs is fully serving all the $200$ IoT devices, the wireless networking system is overloaded with $p>1/N$, thanks to NOMA. Note that $p=1/N$ is the optimal transmission probability in single-cell $p$-persistent slotted Aloha systems without NOMA. The channel access probability is dynamically adapted when the two UAVs cruise upward and downward to exchange roles in the wireless system. At times when both UAVs have associated users, the channel access probability can be seen to spike higher to maintain NOMA overload, as can be observed from Figures 6(a)-(b). It can be seen from these two figures that NOMA’s gain is higher in steady states when all IoT devices are associated with the same UAV, compared to transient states when the two UAVs exchange roles and are both serving IoT devices. This is because it is less likely that the second highest received SINR to a UAV is from within the same cell at times when both UAVs provision wireless service. By deploying multiple UAVs, it is therefore possible to learn a cooperative policy in which UAVs take turns to charge their battery and provision uninterrupted wireless service as Figure 6(b) shows.

The performance comparison of our proposed CDRL algorithm with two other baseline schemes in terms of the achieved mean total network capacity with $95\%$ confidence interval versus flight time is shown in Figure 7(a). The statistical results are based on $32$ roll-outs of the learned policy in the simulated wireless IoT environment with NOMA. It can be seen that the proposed CDRL agent learns a policy whose achieved temporal average network capacity is $82.4\%$ higher compared to the policy learned by the conservative RLWS agent with $\eta_{1}=\eta_{2}=10$. Compared to the RL agent without energy constraints, only $6.47\%$ of the achievable temporal average system capacity is sacrificed in order to maintain energy sustainability of UAVs. In Figure 7(b), the geometric mean of the battery energy of the two UAVs, defined as $\mathcal{G}_{m}[B_{m}(t_{n})]=\big{(}\prod_{m=1}^{M}B_{m}(t_{n})\big{)}^{\frac{1}{M}}$, is plotted versus flight time. Note that the geometric mean is chosen as a single measure to characterize battery energy of the UAVs. If the battery of any UAV is exhausted, $B_{m}(t_{n})=0$, the geometric mean will be $\mathcal{G}_{m}[B_{m}(t_{n})]=0$. It can be seen that the policy learned by the RLWS agent with $\eta_{1}=\eta_{2}=10$ is the most conservative, fully recharging the batteries of both UAVs by the $13$-th minute. On the other hand, the unconstrained agent learns a policy which is indifferent to battery energy, leading to energy depletion of at least one UAV by the $20$-th minute. This is reflected in Figure 8, where the altitude trajectories based on the policies learned by the unconstrained RL agent and the RLWS agents are shown. The policy of the unconstrained agent descends the two UAVs to the lowest allowable altitude of $500$m. This policy is communication-performance bound, achieving the highest network capacity as shown in Figure 7(a). On the other hand, the policy of the RLWS agent with $\eta_{1}=\eta_{2}=10$ attempts to minimize the energy costs and maintain a high battery energy at all times. Hence, the two UAVs hover at the altitude at which solar energy harvesting is the highest. For the RLWS agent with $\eta_{1}=0,\eta_{2}=10$, UAV $2$ ascends to $1500$m where the most solar energy can be harvested to maintain energy sustainability, while UAV $1$ descends down to $500$m to provision wireless service, indifferent to energy sustainability, which leads to its battery depletion by the $19$-th minute as can be seen from Figure 7(b). This is expected as the reward signal for training this agent does not penalize energy costs of UAV $1$. In contrast, the policy learned by the proposed CDRL agent strikes a balance: it ensures energy sustainability while slightly sacrificing the network capacity performance.

Policy Generalizability: To demonstrate the generalizability and robustness of the learned CDRL policy, we test its performance on networks with different initial states and varying number of IoT devices. Note that the CDRL policy has been trained given that $200$ IoT devices are uniformly deployed on ground, and that the two UAVs are initially present at altitudes of $250$m and $750$m. In Figure 9(a), we demonstrate the learned policy performance given different initial altitudes of UAVs. Specifically, we consider two extreme cases: both UAVs are initially deployed at the altitude of $500$m (case A), or at $1500$m (case B). For those two cases, Figure 9(a) shows that the two UAVs tend to fully de-synchronize their vertical flight trajectories, such that when one of them is charging its battery at $1500$m, the other is provisioning wireless service at $500$m. The temporal average system capacity for cases A and B are $2.649*\mathcal{W}$ bit per second (bps) and $2.619*\mathcal{W}$ bps, respectively. Notice that case B achieves a slightly lower temporal average network capacity because both UAVs are initially farther away from IoT devices. In the legend, the temporal geometric mean of the battery energy of each UAV, defined as $\mathcal{G}_{t}[B_{m}]=\big{(}\prod_{n=1}^{H}B_{m}(t_{n})\big{)}^{\frac{1}{H}}$, is reported to demonstrate energy sustainability of UAVs throughout the operating horizon.

In Figure 9(b), we test how the learned CDRL policy scales with varying number of IoT devices. As can be seen from Figure 9(b), the channel access probability is scaled appropriately given the number of IoT devices vary in $\{100,200,600,1000\}$, maintaining comparable temporal average network capacity around $2.67*\mathcal{W}$ bps. In addition, the temporal geometric mean of the battery energy of each UAV is also reported in the legend to demonstrate energy sustainability of UAVs. Last but not least, in Figure 9(c), we test the performance of the learned policy given different horizontal deployment of the two UAVs. We consider three cases: (A) the two UAVs are deployed at $(250,250)$m and $(750,250)$m, as determined by the K-means clustering algorithm, (B) the two UAVs are deployed farthest from each other at $(0,0)$m and $(1000,500)$m, and (C) the two UAVs are randomly deployed on the xy-plane. It can be observed from Figure 9(c) that the mean network capacity is highest when the K-means algorithm is employed to determine the xy-planar deployment of the two UAVs. In addition, it is shown that randomly deploying the two UAVs in the xy-plane, case (C), is slightly lower than that in case (A), whereas the extreme case of deploying the two UAVs on the diagonal, case (B), achieves the lowest network capacity. In all cases, the learned policy still ensures energy sustainability of the two UAVs as indicated by the temporal geometric mean of the battery energy of each UAV, which is reported in the legend.

In this paper, we have applied constrained deep reinforcement learning to study the joint problem of dynamic multi-UAV altitude control and random channel access management of a multi-cell UAV-based wireless network with NOMA, in support of a massive number of IoT devices. To enable an energy-sustainable capacity-optimal IoT network, we have formulated this constrained stochastic control problem as a constrained markov decision process, and proposed an online model-free constrained deep reinforcement learning algorithm to learn an optimal control policy for wireless network management. Our extensive simulations have demonstrated that the proposed algorithm learns a cooperative policy in which the altitude of UAVs and channel access probability of IoT devices are dynamically adapted to maximize the long-term total network capacity while maintaining energy sustainability of UAVs. The policy learned by the proposed algorithm ensures energy sustainable operation of UAVs, and outperforms baseline solutions. In our future work, we will study the design of a constrained multi-agent RL framework to tackle resource management problems in spatially distributed massive wireless networks.

This work was supported in part by the NSF grants ECCS1554576, ECCS-1610874, and CNS-1816908. We gratefully acknowledge the computing resources provided on Bebop, a high-performance computing cluster operated by the Laboratory Computing Resource Center at Argonne National Laboratory.