UAV-Assisted Enhanced Coverage and Capacity in Dynamic MU-mMIMO IoT Systems: A Deep Reinforcement Learning Approach

The current research explores a complex situation in which various users are linked to a gateway via both wired and wireless connections. This configuration is located in a remote region, which is hard to reach directly by BS because of several hindrances like buildings, mountains, etc. Afterward, a UAV is employed as a dual-hop decode-and-forward (DF) relay to connect with the users as illustrated in Fig. 1. Let ($x_{b},y_{b},z_{b}$), ($x_{u},y_{u},z_{u}$), and ($x_{k},y_{k},z_{k}$) denote the location of the BS, UAV relay, and $k^{th}$ IoT user, respectively. We establish the 3D distances for a UAV-assisted mmWave MU-mMIMO IoT system as follows:

where $\tau_{1}$, $\tau_{2,k}$, and $\tau_{k}$ are the 3D distance between UAV & BS, between the UAV and $k^{th}$ IoT user, and between BS and $k^{th}$ IoT user, respectively.

In this system model, we consider BS equipped with $N_{T}$ antennas, UAV relay with $N_{r}$ antennas for receiving and $N_{t}$ antennas for sending information to $K$ single-antenna IoT users scattered in $G$ groups, where $g^{th}$ group has $K_{g}$ IoT users such that $K=\sum_{g=1}^{G}K_{g}$. Both BS and UAV relay utilize HBF architecture. In this setup, the BS includes an RF beamforming stage $\mathbb{F}_{b}\in\mathbb{C}^{N_{T}\times N_{RF_{b}}}$ and BB stage $\mathbb{B}_{b}\in\mathbb{C}^{N_{RF_{b}}\times K}$, where $N_{RF_{b}}$ is the number of RF chains such that $N_{s}\leq N_{RF_{b}}\leq N_{T}$ to guarantee multi-stream transmission. Considering half-duplex (HD) DF relaying, BS sends $K$ data streams $\mathbb{d}=[d_{1},d_{2},\cdots,d_{K}]^{T}$ through channel $\mathbb{H}_{1}\in\mathbb{C}^{N_{r}\times N_{T}}$ in the first time slot. Using $N_{r}$ antennas, UAV receives signals with RF stage $\mathbb{F}_{u,r}\in\mathbb{C}^{N_{RF_{u}}\times N_{r}}$ and BB stage $\mathbb{B}_{u,r}\in\mathbb{C}^{K\times N_{RF_{u}}}$. We assume UAV relay transmits the data in the second time slot using RF beamformer $\mathbb{F}_{u,t}=[\mathbb{f}_{u,t,1}\cdots,\mathbb{f}_{u,t,N_{RF_{u}}}]\in\mathbb{C}^{N_{t}\times N_{RF_{u}}}$ and BB stage $\mathbb{B}_{u,t}=[\mathbb{b}_{u,t,1},\cdots,\mathbb{b}_{u,t,K}]\in\mathbb{C}^{N_{RF_{u}}\times K}$ via channel $\mathbb{H}_{2}\in\mathbb{C}^{K\times N_{t}}$. The HBF design significantly cuts down the number of RF chains, for example, reducing them from $N_{T}$ RF chains to $N_{RF_{b}}$ for BS, and from $N_{t}$($N_{r}$) RF chains to $N_{RF_{u}}$ for UAV, while satisfying the following conditions: 1) $K\leq N_{RF_{b}}\ll N_{T}$; and 2) $K\leq N_{RF_{u}}\ll N_{r}$($N_{t}$). In this case, considering the environment noise follows the distribution $\mathcal{CN}(\mathbb{0},\sigma_{n}^{2})$, the AR for the BS-UAV link can be expressed as follows [21]:

where $\mathbb{Q}_{1}^{-1}=(\sigma_{n}^{2}\mathbb{B}_{u,r}\mathbb{F}_{u,r})^{-1}\mathbb{F}_{u,r}^{H}\mathbb{B}_{u,r}^{H}$, $\bm{\mathcal{H}}_{1}=\mathbb{F}_{u,r}\mathbb{H}_{1}\mathbb{F}_{b}$, and $\mathbb{E}${$\mathbb{dd}^{H}$} $=\mathbb{I}_{K}\in\mathbb{C}^{K\times K}$. In the same manner, the AR between UAV and IoT users is calculated based on the instantaneous signal-to-interference-plus-noise ratio (SINR) as demonstrated by the following expression [21]:

where $g_{k}=k+\sum_{g^{\prime}=1}^{g-1}K_{g^{\prime}}$ represents the IoT user index and $\mathbb{h}_{2,g_{k}}\in\mathbb{C}^{N_{t}}$ is the channel vector between UAV and respective IoT user. Utilizing the instantaneous SINR, the ergodic AR of the second link $R_{2}$, in UAV-assisted mmWave MU-mMIMO systems, can be written as:

We consider the mmWave channel for both links, then using the Saleh-Valenzuela channel model, the channel between BS and UAV can be written as follows [22]:

where $C$ is the total number of groups, $L$ is the total number of paths from BS to UAV, $z_{1_{cl}}\sim\mathcal{CN}(\mathbb{0},\frac{1}{L})$ is the complex gain of $l^{th}$ path in the $c^{th}$ cluster, and $\eta$ is the path loss exponent. In addition, $\mathbb{a}_{1}^{(k)}(.,.)$ represents the respective transmit or receive array steering vector for a uniform rectangular array (URA), which is defined as [7]:

where $k=\left\{r,t\right\}$, $N_{x}$ and $N_{y}$ are the horizontal and vertical size of respective antenna array at BS and UAV, $d$ is the inter-element spacing, $\mathbb{Z}_{1}$ = diag$(z_{1,1}\tau_{1,1}^{-\eta},...,z_{1,L}\tau_{1,L}^{-\eta})\in\mathbb{C}^{L\times L}$ represents the diagonal gain matrix, $\mathbb{A}_{1}^{(r)}\in\mathbb{C}^{N_{r}\times L}$ and $\mathbb{A}_{1}^{(t)}\in\mathbb{C}^{L\times N_{t}}$ are the receive and transmit phase response matrices, respectively. Also, the angles $\theta_{cl}^{(t)}\in[\theta_{c}^{(t)}-\delta_{c}^{\theta(t)},\theta_{c}^{(t)}+\delta_{c}^{\theta(t)}]$ and $\phi_{cl}^{(t)}\in[\phi_{c}^{(t)}-\delta_{c}^{\phi(t)},\phi_{c}^{(t)}+\delta_{c}^{\phi(t)}]$ denote the elevation AoD (EAoD) and azimuth AoD (AAoD) for $l^{th}$ path in channel $\mathbb{H}_{1}$, respectively. $\theta_{c}^{(t)}$ represent the mean EAoD and $\delta_{c}^{\theta(t)}$ is the EAoD spread, while $\phi_{c}^{(t)}$ is mean AAoD with spread $\delta_{c}^{\phi(t)}$. In a similar fashion, the angles $\theta_{cl}^{(r)}$ within the range $[\theta_{c}^{(r)}-\delta_{c}^{\theta(r)},\theta_{c}^{(r)}+\delta_{c}^{\theta(r)}]$ and $\phi_{cl}^{(r)}$ within $[\phi_{c}^{(r)}-\delta_{c}^{\phi(r)},\phi_{c}^{(r)}+\delta_{c}^{\phi(r)}]$ correspond to the elevation AoA (EAoA) and azimuth AoA (AAoA), respectively. Here, $\theta_{c}^{(r)}$ and $\phi_{c}^{(r)}$ represent the mean EAoA and AAoA, with $\delta_{c}^{\theta(r)}$ and $\delta_{c}^{\phi(r)}$ denoting the angular spreads of the elevation and azimuth angles, respectively. The channel vector between UAV and $k^{th}$ IoT user is written as:

where $Q$ is the total number of downlink paths from UAV to users, $z_{2,k_{q}}\sim\mathcal{CN}(0,\frac{1}{Q})$ is the complex path gain of $q^{th}$ path in the second link, and $\mathbb{a}(.,.)\in\mathbb{C}^{N_{t}}$ is the UAV downlink array phase response vector. As given in (7), the obtained downlink channel is comprised of two distinct parts: 1) a fast time-varying path gain vector $z_{2,k}=[z_{2,k_{1}}\tau_{2,k_{1}}^{-\eta},...,z_{2,k_{Q}}\tau_{2,k_{Q}}^{-\eta}]^{T}\in\mathbb{C}^{Q}$; and 2) a slow time-varying downlink array phase response matrix $\mathbb{A}_{2,k}\in\mathbb{C}^{Q\times N_{t}}$ where each row is constituted by $\mathbb{a}(\theta_{kl},\phi_{kl})$. Afterward, the channel matrix for the second link can be expressed as follows:

where $\mathbf{Z}_{2}=[\mathbf{z}_{2,1},\cdots,\mathbf{z}_{2,K}]^{T}\in\mathbb{C}^{K\times Q}$ is the complete path gain matrix for all downlink IoT users.

The RF and BB stages for BS and UAV are designed for the following considerations: 1) maximize the beamforming gain at the desired directions based on the slow time-varying AoD and AoA; 2) reduce the power-hungry RF chains; 3) reduce the CSI overhead; and 4) mitigate multi-user interference (MU-I)¹¹1The details for HBF design for BS and UAV can be found in [21].

DDPG is a sophisticated RL algorithm that combines elements of deep Q-networks (DQN) and policy gradient techniques. Unlike DQN, which is designed for discrete action spaces, DDPG is tailored for continuous action spaces, common in real-world situations. This makes DDPG ideal for complex tasks that demand a spectrum of continuous action values, increasing its effectiveness in diverse and changing environments.

DDPG utilizes actor-critic approach [17], where the actor, denoted as $\mu(\mathbf{s}|\theta^{\mu})$, outputs a deterministic action $\mathbf{a}$ given a state $\mathbf{s}$, where $\theta^{\mu}$ represents the weights of the actor network. In the same manner, critic expressed as $Q(\mathbf{s},\mathbf{a}|\theta^{Q})$, predicts the expected return (value) of taking an action $\mathbf{a}$ in a state $\mathbf{s}$, and it is parameterized by weights $\theta^{Q}$. DDPG uses target networks for both actor and critic, represented as $\mu^{\prime}(\mathbf{s}|\theta^{\mu^{\prime}})$ and $Q^{\prime}(\mathbf{s},\mathbf{a}|\theta^{Q^{\prime}})$ respectively. Here, $\theta^{\mu^{\prime}}$ and $\theta^{Q^{\prime}}$ denote the weights of the target actor and target critic networks. These target networks are essentially slow-paced versions of the primary actor and critic networks, ensuring a more stable and consistent learning process. Moreover, to break the correlation between consecutive experiences ($\mathbf{s}_{t}$, $\mathbf{a}_{t}$, $r_{t}$, $\mathbf{s}_{t+1}$), we use a replay buffer $D$, where $\mathbf{s}_{t}$, $\mathbf{a}_{t}$, $r_{t}$ are the state, action, and reward at timestep $t$, respectively, and $\mathbf{s}_{t+1}$ is the next state. These transitions are stored in the replay buffer, and then at each timestep, the actor and critic are updated by sampling a minibatch of transitions $(\mathbf{s}_{i},\mathbf{a}_{i},r_{i},\mathbf{s}_{i+1})$ uniformly from the buffer. During the training, the weights of the critic are updated by minimizing the mean square error (MSE) loss function as:

where $N$ is the batchsize and $y_{i}$ is defined as:

Here, $\gamma\in[0,1]$ represents the discounting factor. Then, the actor network is updated using the policy gradient method as:

Afterward, the target actor and target critic networks are updated using the soft update approach:

where $\tau$ is a small coefficient denoting how fast the target actor and critic are updated. As in every RL algorithm, we need exploration to achieve the best policy. In this regard, we add a zero-mean Gaussian noise $\mathcal{N}$ to the actor policy as follows:

where $\mathcal{N}$ follows the distribution $\mathcal{CN}(0,\sigma^{2})$.

To apply the DDPG algorithm to our problem, we need to define appropriate states, actions, and rewards for the agent. Moreover, the design of the actor and critic network can significantly influence the performance of the algorithm. Thus, first, we introduce the suitable state, action, and reward for UAV deployment, and then, we discuss the configuration of the actor and critic network.

In this section, we consider IoT users to have a fixed location and discuss two scenarios: 1) narrow-range user distribution; and 2) wide-range user distribution. For narrow-range user distribution, we consider that the BS is located at $(x_{b},y_{b},z_{b})=(0,0,10)$, the UAV initial position is $(x_{u},y_{u},z_{u})=(50,50,20)$, and $K=4$ IoT users are distributed randomly at a far distance from the BS (i.e., $(x_{k},y_{k})\in[90,100]$). UAV starts its initial location at $(50,50,20)$ at the beginning of each episode, and then it explores the environment during the time steps. For wide-range user distribution, we consider the same location and initial position for the UAV, while assuming that $K=4$ IoT users are randomly scattered with $(x_{k},y_{k})\in[50,100]$. Fig. 3 shows the achieved rates for DDPG-UD, PSO-UD, DL-UD, and fixed deployment (FD) (i.e., no optimization). Numerical results reveal that the proposed DDPG-UD can achieve 98.63% of PSO-UD performance for narrow-range user distribution while having a 16 times better performance than FD. Furthermore, DDPG-UD can enhance the performance of PSO-UD by 2.23% for wide-range user distribution while having 3.14 times better AR than the FD. Fig. 3 also demonstrates that although DL-UD can have an acceptable performance when we have narrow-range user distribution while achieving 89.62% AR of PSO-UD, it fails to find the optimal location for wide-range user distribution having only 36.94% AR of the DDPG-UD. This means that for more complex user distributions, DL-UD is not a promising solution. For the next step, we consider that IoT users may change their location during the observation.

In this section, we consider a more practical scenario where IoT users can change their location during the training. This scenario represents dynamic, real-world environments. We assume that the BS is set at $(x_{b},y_{b},z_{b})=[0,0,10]$. Then we set the IoT users’ locations randomly at six different distributions $l\in$ {$l_{1},l_{2},...,l_{6}$} as follows:

where $x_{k,l}$ and $y_{k,l}$ denote the $k^{th}$ IoT user x-axis and y-axis range during $l^{th}$ distribution, respectively. Furthermore, UAV starts its initial location at $(x_{u},y_{u},z_{u})=(50,50,20)$ and finds the optimal deployment $\mathbf{x}_{o}^{(1)}=\{x_{o}^{(1)},y_{o}^{(1)}\}$ for $l_{1}$, which is now used as its initial location for the next user distribution (i.e., $l_{2}$). Fig. 4 shows the accumulated reward and average accumulated reward for the DDPG agent during six different user locations. This figure shows that after the first user location, the DDPG agent maintains the reward and finds the optimal location in less number of episodes. The reason is that the pre-trained networks from previous user locations contain information about the environment, thus helping the DDPG agent find the optimal UAV location for changing user locations in a shorter time. Fig. 5 shows the achieved rate for DDPG-UD, PSO-UD, DL-UD, and FD, which shows that DDPG-UD can achieve a performance close to PSO-UD. For instance, DDPG-UD accomplishes 99.62% of PSO-UD AR for $l_{3}$, and it provides 18.54 bps/Hz AR at $l_{6}$ and achieves 99.42% of PSO-UD performance. Additionally, the capacity is improved by 36.99% and 16.73% for FD for $l_{3}$ and $l_{6}$, respectively. Fig. 5 also supports the previous deduction that DL-UD is not a promising solution for complex scenarios while having only 63.84% and 66.65% of PSO-UD performance for $l_{3}$ and $l_{6}$, respectively.

Fig. 6 displays the time comparison between DDPG-UD and PSO-UD. Here, we provide the runtime for a single user location (i.e., $l\in l_{1}$), three different user locations ($l\in\{l_{1},l_{2},l_{3}\}$), and six distinct user locations ($l\in\{l_{1},\cdots,l_{6}\}$). This figure shows that although for a single user location, DDPG-UD takes 18.04% more than PSO-UD to find the optimal location, for multiple user locations, it takes 76.45% and 68.50% of PSO-UD runtime. This means that as the number of user locations is increased, DDPG-UD will take less time to find the optimal location when compared to PSO-UD, which makes DDPG-UD an efficient solution for dynamic and fast-changing environments in MU-mMIMO IoT systems.