UAV-enabled Collaborative Beamforming via Multi-Agent Deep Reinforcement Learning

As for the global state of the $i$-th agent $s_{i}$, instead of using the concatenation of local observations or the environment-provided global state, we propose an agent-specific global state that combines some agent-related features (e.g., $(\theta_{i},\phi_{i},d_{i})$, $d_{i,o}$, $d_{j,i}$) and the global information provided from the environment which includes the coordinates and excitation current weights of all UAVs, as well as the coordinate of reference point, which is inspired from [42].

According to the local observation of UAVs, each UAV will take an action to fly to a certain location with the optimized excitation current weight for performing CB. Then, the action of each agent can be defined by

where $(x_{i},y_{i},z_{i})$ is the location to which the $i$-th UAV is going to fly, and $I_{i}$ is the excitation current weight of the $i$-th UAV in the next time slot.

The reward function is used to evaluate how well an agent executes the actions. Specifically, a suitable reward function can make the agent learn the best strategy. UCBMOP involves two optimization objectives, which are maximizing the transmission rate of UVAA for communicating with the BS, i.e., $R_{T}$, and minimizing the motion energy consumption of all UAVs, i.e., $E_{total}$. To this end, we design a reward function by considering the following aspects:

1. 1.

   Reward Function for Optimizing $R_{T}$. We find that there are three key components that affect the transmission rate $R_{T}$, which are the transmit power and gain of UVAA, transmission distance, and A2G communication angle. Specifically, the transmit power and gain of UVAA determine the transmission performance of UVAA, the transmission distance indicates the signal fading degree, and the air-to-ground communication angle affects the A2G channel condition. Note that we do not directly set the transmission rate as a reward since it changes significantly with the transmission distance, making the DRL method unstable and decreasing the applicability of the model.

   According to the analysis above, first, we regard the multiplication of the array gain and the total transmit powers as a reward, i.e., $r^{\mathrm{TR}}=GP_{t}$. Note that this reward is shared by all agents. Then, we set the distance between the UAV and BS as a penalty for each agent. However, the UAV only flies within a fixed area, and then the aforementioned distance is replaced by the one between the UAV and the reference point of target BS, which is denoted as $r^{\mathrm{U2B}}_{i}=-d_{i,r}$. Finally, as shown in Eq. (1), the LoS probability is determined mainly by the height of the UAVs. Therefore, we set a reward to optimize the flight altitude of each UAV, denoted as $r_{i}^{\mathrm{H}}=z_{i}$. As such, the designed reward function considers all controllable components of the transmission rate, thereby achieving the optimization of the first objective.

2. 2.

   Reward Function for Optimizing $E_{total}$. To achieve this goal, we regard the energy consumption of each UAV as a penalty, which is denoted as $r_{i}^{\mathrm{EC}}=-E_{i}$. As such, the optimization of the second objective can be achieved by the reward function.

3. 3.

   Reward Function for Coordinating $R_{T}$ and $E_{total}$. To this end, we aim to enhance the connectivity among UAVs. Compared with $R_{T}$, the optimization of $E_{total}$ is more directly tied to the action space (i.e., reducing the movement of the UAVs), which may lead the algorithm to prioritize the optimization of $E_{total}$ and under-optimize $R_{T}$. To overcome this issue, we try to concentrate the array elements (i.e., the UAVs in UVAA) on an applicable distance to achieve a higher CB performance [43] [23], thereby facilitating the optimization of $R_{T}$. Specifically, we design a reward to coordinate the UAVs by minimizing the cumulative distances between the current UAV and other UAVs, as well as between the current UAV and the origin of the UVAA, i.e.,

   $\displaystyle r^{\mathrm{U2U}}_{i}=-\frac{1}{\kappa}(\sum_{j\in\mathcal{N},j\neq i}d_{j,i}+d_{i,o}),$

   (13)

   where $\kappa$ is a large constant for scaling. Since this reward decreases the difficulty of optimizing $R_{T}$, the designed reward function can well coordinate the optimization processes of the two optimization objectives.

HATRPO was proposed by Kuba et al. [16], which has the better performance than other baseline MADRL algorithms in many multi-agent tasks (e.g., Multi-Agent MuJoCo [44], StarCraftII Multi-Agent Challenge (SMAC) [45]). HATRPO applies the trust region learning [46] to MADRL successfully, achieving the monotonic improvement guarantee for joint policy of multiple agents such as the trust region policy optimization (TRPO) algorithm [47]. The detailed theory of HATRPO is introduced in the following.

In the beginning, a cooperative Markov game with $N$ agents exists. A joint policy $\bm{\pi}=(\pi_{1},\dots,\pi_{N})$ to all agents exists. At time slot $t$, the agents are at state $s^{t}$. Each agent utilizes its own policy $\pi_{i}$ to take an action $a^{t}_{i}$, and then the actions of all agents form the joint action $\bm{a}^{t}=(a^{t}_{1},\dots,a^{t}_{N})$. Afterwards, all agents receive a joint reward $r^{t}$, and move to a new state $s^{t+1}$ with probability $P(s^{t+1}|s^{t},\bm{a}^{t})$. The goal of all agents is to maximize the expected total reward:

where $\gamma\in[0,1)$ is the discount factor and $\rho_{\bm{\pi}}$ is the marginal state distribution [16]. Besides, the state value function and the state-action value function are defined as $V^{\bm{\pi}}(s^{t})=\mathbb{E}_{\bm{a}^{t:\infty}\sim\bm{\pi},s^{t+1:\infty}\sim P}\left[\sum_{l=0}^{\infty}\gamma^{l}r^{t+l}\right]$ and $Q^{\bm{\pi}}(s^{t},\bm{a}^{t})=\mathbb{E}_{s^{t+1:\infty}\sim P,\bm{a}^{t+1:\infty}\sim\bm{\pi}}\left[\sum_{l=0}^{\infty}\gamma^{l}r^{t+l}\right]$. The advantage function is written as $A^{\bm{\pi}}(s^{t},\bm{a}^{t})=Q^{\bm{\pi}}(s^{t},\bm{a}^{t})-V^{\bm{\pi}}(s^{t})$.

To implement the above procedure for parameterized joint policy $\bm{\theta}=(\theta_{1},\dots,\theta_{N})$ in practice, in HATRPO, the amendment similar to TRPO is performed. As the maximal KL-divergence penalty $\mathrm{D^{max}_{KL}}(\pi_{\theta^{k}_{i}},\pi_{\theta_{i}})$ is hard to compute, it is replaced by the expected KL-divergence constraint $\mathbb{E}_{s\sim\rho_{\bm{\pi}_{\bm{\theta}^{k}}}}[\mathrm{D_{KL}}(\pi_{\theta^{k}_{i}}(\cdot|s),\pi_{\theta_{i}}(\cdot|s))]\leq\delta$, where $\delta$ is a threshold hyperparameter. Then, at the $k$+$1$-th iteration, given a permutation, all agents can optimize their policy parameters sequentially according to the method as follows:

For the computation of the above equation, similar to TRPO, the linear approximation to the objective function and the quadratic approximation to KL constraint are applied, derivating a closed-form updating scheme that is shown as

$\alpha^{j}\in(0,1)$ is a coefficient that is found via backtracking line search. $\bm{H}^{k}_{i}=\nabla^{2}_{\theta_{i}}\mathbb{E}_{s\sim\rho_{\bm{\pi}_{\bm{\theta}^{k}}}}[\mathrm{D_{KL}}(\pi_{\theta^{k}_{i}}(\cdot|s),\pi_{\theta_{i}}(\cdot|s))]|_{\theta_{i}=\theta^{k}_{i}}$ is the Hessian of the expected KL-divergence. $\bm{g}^{k}_{i}$ is the gradient of the objective in Eq. (19), whose computation requires the estimation about $\mathbb{E}_{\bm{a}_{1:i-1}\sim\bm{\pi}_{\bm{\theta}^{k+1}_{1:i-1}},a_{i}\sim\pi_{\theta_{i}}}[A^{\bm{\pi}_{\bm{\theta}^{k}}}_{i}(s,\bm{a}_{1:i-1},a_{i})]$. According to the derivation in [16], given a batch $B$ of trajectories with length $T$, $\bm{g}^{k}_{i}$ can be computed as

where

$\frac{\bm{\pi}_{\bm{\theta}^{k+1}_{1:i-1}}(\bm{a}^{t}_{1:i-1}|s^{t})}{\bm{\pi}_{\bm{\theta}^{k}_{1:i-1}}(\bm{a}^{t}_{1:i-1}|s^{t})}$ is the compound policy ratio about the previous agents $1:i-1$, which can be computed easily after these agents complete their updates.

Although HATRPO has the excellent performance, it still faces many challenges in solving the UCBMOP. First, the formulated problem involves the complex cooperation among UAVs. Furthermore, only one time slot exists for each episode in the problem, which is different from other traditional MADRL tasks. Thus, it is necessary to take measures to connect the algorithm with the problem closely. Second, there are many UAVs in the problem, which may increase the input dimension of critic network substantially, influencing the critic learning. Finally, the conventional HATRPO uses the Gaussian distribution to sample actions. However, all actions have the finite range in the problem. Thus, using the Gaussian distribution may induce the bias to the policy gradient estimation. Accordingly, these reasons motivate us to propose the HATRPO-UCB, and the details are described in the following sections.

In the proposed HATRPO-UCB, we take conventional HATRPO as the foundation algorithm framework, and propose three techniques that are observation enhancement, agent-specific global state and Beta distribution for policy, to enhance the performance of the algorithm.

Observation enhancement: In MADRL, the environmental information is usually the state that guides the agents to take action. However, in UCBMOP, each episode only has one time slot, and the UVAA may communicate with different remote BSs at various episodes. In this case, the state consisting of the simple Cartesian coordinates of the UAVs and BSs may not precisely characterize their positional relationship. This is because the distance between the UVAA and BSs is significantly larger than the relative distance between the UAVs, and even the same set of UAV coordinates may represent different environmental states when serving various BSs (e.g., transmission angle and distance). To overcome this issue, we propose two observation enhancement strategies as follows. First, we design a spherical coordinate enhancement strategy with dynamically changing origins to represent the UAV positions. Instead of the Cartesian coordinate of each UAV, we set the state to the spherical coordinates with the BS in the current episode as the origin. In this way, the state can adequately express the information about the orientation and distance between the BS and the UVAA. Second, we propose a position representation based on reference points to characterize the BS locations. Specifically, we regard the closest point to the target BS in the monitor area as the reference point and then adopt this reference point to replace the target BS as the state. This adjustment retains the BS information but amplifies the position differences between the UAVs, which better expresses the relative positions of the UAVs. The enhanced state and observation contain representative and adaptive information about the environment, which ensures the proposed method remains effective when the location of the served BS at different episodes changes significantly.

Agent-specific global state: In general, the global state employed by MADRL algorithms has two forms, which are the concatenation of all local observations and environment-provided global state. However, in the considered scenarios, the number of agents (i.e., UAVs) is often large, which means that the input dimension of the critic network grows a lot when using the concatenation of all local observations. This condition will increase the learning burden, thereby making the performance of critic learning degrade. Additionally, the environment-provided global state is also not an appropriate choice because its contained information is not sufficient, which only contains the coordinates and excitation current weights of all UAVs, as well as the coordinates of the reference point. Inspired by [42], we design an agent-specific global state that combines the global information provided by the environment and some features from local observation, such as the spherical coordinate of the target BS that is relative to the UAV $(\theta_{i},\phi_{i},d_{i})$ and the distance between two UAVs $d_{j,i}$, as the input of the critic network. This agent-specific global state can improve the fitting performance of critic learning and thus achieve better MADRL performance.

Beta distribution for policy: In the conventional HATRPO, the actor uses the Gaussian distribution to sample actions. However, the Gaussian distribution has been demonstrated to have the negative impact on reinforcement learning [48]. Specifically, for an action with the finite interval, the Gaussian distribution with infinite support will define the action range out of its boundary. Furthermore, the probability density of all actions is greater than 0, but the probability of actions beyond boundary should equal to 0. Hence, in our scenario where all actions of UAV have the finite interval, it is inevitable that some actions sampled from the Gaussian distribution will be truncated, leading to the boundary effect. Then, the boundary effect may cause bias about policy gradient estimation.

To handle the above issue, the authors of [48] conducted the research on Beta distribution for reinforcement learning. The study indicates that the Beta distribution does not have a bias because the Beta distribution has a finite range (i.e., [0, 1]), and thus the probability density of actions beyond the boundary is guaranteed to be 0. Furthermore, experimental results show that the Beta distribution can make reinforcement learning algorithms converge faster and obtain higher rewards. Thus, in this paper, we choose the Beta distribution instead of the original Gaussian distribution, and then the Beta distribution is defined as

where $\alpha$ and $\beta$ are the shape parameters, and $\Gamma(\cdot)$ is the Gamma function that extends factorial to real numbers. In addition, $\alpha$ and $\beta$ are set to be not less than 1 (i.e., $\alpha,\beta\geq 1$) in the work, which are modeled by softplus and then add a constant 1 [48].

Fig. 4 displays the framework of the proposed HATRPO-UCB algorithm, which has two phases that are the training phase and the implementation phase. Each UAV as an agent has a replay buffer and two networks (i.e., the actor network and critic network). The replay buffer is used to collect the data about the interaction between the UAV and the environment. Then, a central server is deployed with the replay buffers and networks of all agents for training.

In the beginning, the central server will initialize the parameters of networks. Then, at each time slot, the central server will exchange the information with UAVs and then control them, where the amount of the information is very small because it only includes the observation information of UAVs and actions required to be done by UAVs, and the communication distance is very short compared with that between the UAVs and the BS. Hence, for performing the communication tasks, the above communication overhead is very small and acceptable. Then, for illustrating the specific process at each time slot, taking the $i$-th agent as an example, after the central server receives the information, the actor network $\pi_{\theta_{i}}$ utilizes the observation $o_{i}$ to generate a Beta distribution about action, and then an action $a_{i}$ is sampled from the Beta distribution and sent to the UAV. The critic network is used to predict the Q-value of current global state $s_{i}$. After executing the sampled action, the $i$-th agent receives a reward $r_{i}$ from the environment and observes the next local state $o^{\prime}_{i}$ and global state $s^{\prime}_{i}$. After that, the above information tuple $\langle o_{i},s_{i},a_{i},r_{i}\rangle$ will be stored in the replay buffer $\mathcal{B}_{i}$.

All agents will execute more time slots until their replay buffers accumulate enough data, and then the replay buffers will randomly select a mini-batch of tuples for training networks. The sequential updating scheme is adopted to update policies of all agents. In each training, all agents are updated sequentially in a randomly generated order. For the $i$-th agent, the first step is to estimate the gradient $\bm{g}^{k}_{i}$, before that, the advantage function of $i$-th agent $A_{i}(s_{i},\bm{a})$ and the compound policy ratio of the previous agents $\prod^{i-1}_{l=1}\frac{\pi_{\theta^{k+1}_{l}}(a_{l}|o_{l})}{\pi_{\theta^{k}_{l}}(a_{l}|o_{l})}$ are required to compute. Then, the Hessian of the expected KL-divergence $\bm{H}^{k}_{i}$ is computed to approximate the KL constraint. After completing the computation of $\bm{g}^{k}_{i}$ and $\bm{H}^{k}_{i}$, the parameter of actor network $\theta_{i}$ can be updated via backtracking line search. As for the critic network $V_{\phi_{i}}$, the update of network parameter can be achieved by optimizing the following loss function: $\frac{1}{B}\sum^{B}_{b=1}(V_{\phi_{i}}(s^{b}_{i})-R^{b}_{i})^{2}$. Therefore, the above updating procedure will be repeated until HATRPO-UCB converges. The whole training process can be seen in Algorithm 1.

During the implementation stage, only the trained actor network is deployed to the UAV. In each communication mission, all UAVs receive the local observations from the environment, and then select actions through the actor networks. Afterwards, according to the selected actions, all UAVs move to the target locations and adjust the excitation current weights for performing CB. Compared with the training stage, the implementation stage is completed online and does not need much computational resource. In contrast, the training stage costs much computational resource, which is conducted offline in a central server.

In this section, the computational complexity of the training and implementation stage of the proposed HATRPO-UCB is discussed.

Complexity of training. For $n_{\mathrm{uav}}$ agents, each agent is equipped with an actor network and a critic network that are formed by DNNs. A DNN consists of an input layer, $L$ fully connected layers and an output layer, where $z_{0}$, $z_{i}$ and $z_{L+1}$ denote the number of neurons of input layer, $i$-th fully connected layer and output layer, respectively. Then, the computational complexity of DNN at each time slot is $O(\sum^{L+1}_{i=1}z_{i-1}\cdot z_{i})$.

At each training process, a mini-batch of data with $n_{\mathrm{epi}}$ episodes is sampled from the replay buffer, but each episode only contains one time slot. Additionally, the critic network is updated by the stochastic gradient methods, thus the total computational complexity of critic network is $O(n_{\mathrm{epi}}\cdot(\sum^{L+1}_{i=1}z_{i-1}\cdot z_{i}))$. However, the actor network is updated using the backtracking line search. In Eq. (20), the gradient $\bm{g}^{k}_{i}$ is first computed, whose computational complexity is $O(n_{\mathrm{epi}}\cdot(\sum^{L+1}_{i=1}z_{i-1}\cdot z_{i}))$. Then the Hessian of expected KL-divergence $\bm{H}^{k}_{i}$ is computed to approximate the KL constraint. The computational complexity of expected KL-divergence is $O(n_{\mathrm{epi}}\cdot(\sum^{L+1}_{i=1}z_{i-1}\cdot z_{i}))$, and the computational one of Hessian matrix is $O((\sum^{L+1}_{i=1}z_{i-1}\cdot z_{i})^{2})$. Hence, the computational complexity of $\bm{H}^{k}_{i}$ is $O(n_{\mathrm{epi}}\cdot(\sum^{L+1}_{i=1}z_{i-1}\cdot z_{i})+(\sum^{L+1}_{i=1}z_{i-1}\cdot z_{i})^{2})$. For computing $(\bm{H}^{k}_{i})^{-1}\bm{g}^{k}_{i}$, the conjugate gradient algorithm is adopted, which obtains result by searching iteratively until convergence. The computational complexity of each iteration is $O((\sum^{L+1}_{i=1}z_{i-1}\cdot z_{i})^{2})$. Then the total computational complexity of $(\bm{H}^{k}_{i})^{-1}\bm{g}^{k}_{i}$ is $O(n_{\mathrm{epi}}\cdot(\sum^{L+1}_{i=1}z_{i-1}\cdot z_{i})+l_{\mathrm{conj}}\cdot(\sum^{L+1}_{i=1}z_{i-1}\cdot z_{i})^{2})$, where $l_{\mathrm{conj}}$ is the number of iterations. During the backtracking line search, the computational complexity is $O(l_{\mathrm{bt}}\cdot n_{\mathrm{epi}}\cdot(\sum^{L+1}_{i=1}z_{i-1}\cdot z_{i}))$, where $l_{\mathrm{bt}}$ is the search number and $O(n_{\mathrm{epi}}\cdot(\sum^{L+1}_{i=1}z_{i-1}\cdot z_{i}))$ is the computational complexity in each search. Hence, the total computational complexity of actor network is $O(l_{\mathrm{bt}}\cdot n_{\mathrm{epi}}\cdot(\sum^{L+1}_{i=1}z_{i-1}\cdot z_{i})+l_{\mathrm{conj}}\cdot(\sum^{L+1}_{i=1}z_{i-1}\cdot z_{i})^{2})$. Assume that HATRPO-UCB needs to be trained $l_{\mathrm{t}}$ times for convergence, the overall computational complexity in the training phase is $O(l_{\mathrm{t}}\cdot n_{\mathrm{uav}}\cdot(l_{\mathrm{bt}}\cdot n_{\mathrm{epi}}\cdot(\sum^{L+1}_{i=1}z_{i-1}\cdot z_{i})+l_{\mathrm{conj}}\cdot(\sum^{L+1}_{i=1}z_{i-1}\cdot z_{i})^{2}))$.

Complexity of inference. In the implementation phase, all UAVs only use the trained actor networks to make decisions. Therefore, the computational complexity of HATRPO-UCB in the implementation stage is $O(n_{\mathrm{uav}}\cdot(\sum^{L+1}_{i=1}z_{i-1}\cdot z_{i}))$.