Deep Reinforcement Learning for Multi-Agent Power Control in Heterogeneous Networks

In this paper, we study the power control problem for the sum-rate maximization in a typical HetNet, in which multi-tier cells coexist by reusing the same spectrum band. Different from the single-tier scenario, both the maximum transmit power and coverage of each AP in the multi-tier scenario are typically heterogeneous. Main contributions of the paper are summarized as follows:

1. 1.

   We exploit DRL to design a multi-agent power control algorithm in the HetNet. First, we establish a local DNN at each AP and treat each AP as an agent. The input and the output of each local DNN are the local state information and the adopted local action, i.e., transmit power of the corresponding AP, respectively. Then, we propose a novel multiple-actor-shared-critic (MASC) method to train separately each local DNN in an online trial-and-error manner.

2. 2.

   The MASC training method is composed of multiple actor DNNs and a shared critic DNN. In particular, we first establish an actor DNN in the core network for each local DNN, and the structure of each actor DNN is the same as the corresponding local DNN. Then, we establish a shared critic DNN in the core network for these actor DNNs. By feeding historical global information into the critic DNN, the output of the critic DNN can accurately evaluate whether the output (i.e., transmit power) of each actor DNN is good or not from an global view. By training each actor DNN with the evaluation of the critic DNN, the weight vector of each actor DNN can be updated towards the direction of the global optimum. The weight vector of each local DNN can be periodically replaced by that of the associated actor DNN until convergence.

3. 3.

   The proposed algorithm has two main advantages compared with the existing power control algorithms. First, compared with [7], [8], [13], and [14], each AP in the proposed algorithm can independently control the transmit power and enhance the sum-rate based on only local state information, in the absence of instantaneous global CSI and any cooperation with other cells. Second, compared with [14], [15], and [16], the reward function of each agent in the proposed algorithm is the transmission rate between the corresponding AP and its served user, avoiding particular reward function designs for each AP. This may ease the transfer of the proposed algorithm framework into more resource management problems in wireless communications.

4. 4.

   By considering both two-layer and three-layer HetNets in simulations, we demonstrate that the proposed algorithm can rapidly converge to an average sum-rate higher than those of WMMSE and FP algorithms. Simulation results also reveal that the proposed algorithm outperforms WMMSE and FP algorithms in terms of the computational complexity.

DRL originates from RL, which has been widely used for the designs of wireless communications [17]-[22], e.g., user/AP handoff, radio access technology selection, energy-efficient transmissions, user scheduling and resource allocation, spectrum sharing, and etc. In particular, RL estimates the long-term reward of each state-action pair and stores them into a two-dimensional table. For a given state, RL can choose the action subject to the maximum long-term reward to enhance the performance. It has been demonstrated that RL performs well in a decision-making scenario in which the size of the state and action spaces in the wireless system is relatively small. However, the effectiveness of RL diminishes when the state and action spaces become large.

To choose proper actions in the scenarios with large state and action spaces, DRL is proposed by properly integrating DL and RL [23]. In particular, by adopting the DNN which has a strong representation capability, DL has been applied in different areas of wireless communications [24], for example, power control [25], channel access [26], link scheduling [27]. The basic idea of DRL is as follows: Instead of storing the long-term reward of each state-action pair in a tabular manner, DRL uses a DNN to represent the long-term reward as a function of the state and action. Thanks to the strong representation capability of the DNN, the long-term reward of any state-action pair can be properly approximated. The application of DRL in HetNets includes interference control among small cells [15], power control in a HetNet [16], resource allocation in V2V communications [28], caching policy optimization in content distribution networks [29], multiple access optimization in a HetNet [30], modulation and coding scheme selection in a cognitive HetNet [31], joint beamforming/power control/interference coordination in 5G networks [32], UAV navigation [33], and spectrum sharing [34]. More applications of DRL in wireless communications can be found in [35].

The remainder of this paper is organized as follows. We provide the system model in Sec. II. In Sec. III, we provide the problem formulation and analysis. In Sec. IV, we give preliminaries by overviewing related DRL algorithms. In Sec. V, we elaborate the proposed power control algorithm. Simulation results are shown in Sec. VI. Finally, we conclude the paper in Sec. VII.

The channel between an AP and a UE has two components, i.e., large-scale attenuation (including path-loss and shadowing) and small-scale block Rayleigh fading. If we denote $\phi_{n,k}$ as the large-scale attenuation and denote $h_{n,k}$ as the small-scale block Rayleigh fading from AP $n$ to UE $k$, the corresponding channel gain is $g_{n,k}=\phi_{n,k}|h_{n,k}|^{2}$. In particular, the large-scale attenuation is highly related to the locations of the AP and the UE, and typically remains constant for a long time. The small-scale block Rayleigh fading remains constant in a single time slot and changes among different time slots.

According to [36], we adopt the Jake’s model to represent the relationship between the small-scale Rayleigh fadings in two successive time slots, i.e.,

where $\rho$ ($0\leq\rho\leq 1$) denotes the correlation coefficient of two successive small-scale Rayleigh fading realizations, $\omega$ is a random variable represented by a distribution $\omega\sim\mathcal{CN}(0,1-\rho^{2})$, and $h(0)$ is a random variable produced by a distribution $h(0)\sim\mathcal{CN}(0,1)$. It should be noted that, the Jake’s model can be reduced to the independent and identically distributed (IID) channel model if $\rho$ is zero.

If we denote $x_{n}(t)$ as the downlink signal from AP $n$ to UE $n$ with unit power in time slot $t$, the received signal at UE $n$ is

where $p_{n}(t)$ is the transmit power of AP $n$, $\delta_{n}(t)$ is the noise at UE $n$ with power $\sigma^{2}$, the first term on the right side is the received requiring signal from AP $n$, and the second term on the right side is the received interference from other APs. By considering that all the downlink transmissions from APs to UEs are synchronized, the signal to interference and noise ratio (SINR) at UE $n$ can be written as

Accordingly, the downlink transmission rate (in bps) from AP $n$ to UE $n$ is

where $B$ is the bandwidth of the downlink transmission.

DQN establishes a DNN $Q(s,a;\theta)$ with weight vector $\theta$ to represent the expected cumulative discounted (long-term) reward by executing the action $a$ in the environmental state $s$. Then, $Q(s,a;\theta)$ can be rewritten in a recursive form (Bellman equation) as

where $r(s,a)$ is the immediate reward by executing the action $a$ in the environmental state $s$, $\eta\in[0,1]$ is the discount factor representing the discounted impact of future rewards, and ${p_{s,s^{\prime}}}(a)$ is the transition probability of the environment state from $s$ to $s^{\prime}$ by executing the action $a$. The DRL agent aims to find the optimal weight vector $\theta^{*}$ to maximize the long-term reward for each state-action pair. With the optimal weight vector $\theta^{*}$, (6) can be rewritten as

Accordingly, the optimal action policy is

However, it is challenging to directly obtain the optimal $\theta^{*}$ from (7) since the transition probability ${p_{s,s^{\prime}}}(a)$ is typically unknown to the DRL agent. Then, DRL agent updates $\theta$ in an iterative manner. On the one hand, to balance exploitation and exploration, the DRL agent adopts an $\epsilon$-greedy algorithm to choose an action for each environmental state: for a given state $s$, the DRL agent executes the action $a=\arg\max_{a\in\textbf{\emph{A}}}Q(s,a;\theta)$ with the probability $1-\epsilon$, and randomly executes an action with the probability $\epsilon$. On the other hand, by executing actions with $\epsilon$-greedy algorithm, DRL agent can continuously accumulate experience $e=\{s,a,r,s^{\prime}\}$ and store it in an experience replay buffer in a first-in-first-out (FIFO) fashion. After sampling a mini-batch of experiences $\mathcal{E}$ with length $D$ ($\mathcal{E}=\{e_{1},e_{2},\cdots,e_{D}\}$) from the experience replay buffer, the DRL agent can update $\theta$ by adopting a stochastic gradient decent (SGD) method to minimize the expected prediction error (loss function) of the sampled experiences, i.e.,

where $Q^{-}(s,a;\theta^{-})$ is the target DNN and is established with the same structure as $Q(s,a;\theta)$. In particular, the weight vector $\theta^{-}$ is updated by $\theta$ periodically to stabilize the training of $Q(s,a;\theta)$.

It should be noted that DQN needs to estimate the long-term reward for each state-action pair to obtain the optimal policy. When the action space is continuous, DQN is inapplicable and DDPG can be an alternative. DDPG has an actor-critic architecture, which includes an actor DNN $\mu(s;\theta_{\mu})$ and a critic DNN $Q(s,a;\theta_{Q})$. In particular, the actor DNN $\mu(s;\theta_{\mu})$ is the policy network and is responsible to output a deterministic action $a$ from a continuous action space in an environment state $s$, and the critic DNN is the value-function network (similar to DQN) and is responsible to estimate the long-term reward when executing the action $a$ in the environment state $s$. Similar to DQN, to stabilize the training of actor DNN and critic DNN, DDPG establishes an target actor DNN $\mu^{-}(s;\theta_{\mu}^{-})$ and a target critic DNN $Q^{-}(s,a;\theta_{Q}^{-})$, respectively.

Similar to the DQN, the DDPG agent samples experiences from the experience replay buffer to train the actor DNN and critic DNN. Nevertheless, the DDPG accumulates experiences in a unique way. In particular, the executed action in each time slot is a noisy version of the output of the actor DNN, i.e., $a=\mu(s;\theta_{\mu})+\zeta$, where $\zeta$ is a random variable and guarantees continuously exploring the action space around the output of the actor DNN. Note that, the epsilon-greedy method is usually used for action selections in discrete action space scenarios. When the action space is continuous, the noisy version of the output of the actor DNN is preferable for action selections [38] [39].

The training procedure of the critic DNN is similar to that of DQN: by sampling a mini-batch of experiences $\mathcal{E}$ with length $D$ from the experience replay buffer, the DDPG agent can update $\theta_{Q}$ by adopting a SGD method to minimize the expected prediction error of the sampled experiences, i.e.,

Then, the DDPG agent adopts a soft update method to update the weight vector $\theta_{Q}^{-}$ of the target critic DNN, i.e., $\theta_{Q}^{-}\leftarrow\tau\theta_{Q}+(1-\tau)\theta_{Q}^{-}$, where $\tau\in[0,1]$ is the learning rate of the target DNN.

The goal of training the actor DNN is to maximize the expected value-function $Q\left(s,\mu(s;\theta_{\mu});\theta_{Q}\right)$ in terms of the environment state $s$, i.e., $J(\theta_{\mu})=\mathbb{E}_{s}\left[Q\left(s,\mu(s;\theta_{\mu});\theta_{Q}\right)\right]$. Then, the weight vector $\theta_{\mu}$ of the actor DNN can be updated in the sampled gradient direction of $J(\theta_{\mu})$, i.e.,

Similar to the target critic DNN, the DDPG agent updates the weight vector of the target actor DNN by $\theta_{\mu}^{-}\leftarrow\tau\theta_{\mu}+(1-\tau)\theta_{\mu}^{-}$.

It is known that the optimal transmit power of APs is highly related to the instantaneous global CSI. But the instantaneous global CSI is unavailable to APs. Thus, it is impossible for each AP to optimize the transmit power through conventional power control algorithms, e.g., WMMSE algorithm or FP algorithm. From [13] and [14], the historical wireless data (e.g., global CSI, transmit power of APs, the mutual interference, and achieved sum-rates) of the whole network contains useful information that can be utilized to optimize the transmit power of APs. Thus, we aim to leverage DRL to develop an intelligent power control algorithm that can fully utilize the historical wireless data of the whole network. From the aspect of practical implementations, the intelligent power control algorithm should have two basic functionalities:

• •

  Functionality I: Each AP can use the algorithm to complete the optimization of the transmit power at the beginning of the each time slot, in order to guarantee the timeliness of the optimization.

• •

  Functionality II: Each AP can independently optimize the transmit power to enhance the sum-rate with only local observations, in the absence of the global CSI and any cooperations among APs.

To realize Functionality I, we adopt a centralized-training-distributed-execution architecture as the basic algorithm framework as shown in Fig. 2. To be specific, a local DNN is established at each AP, and the input and the output of each local DNN are the local state information and the adopted local action, i.e., transmit power of the corresponding AP, respectively. In this way, each AP can feed local observations into the local DNN to calculate the transmit power in a real-time fashion. The weight vector of each local DNN is trained in the core network, which has redundant historical wireless data of the whole network. Since there are $N$ APs, we denote the corresponding $N$ local DNNs as $\mu_{n}^{\text{(L)}}\left(s_{n};\theta_{n}^{\text{(L)}}\right)$ ($n\in\mathbb{N}$), in which $\theta_{n}^{\text{(L)}}$ is the weight vector of the local DNN $n$ and $s_{n}$ is the local state of AP $n$.

To realize Functionality II, we develop a MASC training method based on the DDPG to update the weight vectors of local DNNs. To be specific, as shown in Fig. 2, $N$ actor DNNs together with $N$ target actor DNNs are established in the core network to associate with $N$ local DNNs, respectively. Each actor DNN has the same structure as the associated local DNN, such that the trained weight vector of the actor DNN can be used to update the associated local DNN. Meanwhile, a shared critic DNN together with the corresponding target critic DNN is established in the core network to guide the training of $N$ actor DNNs. The inputs of the critic DNN include the global state information and the adopted global action, i.e., the transmit power of each AP, and the output of the critic DNN is the long-term sum-rate of the global state-action pair. It should be noted that, there are two main benefits to include the global state and global action in the input of the critic DNN. First, by doing this, the non-stationary radio environment issue the critic DNN faces is only caused by the time-variant nature of wireless channels. This is the major difference compared with the case, in which single-agent algorithm is directly applied to solve a multi-agent problem and the non-stationary issue is caused by both the time-variant nature of wireless channels and unknown actions of other agents. The adverse impact of the non-stationary radio environment issue on the system performance can thus be reduced. Second, by doing this, we can train the critic DNN with historical global state-action pairs together with the achieved sum-rates in the core network, such that the critic DNN has a global view of the relationship between the global state-action pairs and the long-term sum-rate. Then, the critic DNN can evaluate whether the output of an actor DNN is good or not in terms of the long-term sum-rate. By training each actor DNN with the evaluation of the critic DNN, the weight vector of each actor DNN can be updated towards the direction of the global optimum. To this end, the weight vector of each local DNN can be periodically replaced by that of the associated actor DNN until convergence.

We denote $N$ actor DNNs as $\mu_{n}^{(\text{a})}\left(s_{n};\theta_{n}^{(\text{a})}\right)$, ($n\in\mathbb{N}$), where $\theta_{n}^{(\text{a})}$ is the weight vector of actor DNN $n$. Accordingly, we denote $N$ target actor DNNs as $\mu_{n}^{(\text{a-})}\left(s_{n};\theta_{n}^{(\text{a-})}\right)$, ($n\in\mathbb{N}$), where $\theta_{n}^{(\text{a-})}$ is the corresponding weight vector. Then, we denote respectively the critic DNN and the target critic DNN as $Q\left(s_{1},\cdots,s_{N},s_{\text{o}},a_{1},\cdots,a_{N};\theta^{(\text{c})}\right)$ and $Q^{-}\left(s_{1},\cdots,s_{N},s_{\text{o}},a_{1},\cdots,a_{N};\theta^{(\text{c-})}\right)$, where $\{s_{1},\cdots,s_{N},s_{\text{o}}\}$ is referred to as the global state of the whole network including all the local states $s_{n}$ ($n\in\mathbb{N}$) and other global state $s_{\text{o}}$ of the whole network, $\{a_{1},\cdots,a_{N}\}$ is referred to as global action of the whole network including all the actions $a_{n}$ ($\forall\ n\in\mathbb{N}$), $\theta^{(\text{c})}$ and $\theta^{(\text{c-})}$ are the corresponding weight vectors.

Next, we detail the experience accumulation procedure followed by the MASC training method.

In this part, we first design the experience and the structure for each actor DNN and local DNN. Then, we design the global experience and the structure of the critic DNN. Finally, we elaborate the algorithm followed by some related discussions.

To begin with, we provide the hyperparameters of the DNNs in Table I, Table II, and Table III, which are determined by cross-validation [13] [14]. Note that the adopted hyperparameters maybe not the optimal. Since the proposed algorithm with these hyperparameters performs well, we use them to demonstrate the achievable performance rather than the optimal performance of the proposed algorithm. Besides, we pre-process the local/global state information of the proposed algorithm to reduce their variance in the following procedure: we first use the noise power to normalize each channel gain, and then use the mapping function $f(x)=10\log_{10}(1+x)$ to process the data related to the transmit power, channel gain, interference, and SINR.

In the simulations, we consider both two-layer HetNet scenario and three-layer HetNet scenario:

• •

  Two-layer HetNet scenario: In this scenario, there are five APs whose locations are respectively $(0,0)$, $(500,0)$, $(0,500)$, $(-500,0)$, and $(0,-500)$ in meters. Each AP has a disc service coverage defined by a minimum distance $\nu_{\text{min}}$ and a maximum distance $\nu_{\text{max}}$ from the AP to the served UE. AP $1$ is in the first layer and AP $n$ $(n\in\ \{2,3,4,5\})$ is in the second layer. $\nu_{\text{min}}$ is set to be $10$ meters for all APs, and $\nu_{\text{max}}$ of AP $1$ in the first layer is $1000$ meters, and $\nu_{\text{max}}$ of each AP in the second layer is $200$ meters. The maximum transmit power of AP $1$ in the first layer is $30$ dBm, the maximum transmit power of each AP in the second layer is $23$ dBm. The served UE by an AP is randomly located within the service coverage of the AP.

• •

  Three-layer HetNet scenario: In this scenario, there are nine APs whose locations are respectively $(0,0)$, $(500,0)$, $(0,500)$, $(-500,0)$, $(0,-500)$, $(700,0)$, $(0,700)$, $(-700,0)$, and $(0,-700)$ in meters. AP $1$ is in the first layer, and AP $n$ $(n\in\ \{2,3,4,5\})$ is in the second layer, and AP $n$ $(n\in\ \{6,7,8,9\})$ is in the third layer. $\nu_{\text{min}}$ is set to be $10$ meters for all APs, and $\nu_{\text{max}}$ of AP $1$ in the first layer is $1000$ meters, and $\nu_{\text{max}}$ of each AP in the second layer is $200$ meters, and $\nu_{\text{max}}$ of each AP in the third layer is $100$ meters. The maximum transmit power of AP $1$ in the first layer is $30$ dBm, and the maximum transmit power of each AP in the second layer is $23$ dBm, and the maximum transmit power of each AP in the third layer is $20$ dBm. The served UE by an AP is randomly located within the service coverage of the AP.

Furthermore, the transmission bandwidth is set to be $B=10$ MHz, the adopted path-loss model is $120.9+37.6\log 10(d)$ in dB, where $d$ in kilometer is the distance between a transmitter and a receiver [40], the log-normal shadowing standard deviation is $8$ dB, the noise power $\sigma^{2}$ at each UE is $-114$ dBm, the delay $T_{d}$ of the data transmission between the core network and each AP is $T_{d}=50$ time slots, the period $T_{u}$ to update the weight vector of each local DNN is $T_{u}=100$ time slots, and the capacity of the memory replay buffer is $M=1000$.

In this part, we will provide the performance comparison and analysis of the proposed algorithm with four benchmark algorithms in two simulation scenarios. In particular, the simulation of the proposed algorithm has two stages, i.e., training stage and testing stage. In the training stage, the DNNs are trained with the proposed Algorithm 1 in the first $5000$ time slots. In the testing stage, the well-trained DNNs are used to optimize the transmit power in each AP in the following $2000$ time slots. Each curve is the average of ten trials, in which the location of the served UE by an AP is randomly generated within the service coverage of the AP.

Fig. 6 provides the average sum-rate performance of the proposed algorithm and four benchmark algorithms in the two-layer HetNet scenario in the training stage. The channel correlation factor is set to be zero, i.e., IID channel. In the figure, the average sum-rate of the proposed algorithm is the same as the random power algorithm at the beginning of data transmissions since the proposed algorithm has to choose transmit power randomly for each AP to accumulate experiences. Then, the average sum-rate of the proposed algorithm increases rapidly and exceeds the sum-rates calculated by WMMSE algorithm and FP algorithm after around $500$ time slots. Finally, the proposed algorithm converges after $1500$ time slots. This can be explained as follows. On the one hand, both WMMSE algorithm and FP algorithm can only output sub-optimal solutions of the power allocation problem in each single time slot, meaning that the average sum-rate performance of both algorithms are also sub-optimal. On the other hand, the proposed algorithm can explore continuously different power control strategies and accumulates global experiences. By learning from these global experiences, the critic DNN has a global view of the impacts of different power control strategies on the sum-rate. Then, the critic DNN can guide each actor DNN (or each local DNN) to update the weight vector towards the global optimum. Thus, it is reasonable that the proposed algorithm outperforms both the WMMSE algorithm and FP algorithm in terms of the average sum-rate.

Fig. 7 provides the corresponding sum-rate performance of the proposed algorithm in the two-layer HetNet in the testing stage. From the figure, the effectiveness of the proposed algorithm in the two-layer HetNet is demonstrated. It should be noted that, the sum-rate performance of the proposed algorithm in the testing stage is also higher that of the proposed algorithm in the training stage. This is because, the proposed algorithm in the training stage needs to continuously explore the transmit power allocation policy and train DNNs until convergence. The exploration may degrade the sum-rate performance after the convergence of the algorithm. On the contrary, by completely exploiting the well-trained DNNs to optimize the transmit power, the proposed algorithm can further enhance the sum-rate performance in the testing stage.