Beam Selection for Energy-Efficient mmWave Network Using Advantage Actor Critic Learning

The network configuration used in this work consists of a 3D beamforming capable gNB and ${N_{UE}}$ stationary User Equipment (UE). We consider analog beamforming and assume a UE could experience line of sight (LOS) or non-line of sight (NLOS) propagation conditions. Finally, the presence of UDP traffic in the downlink direction is considered.

The 3D beamforming capability is represented by a regular GoB, that can be described through the set of its elevation angles $\Theta=\{\theta_{1},\theta_{2},…,\theta_{N_{\Theta}}\}$ and azimuth angles $\Phi=\{\phi_{1},\phi_{2},…,\phi_{N_{\Phi}}\}$. $\theta$ and $\phi$ are, respectively, the elevation and azimuth angles of the maximum gain in a beam. Each beam is then described by the pair $(\theta_{i},\phi_{j})$, where $i\in[1,N_{\Theta}]$ and $j\in[1,N_{\Phi}]$. $N_{\Theta}$ and $N_{\Phi}$ are the number of values that $\theta$ and $\phi$ could assume. The maximum number of beams available to the system, here defined as $N_{B}$, is $N_{B}=N_{\Theta}\cdot N_{\Phi}$. The full GoB, named $\Upsilon$, is obtained in terms of $\Theta$ and $\Phi$: if $\Upsilon=\Theta\times\Phi$, then:

The 3D GoB beamforming model is depicted in Fig. 1. The overhead in the beam selection process derives from the requirement to exchange pilot signals and evaluate the quality of each beam pair. In highly changing scenarios, it could impact the initial access and mobility procedures and consume channel utilization time. Due to this overhead, constraints on the use of $N_{B}$ beams in the beam selection could be imposed. A subset of the GoB composed of $K_{B}$ beams, where $K_{B}<N_{B}$, must be required in some cases. This constraint can be attended if we limit the number of elevation and azimuth angle options to a subset of $\Theta$ and $\Phi$: $\Theta^{\prime}$ and $\Phi^{\prime}$, where $\Theta^{\prime}\subset\Theta$ and $\Phi^{\prime}\subset\Phi$. Consider that $K_{B}=K_{\Theta}\cdot K_{\Phi}$, where $K_{\Theta}$ and $K_{\Phi}$ are, respectively, the number of options that each beam elevation angle $\theta$ and azimuth angle $\phi$ could assume after the constraint, so as $K_{\Theta}=|\Theta^{\prime}|$ and $K_{\Phi}=|\Phi^{\prime}|$.

The optimization of beam selection and transmission power is done using the RL framework A2C. The foundational goal in RL is to learn a policy based on the sequential actions and environment states that maximize a cumulative future reward. If an action has a high expected reward, it should be given a higher probability of selection for an observed state. A2C differs from other Deep Reinforcement Learning (DRL) frameworks because it proposes a clear separation between two tasks performed by the agent. Each block within the agent (actor and critic) has its dedicated neural network engine to perform the corresponding task.The critic’s task is to judge the actor’s decisions based on the current state of the environment and the outcomes provided as a reward. The actor’s task is to define the best action to be performed on the environment based on the current state and the policy values sent by the critic [17].

Two other versions of the MDP described above are evaluated as baselines. The first one, named A2C-P, intends to cover the case where only power optimization is conducted by the algorithm. In this case, ESB is used to define the constrained GoB. It uses the same state space and reward function presented above, however, the action space is defined as:

A second variant named A2C-B is proposed with the intent to cover the case where only beam selection optimization is conducted. In this case, the baseline power level is used. As A2C-P, it uses the same state space and reward function as A2C, but the action space is defined as:

Finally, we consider the use of ESB and fixed transmission power as a third baseline, covering the case of no dynamic optimization.

rApps are applications designed to automate and control RAN functions and run on top of the SMO platform. rApps are enabled by the Non-Real Time RAN Intelligent Controller (Non-RT RIC), an SMO entity, and operate in control loops of 1 second or more [15]. In accordance with the GoB optimization use case presented in [9], the approach proposed in this work may be deployed as an rApp in the SMO. Note that, the implementation as an rApp is feasible because the global GoB and power optimization proposed here do not require near real-time feedback and interventions.

Fig. 3-a) shows a potential deployment where the proposed solution is used as an rApp in the SMO for Open RAN architecture. In O-RAN, network data gathered by the Open Decentralized Unit (O-DU) is sent to the SMO and Non-RT RIC Framework through O1 interface. Non-RT RIC Framework uses R1 communication to transfer the data to the rApp.

In our case, the A2C-based GoB and power configurations (actions) can be applied to the Open Radio Unit (O-RU) through Open FH M-Plane interface, and to O-DU through O1 interface. Note that, Non-RT RIC rApps are not limited to the Open RAN environment. With the use of proper interfaces, similar deployment could be done on top of Cloud RAN or purpose-built network architectures.

Fig. 3-b) details the functioning of the proposed scheme when it is developed as an rApp. The algorithm configuration is done based on $\Theta$, $\Phi$, $\omega_{C}$, $K_{\Theta}$ and $K_{\Phi}$. In each loop, data collected from O-DU will enable the reward function computation (feeds the critic) and the state report (feeds the actor and the critic). The action selection will be done by the actor and then sent as an rApp policy to be applied to O-RU and O-DU.

We implement our proposed scheme using the 5G LENA module for ns-3 [16]. The main system parameters for the simulation environment are summarized in Table I.

Each simulation is set to last 150 ms, where only 100 ms are used for data traffic purposes. Different scenarios were set according to the intended number of UE, since $N_{UE}=\{5,10,15,20\}$. The results shown in Section V refer to the average value and confidence interval of 40 different seed runs for each scenario configuration. In each run, new channel parameters and UE distribution are set.

A regular GoB where $N_{\Theta}=3$ and $N_{\Phi}=5$, so as $N_{B}=15$, is considered here. The set of available angles for each axis are the following: $\Theta=\{-15^{\circ},-45^{\circ},-60^{\circ}\}$ and $\Phi=\{0^{\circ},45^{\circ},90^{\circ},135^{\circ},180^{\circ}\}$. $K_{\Theta}=1$ and $K_{\Phi}=3$ are defined considering a constraint of $K_{B}=3$. For the A2C based approach, consider $L=3$ and $M=10$. For the ESB strategy with $K_{B}=3$, $\Theta^{\prime}=\{-45^{\circ}\}$ and $\Phi^{\prime}=\{0^{\circ},90^{\circ},180^{\circ}\}$ are considered. Both ESB strategies use the baseline power level ($P_{B}$), so as A2C-B. $P_{B}$ is set as 35 dBm. Five power level change options $\delta$ are defined, so as $N_{P}=5$. The set of power level change options is the following: $\Delta=\{-3,-1.5,0,+1.5,+3\}$ dB.

A2C implementation was done using the python package Stable Baselines 3 [19]. Each A2C episode corresponds to a new simulation round, fed with the action set by A2C, and that returns the states and reward values needed to feed the RL algorithm’s next iteration. The convergence curve for the reward values obtained by A2C in the case where $N_{UE}=5$ is depicted in Fig. 4.

Fig. 5 shows the EE values obtained for the proposed approach (A2C) when compared to the ESB for $K_{B}=3$ and $K_{B}=15$. The cases with only the power or the beam subset optimization (A2C-P and A2C-B) are also shown. $K_{B}=15$ represents the case where $\Upsilon^{\prime}=\Upsilon$, and the beam search over the full GoB is performed.

A2C outperforms ESB for the same number of beams. More importantly, it can improve EE even when compared to the case of greater $K_{B}$ with no power optimization. The ratio of $\varepsilon_{MAX}$ achieved by each strategy is summarized in Table II.

For the same number of beams ($K_{B}=3$), the use of A2C does not cause a significant impact on throughput, as shown in Fig. 6. The results for the complete original GoB ($K_{B}=15$) are shown to demonstrate the achievable performance when no constraints to $N_{B}$ are applied. In the case of throughput, $K_{B}=15$ achieves higher values due to the gains in SINR derived from the use of more beams. However, it is not compliant with the constraints defined in our problem ($K_{B}=3$). $K_{B}=15$ with fixed power is also less energy-efficient than the proposed solution.

Fig. 7 shows that A2C could converge to minimize transmission power at the gNB. If only power optimization is performed (A2C-P), throughput is not improved, and the power levels in use are diminished but are higher than for the global A2C case (Fig. 7). Thus, A2C-P presents poor EE improvement. When only the beam subset optimization is performed (A2C-B), throughput is improved, but since the power is fixed and not optimized, EE gains are discrete. Moreover, A2C-P and A2C-B have similar performances on EE. It is the joint optimization of beam selection and transmission power that produces the best EE results. The proposed approach leveraged the gains from the beam subset optimization with throughput to minimize transmission power and improve EE at the gNB.

As shown in Section IV, the reward function of the proposed approach is designed to improve EE and avoid the presence of coverage holes. Fig. 8 shows how the proposed approach reduces the number of UE with $\hat{\Gamma}<0$ in the scenario when compared to the ESB case with the same number of beams. It happened even though the use of A2C translated into lower levels of power applied in the system.