Online Learning for Autonomous Management of Intent-based 6G Networks

We develop Algorithm 1 with which each CL determines its proposal arm independently. Besides, child agents (CLs) are not allowed to propose every possible arm but the suggested set of arms, which we call active arms, coming from parent agent.

At each phase $p$, there can be a new information denoted as $\tau(p)$ in which a KPI target or number of UEs with a service can change and parent agent provides a set of network configurations that child CLs can propose. Also, child agents takes its associated KPIs with the target value as input (Line 1-2). As the system is dynamic, with changes in the number of UEs or targets in a given phase, child agents are informed of these changes while online learning is ongoing. Each child agent pulls arm from an active set determined denoted as $K_{i}$ by the parent agent and predicts the reward of the proposed action (Line 3-7) by averaging several trials. We note that $\mu_{k,n}^{i}$ is the expected reward of child CL $i$ when it plays arm $k$ at trial $n$. At Line 5, the child agent takes what others can propose as a random factor due to blindness of the child agents. At Line 6, it uses a prediction model (i.e., a machine learning model) $f_{predictor}$ to predict the reward of the proposed arm $k$. $f_{predictor}$ takes the KPI target, arm of the child agents, and other network dynamics such as number of UEs as input, and it predicts the expected KPI value for the given inputs. We use a multi-layer perception for our machine learning model. This process happens multiple times by randomizing the arms from other agents, and the expected score is assigned to the proposed arm $k$ of the child agent (Line 9). This predictor is obtained based on partially observable system before learning process of system starts. The scores for each arm from the active set are returned in vector form to the parent agent which evaluates the effectiveness of the arm from each child agents.

In our MABCR structure the operations of the parent agent is given in Algorithm 2, and the parent agent acts as an evaluation module and makes decisions from an overall perspective.

Initially all possible combinations of child arms are considered as potential final arm that can be applied to the real network. The set $A$ denotes the active set, and an element of set $A$, $a$, is called as parent arm which is one unique combination of child arms. For example, in the case where there are only two child agents with each having $K$ arms, $a$ is a vector including ($P_{1}^{k},P_{2}^{l}$) where $P_{1}^{k}$ and $P_{2}^{l}$ are the particular arms of child agent 1 and 2 respectively where $k,l\in\{1,2,..,K\}$. $Q(a)$ and $N(a)$ are the reward and number of times the arm $a$ played up to time $t$, which are initially set to zero. Also, Algorithm 2 takes the number of child agent $N$, possible active set of arms $A$, the KPI targets of services $\gamma_{target}$. Alg. 2 first checks if there is a new information at a new phase denoted by $\tau(p)$. At this new phase $\tau(p)$, as an example, a KPI target or number of UEs with a service can change and hence we check whether if there is such a change at Line 7-10. If there is a change, the active set is re-initialized, and reset to all possible combination of child arms (Line 11). Parent agent then starts to collect the expected reward from child agents with Algorithm 1 (Line 13). Then the parent agent calculates the weighted average reward of each arm $a$ (Line 14) and $N(a)$ is updated. We note that $w_{i}$ is the weight of intent expectation $i$. At Line 16, by using an Upper Confidence Bound (UCB) algorithm, the arm that maximizes $F(a,t)$ is selected and added to the set $A$. We note that $F(a,t)$ does not only take into account the average reward but also the uncertainty with the arm $a$. Until the parent agents makes a final decision on the arm to be applied to real network (Line 26), the process continues and the set $A$ is generated. Also, the arm with minimum reward is removed in the active set (Line 22). The method we are using to create active set over eliminations is based Fed2-UCB algorithm given in [11]. Alg. 2 keeps the arms that maximizes reward and minimizes the uncertainty, and it eliminates the arms with minimum reward. At final stage, the only the arms that manage to be kept at the set $A$ are sent back to the child agents.

This elimination process continues until there is only one element remaining in the active set (as described in Lines 21-22) and at the decision point $t_{decision}$, parent agent selects the arm in the set $A$ that maximizes the weighted average reward (Line 26-27). Ultimately, the parent CL continuously proposes the best possible arm based on the current network configurations, while the online algorithm strives to converge to the optimal set of arms.

We carried out simulation experiments to assess our proposal and benchmark it against different baseline algorithms. The simulation environment, designed in Python, is based on a in-house network emulator. The network emulator consists of containerized entities including network functions, UEs, and application instances. These individual components are implemented as web servers providing a set of APIs enabling configuration management. Based on these capabilities, the set of actions (arm) that can be executed to reconfigure the network is as follows:

Priority: Changing the priority of a service instance that reconfigures how much resource a service can take.

MBR (Maximum Bit Rate): Changing the maximum throughput that can be achieved by a particular UE.

We implemented two different service types in the network emulator that are: (1) Conversational video and (2) Ultra-reliable low latency communication (URLLC). For the URLLC service, the primary KPIs are latency and reliability (i.e., packet loss). Conversely, conversational video represents an enhanced mobile broadband (eMBB) service, which is not highly sensitive to latency. Instead, this type of service is evaluated using Key Quality Indicators (KQIs) in addition to KPIs to assess user-perceived performance. Therefore, we define Quality of Experience (QoE) as the target KPI, based on both network and application layer metrics.

We first run our in-house network emulator under different configurations and collected realistic data from the emulator, and create a large data set including the actions, the throughput, latency, QoE values for the UEs of different services. First we observed that there is a good correlation between the achieved throughput and the target services KPIs, QoE and latency. Hence, one can estimate the target KPIs from the achieved throughput. To do that, by using this data set, we next attempt to learn the functional relations between latency and the achieved throughput for URLLC service denoted as $f_{u}(th)$, where $th$ is the achieved throughput by a UE of URLLC service and $f_{u}(th)$ represents the latency of the URLLC UE for a given throughput $th$. Also, we find the function $f_{v}(th)$ representing the relation between QoE and the achieved throughput for video service. Those functions are derived by using simple polynomial approximation by using the data set as the relations are not linear. Once we learn these functions and the achieved throughput, one can easily estimate the KPI values. Later, those functions are used in our simplified simulation environment where we test our approach in the rest of this paper, and utilize these functions to learn how much throughput each service needs to achieve to satisfy their KPI requirements such as latency and QoE.

In our simplified environment developed in Python, the throughput of a UE with service $s$ is calculated as follows:

where $B$ is the total system bandwidth. We set $B$ to 10 Mbps, and $MBR$ to 3 Mbps for all UEs in our setup. Also, $P_{s}$ is the priority level of service $s$. In order to be more realistic and also to understand how efficient our proposal is in terms of resource allocation, we also implemented a physical layer model where Physical Resource Blocks (PRBs) are determined based on channel condition. Specifically, for each UE, the Quality Class Identifier (QCI) parameters of channels are randomly generated within a range of 0 to 15 and mapped to the corresponding Modulation and Coding Scheme (MCS) and spectral efficiency values as outlined in Table I. This table presents the Channel Quality Indicator (CQI), MCS index, and spectral efficiency values in 5G New Radio (NR) when the number of layers is two. Under the given channel conditions, the amount of PRBs required can be calculated as [12]:

where $th$ represents the achieved throughput for the UE of a service, and RB represents an adjacent group of 12 subcarriers, where each subcarrier has a bandwidth of 15 KHz, and $SE_{i}$ is the spectral efficiency value of the i-th user calculated based on Table I.

In summary, the control action (i.e., arm) of each services or CL is the service priority levels dynamically determined by our MABCR online algorithm. First, our approach using Algorithm 1 and 2 provides the priority levels, $P_{s}$, for the services (MBR are fixed for all UEs), then we find the throughput, $th$, of the services with the suggested priority levels by our MABCR according to eq. (4). Then, by using the learnt functions $f_{u}(th)$ and $f_{v}(th)$, we determine the achieved KPI levels (latency and QoE). Moreover, by using eq. (5), we decide on how many PRBs are needed to achieve the required throughput level. Later, we asses the efficiency of our solution based on the number of the allocated PRBs. One of our future aim is to make the complete implement and full integration of our approach to our in-house network emulator.

Our experiments compare the performance of three approaches: i-) our MABCR online learning method with UCB, ii-) the greedy child’s approach, and iii-) the MABCR online learning with Thompson sampling (TS). The greedy child’s policy is another version of our MABCR-Online Learning without a parent/evaluator modules. Hence, each child aims to achieve its target greedily and without a holistic control mechanism. The MABCR online learning with Thompson sampling is similar to our proposed approach except the acquisition function [13]. Thompson sampling selects each arm randomly based on its probability of being the optimal choice. On the other hand, UCB in MABCR online learning is designed to find a balance between the most beneficial arms and taking uncertain actions that could potentially have a high reward.

We assume that there are intents containing the KPI expectation of video and URLLC services. For example an intent of URLLC service may indicate an expectation on latency, i.e., latency should be less than 30 ms. Importantly, the intent expectations can vary over time and the network is supposed to adapt these variations. To evaluate the performance of the algorithms, two key measures are used: the total number of SLA violations (deviation from intent expectations) and the amount of allocated PRBs. The cumulative number of SLA violations is calculated by adding up all deviations from the service targets. To effectively quantify the performance of our approach, we create three different dynamic scenarios where the number of UEs with both services and their associated QoE and latency targets vary over time. These scenarios are outlined in Table II, III, and IV, respectively. The total bandwidth, $B$, is set to 10 Mbps and MBR of each UE is set to 3 Mbps.

Specifically, in scenario I given in Table II where $\gamma$ and $\eta$ stands for QoE and latency targets respectively, and there are 5 UEs for both video and URLLC services and delay requirement for URLLC service is set to 60 ms. However, QoE expectation (i.e., target) of each UE of the video services varies at every 20 seconds. For Scenario 2 described in Table III, we now fix the QoE target of video service to 3.4 and changes the delay requirement of URLLC service between 60 and 40 ms. For scenario I and II, we have five phases where at each phase we have a different QoE or delay requirements, and each phase takes 20 seconds in the simulation. At the last scenario with ten phases with 10 seconds of simulation time and given in Table IV, we take a more challenging case where both QoE and delay requirements of video and URLLC services vary over time, respectively.

The horizontal axis in the figures represents the algorithm’s decision stages. The duration parameter in the table is determined based on the total simulation time of 500 loop duration. Multiple sample are conducted to obtain an ensemble average to ensure reliable results and mitigate the impact of randomness. A t-test is used to generate a 95% confidence interval for the cumulative SLA violations graphs.

As illustrated in Figure 2, the algorithms are assessed in terms of QoE changes, which become increasingly challenging with each round. Here, achieving latency targets appears more feasible compared to satisfying QoE targets simultaneously. As a result, the algorithms must compromise on the attainable targets to manage this trade-off effectively. Our proposed algorithm shows the most efficient resolution for this trade-off, managing the QoE satisfaction and latency targets in an adaptive way.

The TS approach seeks to find a compromise between QoE and latency targets. However, as evident from the results in Figure 2, it falls short of achieving a good solution as our proposed algorithm. On the other hand, the greedy approach tends to deviate more compared to the other methods. Notably, greedy child’s focus solely on one target, giving it full importance without considering the trade-off between services. From a single service’s perspective, the greedy approach might appear more successful than TS. However, when evaluating the overall cell of the network, our proposed algorithm outperforms both the greedy child’s and TS approaches in handling the trade-off between QoE and latency targets.

When the targets change, the algorithm adjusts the arms accordingly. In some cases, they may continue with the same arms, incurring minor violations, while in other instances, they may modify their arms slightly after target alterations. However, the exploration-exploitation dilemma faced by online learning algorithms introduces some perturbations in arm selection. The parent, serving as an evaluation module, plays a stabilizing role by considering all system circumstances, allowing the arms to remain consistent for consecutive rounds. As a result, the allocation of PRBs remains more stable in our learning algorithms with parents, in contrast to the greedy child case. Without the evaluator module, we observe a significant level of fluctuation caused by unstable decisions, hindering the discovery of mutually beneficial solutions for our resource allocation problem, as evident in the Figures showing PRB allocation .

Figure 2, 3 and 4 also show that the PRB allocation of the greedy approach fluctuates, causing excessive resource occupation. Our proposed algorithm’s allocation has a small margin compared to the ideal allocation, as seen in Figure 2, 3 and 4. However, in some cases, the TS approach may deviate from the ideal allocation, resulting in violations due to insufficient resource deployment. The margin between these algorithms and the ideal allocation is mainly due to the granularity of bandwidth sharing for each service. Moreover, we add a 0.5 Mbps margin to the ideally required total bandwidth for each phase, and we convey this information to the parent to ensure more realistic constraints.

The greedy child’s approach behaves selfishly, attempting to satisfy all service level agreements without any authority like a parent or evaluator. On the other hand, the results for the MABCR without UCB approach demonstrate the need for an acquisition function to explore the environment and resolve conflicts optimally. Our proposed method offers a comprehensive approach that addresses conflicts between service targets and resource allocation, particularly in terms of PRB allocation.