Attention-based SIC Ordering and Power Allocation for Non-orthogonal Multiple Access Networks

In addition to wireless channel quality, some works have proposed using a static decoding order based on other performance metrics specific to certain problems and scenarios. For example, the descending order of received user signal power [19, 17, 20], the descending order of predicted user throughput [29], the decreasing order of channel gain normalized by noise and interference power [30, 31], the ascending order of average channel gain from user to the base station [32], and the ascending order of a user’s maximum secrecy throughput [18]. For the uplink NOMA scenario, [18] adopted the ascending order of user’s maximum secrecy throughput as the SIC ordering to achieve secrecy transmission in eavesdropper scenarios. [19] used the descending order of received user signal power as the SIC ordering and derived the closed-form expression of the outage probability for a single base station and three-user system. Building on this work, [20] considered the single base station and multiple active user scenario in the case of imperfect channel state information, using the descending order of estimated instantaneous received signal power as the SIC ordering. Although these static SIC ordering methods can achieve excellent performance with small execution latency, they may suffer considerable performance degradation on complex NOMA networks due to the difficulty of finding a suitable metric for ordering.

In addition to works that use a static SIC ordering based on specific problems and scenarios, a few studies have proposed iteratively searching heuristic algorithms to optimize the SIC ordering. For instance, [3] and [14] introduced binary variables to represent the SIC ordering, solving the problem via variable relaxation with compensation for performance degradation and as an integer linear programming problem, respectively. [33] adapt a permutation-based genetic algorithm to optimize the SIC ordering. [21] used the greedy meta-scheduling technique to develop a low-complexity and easy-to-implement SIC ordering algorithm. This algorithm sequentially inserts users into the existing ordering, tries every possible insertion position for each user, and chooses the position that provides the most significant benefit. [34] utilized a heuristic tabu search to optimize the SIC ordering in an iterative process. At each iteration, tabu search swapped the SIC ordering of any two users and selected the best one. To the best of our knowledge, [34] is the state-of-the-art algorithm for dynamic SIC ordering, it achieves the near-optimal performance, but suffers from high computational complexity due to the large number of iterations. While these dynamic SIC ordering methods apply to most NOMA wireless scenarios and achieve better performance than static SIC ordering, they all involve iterative updates and repeatedly solve resource allocation subproblems, resulting in high computational complexity.

As shown in Fig. 1, we consider an uplink NOMA network with a central-located BS and $N$ active users, denoted as a set $\mathcal{N}=\{1,2,\dots,N\}$, where each user has a single antenna. Users have a stable power supply, and all $N$ users simultaneously transmit their information to the BS by NOMA.

The system time is divided into consecutive slots of equal length, smaller than the channel coherence time. We assume that the wireless channel gain is constant in each time slot and may vary across different slots. Without loss of generality, the slot length is normalized for brevity.

The BS employs SIC in a successive order to decode users’ signals. In our framework, we define a function $\xi(n)=i$ and its inverse $\pi(i)=n$, which establish a mapping between a user’s index $n$ and its corresponding decoding order $i$. For instance, $\xi(3)=2$ indicates that the user 3 is decoded second in the sequence. Conversely, $\pi(2)=3$ indicates that the second user to be decoded in the order is user 3. This bi-directional mapping functions serve to clearly delineate the relationship between the decoding sequence and the specific users in the network. Hence, the signal-to-interference-plus-noise ratio of the user $n$ can be expressed as

where $N_{0}$ is the power of the additive Gaussian noise at the BS, $g_{n}$ denotes the wireless channel gain between user $n$ and the BS at a tagged time slot, and $p_{n}$ denotes the transmit power of user $n$ sending its information. So the data rate of user $n$ can be expressed as

where $B$ denotes the communication bandwidth.

We summarize essential notations used throughout this paper in Table I.

This paper aims to achieve weighted proportional fairness across multiple users. Proportional fairness can be achieved by maximizing individual rates with a logarithmic utility function [48]. In addition, considering users’ different priorities, individual weights are assigned to each user to achieve weighted proportional fairness [49, 50, 51]. Thus, the aim is to maximize the weighted sum of the logarithmic throughput of different users by jointly optimizing the SIC ordering $\bm{\pi}$ and the power allocation p, denoted as the network utility $R(\bm{\pi},\textbf{p})$. This optimization problem can be expressed mathematically as:

where $w_{n}$ is the weight of user $n$. $\bm{\pi}=[\pi_{1},\pi_{2},\cdots,\pi_{N}]$ indicates the SIC order, where we denote ${\pi_{i}}=\pi(i)$ for brevity. ${\Pi}$ is the permutation set of all possible SIC orderings with size factorial $N$, represented as $N!$. $\textbf{p}=[p_{1},p_{2},\dots,p_{N}]$ is the power allocation. (3b) is the power constraint for each user $n$, where $P_{n}^{max}$ is the maximum power that user $n$ can achieve. (3c) is the constraint for $\bm{\pi}$.

The problem P0 involves combinatorial optimization and continuous numerical optimization, which is NP-hard. To effectively solve the problem P0, we decompose it into the SIC ordering optimization and the optimization of power allocation under the given SIC ordering, as shown in Fig. 2:

• •

  SIC Ordering: It is computationally expensive to iteratively search for the optimal SIC ordering from $\Pi$ at the $N!$ scale. We tell that one SIC ordering outperforms another one by solving the power allocation problems P1 and comparing their utilities $R(\bm{\pi})$. However, classical comparison-based sorting algorithms cannot perform better than $O(n\log n)$ on average [52], which requires repeatedly solving P1, resulting in long execution latency. In this paper, the deep reinforcement learning method is adopt to generate the SIC ordering before the power allocation.

• •

  Power Allocation: When the SIC ordering $\bm{\pi}$ is determined, we only need to solve the power allocation $\mathbf{p}$, as follows:

  	$\displaystyle\textbf{P1}:R(\bm{\pi})=\max\limits_{\textbf{p}}\$	$\displaystyle\sum\limits_{n=1}^{N}{w_{n}\ln{R_{n}}}$		(4a)
  	$\displaystyle s.t.\$	$\displaystyle 0<p_{n}\leq P_{n}^{max},\forall n\in\mathcal{N}.$		(4b)

  We can solve this power allocation sub-problem P1 by converting it to a convex problem and using the inter-point method.

In the next Section, these two subproblems are solved by taking advantage of DRL and convex optimization.

Fig. 3 shows the schematics of the proposed ASOPA framework. It uses an encoder-decoder-based actor network $\theta$ to generate the SIC ordering sequentially and uses optimization techniques to optimize all users’ transmit power. The design of the actor network follows from the pointer network [54, 53] for routing problems. In Fig. 3, the black solid lines depict the inference process of ASOPA, which necessitates real-time network parameters. These parameters include each user’s weight $w_{n}$, maximum power $P_{n}^{max}$, and channel gain $g_{n}$. The complete set of users’ information, denoted by $\mathbf{X}=\{(w_{i},P_{i}^{max},g_{i})\}_{i\in\mathcal{N}}$, is fed into the actor network to generate the SIC ordering $\bm{\pi}$. Following the determination of the SIC ordering $\bm{\pi}$, we solve the sub-problem P1 for the optimal power allocation $\mathbf{p}$ through convex optimization. Then, ASOPA outputs the network decision, represented as $(\bm{\pi},\mathbf{p})$, based on the instantaneous user information $\mathbf{X}$.

The red dotted lines in Fig. 3 illustrate the policy update process in ASOPA. We employ a replica of the actor network, referred to as the baseline network $\theta^{\prime}$, and train the actor network using the REINFORCE algorithm. Each users’ information $\bf{X}$ is fed into both the actor and baseline networks. This process yields the output SIC orderings $\bm{\pi}$ and the baseline SIC orderings $\bm{\pi}^{BL}$. Subsequently, $R\left({\bm{\pi}|{\bf{X}}}\right)$ and $R\left(\bm{\pi}^{BL}|\bf{X}\right)$ are obtained by solving problem P1, which are used to compute the loss function. The backpropagation method is employed to update the parameters $\theta$ of the actor network. The procedures of ASOPA are detailed in the following subsections.

As shown in Fig. 3, the SIC ordering $\bm{\pi}$ is generated by the actor network composed of an encoder and a decoder as follows:

1. 1.

   The encoder takes users’ information $\mathbf{X}=[\mathbf{x}_{1},\mathbf{x}_{2},\dots,\mathbf{x}_{N}]$ as input and outputs users’ embedding $\mathbf{E}=[\mathbf{e}_{1},\mathbf{e}_{2},...,\mathbf{e}_{N}]$ using self-attention layers. The global embedding ${\overline{\mathbf{e}}}$ is then calculated as the average of $\mathbf{E}$.

2. 2.

   The decoder generates the SIC ordering in an iterative process. At each iteration $t$, by utilizing the cross-attention layers, the decoder generates the probability of all users according to $[{\overline{\mathbf{e}}},{\mathbf{e}}_{\pi_{t-1}}]$ and $\mathbf{E}$. By masking the previously selected users, the probability of the remaining users being selected is calculated using the softmax function, allowing the decoder to determine the current user’s index $\pi_{t}$. At the end of each iteration, the decoder updates users selected for masking and takes ${\mathbf{e}}_{\pi_{t}}$ as input to next iteration. It iterates $N$ times to obtain the complete SIC ordering $\bm{\pi}=[\pi_{1},\pi_{2},...,\pi_{N}]$.

The integration of an attention scheme in the encoder and an iterative decoding scheme in the decoder empowers ASOPA to effectively manage a varying number of users in NOMA networks. This approach ensures adaptability and responsiveness to user dynamics, maintaining optimal performance across diverse network scenarios.

In this subsection, we design a convex transformation algorithm for the power allocation problem under given SIC ordering. Referring to Fig. 3, the actor network of ASOPA only generates SIC ordering without considering transmit power allocation and the corresponding constraints. To obtain the corresponding power allocation and evaluate the SIC ordering, we solve the power allocation subproblem and obtain the achieved network utility as follows.

Upon determining the SIC ordering $\bm{\pi}$, the sub-problem P1 is addressed to identify the optimal power allocation satisfying the constraints and subsequently calculate the network utility $R(\bm{\pi}|\mathbf{X})$ for the specified SIC ordering. Given that the data rate $R_{n}$ for each user is non-convex, P1 is inherently a non-convex problem. To tackle this, we employ variable substitution to transform P1 into a convex problem. The details of this transformation and the proof of its convexity are provided in Appendix B. We solve the transformed version of P1 using the interior-point method [56] of the CVX solver. The solution of P1 yields the optimal power allocation $\mathbf{p}$ under the given SIC ordering $\bm{\pi}$ satisfying the constraints. Following this, ASOPA outputs the network decision $(\bm{\pi},\mathbf{p})$ based on the instantaneous user information $\mathbf{X}$.

Consequently, the network utility $R(\bm{\pi}|\mathbf{X})$ can be calculated. This calculated utility provides essential feedback on the effectiveness of the SIC ordering under the current policy. As such, the resource allocation module acts as a critic in training the actor network, playing a crucial role in evaluating these generated SIC orderings.

Notice that ASOPA can be migrated to any other NOMA optimization problem with the required resource allocation problem. The extensibility of ASOPA is discussed in following Sec. 5.

The red dotted lines in Fig. 3 represent the policy update process. For an input instance $\mathbf{X}$, our goal is to maximize the expected sum of weighted users’ logarithmic throughput, as

where $z_{\theta}\left(\bm{\pi}|\mathbf{X}\right)=\prod_{t=1}^{N}\ell_{t,\pi_{t}}$ is a measure of the likelihood that the generated SIC decoding order $\bm{\pi}$ is the optimal sequence given the current network state $\mathbf{X}$.

To explore more SIC orderings in the training phase, at the selection module of the decoder network, the user based on the probability $\bm{\ell}_{t}$ at the iteration $t$ is sampled [53] as

After iterating $t\in\left[1,2,...,N\right]$, we obtain the SIC ordering $\bm{\pi}$. Then, the gradient of the actor network $\theta$ can be formulated by the REINFORCE algorithm [57] as

However, the REINFORCE algorithm may be of high variance and thus produce slow learning [59]. To improve the performance of DRL, the REINFORCE algorithm with baseline [59] is adopted in this paper. As illustrated in Fig. 3, the baseline network $\theta^{\prime}$, which uses the actor network parameters from the previous epoch, serves as a baseline to generate SIC orderings $\bm{\pi}^{BL}$ and subsequently calculates the network utility $R(\bm{\pi}^{BL}|\mathbf{X})$. The difference between $R(\bm{\pi}^{BL}|\mathbf{X})$ and $R(\bm{\pi}^{BL}|\mathbf{X})$ is utilized to train the current actor network $\theta$.

Consequently, the gradient of the REINFORCE algorithm with baseline can be expressed and then approximated by Monte Carlo sampling as

where $|\bm{\tau}|$ is the batch size, $\mathbf{X}_{i}$ is the $i$-th input, $\bm{\pi}_{i}$ is the SIC ordering produced by the actor network based on (15), and $\bm{\pi}_{i}^{BL}$ is the $i$-th SIC ordering produced by the baseline network. In practice, the expectation in Equation (17) is approximated by averaging over a batch of uniformly sampled input instances ${\left\{{{{\bf{X}}_{\tau}}}\right\}_{\tau\in\bm{\tau}}}$, where $\bm{\tau}$ represents the index set of these sampled instances. After obtaining the gradients (17), Adam [58] is applied as the optimizer to update the actor network’s parameters ${\theta}$.

Our reinforcement learning algorithm for training the actor network is outlined in Algorithm 1. To facilitate this process, an empty memory with limited capacity is established to store past samples. As new samples are received in each time slot, policy updates are executed infrequently. For every policy update, a random batch of samples is selected from this memory to train the actor network. The baseline network, on the other hand, undergoes updates every epoch which consists of $M$ times policy update. This update process involves copying the parameters from the actor network to the baseline network, as denoted by $\theta^{\prime}=\theta$. This systematic approach ensures continuous adaptation and optimization of the actor network’s performance based on the latest data.

ASOPA operates through two distinct processes: the inference process and the policy update process. During each time slot, ASOPA’s inference process is activated to generate Successive Interference Cancellation (SIC) orderings and power allocations. Contrarily, the policy update process can be carried out less frequently, and can also be executed in parallel on different servers in practical applications. Given the crucial role of inference delay in determining the feasibility of field deployment, the inference complexity of ASOPA is a key area of interest.

The inference process is detailed in lines 4-5 of Algorithm 1. Line 4 involves generating the SIC ordering from the actor network, where the computational complexity is primarily driven by matrix multiplications in the attention mechanism. The complexity of interactions between the query, key, and value is $O(HDN^{2})$, with $H$ representing the number of heads in multiple attention mechanisms, $D$ the dimension of these components, and $N$ the number of users. Line 5 addresses the generation of power allocation by solving subproblem P1, which is reformulated into a convex problem P2 and solved using the cvxopt solver with the Interior Point Method. The computational complexity of this method is $O(N^{3.5})$ [60]. Therefore, the overall inference complexity of ASOPA is $O(N^{3.5})$. In Section 6.3, we will numerically demonstrate that ASOPA meets the real-time requirements of NOMA networks.

In this subsection, we evaluate ASOPA in NOMA networks with multiple-antenna setups, since multiple-antenna technology has advantages in improving spectrum and energy efficiency [61, 62]. The uplink multiple-antenna system consists of a ${M_{r}}$ antennas BS and $N$ single-antenna users as shown in Fig. 5. The received signal ${\bf{Y}}\in{\mathbb{C}^{{M_{r}}\times 1}}$ of BS is

where ${\bf{H}}\in{\mathbb{C}^{{M_{r}}\times N}}$ denotes the channel state from users to the receive antennas of BS, ${\bf{S}}={\left[{{s_{1}},{s_{2}},...,{s_{N}}}\right]^{T}}$ denotes the transmit signal matrix, and ${\bf{n}}\sim{\cal{CN}}\left({{\bf{0}},{\sigma^{2}}{\bf{I}}}\right)$ represents the additive white Gaussian noise at the BS side. ${\cal{CN}}(0,\sigma^{2})$ denotes the complex Gaussian distribution with mean zero and the variance $\sigma^{2}$. The linear equalization, such as zero-forcing (ZF) or minimum-mean-square-error (MMSE), is used in multiple-antenna scenarios to symbol-by-symbol detection. According to the states of multiple-antenna scenario, ASOPA correspondingly generates the SIC ordering and power allocation. The specific steps are as follows.

The equalization matrix ${\bf{V}}\in{\mathbb{C}}^{{N}\times{M_{r}}}$ for ZF [61] or MMSE [62] can be expressed as [63]

where ${\bf{P}}$ denotes the diagonal matrix ${diag}(p_{1},p_{2},…,p_{N})$.

Then the received estimated signal is

and the estimated value of user $n$ is

where ${{\bf{h}}_{n}}\in{\mathbb{C}^{{M_{r}}\times 1}}$ denotes the channel states from user $n$ to the receive antenna of the BS, and ${\bf{v}}_{n}\in{\mathbb{C}^{1\times{{M_{r}}}}}$ denotes the $n$-th row of ${\bf{V}}$, which can be given by

According to the estimated value of user $n$, its achieved transmit rate can be calculated by

When using ZF method, the equalization matrix $\bf{V}$ in (22) is independent of $\bf{p}$, so that the power allocation problem can be transformed into a convex as appendix E and solved by the CVX solver.

However, when using the MMSE method, the equalization matrix $\bf{V}$ depends on the variable $\bf{p}$, making the power allocation problem of MMSE intractable and non-convex. To address this difficulty, alternative optimization is employed to decompose the non-convex problem into two subproblems: one for $\bf{V}$ and one for $\bf{p}$. When $\bf{V}$ is fixed, the power allocation problem can be transformed into a convex problem, as shown in Appendix E. Therefore, the alternative optimization starts with an initial power allocation to calculate the corresponding equalization matrix. Using this equalization matrix, it then applies the convex method to determine a new power allocation. With this updated power allocation, the equalization matrix is recalculated. This process repeats until the difference between successive power allocations is smaller than the set threshold.

Different from the single antenna scenario, the channel gain between the BS and a user $n$ is a complex value, whose real and imaginal parts are denoted as $\bm{h}^{r}_{n}=\{h^{r}_{1,n},...,h^{r}_{M_{r},n}\}$ and $\bm{h}^{c}_{n}=\{h^{c}_{1,n},...,h^{c}_{M_{r},n}\}$, respectively. To tackle multiple-antenna scenarios, ASOPA modifies its input from ${\bf{X}}={\{({w_{n}},P_{i}^{max},{g_{n}})\}_{n\in{\cal N}}}$ to ${{\bf{X}}}={\{({w_{n}},P_{n}^{max},{\bm{h}^{r}_{n}},\bm{h}^{c}_{n})\}_{n\in{\mathcal{N}}}}$. The rest of the ASOPA structure remains the same as one illustrated in Fig. 3.

In this subsection, we assess the impact of estimation errors on the performance of ASOPA. For the perfect CSI scenario, the channel gain is expressed as $g_{n}=\overline{g}_{n}|\alpha_{n}|^{2}$, where $\overline{g}_{n}=A_{d}\left(\frac{3\cdot 10^{8}}{4\pi f_{c}b_{n}}\right)^{b_{e}}$ and $\alpha_{n}~{}\sim~{}{\cal{CN}}\left({0,1}\right)$ account for path loss power gain and the Rayleigh fading channel coefficient between BS and $n$-th user, respectively. Since the path loss coefficient are large-scale fading factors and are slowly varying, we assume that the path loss coefficient $\overline{g}_{n}$ between BS and each user can be estimated perfectly. However, in dynamic and complex wireless environments, accurately acquiring time-varying Rayleigh fading channel gains is challenging. Following the approaches in [64, 65], the Rayleigh fading channel gain is modeled as

where ${\alpha_{n}}$ is the realistic Rayleigh fading channel coefficient between BS and $n$-th user, ${\hat{h}_{n}}\sim{\cal{CN}}\left({0,1-\sigma_{\epsilon}^{2}}\right)$ denotes the estimated channel coefficient, and ${\mathcal{\epsilon}_{n}}~{}\sim~{}{\cal{CN}}\left({0,\sigma_{{{\mathcal{\epsilon}}}}^{2}}\right)$ is the estimated error. Note that the parameter $\sigma_{{{\mathcal{\epsilon}}}}^{2}$ indicates the quality of channel estimation, and keeps constant as [66, 67]. We assume that ${\hat{\alpha}_{n}}$ and ${\epsilon_{n}}$ are uncorrelated.

If the perfect CSI is known, the maximum achievable data rate between BS and $n$-th can be written as

where

In (26), $\phi_{n}$ denotes the signal-to-interference-plus-noise ratio (SINR) of the user $n$. In practice, the BS can only obtain the estimated fading channel coefficient ${\hat{\alpha}}_{n}$. The scheduled data rate with imperfect CSI can be expressed as

where

However, the scheduled data rate with imperfect CSI may easily exceed the maximum achievable data rate, i.e., $r_{n}>c_{n}$. To measure the performance of this case, we introduce outage probability as a metric [68, 69]. Therefore, the weighted proportional fairness function with outage probability can be expressed as $\sum\limits_{n=1}^{N}{w_{n}\ln r_{n}\Pr[r_{n}\leq c_{n}|{\hat{\alpha}}_{n}]}$. $\Pr[r_{n}\leq c_{n}|{\hat{\alpha}}_{n}]$ denotes the probability of a case when the scheduled data rate $r_{n}$ is less than or equal to the maximum data rate $c_{n}$ under the estimated channel coefficient ${\hat{\alpha}}_{n}$. The optimization problem can be reformulated as

where (29b) is introduced to satisfy the channel outage probability requirement $\epsilon_{out}$ for all users in the imperfect CSI scenario. Due to the probability constraints (29b), this problem (29) turns into a non-convex problem and cannot easily be optimally solved in polynomial time [68]. To tackle this problem efficiently, we transform the probabilistic mixed problem into a non-probability problem as

where ${\tilde{r}}_{n}=W\log_{2}(1+{\tilde{\phi}}_{n})$, and the transformed SINR ${\tilde{\phi}}_{n}$ can be expressed as

where $F_{|g_{n}|^{2}}^{-1}(\epsilon_{out}/2)$ denotes the inverse cumulative distribution function of a noncentral chi-square random variable with 2 degrees of freedom and non-centrality parameter $2|{\hat{g}}_{n}|^{2}/\sigma^{2}_{\epsilon}$. The details of the probabilistic mixed problem transformation are shown in Appendix D.

Notice that once the SIC ordering is determined, the power allocation problem of (61) can be transformed into a convex problem as well as Appendix B and solved by the CVX solver. Thus, ASOPA can be applied to solve it by simply modifying its input from ${\bf{X}}={\{({w_{n}},P_{i}^{max},{g_{n}})\}_{n\in{\cal N}}}$ to ${{\bf{X}}}={\{({w_{n}},P_{n}^{max},{|{{\hat{\alpha}}_{n}|}^{2}{\overline{g}}_{n}})\}_{n\in{\mathcal{N}}}}$

In this subsection, we assess the performance of ASOPA in NOMA networks with multiple BSs, represented by the set $\mathbf{B}$, each containing $N_{b}$ users where $b\in\mathbf{B}$. In scenarios involving multiple BSs, each user experiences inter-cell interference from users linked to other BSs. To quantify this interference, the channel gain from a user $n$ in BS $b$ to another BS $b^{\prime}$ is defined as $h^{({b})}_{n,b^{\prime}}$. Specifically, the superscript $(b)$ indicates that user $n$ is associated with BS $b$.

To accommodate the multiple-BS scenario, it is straightforward to adjust the input of ASOPA to ${\bf{X}}=\{(w_{n}^{(b)},P_{n,max}^{(b)},g_{n}^{(b)},\bm{h}^{(b)}_{n})\}_{n\in\mathcal{N}_{b},b\in\mathbf{B}}$, where $\bm{h}^{(b)}_{n}=\{h^{(b)}_{n,b^{\prime}}\}_{b^{\prime}\in\mathbf{B}\setminus\{b\}}$. The resource allocation problem is still convex and can be solved using the CVX solver. The detailed setup of the multiple-BS scenario is in Appendix E.

The addition of inter-cell interference, however, adds complexity to the SIC ordering problem. In ASOPA, the next decoding user is iteratively chosen based on the highest probability from Equation (12), under the assumption that all users are within the same BS. However, in scenarios involving multiple BSs, the SIC decoding order is specific to each BS. Comparing the probabilities of users from different BSs lacks physical insight, as such comparisons are not meaningful in this context. To overcome this issue, we introduce enhance the decoding process by introducing an additional masking mechanism in the decoder. This mechanism allows for the generation of appropriate SIC orderings for users across all BSs. Specifically, the decoder generates the SIC ordering for users associated with the current BS while effectively masking the users of other BSs. This approach ensures efficient decoding order determination in multi-BS NOMA networks without needing to modify Equation (12).

To demonstrate the effectiveness of the enhanced mask mechanism in ASOPA, let’s consider a case study outlined in Table III. As depicted in Fig. 6, we consider a dual-BS NOMA network, where each BS contains three users, as $N_{b}=3$ for all $b\in\mathbf{B}=\{1,2\}$. Consequently, it takes six iterations for ASOPA to establish the SIC ordering for all users. Utilizing the mask mechanism, ASOPA initially decodes the users in BS 1 during iterations $t=1$, 2, and 3, and then shifts to decoding users in BS 2 for iterations $t=4$, 5, and 6. In the first iteration ($t=1$), the algorithm calculates the probabilities $\bm{\ell}^{(1)}_{1}$ for users in BS 1 from Equation (12) and simultaneously masks the probabilities of users in BS 2 by setting $\bm{\ell}^{(2)}_{1}=0$. Given that user 1 in BS 1 has the highest probability of 0.3997, it is selected as the first decoded user, $\pi^{(1)}_{1}=1$. In the next two iterations, users in BS 2 remain masked, indicated by $\bm{\ell}^{(2)}_{2}=0$ and $\bm{\ell}^{(2)}_{3}=0$. Conversely, when decoding the SIC ordering for users in BS 2 during iterations $t=4$, 5, and 6, the users in BS 1 are masked with $\bm{\ell}^{(1)}_{t}=0$. This mechanism enhances the efficiency and accuracy of ASOPA in multi-BS NOMA networks by systematically focusing on one BS at a time, thereby streamlining the decoding process.

In Fig. 7, we evaluate the effect of different parameters on the convergence performance of ASOPA, including different learning rates, batch sizes, and embedding dimensions.

Fig. 7(a) shows the effect of different learning rates. We can see that a significant learning rate (1e-2 or 1e-3) causes the algorithm to converge to a local optimum, but a small learning rate (1e-5) results in slow convergence. Hence, the learning rate is set as 1e-4.

Fig. 7(b) shows the effect of different batch sizes. A small batch size (8 or 16) leads to high variance in the network utility. The larger the batch size, the more memory space the algorithm consumes. Also, a large batch size may reduce the randomness of gradient descent and lead to the local minimum value. Hence, the batch size is set to 64.

Fig. 7(c) shows the effect of different embedding dimensions $d_{e}$. A small embedding dimension (16) cannot adequately characterize features and thus degrades the performance and convergence speed. A large embedding dimension (256) may overfit the training set, resulting in unstable performance. Hence, the embedding dimension is set as $d_{e}=128$.

Overall, the simulation results in Fig. 7 show that the proposed ASOPA can converge under the set parameters.

To evaluate the SIC ordering generated by ASOPA, we compare it with five baseline algorithms:

1. 1.

   Exhaustive search [24]: This scheme calculates the network utility for all $N!$ SIC orderings and obtains the optimal network utility.

2. 2.

   Tabu search [34]: This scheme initiates a SIC ordering and swaps any two users’ ordering to search. For each search iteration, it tries all possible swapping of two users for a SIC ordering and selects the best one for the next search iteration. To the best of our knowledge, the Tabu search algorithm presented in [34] is the state-of-the-art algorithm for dynamic SIC ordering, albeit at the cost of high computational complexity.

3. 3.

   Meta-scheduling [21]: This scheme sequentially adds and inserts each user into an order. It tries every possible insertion position for each insertion and greedily chooses the one with the greatest utility gain.

4. 4.

   Weight descending [23]: The static SIC ordering follows the descending order of users’ weights.

5. 5.

   Channel descending [25]: The static SIC ordering follows the descending order of users’ channel gains.

After those baseline algorithms determine the SIC ordering, the optimal transmit power is determined by the power allocation method proposed in Section 4.3.

Fig. 8 presents the network utility achieved by different algorithms for varying numbers of users $N$. Exhaustive search achieves optimal performance with $N=5$ and $N=8$ but not $N=10$ due to the unacceptable running time for enumerating $10!$ possible SIC ordering. When $N=5$ and $N=8$, through sufficient search iteration, Tabu search achieves 99.61% and 99.19% of the optimal performance obtained by exhaustive search. ASOPA achieves 97.54% and 97.60% of the optimal performance, which is close to the performance of Tabu search. When $N=5$, $N=8$, and $N=10$, ASOPA is over 10% higher in network utility than the other three baseline algorithms besides Tabu search, respectively.

Fig. 9 provides further evaluation of ASOPA in large-scale scenarios, specifically where the number of users $N$ varies from 10 to 20. In Fig. 9(a), ASOPA demonstrates a consistent convergence rate of around 50 epochs, regardless of the specific values of $N$. Meanwhile, Fig. 9(b) illustrates that ASOPA consistently achieves average 95% performance of Tabu search, and outperforms the other three baseline algorithms in these large-scale scenarios, aligning with the observations from Fig. 8. An interesting observation is that for $N>16$, the channel descending algorithm surpasses Meta-scheduling in terms of network utility. This comparison further underscores that network utility tends to decline when the number of users in a NOMA system exceeds a certain threshold, particularly when $N>16$. The decline of network utility can be attributed to the concave logarithm throughput function $\ln{R_{n}}$ in network utility in (3). Intuitively, as the number of users increases, the sum rate $\sum_{n=1}^{N}R_{n}$ tends to saturate, leading to a decrease in the sum of logarithms $\sum_{n=1}^{N}\ln R_{n}$ due to Jensen’s inequality. Overall, the results depicted in Fig. 9 confirm ASOPA’s effectiveness in handling large-scale scenarios and its superiority over all baseline algorithms.

In Fig. 10, we further compare the performance of ASOPA and baseline algorithms over 1000 independent samples when $N=5$. Fig. 10(a) displays the mean, median, confidence interval, and outliers of the normalized network utility for different algorithms. The normalized network utility is the ratio of the network utility achieved by an algorithm to the optimal network utility obtained by exhaustive search. We observe that the medians of Tabu search and ASOPA are close to 1, and the confidence intervals of Tabu search and ASOPA are over 99% and 97%, respectively. Although some outliers affect the mean of ASOPA, it still outperforms the baseline algorithms besides Tabu search. In Fig. 10(b), we present the hit rate of the top 10 maximum network utilities for ASOPA and baseline algorithms. The hit rate is defined as the percentage of times that an algorithm generates an SIC ordering that appears in the top 10 maximum network utilities obtained by exhaustive search. We observe that ASOPA achieves hit rates of over 55% and 70% for the top 5 and top 10 maximum network utilities, respectively. The results in Fig. 10 further confirm that ASOPA can achieve near-optimal network utility performance.

Fig. 11 shows the network utility under different maximum distances of users to the BS. The network utility achieved by all algorithms decreases slightly as the maximum distance increases. Within the distance range $[100,300]~{}\text{m}$, ASOPA achieves performance close to that of Tabu search algorithm and outperforms the other three baseline algorithms by an average of 10%.

Fig. 12 demonstrates how the network utility varies with different levels of noise power spectral densities. As the noise power spectral density increases, the network utility of all algorithms decreases, and the difference between ASOPA and baseline algorithms diminishes. At a noise power spectral density of -144 dBm/Hz, ASOPA and the channel descending algorithm exhibit the slightest difference of 4.27%. This result indicates that when the users’ channel quality is inferior, the users’ channel state significantly impacts the SIC ordering.

In order to meet the real-time requirement of NOMA networks, the execution time of the SIC ordering and power allocation algorithm need be much smaller than the slot duration, i.e., two seconds [43]. To evaluate the efficiency of ASOPA and baseline algorithms, we test the average execution time under different numbers of users, and the results are shown in Fig. 13 and Table. IV.

The execution time of ASOPA is close to that of the weight descending algorithm and the channel descending algorithm, i.e., 24 ms, 25 ms, and 23 ms for a ten-user NOMA network. The execution delay is scalable with the network size $N$ and is acceptable for field deployment. ASOPA only takes 152 ms even for a twenty-user NOMA network. However, the execution latency of Meta-scheduling and Tabu saerch significantly increases with the network size $N$, consuming 771 ms and $5643$ ms for a ten-user NOMA network, respectively. The execution time of exhaustive search is exponentially increasing with $N$. It takes 823 ms and $6630$ ms even for NOMA networks with five-user and six-user, respectively. According to Fig. 13 and Table. IV, Tabu search fails to cope with real-time execution when $N>7$, while Meta-scheduling fails at $N>10$. In contrast, ASOPA maintains the same latency as the static algorithm for all the number of users. In particular, is three orders of magnitude lower than Tabu search and two orders of magnitude lower than Meta-scheduling when $N=20$.

ASOPA uses the actor network to generate the SIC ordering, whose time consumption is negligible. The primary time overhead of various algorithms comes from solving the power allocation problem by the interior-point method. Exhaustive search solves the power allocation problem $N!$ times. Meta-scheduling solves the power allocation ${N(N+1)}/{2}$ times. Tabu search solves the power allocation ${IN(N+1)}/{2}$ times, where $I$ denotes the number of search iterations. ASOPA, the weight descending algorithm, and the channel descending algorithm solve the power allocation problem once. Therefore, the proposed ASOPA executes efficiently like the static SIC ordering algorithms, while performing as well as the exhaustive search algorithm.

Regarding the training latency, ASOPA’s policy update is conducted infrequently and in parallel with the inference process, as detailed in Algorithm 1. Extensive evaluations have shown that the duration of a single policy update is approximately one second when the number of users $N$ is 10 and training batch size is 64. On average, the duration of the policy update process is less than 20 ms for each sample. Therefore, the policy update process of ASOPA can feasibly be executed online for NOMA networks, ensuring that the system remains up-to-date and responsive to changing network conditions.

Fig. 14 presents the network utility achieved by different algorithms for varying numbers of users $N$ in multiple-antenna scenario. Specifically, we consider a NOMA network with two antennas at the BS. All algorithms use the minimum-mean-square-error (MMSE) [62] linear equalization to detect symbols. For $N=5$, the optimal network utility was calculated using exhaustive search. Remarkably, ASOPA achieves 90.94% of the optimal network utility, with only a 5% performance degradation compared to the Tabu search algorithm. Furthermore, ASOPA consistently outperforms the other three baseline algorithms across all settings, which agrees the observation in Fig. 8. Due to the complexity of user state in multiple antenna scenarios, the performance of ASOPA can be further optimized in future work.

Fig. 15 shows how network utility varies with different variances of estimated error under imperfect CSI conditions with five users. ASOPA consistently achieves over 98% of the optimal performance obtained through exhaustive search and achieves 99% performance of Tabu search. Specifically, when $\sigma_{{\epsilon_{n}}}^{2}=0.025$, ASOPA’s network utility is 6.75%, 12.49%, 13.44% higher than that of Meta-scheduling, channel descending and weight descending, respectively. These results demonstrate ASOPA’s robustness and effectiveness in scenarios with imperfect CSI, highlighting its ability to adapt to varying degrees of channel estimation errors.

Fig. 16 presents the network utility achieved by various algorithms for different pairs of users $N_{1}$-$N_{2}$ in dual-BS NOMA networks. From the figure, it’s evident that ASOPA performs comparably to the exhaustive search method and surpasses other benchmark algorithms. Notably, Tabu search in [34] and Meta-scheduling proposed in [21] is not applicable in this scenario and thus is not included in the comparison. Particularly, when each BS has three or four users, ASOPA achieves 99.45% and 99.80% of the optimal performance determined by exhaustive search, respectively. The network utility reaches its peak when each BS is serving six users. This optimal network utilization can be attributed to the increasing inter-cell interference with the number of users and the logarithmic throughput function in network utility. As the number of users increases, although the sum throughput saturates, some users’ weighted logarithmic throughput even becomes a small negative value (say $-10$), which leads to a decline in network utility. As the number of users per BS rises to ten, ASOPA’s performance advantage becomes more pronounced, showing a 27.69% higher network utility compared to the channel descending algorithm. These results highlight ASOPA’s effectiveness in adapting to varying user densities in multi-BS NOMA networks.