Robust Network Slicing: Multi-Agent Policies, Adversarial Attacks, and Defensive Strategies

WNV is well-known for enhancing the spectrum allocation efficiency. As shown in Fig. 1, infrastructure providers (InPs) own the physical infrastructure (e.g., base stations, backhaul, and core network), and operate the physical wireless network. WNV virtualizes the physical resources of InPs and separates them into slices to efficiently allocate these virtualized resources to the mobile virtual network operators (MVNOs). Therefore, MVNOs deliver differentiated services via slices to user equipments (UEs) with varying transmission requirements.

In this paper, we consider a service area that contains $N_{B}$ base stations of InPs with inter-base-station interference, and MVNOs that require services from the InPs. These MVNOs provide services to $N_{u}$ UEs that are within the coverage area of at least one base station, and may move from the coverage of one base station to that of another. Users in the coverage area of multiple base stations that serve the MVNO can be assigned to any of these base stations. Without loss of generality, we discuss the station-user pair for a single MVNO in the remainder of this paper. Next, we introduce the dynamic channel environment.

We consider a dynamic environment where $N$ channels are available for each MVNO. The fading coefficient of the link between base station $b$ and a UE $u$ in a certain channel $c$ at time $t$ is denoted by $h_{c}^{b,u}(t)\sim\mathcal{CN}(0,1)$, an independent and identically distributed (i.i.d.) circularly symmetric complex Gaussian (CSCG) random variable with zero mean and unit variance. Each fading coefficient varies every $T$ time slots as a first-order complex Gauss-Markov process, according to the Jakes fading model [30]. Therefore, at time $(n+1)T$, the fading coefficient can be expressed as

where $e_{c}$ denotes the channel innovation process, which is also an i.i.d. CSCG random variable. Furthermore, we have $\rho=J_{0}(2\pi f_{d}T)$, where $J_{0}$ is the zeroth order Bessel function of the first kind and $f_{d}$ is the maximum Doppler frequency.

In the aforementioned environment, the resource allocation is performed in a first-come first-served fashion. Each base station may serve at most $N_{r}$ requests from different users simultaneously, and the request queue can stack at most $N_{q}$ requests. For each request, multiple channels can be allocated, and the transmission power is $P_{B}$ in each channel. Different requests served by the same base station do not share the same channels, while requests to different base stations may share the same channel at the cost of inflicting interference. For all requests to each base station, transmission is allowed in no more than $N_{c}$ channels, and consequently the total power is limited to $N_{c}P_{B}$. In this setting, if request $k$ is allocated the subset $C_{k}$ of channels at base station $b$, the sum rate $\mathsf{r}_{k}$ for this request is

with

where $c$ denotes the index of a selected channel, $\mathsf{r}_{c}^{b,u}$ is the transmission rate from base station $b$ to UE $u$ in channel $c$, $\mathcal{N}^{b,b^{\prime}}_{c}$ is the interference from base station $b^{\prime}$ in channel $c$, $\mathbf{1}_{c}^{b,b^{\prime}}$ is the indicator function for transmission at base stations $b$ and $b^{\prime}$ sharing channel $c$, $\sigma^{2}$ is the variance of the additive white Gaussian noise, and $L^{b,u}$ is the path loss:

where $h_{B}$ is the height of each base station, $\alpha$ is the path loss exponent, and $\{x_{b},y_{b}\}$ and $\{x_{u},y_{u}\}$ are the 2-D coordinates of base station $b$ and user $u$, respectively.

Therefore, for each base station, the resource assignment including the selected subsets $C_{k}$, the number of selected channels $n_{c}$, the number of served requests $n_{r}$, and the number of requests in queue $n_{q}$ must follow the following constraints:

Above, (6) indicates that no channel is shared among different requests at the same base station, and (7) defines the number of channels being used. The constraints in (8)–(10) are the upper bounds on the number of channels, the number of requests being served simultaneously, and the number of requests in the queue, respectively. If $n_{r}=N_{r}$ and $n_{q}=N_{q}$ at a base station, any incoming request to that base station will be denied service.

The features of each request $k$ consist of minimum transmission rate $m_{k}$, lifetime $l_{k}$, and initial payload $p_{k}$ before the transmission starts. At each time slot $t$ during transmission, the constraints are given as follows:

where $l_{k}(t)$ denotes the remaining lifetime at time $t$. Note that with this definition, we have $l_{k}(0)=l_{k}$. If any request being processed fails to meet the constraints (11) or (12), it will be terminated and be marked as failed. Any request in the queue that fails to meet constraint (12) will also be removed and marked as failure. Otherwise, the lifetime will be updated for each request being processed and each request in the queue as follows:

Each request being processed will transmit $\mathsf{r}_{k}(t)$ bits of the remaining payload:

Note that the initial payload is $p_{k}(0)=p_{k}$. If the payload is completed within the lifetime (i.e., the remaining payload is $p_{k}(t+1)=0$ for $t+1\leq l_{k}$), the request $k$ is completed and marked as success.

For all aforementioned cases, if the request is completed successfully in time, the network slicing agent at the base station $b$ receives a positive reward $R_{k}$ equal to the initial payload $p_{k}$. Otherwise, it receives a negative reward of $R_{k}=-p_{k}$. Each user can only send one request at a time.

Afterwards, base station $b$ records the latest transmission rate history of $\mathsf{r}_{c}^{b,u}$ into a 2 dimensional matrix $H^{b}\in\mathbb{R}_{\geq 0}^{N_{u}\times N}$. In each time slot $t$, for request $k$ from user $u$ that is allocated subset $C_{k}$ of channels at base station $b$, the base station updates the entries corresponding to the channels that were selected for transmission:

Note that the location $\{x_{u},y_{u}\}$ of user $u$, the fading coefficient $h_{c}^{b,u}$, and the interference corresponding to $\mathbf{1}_{c}^{b,b^{\prime}}$ vary over time, and therefore $H_{b}$ is only a first-order approximation of the potential transmission rate $\mathsf{r}_{c}^{b,u}(t+1)$.

The goal of the network slicing agent at each base station is to find a policy $\pi$ that selects $n_{c}$ channels and assigns them to $n_{r}$ requests, so that the sum reward of all requests at all base stations is maximized over time:

where $K^{\prime}_{b}$ is the set of completed or terminated requests for base station $b$ at time $t$, and $\gamma\in(0,1)$ is the discount factor.

In this section, we briefly discuss the actor-critic algorithm [31]. This deep RL algorithm utilizes two neural networks, the actor and critic. The two networks have separate neurons and utilize separate backpropagation, and they may have separate hyperparameters.

The multi-agent extension of actor-critic algorithm where each agent has separate actor and critic that aim at maximizing each individual reward is referred to as independent actor-critic (IAC) [15, 16]. When the action of each agent interferes with the others and the goal is to maximize the sum reward over all agents, the deep RL with MACC may be preferred [16]. In this framework, the decentralized actors with parameter $\theta$ and policy $f^{\theta}(\mathcal{O}_{b})$ are the same as in IAC, while there is only one centralized critic with parameter $\phi$ and policy $g^{\phi}(\mathcal{O})$ aiming at the sum reward by updating each decentralized actor. Therefore, MACC agents are more likely to learn coordinated strategies through interaction between agents and mutual environment, despite the scarcity of information sharing at the training phase. In the framework of MACC, the critic maps $\mathcal{O}$ to a single temporal difference (TD) error

where $K_{b}$ is the set of requests being processed and requests in the queue at time $t$, and $\gamma\in(0,1)$ is the discount factor. Then, the critic parameters $\phi$ and actor parameters $\theta$ are optimized by considering the least square temporal difference and policy gradient, respectively, as follows:

For a wireless resource allocation problem, fast convergence in the training of neural networks is highly important. Therefore in our implementation, we not only recommend offline pre-training via simulations, but also speed up learning by sharing the neural network parameters among the agents, i.e., we use all the information to update one actor and one critic (as in IAC) during training and share the parameters among all agents.

In this section, we introduce the pointer network to implement the actor policy $f^{\theta}$ for IAC and MACC.

Traditional sequence-processing recurrent neural networks (RNN), such as sequence-to-sequence learning [32] and neural Turing machines [33], have a limit on the output dimensionality, i.e., the output dimensionality is fixed by the dimensionality of the problem so it is the same during training and inference. However in a network slicing problem, the actor’s input dimensionality $\{N,n_{c},n_{r}\}$ may vary over time, and thus the expected output dimension of $\mathcal{A}_{b}$ also varies according to the input. As opposed to the aforementioned sequence-processing algorithms, pointer network learns the conditional probability of an output sequence with elements that are discrete tokens corresponding to positions in the input sequence [18], and therefore the dimension of action $\mathcal{A}_{b}$ in each time slot depends on the length of the input.

Another benefit of pointer networks is that they reduce the cardinality of $\mathcal{A}_{b}$. General deep RL algorithms typically list out each combination as an element in action $\mathcal{A}_{b}^{\prime}$, and each element indicates picking $n_{c}$ channels out of $N$ channels and assigning each to one of $n_{r}$ requests. Therefore $\mathcal{A}_{b}^{\prime}$ has the dimension of $\binom{N}{n_{c}}n_{r}^{n_{c}}$. When the dimensions of $N$, $n_{c}$ and $n_{r}$ are high, this will require a prohibitively large network to give such an output, and require exponentially longer time to train. Comparatively, $\mathcal{A}_{b}$ generated from our proposed pointer network actor has only $n_{r}N$ output elements.

Therefore, we here introduce the pointer network as the novel architecture of the actor of the proposed agents as shown in Fig. 2. Similar to sequence-to-sequence learning, pointer network utilizes two RNNs called encoder and decoder whose output dimensions are $h_{e}$ and $h_{d}$. The former encodes the sequence $\{\mathcal{O}_{b}^{e1},\mathcal{O}_{b}^{e2},\ldots,\mathcal{O}_{b}^{e{n_{r}}}\}$ with $\mathcal{O}_{b}^{ek}\subset\mathcal{O}_{b}$ (where the index $k\in\{1,2,\ldots,n_{r}\}$ corresponds to requests), and produces the sequence of vectors $\{e_{1},e_{2},\ldots,e_{n_{r}}\}$ with $e_{k}\in\mathbb{R}^{h_{e}}$ at the output gate in each step. The decoder inherits the encoder’s hidden state and is fed by another sequence $\{\mathcal{O}_{b}^{d1},\mathcal{O}_{b}^{d2},\ldots,\mathcal{O}_{b}^{d{N}}\},\mathcal{O}_{b}^{dc}\subset\mathcal{O}_{b}$ (where the index $c\in\{1,2,\ldots,N\}$ corresponds to channels), and produces the sequence of vectors $\{d_{1},d_{2},\ldots,d_{N}\},d_{c}\in\mathbb{R}^{h_{d}}$.

Furthermore, pointer network computes the conditional probability array $\mathcal{P}^{c}\in[0,1]^{n_{r}}$ where

using an attention mechanism as follows:

where softmax normalizes the vector of $u^{c}_{k}$ to be an output distribution over the dictionary of inputs, and vector $v$ and matrices $W_{1}$ and $W_{2}$ are trainable parameters of the attention model.

We note that the $n_{r}\times N$ matrix of the pointer network output $\mathcal{P}=[\mathcal{P}^{1},\mathcal{P}^{2},\ldots,\mathcal{P}^{N}]$ typically consists of non-integer values, and we obtain the decision of $\mathcal{A}_{b}$ by picking $n_{c}$ maximum elements in each column:

In our implementation, we use two Long Short Term Memory (LSTM) [34] RNNs of the same size (i.e., $h_{e}=h_{d}$) to model the encoder and decoder. During the backpropagation phase of neural network training, the partial derivative of error functions with respect to weights in layers (encoder and decoder in our case) that are far from the output layer may be vanishingly small, preventing the network from further training, and this is called vanishing gradient problem [35]. However, the encoder and decoder are cardinal for the function of the pointer network while we demand fast convergence, so we set $W_{1}$ and $W_{2}$ in (22) as trainable vectors and set $v$ as a single constant to reduce the depth of the pointer network and speed the training on both LSTMs.

In this subsection, we introduce the action and state spaces for MACC deep RL with pointer networks employed for the aforementioned network slicing problem.

As shown in Fig. 3, we in the experiments consider a service area with $N_{B}=5$ base stations and $N_{u}=30$ users, and each cell radius is 2.5km. There are $N=16$ channels available, and each base station picks at most $N_{c}=8$ channels for transmission. The ratio between transmission power and the noise in each channel is $P_{B}/\sigma_{k}^{2}=6.3$. Each base station allows the processing of $N_{r}=4$ requests, and keeps at most $N_{q}=2$ requests in the queue. When a request from one user is terminated or completed, the user will not be able to send another request within $T_{r}=2$ time slots.

The channel varies with maximum Doppler frequency $f_{d}=1$ and dynamic slot duration $T=0.02s$, and the location of users follows a random walk pattern. Each user may transfer between the coverage of different base stations, while staying within the coverage area of at least one of the base stations. All other parameters are listed in Table I.

In Fig. 4, we compare the moving average of the sum reward achieved by the network slicing agents utilizing the proposed MACC with pointer networks against other algorithms. Similarly as in [36], we introduce three statistical algorithms including max-rate which always executes the max-rate mode, FIFO that selects channels with high channel rate history but always assigns enough channel resources to the requests that has arrived earlier, and hard slicing that assigns channels with high channel rate history and the number of channels assigned to each request is proportional to the corresponding minimum transmission rate $m_{k}$. We also compare four deep RL agents, each of which deploys IAC or MACC frameworks using either FNN and pointer networks as actor. While the proposed MACC with pointer network agent starts with a lower performance at the beginning of the training phase due to random exploration, it outperforms all other algorithms during the test phase. In the initial training phase of $t<50000$, the proposed agent starts with random parameters and employs the $\epsilon$-$\epsilon_{m}$-greedy policy, which starts with a large probability $\epsilon(1-\epsilon_{m})$ to explore with random actions, and consequently the performance starts low but gradually improves. We observe that in the test phase from $t=50000$ to $t=200000$, the sum reward eventually converges to around 20 bits/symbol, while slightly oscillating due to the challenging dynamic environment with random channel states, varying user locations, and randomly arriving requests. We further note that the proposed network slicing agents complete over $95\%$ of the requests, and hence attain a very high completion ratio as well.

In the considered channel model, the fading coefficient of the link from the jammer to the user equipment (UE) $u$ in a certain channel $c$ is denoted by $h_{c}^{J,u}$, and the fading coefficient of the link from the base station $b$ to the jammer in a certain channel $c$ is denoted by $h_{c}^{b,J}$. We consider $h_{c}^{b,u}$, $h_{c}^{J,u}$ and $h_{c}^{b,J}$ to be independent and identically distributed (i.i.d.) and vary over time according to the Jakes fading model [30]. Once the jammer is initialized at horizontal location $\{x_{J},y_{J}\}$ with height $h_{J}$, it can choose in any given time slot one of the two operational phases: jamming phase and listening phase.

The jammer aims at minimizing the performance of victim users, but it does not have any information on channel fading, UE locations or rewards provided to different requests. Therefore, the jamming location is optimized by minimizing the expected sum transmission rate for given UE $u$ integrated over the service area when the channels for transmission coincide with the channels being jammed. More specifically, we have the following optimization:

where the expectation with respect to the set of fading coefficients $\{h\}$ considers $\forall b,u,c:h_{c}^{b,u},h_{c}^{J,u},h_{c}^{b,J}\overset{\text{i.i.d.}}{\sim}\mathcal{CN}(0,1)$, $D^{b}_{h}$ is the subset of coverage area with maximal transmission rate $\mathsf{r}_{c,J}^{b,u}$ from base station $b$ given $\{h_{c}^{b,u},h_{c}^{J,u},h_{c}^{b,J}\}$, and $\forall b,b^{\prime}:\mathbf{1}_{c}^{b,b^{\prime}}=0,\mathbf{1}_{c}^{b,J}=\mathbf{1}_{c}^{b}=1$. Additionally, we note that $P_{B}$, $h_{B}$, $|\alpha|$, and $\sigma$ with arbitrary positive values will not affect the optimized jamming location.

After the jammer is initialized in the true environment and has observed/listened the interference power $\mathcal{N}^{\text{listen}}_{c}(t-1)$ within the listening phase at time $t-1$, it will decide the subset of channels $\mathcal{C}_{J}(t)\subset\mathcal{C}$ to jam during jamming phase at time $t$, where $\left|\mathcal{C}_{J}(t)\right|=n_{J}(t)$ and $\mathcal{C}$ is the set of all channels $\{1,2,\ldots,N\}$. According to (30), channels with higher $\mathcal{N}^{\text{listen}}_{c}$ are more likely to be better choices, but this information is not available in the jamming phase at time $t$ (since jamming is performed rather than listening). Therefore, $\mathcal{C}_{J}(t)$ can only be evaluated via $\mathcal{N}^{\text{listen}}_{c}(t-1)$ and $\mathcal{N}^{\text{listen}}_{c}(t+1)$. Note that $\mathcal{N}^{\text{listen}}_{c}(t+1)$ is not available at time $t$ and this challenge will be addressed via deep RL in the next subsection (i.e., by essentially introducing a reward to train the neural network at time $t+1$ and having that reward depend on $\mathcal{N}^{\text{listen}}_{c}(t-1)$ and $\mathcal{N}^{\text{listen}}_{c}(t+1)$.) Hence, with the deep RL approach, the action will depend only on observation before time $t$.

In the absence of information on the requests, we assume a model in which each request arrives and is completed independently. Thus, the state of current time slot $t$ can be estimated as a linear interpolation (or a weighted average) of $\mathcal{N}^{\text{listen}}_{c}(t-1)$ and $\mathcal{N}^{\text{listen}}_{c}(t+1)$. Therefore, the optimized subset of channels to jam can be determined from

where

and $\beta(t)$ describes the impact of the jammer on the victims. Typically, a request takes multiple time slots to get completed. When it is jammed, there are two possibilities. On the one hand, it may fail to meet the minimum transmission rate limit and get terminated immediately. In such a case, the next request in the queue is processed, and the network slicing agent rearranges and distributes channels into a new set of slices to be allocated to different requests from different users, and thus the listened interference $\mathcal{N}^{\text{listen}}_{c}(t+1)$ may change dramatically. In this case, we are likely to have $\beta(t)>1$. On the other hand, if the request under attack has a lower transmission rate but the minimum transmission rate limit and the lifetime constraint are satisfied, the transmission will last longer, and the interference $\mathcal{N}^{\text{listen}}_{c}(t+1)$ is less likely to vary from that at time $t$. In this case, we are likely to have $\beta(t)<1$. Therefore, the value of $\beta(t)$ should be determined via experience:

where

and $\mathcal{T}^{\prime\prime}$ is a set of time points in the jamming phase where each $t^{\prime\prime}\in\mathcal{T}^{\prime\prime}$ is close to time $t$, and $\mathcal{T}^{\prime}$ is a set of time points where each $t^{\prime}\in\mathcal{T}^{\prime}$ are in successive listening phases without jamming attack. If $T_{J}>3$, $d^{\text{listen}}_{c}(\mathcal{T}^{\prime})$ can be the set of successive listening phases in every period during training. Otherwise if $T_{J}\leq 3$, $d^{\text{listen}}_{c}(\mathcal{T}^{\prime})$ has to be collected before jamming starts.

Again, it is important to note that when the jammer agent makes decisions at time $t$, $\mathcal{N}^{\text{listen}}_{c}(t+1)$ is not available. To address this, we propose a deep RL agent that uses the actor-critic algorithm to learn the policy.

Our proposed jammer agent utilizes an actor-critic deep RL algorithm to learn the policy that optimizes the output $\mathcal{C}_{J}(t)$ to minimize the victims’ expected sum rate. The jammer works with a period $T_{J}$, and only switches from the listening phase to the jamming phase at the end of each period and uses the policy to make the decision $\mathcal{C}_{J}(t)$. We next introduce the observation, action, reward, and the actor-critic update of this agent.

As shown in Fig. 5, we in the experiments consider a service area with $N_{B}=5$ base stations, $N_{u}=30$ users, and a jammer. The theoretically optimized location of the jammer $\{x_{J}^{*},y_{J}^{*}\}$ is determined according to (31) and it lies at the center $\{0,0\}$, while the actual location is slightly moved away from the base station tower. There are $N=16$ channels available, and the jammer picks at most $N_{J}=8$ channels for jamming. The jamming power in each channel equals the transmission power in each channel: $P_{J}=P_{B}$. The jammer has phase switching period of $T_{J}=2$ time slots.

In the experiments, the network slicing agents are initialized as the well-trained MACC agents with pointer network actors as detailed in Section III, and time is initiated at $t=0$. The jammer’s actor and critic policies are implemented as two feed-forward neural networks (FNNs), and both have one hidden layer with 16 hidden nodes. The jammer is initialized at $t=100$ and begins jamming and performs online updating. During the training phase $100\leq t<10000$, the jammer follows an $\epsilon$-greedy policy to update the neural network parameters $\theta_{J}$ and $\phi_{J}$ with learning rate $10^{-5}$. It starts by fully exploring random actions, and the probability to choose random actions linearly decreases to $0.01$, thus eventually leading the agent to mostly follow the actor policy $f^{\theta_{J}}(\mathcal{O}_{J})$. This probability is fixed to $0.01$ in the testing phase from $t=10000$ to $t=20000$.

In Fig. 6, we compare the performance of the proposed actor-critic jammer that approximates $\hat{\mathcal{N}}^{\beta}_{c}(t)$ with four other scenarios in terms of victim sum reward in the testing phase. The figure shows the testing phase after the victim agent adapts to the jamming attack, and as a result its action policy becomes stable. The slight fluctuations in the curves are due to randomly arriving requests and varying channel states. We also note that the vertical axis is the victim’s reward, and hence the jammer with the better performance will lead to lower victim reward. The first case is the setting with no jammer and hence the performance is that of the original network slicing agent in terms of the sum reward in the absence of jamming attacks. The second scenario is with a last-interference jammer agent that is positioned at the same location (i.e., the origin $\{0,0\}$) with the same power budget. However, this jammer agent does not utilize any machine learning algorithms, and chooses the subset of channels with the highest observed/listened interference power levels in the last listening phase:

Consequentially, the last-interference jammer which focuses on the last time slot is equivalent to the proposed jammer when $\beta\rightarrow\infty$. The third scenario is with the next-interference jammer agent, which is also an actor-critic jammer agent but whose reward has $\beta=0$, so it concentrates on the next time slot. Furthermore, we provide performance comparisons with a max-rate jammer that is assumed to know the channel environment between the base stations and users perfectly (which implies a rather strong jammer), and thus it obtains every potential channel rate regardless of the interference from other users, and picks the maximum via

Given $N$ maximum potential rates in $N$ different channels, the jammer randomly picks channel $c$ to jam with probability $r^{max}_{c}/\sum_{c^{\prime}}^{N}{r^{max}_{c^{\prime}}}$. To better evaluate the performance among different jammers, we assume that all jammers have the same power budget. We observe that all other jammers are less efficient in suppressing the victim sum reward compared to our proposed actor-critic jammer, which aims at estimating $\hat{\mathcal{N}}^{\beta}_{c}(t)$ and therefore has better performance. Specifically, we observe in Fig. 6 that the proposed jammer results in the smallest sum reward values for the network slicing agents and has the most significant adversarial impact. In order to give a comprehensive comparison, we in Fig. 7 also illustrate the performance of the same set of jammers where the jamming power is 60dB higher than the transmission power, i.e. $P_{J}={10}^{6}P_{B}$. We notice that since the victim user receives negative reward when the request fails, some curves have negative values at times in this harsh jamming interference environment.

While it is hard to distinguish some curves with similar performance, we provide the numerical results of average reward and request completion ratio in both experiments in Table II. We observe that the victim under the actor-critic jamming attack completes much less requests and attains the smallest average reward values in both scenarios, and thus the actor-critic jammer outperforms all other jammers including the max-rate jammer that knows the channel status. Additionally in the first scenario where $P_{J}=P_{B}$, we notice that the base station at coordinates $(0,0)$ in Fig. 5, which is the closest one to the jammer’s location, only completes 67.25% of the requests under the proposed jamming attack. In comparison, this base station under other types of attack completes about 80% of the requests. This base station also completes less requests under our proposed attack when $P_{J}={10}^{6}P_{B}$. These observations indicate that the proposed jammer agent is more likely to learn from the received interference.

We first describe the details of the NesPE as a scheme that enhances the performance of a player in a multi-player zero-sum stochastic game [37] in which the player does not have direct observation of the opponents’ action choices. The actions of the optimal mixed strategy depend on the prior observation, and this relationship is to be determined via deep RL or other machine-learning-based algorithms. To train the policy ensemble with the deep RL algorithm of the player and to determine the optimal mixed strategy profile, we consider the tuple $\mathcal{G}=\left(\mathcal{O},\mathcal{E},(f^{\theta_{e}})_{e\in\mathcal{E}},\hat{e},\mathcal{L},(\mathcal{O}_{l})_{l\in\mathcal{L}},u\right)$, where

• •

  $\mathcal{O}$ is the prior observation before decision-making.

• •

  $\mathcal{E}$ is a finite index set of policies, i.e., $\mathcal{E}=\{1,\ldots,E\}$.

• •

  $f^{\theta_{e}}$ is the actor function of policy $e$ with parameter $\theta_{e}$, and is regarded as the player’s strategy. It maps the prior observation to the player’s action $\mathcal{A}_{e}=f^{\theta_{e}}(\mathcal{O})\in\{0,1\}^{N}$ where the chosen elements have value $1$ and others are set to $0$.

• •

  $\hat{e}$ is the index of the chosen policy to execute the action.

• •

  $\mathcal{L}$ is the finite index set of subsequent observations obtained after decision-making, i.e. $\mathcal{L}=\{1,\ldots,L\}$.

• •

  $\mathcal{O}_{l}$ is the subsequent observation, and $l$ indirectly represents the opponents’ action choice. Thus it is regarded as the opponents’ strategy.

• •

  $u:f^{\theta_{e}}(\mathcal{O})\times\mathcal{O}_{l}\rightarrow\mathbb{R}$ is the utility (or payoff, reward) function of the player when it chooses policy $e$ and the interaction with the opponents results in the observation $\mathcal{O}_{l}$. The mapping function is unknown to both players, and its outcome is available only to the considered player after decision-making.

According to [38], every finite strategic-form game has a mixed strategy equilibrium. In this stochastic game without prior knowledge of the utility matrix, the considered player can alternatively obtain the utility matrix by recording the experienced utility and taking the average. We consider a matrix $\mathcal{H}$ of queues with finite length (specifically, each element of this matrix is a queue with a few recorded utilities), and the matrix has $E$ rows and $L$ columns. In each time slot, the experienced utility $u_{\hat{e},l}(t)$ is appended to the queue $\mathcal{H}(\hat{e},l)$. At the beginning of the next time slot, the player uses the average of each queue to form a new matrix with $E\times L$ elements as the utility matrix for calculating the optimal profile at Nash equilibrium $\sigma=\{\sigma_{1},\ldots,\sigma_{E}\}$, which is the set of probabilities to choose and execute each strategy at a time.

However, although the probability to choose each given policy is optimized, the parameters of the policies $({\theta_{e}})_{e\in\mathcal{E}}$ may not be optimized, and need further training with the received utility $u_{\hat{e},l}(t)$ after each time slot $t$. Most existing works of policy ensemble use similar rewards to train different policies, thus there is no guarantee that the policy ensemble explores different strategies. In NesPE, we consider a dual problem that takes both utility reward and the correlation of policy ensemble [23] into account. The expected correlation between policy $e_{1}$ and policy $e_{2}$ is defined as

Therefore, the correlation between the two policies indicates the similarity of decision-making patterns between them. To fully explore all possible strategies and increase the policy variety to feed the mixed strategy, lower correlations between the policies are desired.

Thus, in order to maximize the expectation of utility reward $u_{\hat{e},l}$ and limit the correlation below a certain threshold $D$, we at each time slot $t$ consider a non-linear programming problem for policy $e$: