Deep Echo State Q-Network (DEQN) and Its Application in Dynamic Spectrum Sharing for 5G and Beyond

RL is one type of machine learning method that provides a flexible architecture for solving many types of practical problems because it does not need to model complex systems or to label data for training. In RL, an agent learns how to select actions to maximize the cumulative reward in a stochastic environment. The dynamics of the environment is usually modeled as a Markov decision process (MDP), which characterized by a tuple $(\mathcal{S},\mathcal{A},\mathcal{P},R,\mathcal{\gamma})$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $\mathcal{P}$ is the state transition providing $\Pr(s_{t+1}|s_{t},a_{t})$, $R$ is the reward function providing $r_{t}=R(s_{t},a_{t})$, and $\gamma$ is a discount factor for calculating cumulative reward. Specifically, at time $t$, the state is $s_{t}\in\mathcal{S}$, the RL agent selects an action $a_{t}\in\mathcal{A}$ by following a policy $\pi(s_{t})$ and receives the reward $r_{t}$, and then the system shifts to the next state $s_{t+1}$ according to the state transition probability. Note that the action $a_{t}$ affects both the immediate reward $r_{t}$ and the next state $s_{t+1}$. Consequently, all subsequent rewards are affected by the current action. The goal of RL agent is to find a policy $\pi$ to maximize the cumulative reward, $\mathbb{E}_{\pi}\left[\sum_{t=1}^{\infty}\gamma^{t-1}r_{t}\right]$.

In RL, a model-free algorithm does not require state transition probability for learning, which is useful when the underlying system is complicated and difficult to model. Q-learning [15] is the most widely used model-free RL algorithm that aims to find the Q-function of each state-action pair for a given policy, which is defined as

Q-function represents the cumulative reward when taking action $a_{t}$ in the state $s_{t}$ and then following policy $\pi$. Q-learning constructs a Q-table to estimate the Q-function of each state-action pair by iteratively updating each element of the Q-table through dynamic programming. The update rule of the Q-table is given as follows:

where $\alpha\in(0,1)$ is the learning rate. The policy $\pi$ that selects action is the $\epsilon$-greedy policy as follows:

where $\epsilon\in[0,1]$ is the exploration probability. However, Q-learning performs poorly when the dimension of the state is high because updating a large Q-table makes training difficult or even impossible.

Deep Q-Networks (DQN) [1] is introduced to solve high-dimensional state problems by leveraging a neural network as the function approximator of the Q-table, which is referred to as the Q-network. Specifically, the Q-network takes the state $s_{t}$ as input and outputs the estimated Q-function of all possible actions. One key approach of DQN to improve the training stability is by creating two Q-networks: the evaluation network $Q(s,a;\theta)$ and the target network $Q(s,a;\theta^{-})$. The target network is used to generate the targets for training the evaluation network while the evaluation network is used to determine the actions. The loss function for training the evaluation network is written as

where $r_{t}+\gamma\max\limits_{a}Q(s_{t+1},a;\theta^{-})$ is the target Q-value. The weights of the target network $\theta^{-}$ is periodically synchronized with the weights of the evaluation network $\theta$. The purpose is to fix targets temporarily during training to improve the training stability of the evaluation network.

An improvement of DQN to prevent overestimation of Q-values is called double Q-learning [16], where the evaluation network is used to select the action when computing the target Q-value, but the target Q-value is still generated by the target network. Specifically, the target Q-value for the evaluation network is calculated by

where $a^{\prime}=\operatorname*{argmax}\limits_{a}Q(s_{t+1},a;\theta)$. Double Q-learning can improve the accuracy in estimating Q-function, thereby improves the learned policy.

DRL-based methods have recently been applied in dynamic spectrum access (DSA) networks [17, 18, 19] where the focus is exclusively on the ”access” part of the problem with over-simplified network setup. To be specific, [17] considers single SU selects one channel to access in the multichannel environment, and the goal is to maximize the number of selecting good channels for access. [18] assumes that the available spectrum channels are known a priori and develops a centralized spectrum access algorithm for multi-user access. Both [17] and [18] assumes that one channel can only be used by one user at any particular time. Although [19] considers multiple SUs can access a channel at the same time, a SU cannot access a channel that a PU is using. [19] also assumes that each SU can sense all channels simultaneously and the collision between a PU and a SU can be perfectly detected. In this work, in order to provide a comprehensive study for the impact of DEQN on relevant DSS networks for 5G, we consider practical situations of DSS where 1) mobile users cannot conduct spectrum sensing perfectly. 2) mobile users cannot sense multiple channel at a particular time. 3) there are multiple PUs and multiple SUs in a DSS network. 4) A channel can be shared by multiple users if the interference between them is weak. Furthermore, unlike previous work which utilizes binary ACK/NACK feedback as the reward function, we calculate the practical reward based on the spectral-efficiency of each mobile link. To be closely in line with the real wireless environment, the spectral-efficiency of a mobile link is calculated using the transmission procedure defined in the telecommunication standard. In this way, we can train and evaluate the underlying DEQN-based DRL strategies in realistic 5G application scenarios. It is important to note that in our work we treat the unprocessed soft spectrum sensing information as the input states of the DRL agent. Soft spectrum sensing information can be directly obtained from spectrum sensing sensors. Through the soft spectrum sensing input, the DRL agent will learn an appropriate detection criterion for each SU that adapts to different mobile wireless environments, geometry of mobile users, and activities of mobile users. This is indeed the first work to study DSS that combines soft spectrum sensing information and spectrum access strategies through the DRL framework.

We now formulate the DSS problem using the DRL framework, where all SUs in the secondary system learn their spectrum access strategies in a distributed fashion through the interactions with the mobile wireless environment. To be specific, we assume that each SU has a DRL agent that takes its observed state as the input and learns how to perform spectrum sensing and access actions in order to maximize its cumulative reward. The reward for each SU is designed to maximize its spectrum efficiency and to prevent harmful interference to PUs.

The state of SU $n$ in the $k^{\text{th}}$ sensing and transmission period is denoted by

where $k$ is a non-negative integer, $E^{n}[k]$ is the energy of received signals, and $Q^{n}[k]$ is a one-hot $M$-dimensional vector indicating the sensed channel from time slots $kT+1$ to $kT+T_{s}$. If the index of the sensed channel is $m$, then the $m^{\text{th}}$ element of $Q^{n}[k]$ is equal to one while other elements of $Q^{n}[k]$ are zeros. On the other hand, $E^{n}[k]$ is equal to $E^{n}_{m}[kT]$ that is calculated by Equation (2).

The action of SU $n$ in the $k^{\text{th}}$ sensing and transmission period is denoted by

where $q^{n}[k]\in\{0,1\}$ represents SU $n$ will either access the current sensed channel ($q^{n}[k]=1$) or be idle ($q^{n}[k]=0$) during the data transmission part of the $k^{\text{th}}$ period (from time slots $kT+T_{s}+1$ to $(k+1)T$), $z^{n}[k]\in\{1,...,M\}$ represents SU $n$ will sense channel $z^{n}[k]$ during the sensing part of the $(k+1)^{\text{th}}$ period (from time slots $(k+1)T+1$ to $(k+1)T+T_{s}$). In other words, SU $n$ makes two decisions: $q^{n}[k]$ decides whether to conduct data transmission in the current sensed channel of the $k^{\text{th}}$ period and $z^{n}[k]$ decides which channel to sense in the $(k+1)^{\text{th}}$ period. Therefore, the dimension of each SU’s action space is $2M$. Note that the sensed channel in the $k^{\text{th}}$ period may be different from that in the $(k+1)^{\text{th}}$ period

In our work, we use a discrete reward function which is similar to the existing DRL-based DSS methods. Compared to a simple binary reward ($0$ and $+1$, $-1$ and $+1$) in [17] and [18], we consider a more relevant and comprehensive reward design that is based on the underlying achieved modulation and coding strategy (MCS) adopted in the 3GPP LTE/LTE-Advanced standard [20]. To be specific, a receiver measures SINR to evaluate the quality of the wireless connection and feedback the corresponding Channel Quality Indicator (CQI) to the transmitter [21]. In this work, we follow the method presented in [22] to map the received SINR to the CQI. After receiving the CQI, the transmitter determines the MCS for data transmission based on the CQI table specified in the 3GPP standard [20]. The SINR and CQI mapping to MCS is given in Table I for reference. Accordingly, the achieved spectral-efficiency can be calculated by (bits/symbol) = (modulation’s power of 2) $\times$ (code rate) representing the average information bits per symbol. This critical metric is utilized as the reward function of our design.

To jointly consider the performance of the primary and the secondary systems, the reward function corresponding to SU $n$ accessing channel $m$ depends on both the spectral-efficiency of SU $n$ and PU $m$. During time slots $kT+T_{s}+1$ to $(k+1)T$, the average spectral-efficiency of SU $n$, $\bar{e}^{n}[k]$, and the average spectral-efficiency of PU $m$, $\bar{e}^{m}[k]$, are calculated by

where $e_{m}^{n}[t^{\prime}]$ and $e_{m}^{m}[t^{\prime}]$ represent the spectral-efficiency of SU $n$ and PU $m$ on channel $m$ at time slot $t^{\prime}$, respectively.

The reward of SU $n$ in the $k^{\text{th}}$ transmission period is defined as

To enable the protection for the primary system, PU $m$ will broadcast a warning signal if its average spectral-efficiency is below $1.5$, and then the reward received by SU $n$ that accesses channel $m$ is set to $-2$. To motivate SUs to explore spectrum opportunities, the reward $r^{n}[k]$ is set to $-1$ if SU $n$ decides to be idle in the $k^{\text{th}}$ transmission period. When PU $m$ does not suffer from strong interference (the average spectral-efficiency of PU $m$ is larger than $1.5$), we increase the reward $r^{n}[k]$ from $0$ to $3$ as the average spectral-efficiency of SU $n$ increases (see Equation (13)). Note that the low spectral-efficiency of a PU or a SU does not necessarily mean collisions because the underlying wireless channels are changing dynamically over time. If the channel gain of the wireless link is small, the spectral-efficiency of the user will be low even if there is no collision. Therefore, the reward function and the warning signal are introduced since it is impossible to detect collisions perfectly in practical wireless environments.

To capture the activity patterns of PUs, which are usually time-dependent, applying DRQNs is a natural choice. Although DQNs are able to learn the temporal correlation by stacking a history of states in the input, the sufficient number of stacked states is unknown because it depends on PUs’ behavior patterns. RNNs are a family of neural networks for processing sequential data without specifying the length of temporal correlation.

However, the training of RNNs is known to be difficult that suffers from vanishing and the exploding gradients problems. Furthermore, the required amount of training data for achieving convergence is large in the DRL scheme, since there are no explicit labels to guide the training and the agents have to learn from interacting with its environment. In the wireless environment, the channel gain of a wireless link changes rapidly, which is shown in Figure 3. Note that the environment observed by a SU is affected by other SUs’ access strategies because of possible collisions between SUs, and all SUs are dynamically adjusting their DSS strategies during their training processes. As a result, in the DSS problem, the duration for a learning environment being stable is short and the available training data is very limited.

The standard training technique for RNNs is to unfold the network in time into a computational graph that has a repetitive structure, which is called backpropagation through time (BPTT). BPTT suffers from the slow convergence rate and needs many training examples. DRQN also requires a large amount of training data because a learning agent finds a good policy by exploring the environment with different potential policies. Unfortunately, in the DSS problem, there are only limited training data for a stable environment due to dynamic channel gains, partial sensing, and the existence of multiple SUs. To address this issue, we use ESNs as the Q-networks in the DRQN framework to rapidly adapt to the environment. ESNs simplify the training of RNNs significantly by keeping the input weights and recurrent weights fixed and only training the output weights.

We denote the sequence of states for SU $n$ by $\{s^{n}[1],s^{n}[2],...\}$. Accordingly, the sequence of hidden states, $\{h^{n}[1],h^{n}[2],...\}$, is updated by

where $W^{n}_{in}$ is the input weight, $W^{n}_{rec}$ is the recurrent weight, $\beta\in[0,1]$ is the leaky parameter, and we let $h^{n}[0]=\boldsymbol{0}$. The output sequence, $\{o^{n}[1],o^{n}[2],...\}$, is computed by

where $u^{n}[k]$ is a concatenated vector of $s^{n}[k]$ and $h^{n}[k]$, and $W^{n}_{out}$ is the output weight. Note that the output vector $o^{n}[k]$ is a $2M$-dimensional vector, where each element of $o^{n}[k]$ corresponds to the estimated Q-value of selecting one of all possible actions given the state $s^{n}[1],...,s^{n}[k]$.

The double Q-learning algorithm [16] is adopted to train the underlying DEQN agent of each SU. As discussed in Section III-A, each DEQN agent has two Q-networks: the evaluation network and the target network. Let the output sequence from the evaluation network and the target network be $\{o^{n}_{\theta}[1],o^{n}_{\theta}[2],...\}$ and $\{o^{n}_{\theta^{-}}[1],o^{n}_{\theta^{-}}[2],...\}$, respectively. The loss function for training the evaluation network of SU $n$ is written as

where $o^{n}_{y,\theta^{-}}[k+1]$ and $o^{n}_{y,\theta}[k]$ are the $y^{\text{th}}$ element of $o^{n}_{\theta^{-}}[k+1]$ and $o^{n}_{\theta}[k]$, respectively, $y$ is the index of the maximum element of $o^{n}_{\theta}[k+1]$, $r^{n}[k]+\gamma o^{n}_{y,\theta^{-}}[k+1]$ is the target Q-value. To stabilize the training targets, the target network is only periodically synchronized with the evaluation network.

The input weights and the recurrent weights of ESNs are randomly initialized according to the constraints specified by the Echo State Property [23], and then they remain untrained. Only the output weights of ESNs are trained so the training is extremely fast. The main idea of ESNs is to generate a large reservoir that contains the necessary summary of past input sequences for predicting targets. From Equation (14), we can observe that the hidden state $h^{n}[k]$ at any given time slot $k$ is unchanged during the training process if the input weights and recurrent weights are fixed. In contrast to conventional RNNs that usually initialize the hidden states to zeros and waste some training examples to set them to appropriate values in one training iteration, the benefit of ESNs is that the hidden states do not need to be reinitialized in every training iteration. Therefore, the training process becomes extremely efficient, which is especially suitable for learning in a high dynamic environment. Compared to storing $(s[k],a[k],r[k],s[k+1])$ in conventional DRQN framework, we also store hidden states $(h[k],h[k+1])$ because hidden states are unchanged. In this way, we do not have to waste lots of training time and data to recalculate hidden states in every training iteration. It largely boosts the training efficiency in the highly dynamic environment since we can avoid using BPTT and only update the output weights of networks. Furthermore, we can randomly sample from the replay memory to create a training batch, while conventional DRQN methods have to sample continuous sequences to create a training batch. Thus the training data can be more efficiently used in our DEQN method. The training data stored in the buffer will be refreshed periodically in order to adapt to the latest environment. Therefore, our training method is an online training algorithm that keeps updating the learning agent. The training algorithm for DEQNs in the DSS problem is detailed in Algorithm 1.

We set the number of PUs and SUs to 4 and 6, respectively, and the locations of PUs and SUs are randomly defined in a 2000m$\times$2000m area. The distance between the transmitter and the receiver of each desired link is randomly chosen from 400m-450m. Figure 4 shows the geometry of the DSS network, where PUT/SUT represent the transmitters of PU/SU and PUR/SUR represent the receivers of PU/SU. The channel gains of desired links, interference links, and sensing links are generated by the WINNER II channel model widely used in 3GPP LTE-Advanced and 5G networks [14]. In this case, there are 4 desired links for PUs, 6 desired links for SUs, 30 interference links between different SUs, 24 interference links between SUTs and PURs, 24 interference links between PUTs and SURs, and 24 sensing links between PUTs and SUTs. Totally, 112 wireless links are generated in our simulation, which establishes a more complicated scenario than existing DRL-based DSS strategies [17, 18, 19]. Specifically, [17] considers each channel only has two possible states (good or bad) without modeling the true wireless environment; [18] assumes that the collision between users can be perfectly detected without considering the dynamics of interference links; [19] assumes that SUs are forbidden to access a channel when a PU is using without considering the actual interference links between PUs and SUs.

For each channel, the bandwidth is set to 5MHz and the variance of the Gaussian noise is set to -157.3dBm. The transmit power of PUs and SUs are both set to 500mW. We set the sensing and transmission period $T$ to 10 time slots and the sensing duration $T_{s}$ to 2 time slots, where one time slot represents interval of 1ms. We list all the parameters to generate the wireless environment in Table II.

For the activity pattern of PUs, we let two PUs be in Active state every $3T$ (PU1 and PU3) and two PUs be in Active state every $4T$ (PU2 and PU4). Each SU trains its DEQN agent and updates the policy accordingly after collecting 300 samples in the buffer. The buffer will be refreshed after training so we only use training data from the latest 3 sec. The total number of training data is 60000, which requires 600 sec to collect all the training data. The initial exploration probability $\epsilon$ is set to 0.3, and then it will gradually decrease until $\epsilon$ is 0. We first train the Q-network with learning rate 0.01, and then the learning rate decreases to 0.001 when $\epsilon$ is less than 0.2.

As shown in Figure 5, our DEQN network consists of $L$ reservoirs for extracting the necessary temporal correlation to predict targets. The number of neurons in each reservoir is set to 32 and the leaky parameter $\beta$ is set to 0.7 in Equation (14). During the training process, the input weights $\{W_{in}^{(1)},...,W_{in}^{(L)}\}$ and the output weights $\{W_{rec}^{(1)},...,W_{rec}^{(L)}\}$ are untrained. To find a good policy, only the output weight $W_{out}$ is trained to read essential temporal information from the input states and the hidden states stored in the experience replay buffer. Existing research shows that stacking RNNs automatically creates different time scales at different levels, and this stacked architecture has better ability to model long-term dependencies than single layer RNN [24, 25, 26]. We also find that stacking ESNs can indeed improve the performance in our experiment.

We evaluate our introduced DEQN method with three performance metrics: 1) The system throughput of PUs. 2) The system throughput of SUs. 3) The required training time. The throughput represents the number of transmitted bits per second, which is calculated by (spectral-efficiency) $\times$ (bandwidth), and the system throughput represents the sum of users’ throughput in the primary system or secondary system. A good DSS strategy should increase the throughput of SUs as much as possible, while the transmissions of SUs do not harm the throughput of PUs. Therefore, each SU has to access an available channel by predicting activities of other mobile users. We compare with conventional DRQN method that uses Long Short Term Memory (LSTM) [27] as the Q-network. For a fair comparison, we also set the number of neurons in each LSTM layer to 32. The training algorithm of DRQNs is BPTT and double Q-learning with the same learning rate as DEQNs. Since each SU updates its policy for every 300 samples, we show all of our curves in figures by calculating the moving average of 300 consecutive samples for clarity.

DEQN1 and DEQN2 are our DEQN method with one and two layers, respectively, and DRQN1 and DRQN2 are the conventional DRQN method with one and two layers, respectively. The system throughput of PUs is shown in Figure 6 and the system throughput of SUs is shown in Figure 7. We observe that DEQNs have more stable performance than DRQNs, which empirically proves that the DEQN method can learn efficiently with limited training data. Note that one experience replay buffer only contains 300 latest training samples. After updating the learning agent of each SU using the 300 data in the buffer, DSS strategy of each SU changes so the environment observed by one SU also changes. Therefore, we have to erase the outdated samples from the buffer and let SUs collect new training data from the environment. Figure 8 shows the average reward of SUs versus time. We observe extremely unstable reward curves of both DRQN1 and DRQN2 so it proves that DRQNs cannot adapt to this dynamic 5G scenario well with few training data.

We observe that DEQN2 has better performance than DEQN1 in both the system throughput of PUs and SUs, which shows that deep structure (stacking ESNs) indeed improves the capability of the DRL agent to learn long-term temporal correlation. As for DRQNs, we observe that DRQNs do not have improved performance as we increase the number of layers in the underlying RNN. The main reason is that more training data are needed for training a larger network but even DRQN with one layer cannot be trained well.

The top priority of designing a DSS network is to prevent harmful interference to the primary system. To analyze the performance degradation of the primary system after allowing the secondary system to access, we show the system throughput of PUs when there is no SU exist in Figure 6. We observe that DEQN2 can achieve almost the same performance of the system throughput of PUs. A PU broadcasts a warning signal if its spectral-efficiency is below a threshold. For each PU, we record the frequency of (the PU sends a warning signal and it is received by some SUs) / (number of the PU’s access), which is called as the warning frequency. Figure 9 shows the average warning frequency of each PU versus time. We observe that the every PU decreases its warning frequency over time, meaning that each SU learns not to access the channel that will cause harmful interference to PUs.

We compare the training time of different approaches in Table III when implemented and executed on the same machine with 2.71 GHz Intel i5 CPU and 12 GB RAM. The required training time for DRQN1 is 23.4 times the training time for DEQN1, and the required training time for DRQN2 is 42.8 times the training time for DEQN2. This huge difference shows the training speed advantage of our introduced DEQN method against the conventional DRQN method. DRQN suffers from high training time because BPTT unfolds the network in time to compute the gradients, but DEQN can be trained very efficiently because the hidden states can be pre-stored for many training iterations.