Fast Reinforcement Learning
for Anti-jamming Communications

Wireless communications are highly vulnerable to jamming attacks due to the open and sharing nature of wireless medium [1]. In general, attackers jam the ongoing transmissions via injecting malicious signal to wireless channel in use. To deal with the jamming attacks, the frequency hopping was proposed by selecting “good” frequencies in an ad-hoc way and avoiding the jammed frequency [2, 3]. But, it is no guidance for the random frequency hopping techniques to select a channel that is not blocked. Soon afterwards, Wang et al. [4] proposed an uncoordinated frequency hopping method by using the online learning theory to mitigate this problem. However, this conventional online learning method achieves only asymptotic optimum.

Recently, the success of reinforcement learning (RL) in decision-making problem attracts researchers to apply Q-learning (QL) to anti-jamming wireless communications. For instance, Xiao et al. [5, 6] employ QL to choose an appropriate transmission power. In [7, 8], QL is applied to select optimal frequency hopping channel. However, as the dimension of the actions increases, the Q-table of QL will become too large to quickly derive the optimal policy. Subsequently, deep Q-network (DQN) [9] was proposed to solve such a high-dimensional optimization problem. Different from QL, the Q value of DQN is not calculated by the state-value function, but learned by the neural networks such as convolutional neural network (CNN) and recurrent neural network (RNN). For example, in order to accelerate the learning speed and enhance the anti-jamming communication performance, Han et al. [10] apply DQN to choose the optimal communication channel, and Liu et al. [11] combine DQN and spectrum waterfall to improve the ability of exploring the unknown environment. With the rapid development of RL, double DQN (DDQN) [12] has been proposed, which combines two deep networks to avoid the dependency between the Q value calculation and network update. During the backpropagation of DDQN, only one network used for action selection is updated and its parameters are directly copied to the other network with a constant frequency. Hence, to some extent, DDQN mitigates the overestimation problem. Furthermore, prioritized experience reply based DQN (PDQN) [13] was developed to optimize the experience replay of DQN. Interestingly, researchers promptly set out to combine DDQN and PDQN together (a.k.a. PDDQN) to improve the convergence rate and avoid the overestimation problem, such as in game design [14]. Besides, the action repetition theory [15] was proposed to further enhance the convergence rate of DQN, whereas optimizing the action repetition rate becomes difficult. It is important and challenging to apply PDDQN to solve the optimization problem with the anti-jamming wireless communications.

In this letter, we present a fast reinforcement learning algorithm called $(\tau,\epsilon)$-greedy. The key idea of $(\tau,\epsilon)$-greedy is to preserve previous action with probability $\tau$ and apply $\epsilon$-greedy with probability $1-\tau$. Specifically, probability $\tau$ is formulated as a Gaussian-like function such that the greater value the previous action has, the larger value $\tau$ takes. By doing so, the optimal action can be selected with a higher probability and thus the learning process is significantly accelerated. As a concrete example, the proposed algorithm is implemented in a wireless communication system against multiple jammers. Experimental results show the superiority of the proposed algorithm over existing value-based RLs including QL, DQN, DDQN, and PDDQN. Especially, the $(\tau,\epsilon)$-greedy based PDDQN achieves the best performance in terms of signal-to-interference-plus-noise ratio (SINR) and convergence rate.

The rest of this letter is organized as follows. In Section II, we introduce an anti-jamming wireless communication model. The proposed algorithm is described in detail in Section III. Section IV provides experimental results and conclusions are drawn in Section V.

As depicted in Fig. 1, we consider a wireless communication system where a sender transmits data to the receiver with a power $P_{S}(s)$, while there are $J$ jammers who can simultaneously launch jamming attacks with $L$ different power levels on channels. The $L$ jamming power levels are denoted as $P_{J}(l)\in\{P_{J}(1),P_{J}(2),...,P_{J}(L)\}$. In order to resist this complicated attack, the sender dynamically chooses an appropriate $P_{S}(s)$ from $S$ different power levels $\{P_{S}(1),P_{S}(2),...,P_{S}(S)\}$, which in general is shown to be superior to the use of constant power under the constraint of the same average power [16]. Note that both the sender and jammers share the same $N$ frequency channels. At time slot $k$, the channels chosen by the sender and $J$ jammers are respectively denoted by $x^{(k)}$ and $\{y_{1}^{(k)},y_{2}^{(k)},...,y_{J}^{(k)}\}$. $h_{s}$ and $h_{j}$ denote the channel power gains from the sender and the $j$th jammer ($j\in\{1,...,J\}$) to the receiver, respectively.

In the above communication system, after the receiver gets the signal at time slot $k$, the SINR is calculated by Eq. (1) and returned to the sender in the feedback channel.

where $\beta$ is the receiver noise power, $P_{J}^{j}(l)$ denotes the jamming power chosen by the $j$th jammer, and $f(\xi)$ is an indicator function that equals 1 if $\xi$ is true and 0 otherwise. If the receiver did not get the signal at time slot $k$, the transmission channel is considered to be blocked by the jammers. In this case, the receiver informs the sender about the channel state information through the feedback channel. Then the sender retransmits the signal. The additional cost caused by the retransmission is denoted as $C_{m}$. For simplicity, we assume that the feedback channel cannot be attacked and the transmission channel is considered to be blocked if $P_{J}(l)$ takes the maximum value $P_{J}(L)$. Besides, the cost of the unit transmit power is denoted as $C_{s}$. In [17], the authors formulated the utility function by considering the transit cost is proportional to the transmit power. Based on the SINR, retransmission cost $C_{m}$, and transit cost, in this letter the utility $u_{s}^{(k)}$ at time slot $k$ is defined by

It is known that the anti-jamming wireless communications aim to maximize the communication performance while at the same time save energy as much as possible. By referring to Eq. (2), our designed utility function has the ability of making a good tradeoff between the cost and communication performance.

The proposed algorithm consists of three modules: $(\tau,\epsilon)$-greedy action policy, double deep network structure, and prioritized experience replay, as shown in Fig. 2. The system state at time slot $k$ is denoted as $s^{(k)}=\textrm{SINR}^{(k-1)}$, where $\textrm{SINR}^{(k-1)}$ is the SINR at time slot $k-1$. Based on the current state $s^{(k)}$, the sender selects an action containing the selected transmission channel $x^{(k)}$ and power level $P^{(k)}_{S}(s)$ (denoted as $\textbf{a}^{(k)}=[x^{(k)},P^{(k)}_{S}(s)]$). After the action is adopted, the sender receives a reward (denoted as $u^{(k)}_{s}$). In the following, we describe each module in detail.

In this section, a number of experiments is conducted to evaluate the proposed method. In our experimental setup, the parameters are set as $J=2$, $N=32$, $P_{S}=[1,5,10]$W, $P_{J}=[0,4,8,10]$W, $h_{s}=h_{j}=0.5$, $\beta=1$, $C_{m}=1$, $C_{s}=0.2$, $\gamma=0.6$, $T=5$, $\sigma_{1}=0.8$, $\sigma_{2}=85$, $W=32$, $M=10$, $f=10$, and epoch=400.

First, we implement the existing value-based RL methods including QL, DQN, DDQN, and PDDQN in a wireless communication system. These four methods are all based on $\epsilon$-greedy. The SINR performance and convergence rate are analyzed and the results are shown in Fig. 3(a). As expected, PDDQN performs the best among the four methods. Hence, PDDQN is selected. This is the first time to our knowledge that PDDQN is applied to the anti-jamming wireless communications. Next, we compare the SINR performance between the variable and constant transmission power models under the same average power $\overline{P}$ condition. For a fair comparison, both of the models use PDDQN and the comparison results are given after the convergence. According to our experiment, PDDQN becomes completely convergent after 200 time slots, and the average power $\overline{P}$ is 6.19W in the variable power model. We can see from Fig. 3(b) that the variable power model obtains much higher SINR than the constant power model under the condition of the same average power $\overline{P}=6.19$W. Therefore, the variable transmission power model is adopted. Finally, we apply the proposed $(\tau,\epsilon)$-greedy to the four value-based RL methods, which are named $(\tau,\epsilon)$-QL, $(\tau,\epsilon)$-DQN, $(\tau,\epsilon)$-DDQN, and $(\tau,\epsilon)$-PDDQN. The curves of SINR performance and convergence rate are drawn in Fig. 4. It is clear that the convergence rate with the proposed $(\tau,\epsilon)$-greedy is greatly improved for the four RL methods. Moreover, the communication performance (measured by SINR) with $(\tau,\epsilon)$-greedy is also much better than those without $(\tau,\epsilon)$-greedy. This is mainly due to the fact the proposed algorithm can not only select the more valuable action in most cases but also seek for the most valuable action at a faster speed.

In this letter, we have presented a $(\tau,\epsilon)$-greedy RL algorithm. Our proposed $(\tau,\epsilon)$-greedy can replace the existing $\epsilon$-greedy which is used in almost all the value-based RL methods. We have implemented the proposed algorithm in a wireless communication system against multiple jammers. The experimental results show that the proposed algorithm accelerates the learning speed of the wireless device and significantly improves the performance of existing RL methods in terms of SINR and convergence rate. In the future, this algorithm will be implemented in the anti-jamming wireless communication scenario for continuous action set.