Spectrum Intelligent Radio: Technology, Development, and Future Trends

Spectrum sensing is a physical operation that measures the surrounding radio spectrum states based on various signal processing methods, including matched filtering, cyclostationary, energy detection for narrow-band scenario, and sub-Nyquist techniques for wideband case. However, these methods solely focus on a single parameter, the occupancy of a specific spectrum. In general, false alarm and detection probability are the most essential performance metrics. It is clear that a rich ensemble of RF activities and variations in the physical world are ignored. On the other hand, these methods usually assume a homogeneous spectrum state, where the propagation range of a PU is large enough to cover all SUs, and the whole network shares one common sparse spectrum [4]. This single-parameter sensing approach, combined with the unrealistic assumption on homogeneity, is incapable of handling complex RF environments of future massive connectivity.

Decision making consists of a series of logical operations, including sensing scheduling (when to perform sensing and which channels to sense), and channel access strategy (whether and which channels to access based on the spectrum sensing results). In addition, when only a small portion of the spectrum can be sensed at one time by low-cost battery-powered devices, spectrum sensing and channel access strategies can be treated jointly under a unified framework. Conventional studies use model-dependent approaches such as myopic (greedy) strategy, classical game theory, congestion control, matching theory, and graph coloring [5] to obtain structured solutions. Essentially, the model-dependent methods tackle a global constrained optimization problem by decomposition/decoupling, requiring the knowledge of the a priori parameters in the network, such as user activation dynamics in certain spectrum.

In practice, the complexity of spectrum environment often makes it impossible to gain enough knowledge in advance to develop the network evolution model, especially for the fast-changing RF dynamics, whose statistics is costly to obtain accurately due to its inherent complexity and practical limitations such as sensing time, hardware capability, and computational power. Furthermore, in the envisioned mMTC applications, decision making should cater for a decentralized self-organized multi-agent (multiple SUs) scenario, which is extremely challenging. All these limitations urge us to look for model-free decision-making [6]. A model-free method helps a decision-making entity interact with the environment, eliminating the need for a pre-defined network model. To achieve this, techniques more advanced beyond handcrafted approach are required, resorting to the contemporary interdisciplinary techniques, such as ML-based approaches to directly interact with the unknown RF environment.

The first level involves the autonomous multiple feature identification of signals in an unknown complicated RF environment. This allows an SU to observe network heterogeneity and dynamics from different perspectives. It extends a single feature identification in Stream 1 to a multiple one.

As discussed before, the spectrum sensing in Stream 1 mainly determines a detection threshold $\theta$ shown in Fig. 2, which is readily solvable. For example, given the target probability of false alarm and the noise power, $\theta$ can be simply determined by the Neyman-Pearson criterion. The work in [7] lifts this barrier, and considers different PU signal power levels as a new feature, potentially enabling the design of a more efficient spectrum access strategy. The goal of the multi-level case is to jointly determine multiple thresholds $\{\theta_{i}\}_{i=1}^{L-1}$ to differentiate $L$ multiple power levels, which is far more complicated than the binary one. In essence, there are $L(L-1)$ possibilities of errors that are intertwined to exacerbate the complexity in threshold calculation. In [7], closed-form thresholds were determined by assuming the same cost for all the errors. Note that a number of parameters (power mode) are required a priori, including the number of power levels, the value of powers, the noise power, and the prior probability of hypotheses. In theory, this problem can be solved by labelling the collected samples from different PU power levels [7]. However, this is not very practical in reality.

To remove the limitation of the existing work and achieve multi-level spectrum sensing with no or minimal prior information, we recently proposed a data-driven/ML based scheme [8]. It is fully blind in the sense that the SU does not require any prior knowledge of the power mode. Specifically, the proposed spectrum sensing scheme spans across two stages as shown in Fig. 2. In Stage I (spectrum learning, a.k.a the training phase in ML), the SU collects a multitude of signals long enough to ensure all the possible PU transmit power levels have been operated with a high probability. Then, a Gaussian mixture model (GMM) is used to capture the multi-level power characteristics of the signals. Finally, we introduce a Bayesian nonparametric method, referred to as conditionally conjugate Dirichlet process GMM (CCDPGMM), to automatically cluster the signals with the same PU transmit power and infer the model parameters (GMM parameters and PU power level duration distribution parameters). With the model parameters inferred in Stage I, the prediction part in Stage II can easily identify the current PU power level from the collected PU signal samples. In this way, Stage I together with the prediction part in Stage II achieve fully blind multi-level spectrum sensing.

The simulation results are presented in Fig. 3 to compare the accuracy of PU power level identification $P_{c}$ for the proposed CCDPGMM, expectation maximization GMM (EMGMM), and mean shift (MS) with respect to the average received SNR $\gamma_{st}$. The EMGMM is a parametric clustering method that requires the prior knowledge of $L$, while the CCDPGMM and MS belong to a nonparametric class without the need for such prior knowledge. We set the power level $L=4$ and the probability of each hypothesis ${\rm P_{r}}(H_{l})$ = 0.25. It is assumed that the noise variance is 1, the PU transmit powers $P_{1}:P_{2}:P_{3}=1:2:3$, and $P_{4}=0$. It is shown that the proposed CCDPGMM significantly outperforms MS (for example, a 2.2 dB gain at $P_{c}=0.6$), and is only slightly inferior to EMGMM. At $\gamma_{st}=-12$ dB, CCDPGMM achieves $P_{c}=0.96$. Note that the accurate estimation of the power levels is important, or else it will degrade the performance in Stage II.

Clearly, in order to achieve a full RF cognition, future networks demand automated extraction of far more features with no or minimal prior information. Preferably, the physical layer information (spectrum occupancy, transmit power level, modulation, constellation, and channel coding) and upper layer features (application types, network topology, and communication protocols) should be mined under a unified framework. In this line, a deep convolutional neural network (CNN), built on the layers of convolving trainable filters, can perform nonlinear approximations and emerge as a tool of the future to automate the extraction of a multitude of features. This represents a new trend for RF landscape perception.

The second level learns the intricate structure of the RF environment in a large-scale complex network, and establishes the ongoing RF activity map. This may include understanding large and small-scale wireless channel propagation conditions and identifying non-collaborative wireless systems. Network complexity is mainly embodied in heterogeneity and dynamics. For example, a future network could include multiple PUs across a large area, and the spectrum states could vary significantly from place to place within the network. This heterogeneous property stymies the accurate spectrum sensing and availability prediction for a given location. Considering the potentially large number of spectrum states, one can deploy many static SUs at different geographic locations to carry out spectrum sensing simultaneously, and then collate all the sensing results to identify all the different spectrum states. However, this is highly undesirable due to the high deployment cost.

To understand the RF environment in a more cost-efficient way, we exploit the mobility nature inherent to most wireless devices to explore the spectrum footprint across a network [9]. Specifically, a small number of SUs simultaneously sample the RF channels while moving along, and each SU sends its sensing results to its nearby cluster head (CH). These CHs are interconnected and exchange information with each other, and thus the global spectrum states are derived cooperatively. This basic collaborative concept is illustrated in Fig. 4. Mathematically, we proposed a beta process sticky hidden Markov model (BP-SHMM) to represent and capture spatial-temporal correlation among RF samples. Then, Bayesian inference is adopted to classify sensing samples into different classes in an unsupervised manner. In essence, all the spectrum samples in each class have the same spectrum state. Then the locations of PUs and their transmission ranges can be inferred from the classification results. In specific, our calculation indicates that the coverage radius prediction error is 2.8%. Consequently, a newly joined SU could have almost instantaneous access to the available frequency channels without sensing. The accuracy is much higher than individual SU sensing, and it also brings an important benefit of low latency.

The superiority of our proposed method is illustrated by an example in Fig. 5. It is clearly shown that all the six spectrum states (each represented by a different color) were identified successfully and the coverage areas predicted, both with high accuracy.

The proposed method mainly focuses on the spectrum heterogeneity. Handling the envisioned scenario with fast-changing dynamics and interference is still an open problem. A potential way is to design two coupled SHMMs. The first one models the latent statistical correlation within each mobile SU’s sequential sensing samples in terms of the physical layer parameters, while the second one targets the application layer parameters interpreting the dynamics from an application perspective and identifying interference. This may allow us to gauge the mutual influence of the physical layer and application layer parameters simultaneously.

At the third level, SU can perceive its surrounding spectrum environment and make use of its understanding of environmental features to reason how to seamlessly access the spectrum resources shared among devices, minimizing the interference to the PU. Essentially, multi-user dynamic spectrum access is a highly challenging distributed decision-making problem [10], as the network utility should be maximized without any online coordination or inter-user information exchange. As explained before, when facing a complex dynamic network, model-dependent decision-marking approaches are no longer effective. In contrast, model-free learning enables the SU to interact with the environment directly and then adjust its behavior by reinforcement learning, the key to realizing intelligent spectrum.

In practice, a battery-powered SU can only perceive a small portion of the available channels at an instance, and needs to select a possible idle channel(s). As such, dynamic multi-channel access problem can be modeled as a partially observable Markov decision process (POMDP) [10], which is the generalization of an MDP. When the network dynamics (the state transition probability of the channel) is unknown and the channels are correlated, the POMDP becomes very challenging due to high-dimensional and large state-space issues. The popular reinforcement learning approaches, such as Q-learning, are intractable to solve it. To address this issue, the work in [11] uses a deep neural network (DNN) approximate Q-values, replacing the conventional look-up Q table. However, DRL usually requires a large number of training samples to tap the strong mapping capability of DNNs [12]. However, only a single sensing sample for one channel can be collected in each time slot, and it results in a low learning rate and channel selection accuracy.

We propose a novel model-free Gaussian process reinforcement learning (GPRL) based solution. As a Q-function approximator in RL, the GP is embedded to measure data correlation using a kernel function and update the posterior Q-value distribution via the Bayesian theory. On this basis, the past learning experience can be utilized in a more efficient way compared to the DNN-based Q-function approximators. This accelerates learning and eliminates the need for a large number of training samples. Fig. 6 compares the channel selection accuracy of the proposed GPRL, DRL, Whittle index (WI), and the optimal (ideal) method for a total of 16 channels. The results of DRL and GPRL are obtained via an online learning process with learning and testing phases of 120 and 30 time spans, while WI and the optimal method are directly applied in the testing phase. It is clearly shown that GPRL outperforms DRL significantly for increased $U$, and achieves an accuracy close to the optimal one.

However, GPRL and DRL only suit a single-user scenario. The multi-user setting is significantly different in environment dynamics, network utility, and algorithm design, which is much more challenging to solve than the single-user case. Due to interactions among users, it is highly desirable to develop a model-free distributed multi-user method without coordination or message exchange among users.