Self-Evolving Wireless Communications: A Novel Intelligence Trend for 6G and Beyond

In current wireless communications, adaptive technology is one of the main solutions to minimize human involvement and enhance automation. It can adjust system parameters to improve communication quality according to different environments, such as channels and interference. It has been widely used in the fourth generation (4G) for resource allocation and network optimization[2]. The optional modulation coding scheme (MCS) is added as a set of user-selectable waveform parameters in 5G. It is still very difficult to adapt links to meet diverse constraints and requirements, including rate, delay, and reliability. In 6G, more diverse types of terminals will work in intelligence-to-intelligence communication scenarios, requiring timely information on the terminal, base station, and network sides. Furthermore, the 6G network will be changed from signal-data content awareness into scenario-semantic awareness to support the connected intelligent devices. Traditional communication networks simply address the services around people, such as voice, image, and video. Machine-to-machine communications are expected to be dominant in 6G. Therefore, more sensing-fusion-decision-data-knowledge-based learning is needed to realize various tasks, such as autonomous collaboration of different types of robots/unmanned systems from different vendors. At this point, the purpose of the future network is to connect multiple intelligence together rather than simply communications. In adaptive communications, intelligence is mainly centralized or semi-centralized, which can’t adapt to the future fully distributed terminal communication scenarios [3]. Therefore, network architecture, resource management, and physical layer transmit technologies in future communication will be inherently intelligent and self-evolving, which is different from previous adaptive systems. In Table I, we present the comparisons between adaptive communications and future communications.

The term “self-evolution” first appeared in [4] in 1997. Compared to the sparse distributed memory which is a static and inflexible system, the authors removed some problematic features of it and directly learned input patterns to achieve robustness and self-evolution. However, the mechanism and process of self-evolution were not clearly presented in [4]. The meaning of “self-evolution” was first elaborated in 1998 in [5]. Self-evolution there refers to an evolutionary process in a system where the evolutionary behavior was carried out by the individuals of the system itself. The authors explored self-evolution in a binary string system in extinction, emergences of an organization, exploration-innovation, and evolution processes. Their work was motivated by chemical reaction dynamics, where molecular collisions and interactions produce new molecules that form metabolic networks. Instead of using mutation, selection, or recombination operations, a series of logical operations, e.g., AND, OR, NOT, EXOR, and EQ, and reactor algorithms were utilized in [5], where evolutionary phenomena were observed. A self-evolving knowledge base was proposed in 2005 [6], and it indicates the ability to adjust the system configuration to improve its performance over time. It was originally developed for military systems that exploit a dynamic knowledge base for adaptation to address various tasks, such as self-learning, self-management, and self-diagnosis. The characteristics and structure of self-evolving systems were first shown in [6]. These work laid the basic prototype of the self-evolving technologies.

There are various evolutionary algorithms according to the Darwinian theory of evolution and the main branches of them are genetic algorithms, evolution strategies, evolutionary programming, generic programming, and classifier systems [7]. These genetic algorithms can be used to search for optimal or sub-optimal solutions in specific optimization problems. A self-evolutionary unicast routing protocol was proposed in [8], where the cellular attractor selection mechanism was used to select the next hop until the best routing path was found. Self-evolution is reflected in the adaptability and robustness of this protocol. Ye et al. [9] proposed a self-evolving service composition approach to deal with the evolving environments rapidly over time. It comprised five stages: service discovery, candidate selection, service negotiation, task execution, and network evolution. They were jointly optimized to make this service composition approach more efficient. In [10], a deep self-evolution clustering was proposed to jointly learn representations and cluster data. The proposed self-evolution mechanism is reflected in the pair-wise patterns selection. With the progress of learning, the algorithm gradually chose those patterns with maximum likelihood to improve the cluster accuracy and refine the model. An alternating iterative optimization algorithm, self-evolution clustering training, was also studied in [10] to select more accurate pair-wise patterns for training. Simulation results had shown superior performance compared with other cluster algorithms. Apart from these, an algorithm called deep deterministic policy gradient (DDPG) was utilized in [11]. The network of DDPG is trained off-policy with samples from the replay buffer to minimize sample correlation. At each time step, the self-evolution of the network is accomplished by updating the samples in the replay buffer when the buffer reaches its capacity. In multi-task optimization problems, the uncertainty and challenges may increase, which can cause negative effects on optimization performance. A multisource knowledge transfer mechanism was proposed in [12] to mitigate the effects. Based on the evolution experience, it estimated the likelihood of applying knowledge transfer adaptively and balanced the self-evolution in each task and knowledge transfer within multiple tasks. A novel self-evolving and transformative (SET) protocol was proposed in [13] to support various services in 6G. The decomposition of feasible solutions can provide a library of control blocks to solve a wide range of problems. When an existing function cannot solve a new situation, new control functions can be created. The expanding function library causes SET protocols to self-evolve. The work presented in [3] introduces a novel self-evolving network framework based on the intelligent vertical heterogeneous network architecture. The proposed framework addresses conflicts and manages coordination among multiple automated network entities while satisfying Quality of Experience (QoE) requirements of diverse user equipment (UEs). By leveraging artificial intelligence and machine learning, the framework enables dynamic and intelligent management of heterogeneous networks in an integrated network.

To sum up, the above self-evolution-related work has a common mechanism: first, detect the new environment or service, then select the individuals who achieve the best performance in the new environment or service, and then replicate them, so as to achieve system self-evolution. Next, we will extend the concept of evolutionary communications based on this idea.

To develop self-evolving communication systems, let us start with the basic meaning of evolution. Originally, evolution refers to the gradual change, development, and transition of things from one state to another, usually towards the direction of being more advanced, more complex, and more perfect. For example, we can regard the source code, channel code, modulation, and demodulation as the “genes” of a communication system. The process of learning and updating communication knowledge from the environment can thus be regarded as a process of “genetic mutation”, “genetic variation”, and evolution. Gradually, the genome of a self-evolving communication system will become entirely new and can better adapt to new environments.

For self-evolving communications, knowledge acquired from the environment plays an important role in the evolutionary process. To better illustrate the process of knowledge acquisition, we borrow some concepts from Szpankowski [14]. The characteristic parameters in communication systems, such as the distribution and model of the signal or noise, are extracted from the sampled signal data and synthesized into valuable information. The learning algorithms then aim to learn some actionable insights from this information and integrate these insights into knowledge.

Based on the above discussion, a self-evolving communication system can adapt to different unknown environments, conduct cognitive reasoning, and continuously update the knowledge base to improve communication performance over time. It is typically unsupervised.

In Fig. 1, we compare self-evolving communication systems with other communication systems. In addition to the adaptive technologies previously discussed, cognitive radio is also capable of sensing the environment and adapting its transmission parameters to smartly allocate resources and improve communication performance [15]. Cognitive radio only has some degree of automation and adaptation. It still requires manual assistance and lacks full intelligence.

In the past 20 years, machine learning, especially deep neural networks, has flourished and has led to many AI-based cognitive radio or adaptive intelligent communication frameworks. However, intelligent communication frameworks are still at the level of communication signal processing and network management and are unable to process massive data and acquire the information and knowledge to evolve communication systems and networks. Therefore, they cannot adapt to broad intelligent application scenarios in the 6G and beyond. More than that, current AI-based communication frameworks focus on observation and adaptation and lack endogenous learning and cognitive reasoning capabilities [15]. For example, AI-based adaptive technologies can only use limited information to choose fixed policies in terms of decision-making. However, the self-evolving communication system proposed in this paper is able to cognitively recognize information about various types of intelligent application scenarios and can continuously learn, reason, accumulate, and update environment and policy knowledge. On top of that, it can modify and improve the policy base for different environments through contention.

In Fig. 2, we illustrate a self-evolving communication system. It includes the data layer, the information layer, and the knowledge layer. The data layer consists of conventional communication modules, such as transceiver, channel, and noise. The information layer includes four modules: application scenarios, environment sensing, intelligent decision-making, and intelligent waveform generation. The knowledge layer consists of four modules: knowledge generation, knowledge evaluation, knowledge reconstruction, and knowledge utilization. Information acquisition, knowledge learning, knowledge externalisation, and information externalization are accomplished through interaction among the three layers.

As shown in Fig. 2, the data layer includes conventional communication modules and addresses the problems of reliable transmission of signals by using information theory and signal theory. The data layer also allows us to acquire information about environments, signal waveforms, etc., from various application scenarios. The measured status data from the communication modules will be fed into environment sensing modules, intelligent decision-making modules, and intelligent waveform generation modules in the information layer, where they are analyzed to obtain useful information. The circular process of acquiring information between these three modules and application scenarios is called the information chain. The obtained information is used in the learning phase of these modules.

The information layer modules cycle through learning iteratively and generate new knowledge for the knowledge layer. The knowledge layer conducts cognitive reasoning by evaluating the validity of the knowledge, adding new useful knowledge, or removing aged one through knowledge evaluation and reconstruction. The new knowledge at the knowledge layer will be utilized in the information layer and eventually the data layer for optimizing communication systems. As shown in Fig. 2, these processes are called knowledge and information externalization, respectively. The circular process from knowledge generation to knowledge utilization in the knowledge layer is called the knowledge chain. The feedback from the knowledge layer or knowledge externalization determines the new direction in which the communication system will evolve.

Based on the above discussion, a self-evolving communication system may briefly entail the following features:
$\bullet$ Self-learning: Through cognitive reasoning, acquiring critical knowledge that affects system performance and can adjust system parameters timely for improving performance under different and probably new scenarios.
$\bullet$ Self-evaluation: Evaluating the performance of the corresponding communication strategies and then determining a communication strategy that best suits the new environments.
$\bullet$ Self-management: Dynamically updating knowledge in the knowledge base. Removing the knowledge that will lead to poor system performance or has a low probability of being used. Adding the new knowledge resulting from interaction with new environments for evaluation and application.
$\bullet$ Automatic: The self-evolving communication system may initially be able to organize the data itself through some approaches. Or, for the sake of efficiency, to manually observe and process some data first and provide some useful knowledge to the system. But eventually, there should be little manual intervention in the system.

To demonstrate the power of self-evolving modules, we compare the performances of a communication system with and without evolution. We first take the ELM technique as an example, which has a single hidden layer neural network (NN) that discovers relationships between input and output data by imitating human neurons. It not only has a relatively simple network structure but also needs no updating on network parameters iteratively. It works in two steps: feature extraction and mapping. Feature on the input data is extracted and the parameters between the input layer and the single hidden layer are assigned randomly. Then, the parameters between the single hidden layer and the output layer are calculated through matrix operations and finally are mapped to the output.

Due to the random parameter assignment in the first step, the effect of feature extraction cannot be well guaranteed. Therefore, the self-adaptive evolutionary extreme learning machine (SaE-ELM) has been proposed in [16] to extract features more effectively by utilizing the self-adaptive differential evolution algorithm. The idea is to predefine a parameter set and a policy set. In each iteration, a policy and a parameter will be selected from the policy set and the parameter set to update the current parameters, finally making the network evolve continuously and obtain better performance.

Consider a point-to-point communication system with one single antenna at both transmitter and receiver and with binary phase shift keying (BPSK). The channel is with Rayleigh fading. The SaE-ELM and ELM are adopted as detection modules, which represent communication systems with and without self-evolving elements, respectively. At each signal-noise ratio point, we generate 100,000 training data sets, including channel matrix as input and BPSK symbols as output, of which 70% are used for training and 30% for testing. We adopt four parameter-changing strategies in [16] and set $\xi$ to 0.01, which is a small positive constant to avoid the possible null success rate. The number of hidden neurons of both SaE-ELM and ELM is 1000, and the batch size is 100. The number of iterations of SaE-ELM is set to 50, and the size of the parameter set is set to 100. We assume a classical detection method (Zero-forcing) as a baseline without machine learning. From Fig. 3, there is nearly a 6 dB bit error rate (BER) performance gap between the baseline and SaE-ELM method. The BER of the SaE-ELM method with the adaptive differential evolutionary algorithm is about 2 dB higher than that of the ELM method. The above results indicate that the SaE-ELM method can effectively avoid the limitations of parameter control in the ELM method and further improve the generalization performance of the network by introducing the adaptive differential evolutionary algorithm to optimize the network parameters.

In addition to communication system detection performance, we also investigate the rendezvous performance of unmanned aerial vehicle (UAV) systems with and without self-evolution in this work. To achieve joint space-frequency rendezvous for a swarm of UAVs, a Q-learning-based algorithm that directly optimizes the policy actions of the UAVs using reinforcement learning techniques is proposed. The algorithm maintains and updates a value table or value function through the Bellman equation, whereby the action with the greatest value is selected.

However, due to the difficulty of Q-learning in handling massive or continuous state and action spaces, the algorithm may fail to converge. To address this limitation, we propose a self-evolving and Q-learning-based (SE-QL) joint space-frequency rendezvous method for UAV swarms. The method uses the state at the end of each Q-learning round as part of the initial population, randomly generates states as other individuals, and then performs selecting, copying, exchanging, mutation, and updating. The individual with the best performance in the population is selected using a genetic algorithm and used as the initial state for the next round of Q-learning, leading to improved space-frequency rendezvous performance and accelerated convergence compared to the classic Q-learning method without evolution.

The proposed method is evaluated on a simple UAV rendezvous scenario where multiple UAVs form clusters in an open two-dimensional region, and UAVs within the same cluster need to rendezvous at the same frequency to establish communication. We randomly generate the initial positions of 15 UAVs and set the number of channels to 3, the learning rate $l_{r}$ of Q-learning to 0.85, and the decay factor to 0.99. Further, we use the Q-learning method without self-evolution as the baseline and compare it with the Q-learning method with self-evolution. In Fig. 4, results indicate that for a sensing range of 25m, the Q-learning method with self-evolution uses 80% fewer cycles than the baseline to reach convergence, resulting in improved algorithm efficiency. Additionally, for a sensing range of 35m, the algorithm converges more rapidly. Notably, for a sensing range of 15m, the convergence rate of the baseline is still twice as fast, highlighting the robustness of the proposed algorithm. Our findings demonstrate that the SE-QL method is effective in circumventing the issue of poor algorithm convergence in larger state spaces and can better leverage the airspace motion capability to manipulate the network topology and improve the efficiency of UAV swarm frequency rendezvous.

Characterization of environments refers to the representation of environments by some characteristic parameters. In the past, we usually characterize electromagnetic environments by the six domains, including space, time, frequency, energy, polarization, and modulation. However, they will not be enough for future communications. For example, the means for jamming will be more diversified and intelligent. Traditional six-domain characterization can hardly portray such a complex electromagnetic environment well in the future. In addition to the above, the future electromagnetic environment should also include signal density, variety, dynamism, and intense confrontation.

The classification and quantification method of the electromagnetic environment is in the standard IEC-TR-61000-2-5 proposed by the International Electrotechnical Commission (IEC). It begins with the electromagnetic environment by counting the values of specific measurement parameters (e.g., frequency band, signal rise time, voltage) based on different electromagnetic phenomena in different platforms. This solution focuses on environmental effects. From the point of view of application requirements, the cognition of electromagnetic signals can be divided into the signal level and information level macroscopically. At the signal level, parameters, such as time, frequency, motion, polarization, waveform, modulation, and baseband signal, can be analyzed to solve application requirements, such as spectrum management, signal identification and analysis, and feature measurement. At the information level, the information of the electromagnetic environment can be grasped by analyzing the encoding and encryption of signals so as to facilitate decision-making and solve the application requirements of information acquisition, and management control.

There are two approaches to establishing knowledge bases. First, a knowledge base can be designed by a group of experts or formed using known expertise [22]. Such a knowledge base is typically well-curated, professional, and capable of producing highly accurate results. However, it relies too much on experts or expert knowledge, and is hard to adapt to new environments, applications, and technologies. In real communication systems, we often encounter changing environments and a wide variety of data, including signal data, channel information, network routing information, etc., in which case expert knowledge may not be applicable. Knowledge bases can also be constructed by purely machine learning means (e.g., deep reinforcement learning (DRL) [19]) based on real-time communication data, relevant policy evaluation, and real-time simulation results. Automatic knowledge base construction approaches cater to the future big data era and are gaining more and more attention. Since the DRL algorithm may suffer from the model mismatch, exploration risk, and slow convergence, digital twins can be utilized as a potential technique to mitigate these risks by creating a real-time high-fidelity digital visualization model of physical objects [19]. In addition, with the help of relational machine learning, knowledge graphs can effectively discover new facts about physical objects or new relationships among objects, which can be utilized to construct the knowledge base [22].

As communication networks gradually evolve towards 6G, higher frequency spectrum (e.g., tera-hertz band) will be utilized, the data rate will reach tera-bit per second, the required end-to-end (E2E) delay will be less than one millisecond, system reliability requirements will be stricter. At the same time, a large number of intelligent transceiver devices will appear. As a result, the signal processing complexity will become higher, and optimization problems will be more elaborate. All the above issues demand the computing ability of future communication systems.

Mobile edge computing (MEC), a new computing platform, can provide services and cloud computing capabilities required by users on the wireless side. It is considered as a potential for ultra-reliable and low-latency communications (URLLCs) [19]. MEC has lower delays than centralized cloud computing and may achieve one-millisecond latency with guaranteed reliability. Recently, distributed learning (DL) and federated learning (FL) have been investigated to mitigate the high information exchange overhead between the central controller and other base stations and the high computation overhead of each [19]. In contrast to the above approaches, quantum computing has been also widely recognized as a key enabling technology for implementing complex computing systems and solving complex optimization problems because of its powerful large-scale parallel computation capability by using quantum information units [1].

As the number of intelligent agents increases in the future, the methods for adversarial attackers become more intelligent and less easily detectable. Such new types of security intelligent problems (e.g., deep learning-assisted attackers) are called adversarial security. They act intelligently and simultaneously to evaluate the stability of the system, spreading false approximation information to make the system adopt the wrong policies. Current decision networks of intelligent entities at the physical and network layers appear vulnerable to such real-time decision perturbations. Also, adaptive modulation techniques are at the risk of indirectly exposing link information due to their need for channel state information or transmitter information. There are two possible solutions to this type of malicious adversarial attack: building a real-time attack detector and building a learning-based transceiver to resist malicious attacks and reduce their impact. For example, the automated learning-based transceiver, which does not require knowledge of the channel model, can undergo adversarial training [23] to generate a more robust transmit network or decision network.

Even if the current supervised learning methods have achieved good results, they generally require a lot of labeling work. Moreover, they are less applicable to new categories (e.g., new communication devices) because they require the labels of the existing categories. In order to obtain faster adaptation in such cases, we can use few-shot or zero-shot learning, which aims to find historical tasks similar to the current one, and use meta-learning to obtain the initial values of the neural networks. It allows fast adaptation of the model with very few samples. In addition to this, we can use GANs to synthesize data artificially when very limited data is available in real systems or communications, such as image generation, signal generation, end-to-end latency, and channel gains [19].