Towards Crossing the Reality Gap with Evolved Plastic Neurocontrollers

The current widely used Artificial Neural Networks (ANNs) follow a computation cycle of multiply-accumulate-activate. The neuron model consists of two components: a weighted sum of inputs and an activation function generating the output accordingly. Both the inputs and outputs of these neurons are real-valued. While ANN models have shown exceptional performance in the artificial intelligence domain, they are highly abstracted from their biological counterparts in terms of information representation, transmission and computation paradigms.

SNNs, on the other hand, carry out computation based on biological modeling of neurons and synaptic interactions, and have been of great interest in the computational intelligence community in recent decades. Applications have been both non-behavioral (Abbott et al., 2016) and behavioral (Vasu and Izquierdo, 2017; Qiu et al., 2018). Information transmission in SNNs is by means of discrete spikes generated during a potential integration process. Such spatiotemporal dynamics are able to yield more powerful computation compared with non-spiking neural systems (Maass, 1997). Moreover, neuromorphic hardware implementations of SNNs are believed to provide fast and low-power information processing due to their event-driven sparsity (Bouvier et al., 2019), which perfectly suits embedded applications such as UAVs.

As shown in Fig. 1, spikes are fired at certain points in time, whenever the membrane potential of a neuron exceeds its threshold. They will travel through synapses from the presynaptic neurons and arrive at all forward-connected postsynaptic neurons. The information measured by spikes is in form of timing and frequency, rather than the amplitude or intensity.

In order to assist the process of designing our spiking controller, we have developed the eSpinn software package. The eSpinn library stands for Evolving Spiking Neural Networks. It is designed to develop controller learning strategies for nonlinear control models by integrating biological learning mechanisms with neuroevolution algorithms. It is able to accommodate different network implementations (ANNs, SNNs and hybrid models) with specific dataflow schemes. eSpinn is written in C++ and has abundant interfaces to easily archive data through serialization. It also contains scripts for data visualization and integration with MATLAB and Simulink simulations.

To date, there have been different kinds of spiking neuron models. When implementing a neuron model, trade-offs must be considered between biological reality and computational efficiency. In this work we use the two-dimensional Izhikevich model (Izhikevich, 2003), because of its capability of exhibiting richness and complexity in neuron firing behavior (detailed in (Izhikevich, 2003)) with only two ordinary differential equations:

with after-spike resetting following:

Here $v$ represents the membrane potential of the neuron; $u$ represents a recovery variable; $\dot{v}$ and $\dot{u}$ denote their derivatives, respectively. $I$ represents the synaptic current that is injected into the neuron. Whenever $v$ exceeds the threshold of membrane potential $v_{t}$, a spike will be fired and $v$ and $u$ will be reset following Eq. (2). $a,b,c$ and $d$ are dimensionless coefficients that are tunable to form different firing patterns (Izhikevich, 2003). The membrane potential response of an Izhikevich neuron is given in Fig. 2, with an injected current signal.

A spike train is defined as a temporal sequence of firing times:

where $\delta(t)$ is the Dirac $\delta$ function; $t^{(f)}$ represents the firing time, i.e., the moment of $v$ crossing threshold $v_{t}$ from below.

We use a three-layer architecture that has hidden-layer recurrent connections, illustrated in Fig. 3. The input layer consists of encoding neurons which act as information converters. Hidden-layer spiking neurons are connected via unidirectional weighted synapses among themselves. Such internal recurrence ensures a history of recent inputs is preserved within the network, which exhibits highly nonlinear functionality. Output neurons can be configured as either activation-based or spiking. In this work a linear unit is used to obtain real-value outputs from a weighted sum of outputs from hidden-layer neurons. A bias neuron that has a constant output value is able to connect to any neurons in the hidden and output layers. Connection weights are bounded within [-1, 1]. The NEAT topology and weight evolution scheme is used to form and update network connections and consequently to seek functional network compositions.

In a rate coding scheme, neuron output is defined as the spike train frequency calculated within a given time window. Loss of precision during this process is likely to happen. eSpinn configures a decoding method with high accuracy to derive continuous outputs from discrete spike trains. The implementation involves direct transfer of intermediate membrane potentials as well as decoding of spikes in a rate-based manner.

In neuroscience, studies have shown that synaptic strength in biological neural systems is not fixed but changes over time (Kandel and Hawkins, 1992) – connections between pre- and postsynaptic neurons are updated according to their degree of causality, which involves changes of synaptic weights or even formation/removal of synapses. This phenomenon is often referred to as Hebbian plasticity as inspired by the Hebb’s postulate (Hebb, 1949).In our work, plastic behaviors are determined by leveraging evolutionary algorithms to optimize plastic rule coefficients, such that each connection is able to develop its own plastic rule.

Modern Hebbian rules generally describe weight change $\Delta w$ as a function of the joint activity of pre- and postsynaptic neurons:

where $w_{ij}$ represents the weight of the connection from neuron $j$ to neuron $i$; $u_{j}$ and $u_{i}$ represent the firing activity of $j$ and $i$, respectively.

In a spike-based scheme, we consider the synaptic plasticity at the level of individual spikes. This has led to a phenomenological temporal Hebbian paradigm: Spiking-Timing Dependent Plasticity (STDP) (Gerstner and Kistler, 2002), which modulates synaptic weights between neurons based on the temporal difference of spikes.

While different STDP variants have been proposed (Izhikevich and Desai, 2003), the basic principle of STDP is that the change of weight is driven by the causal correlations between the pre- and postsynaptic spikes. Weight change would be more significant when the two spikes fire closer together in time. The standard STDP learning window is formulated as:

where $A_{+}$ and $A_{-}$ are scaling constants of strength of potentiation and depression; $\tau_{+}$ and $\tau_{-}$ represent the time decay constants; $\Delta t$ is the time difference between pre- and post-synaptic firing timings:

In eSpinn we have introduced a rate-based Hebbian model derived from the nearest neighbor STDP implementation (Izhikevich and Desai, 2003), with two additional evolvable parameters:

where $k_{m}$ is a magnitude term that determines the amplitude of weight changes, and $k_{c}$ is a correlation term that determines the correlation between pre- and postsynaptic firing activity. These factors are set to as evolvable so that the best values can be autonomously located. Fig. 4 shows the resulting Hebbian learning curve. The connection weight has a stable converging equilibrium at $u_{\theta}$, which is due to the correlation term $k_{c}$. This equilibrium corresponds to a balance of the pre- and postsynaptic firing.

While gradient methods have been very successful in training traditional MLPs (Demuth et al., 2014), their implementations on SNNs are not as straightforward because they require the availability of gradient information. Instead, eSpinn has developed its own version of a popular neuroevolution approach – NEAT (Stanley and Miikkulainen, 2002), which can accommodate different network implementations and integrate with Hebbian plasticity, as the method to learn the best network controller.

NEAT is a popular neuroevolution algorithm that involves network topology and weight evolution. It enables an incremental network topological growth to discover the (near) minimal effective network structure.

The basis of NEAT is the use of historical markings, which are essentially gene IDs. They are used as a measurement of the genetic similarity of network topology, based on which, genomes are clustered into different species. Then NEAT uses an explicit fitness sharing scheme (Eiben et al., 2015) to preserve network diversities. Meanwhile, these markings are also used to line up genes from variant topologies and allow crossover of divergent genomes in a rationale manner.

eSpinn keeps a global list of innovations (e.g., structural variations), so that when an innovation occurs, we can know whether it has already existed. This mechanism will ensure networks with the same topology will have the exactly same innovation numbers, which is essential during the process of network structural growth.

With the identified model we have developed according to Eq. 8, we begin to search for optimal network compositions by evolving SNNs using our NEAT implementation. By ‘optimal,’ we mean the SNN controller is defined to be able to drive the plant model to follow a reference signal with minimal error in height during the course of flight. Each simulation (flight) lasts 80 s and is updated every 0.02 s.

To speed up the evolution process, the whole simulation in this part is implemented in C++, with our eSpinn package. At the beginning, a population of non-plastic networks are initialized and categorized into different species. These networks are feed-forward, fully-connected with random connection weights. The initial topology is 2-4-1 (input-hidden-output layer neurons), with an additional bias neuron that is connected to all hidden and output layer neurons. The two inputs consist of error of position in z-axis $e_{z}$ and vertical velocity $v_{z}$, other than which, the system’s dynamics are unknown to the controller. Output of the controller is thrust command that will be fed to the plant model.

Encoding of sensing data is done by the encoding neurons in the input layer. Input data are first normalized within the range of [0,1], so that the standardized signal can be linearly converted into a current value (i.e., $I$ in Eq. 1). This so-called ‘current coding’ method is a common practice to provide a notional scale to the input metrics.

After initialization, each network will be iterated one by one to be evaluated against the plant model. A fitness value will be assigned to each of them based on their performance. Afterwards, these networks will be ranked within their species according to their fitness values in descending order. A newer generation will be formed from the best parent networks using NEAT: only the top 20% of parents in each species are allowed to reproduce, after which, the previous generation is discarded and the newly created children will form the next generation. During evolution, hidden layer neurons will increase with a probability of 0.005, connections will be added with a probability of 0.01. Connection weights will be bounded within [-1, 1].

The program terminates when the population’s best fitness has been stagnant for 12 generations or if the evolution has reached 50 generations¹¹1empirically determined. During the simulation, outputs of the champion will be saved to files for later visualization. The best fitness will also be saved. Upon completion of simulation, data structure of the whole population will be archived to a text file, which can be retrieved to be constructed in our later work.

Note the control system to be solved is a Constraint Problem (Michalewicz and Schoenauer, 1996), because the height of the UAV must be bounded within some certain range in the real world. However, constraint handling is not straightforward in NEAT – invalid solutions that violate the system’s boundary can be generated, even if their parents satisfy these constraints. Therefore, in this paper we use the feasibility-first principle (Michalewicz and Schoenauer, 1996) to handle the constraints.

We divide the potential solution space into two disjoint regions, the feasible region and the infeasible, by whether the hexacopter is staying in the bounded area during the entire simulation. For infeasible candidates, a penalized fitness function is introduced so that their fitness values are guaranteed to be smaller than those feasible.

We define the fitness function of feasible solutions based on the mean normalized absolute error during the simulation:

where $\lvert e_{n}\rvert$ denotes the normalized absolute error between actual and reference position. Since the error is normalized, desired solution will have a fitness value close to 1.

For infeasible solutions, we define the fitness based on the time that the hexacopter stays in the bounded region:

where $t_{i}$ is the steps that the hexacopter successively stays in the bounded region, and $t_{t}$ is the total amount of steps the entire simulation has. Penalty is applied using a scalar $k$ of 0.2.

Once the above step is done to discover the optimal network topology, we proceed to consider the plasticity rules. The champion network from the previous step is loaded from file, with the Hebbian rule activated. It is spawned into a NEAT population, where each network connection has randomly initialized Hebbian parameters (i.e., $k_{m}$ and $k_{c}$ in Eq. 7).

Networks are evaluated as previously stated. The best parents will be selected to reproduce. During this step, all evolution is disabled except for that of the plasticity rules, e.g. the EA is only used to determine the optimal configuration of the plasticity rules.

Upon completion of the previous steps, the final network controller is obtained and ready for deployment. To construct the controller in the Simulink hexacopter model, it is implemented as a C++ S-function block.

Table 1 shows the fitness changes of the best controller during the course. From left to right are non-plastic networks controlling the identified model, plastic networks controlling the identified model and plastic networks controlling the hexacopter model, respectively. The fitness values are averaged among the 10 runs.

As stated in 6.1, evolution would be terminated if the performance does not improve for 12 consecutive generations before the threshold of 50. For non-plastic controllers, only one of the 10 runs has reached the threshold, and its fitness has only increased by 0.0034 in the last 15 generations. This indicates the evolutionary runs of non-plastic controllers have plateaued and further evolution is unlikely to find better solutions. On the other hand, when plasticity is enabled, an increase in fitness can be clearly observed when controlling the same identified model. The plastic controllers demonstrate better performance even when transferred to control the hexacopter model that has different dynamics.

A second comparison is conducted between non-plastic and plastic controllers on the hexacopter model. Results are given in Table 2. For 9 out of the 10 runs, we can see a performance improvement when plasticity is enabled. The only one not being better, still has a close fitness value. Statistic difference is assessed using the two-tailed Mann-Whitney U-test between the two sets of data. The $U$-value is 21, showing the plastic controllers are significantly better than the non-plastics at $p<0.05$.

Fig. 9 shows a typical run using the non-plastic and plastic controller. We can see the plastic control system has a faster response as well as smaller steady error. It is clear that plasticity is a key component to bridge the gap between two models.

To verify the contribution of the proposed Hebbian plasticity, we extract the evolved best plastic rule and applied it to other networks that have sub-optimal performance. With plasticity enabled, a sub-optimal network is selected to repetitively drive the hexacopter model to follow the same reference signal. Fig. 10 shows the progress of 4 consecutive runs when a) plasticity is disabled; b-d) plasticity is enabled.

We can see that in Fig. a), there is a considerable steady system output error. When plasticity is turned on, connection weights begin to adjust themselves gradually. The system follows the reference signal with a decreasing steady error until around 0.005 m. Meanwhile a fitness increase is witnessed from a) 0.921296, b) 0.927559, c) 0.932286 to d) 0.933918.

The same results can be obtained when the rule is assigned to other near-optimal networks, while for those with poor initial performance, plasticity learns worse patterns. This analysis has justified our evolutionary approach to search for the optimal plastic function, demonstrating that plasticity narrows the reality gap for evolved spiking neurocontrollers.

PID control is a classic linear control algorithm that has been dominant in engineering. The aforementioned PID height controller is taken for comparison. Note here the PID controller is designed directly based on the hexacopter model, whereas the SNN controller only relies on the identified model and utilizes Hebbian plasticity to adapt itself to the new plant model. System outputs of the two approaches is given in Fig. 11. Evidently our controller has smaller overshoot and steady output error. The PID controller has a mean absolute error of 0.108 m during the course of flight, while our plastic SNN controller has a value of 0.090 m.