Ensembles of Compact, Region-specific & Regularized
Spiking Neural Networks for Scalable Place Recognition

The neuromorphic computing field develops hardware, sensors and algorithms that are inspired by biological neural networks, with the aim of exploiting their advantages including robustness, generalization capabilities, and incredible energy efficiency [33, 4]. Spiking neural networks are one class of algorithms considered within neuromorphic computing.

Such spiking neural networks can be trained via unsupervised methods such as Spike-Timing-Dependent-Plasticity [34], or by converting pre-trained conventional artificial neural networks to spiking networks [35, 36, 37]. ANN-to-SNN conversion approaches have demonstrated comparable performance to their ANN equivalents; however, these approaches typically cannot fully exploit the advantages of SNNs. We note that the non-differentiable nature of spikes in SNNs prevents direct application of supervised techniques such as back-propagation; however, some recent works proposed solutions to approximate back-propagation for SNNs [38, 39].

Thanks to their desirable characteristics, SNNs have gathered interest in a range of robotics applications, including control [5, 6, 7, 8], manipulation [9, 10], scene understanding [11], and object tracking [12]. Key works that use spiking networks for robot localization, the task considered in this paper, include an energy-efficient uni-dimensional SLAM [40], a robot navigation controller system [41], a pose estimation and mapping system [42], and models of the place, grid and border cells of rat hippocampus [43] based on RatSLAM [44].

However, thus far the performance of these methods have only been demonstrated in simulated [43, 42], constrained indoor [40, 41], or small-scale outdoor environments [14]. The most similar prior work is [14] which introduced a high-performing SNN for VPR. However, [14] was limited to recognizing just 100 places, compared to several thousand places in our proposed ensemble spiking networks.

Ensemble neural networks contain multiple ensemble members, with each ensemble member being responsible for a simple sub-task [25, 26, 27]. In this paper, we decompose the learning data so that each ensemble member is trained in parallel on a disjoint subset of the data [27].

Various ensemble schemes have been used in SNN research, including unsupervised ensembles for spiking expectation maximization networks [45]. The most similar ensemble SNN is that of [46]. However, each ensemble member in [46] learns a portion of an input image, opposed to different sections of the input data as in our work.

Visual place recognition (VPR) is the task of recognizing a previously visited place despite changes in appearance and perceptual aliasing [15, 16]. VPR is often considered as a template matching problem, where the query image is matched to the most similar reference image.

Recent works on VPR are dominated by deep learning. A widely-known deep learning approach is NetVLAD [28], which is based on the Vector of Locally Aggregated Descriptors (VLAD) [47], trained end-to-end thanks to a differentiable pooling layer. Many works extended NetVLAD in several directions [48, 49, 50, 51]. As NetVLAD still performs competitively, we use it for benchmarking in this work.

Bio-inspiration has a long history in VPR research: The hippocampus of rodent brains has inspired RatSLAM [44], 3D grid cells and multilayer head direction cells inspired [52], and cognitive processes of fruit flies inspired [53]. Other works are based on spatio-temporal memory architectures [54, 55]. We note that the detection and removal of hyperactive neurons in our approach is conceptually similar to using salient features of a place representation [56].

Our ensemble is homogeneous, i.e. each expert within the ensemble has the same architecture and uses the same hyperparameters. The experts only differ in their training data, which consists of geographically non-overlapping regions of the environment. The training of a single expert spiking network follows [57, 14] and is briefly introduced for completeness in this subsection.

Network structure: Each expert module consists of three layers: 1) The input layer transforms each input image into Poisson-distributed spike trains via pixel-wise rate coding. The number of input neurons $K_{P}$ corresponds to the number of pixels in the input image: $K_{P}=W\times H$, where $W$ and $H$ correspond to the width and height of the input image respectively. 2) The $K_{P}$ input neurons are fully connected to $K_{E}$ excitatory neurons. Each excitatory neuron learns to represent a particular stimulus (place), and a high firing rate of an excitatory neuron indicates high similarity between the learned and presented stimuli. Note that multiple excitatory neurons can learn the same place. 3) Each excitatory neuron connects to exactly one inhibitory neuron. These inhibitory neurons inhibit all excitatory neurons except the excitatory neuron it receives a connection from. This enables lateral inhibition, resulting in a winner-takes-all system.

Neuronal dynamics: The neuronal dynamics of all neurons are implemented using the Leaky-Integrate-and-Fire (LIF) model [2], which describes the internal voltage of a spiking neuron in the following form:

where $\tau$ is neuron time constant, $E_{\text{rest}}$ is the membrane potential at rest, $E_{\text{exc}}$ and $E_{\text{inh}}$ are the equilibrium potentials of the excitatory and inhibitory synapses with synaptic conductance $g_{e}$ and $g_{i}$ respectively.

Network connections: The connections between the inhibitory and excitatory neurons are defined with constant synaptic weights. The synaptic conductance between input neurons and excitatory neurons is exponentially decaying, as modeled by:

where the time constant of the excitatory postsynaptic neuron is $\tau_{g_{e}}$. The same model is used for inhibitory synaptic conductance $g_{i}$ with the inhibitory postsynaptic potential time constant $\tau_{g_{i}}$.

Weight updates: The biologically inspired unsupervised learning mechanism Spike-Timing-Dependent-Plasticity (STDP) is used to learn the connection weights between the input layer and excitatory neurons. Connection weights are increased if the presynaptic spike occurs before a postsynaptic spike, and decreased otherwise. The synaptic weight change $\Delta w$ after receiving a postsynaptic spike is defined by:

where $\eta$ is the learning rate, $\textit{x}_{\text{pre}}$ records the number of presynaptic spikes, $\textit{x}_{\text{tar}}$ is the presynaptic trace target value when a postsynaptic spike arrives, $w_{\text{max}}$ is the maximum weight, and $\mu$ is a ratio for the dependence of the update on the previous weight.

Local regularization: To prevent individual neurons from dominating the response, homeostasis is implemented through an adaptive neuronal threshold. The voltage threshold of the excitatory neurons is increased by a constant $\Theta$ after the neuron fires a spike, otherwise the voltage threshold decreases exponentially. We note that the homeostasis provides regularization only on the local, expert-specific scale, not on the global ensemble-level scale.

Neuronal assignment: The network training encourages the network to discern the different patterns (i.e. places) that were presented during training. As the training is unsupervised, one needs to assign each of the $K_{E}$ excitatory neurons to one of the $L$ training places ($K_{E}\gg L$). Following [57], we record the number of spikes $S_{e,i}$ of the $e$-th excitatory neuron when presented with an image of the $i$-th place. The highest average response of the neurons to place labels across the local training data is then used for the assignment $A_{e}$, such that neuron $K_{e}$ is assigned to place $l^{*}$ if:

Place matching decisions: Following [57], given a query image $q$, the matched place $\hat{l}$ is the place $l$ which is the label assigned to the group of neurons with the highest sum of spikes to the query image $\big{(}\text{i.e.~{}}A_{e}=\hat{l}\big{)}$. Formally:

The previous section described how to train individual spiking networks following [14]. In this section, we present our novel ensemble spiking network, which consists of a set of $\mathcal{M}=\{M_{1},\dots,M_{i},\dots,M_{N}\}$ experts. The $i$-th expert is tasked to learn the places contained in non-overlapping subsets $R_{i}\in\mathcal{R}$ of the reference database $\mathcal{R}$, whereby

All subsets are of equal size, i.e. $|R_{i}|=\kappa$. Therefore, at training time the expert modules are independent and do not interact with each other, a key enabler of scalability.

At deployment time, the query image $q$ is provided as input to all experts in parallel. The place matching decision is obtained by considering the spike outputs of all ensemble members, rather than just a single expert as in Eq. (5).

The basic fusion approach that considers all spiking neurons of all ensemble members is problematic. As the expert members are only ever exposed to their local subset of the training data, there is a lack of global regularization to unseen training data outside of their local subset. In the case of spiking networks, this phenomenon leads to “hyperactive” neurons that are spuriously activated when stimulated with images from outside their training data. We decided to detect and remove these hyperactive neurons.

To detect hyperactive neurons, we do not require access to query data. We use the cumulative number of spikes $S_{e,l}^{i}$ fired by neurons $K_{e}^{i}$ of each module $M_{i}\in\mathcal{M}$ in response to the entire reference dataset $\mathcal{R}$. $S_{e,l}^{i}$ indicates the number of spikes fired by neuron $K_{e}$ of module $M_{i}$ in response to image $l\in\mathcal{R}$. Neuron $K_{e}^{i}$ is considered hyperactive if

where $\theta$ is a threshold value that is determined as described in Section III-D. The place match is then obtained by the highest response of neurons that are assigned to place $\hat{l}$ after ignoring all hyperactive neurons:

where the indicator function $\mathds{1}$ filters all hyperactive neurons.

We use a grid search to tune the network’s hyperparameters: the time constant of the inhibitory synaptic conductance $\tau_{gi}$, and the threshold value to detect the hyperactive neurons $\theta$. We train the modules multiple times using the reference images $\mathcal{R}$ introduced in Section III-B, and vary $\tau_{gi}$ and $\theta$. We then observe the performance in response to a query set $\mathcal{C}$ which is geographically separate from the test set $\mathcal{T}$.

Specifically, for each combination of the hyperparameter values, we evaluate the performance of the ensemble SNN model on the $\mathcal{C}$ calibration images using the precision at 100% recall metric (see Section IV-C). We select the $\tau_{gi}$ and $\theta$ hyperparameter values that lead to the highest performance and use these values for all test images at deployment time.

We implemented our ensemble spiking neural network in Python3 and the Brian2 simulator [58]. We pre-processed all reference and query input images by resizing images to $W\times H=28\times 28$ pixels, and patch-normalizing images [30] using patches of size $W_{P}\times H_{P}=7\times 7$ pixels.

We use rate coding to convert input images to Poisson spike trains. The number of $K_{P}=784$ neurons in the input layer corresponds to the number of pixels in the input image. We used $\kappa=25$ consecutive places to train each ensemble member. The hyperparameter search in Section III-D resulted in $\tau_{gi}=0.5$ and $\theta=100$ for the Nordland dataset, and $\tau_{gi}=0.5$ with $\theta=180$ for the Oxford RobotCar dataset. Given an input image, the number of spikes of the $K_{E}=400$ excitatory (output) neurons in the last 10 epochs are recorded. The SNN modules were trained in parallel for 60 epochs irrespective of the dataset.

We evaluated our ensemble spiking neural network on two widely used VPR datasets, Nordland [31] and Oxford RobotCar [32]. The Nordland dataset [31] captures a 728 km train journey in Norway where the same traverse is recorded during spring, summer, fall and winter. As in prior works [59, 60, 49] tunnels and sections where the train travels below 15 km/hr were removed. We trained our model on the spring and fall traverses, and we used summer traverse as the query dataset. We subsampled places every 8 seconds (about 100 meters) from the entire dataset, resulting in 3300 places. The Oxford RobotCar dataset [32] contains over 100 traversals captured under varying weather conditions, times of the day and seasons. As in [59], our reference dataset consists of sun (2015-08-12-15-04-18) and rain (2015-10-29-12-18-17) traverses, and our query dataset is the dusk (2014-11-21-16-07-03) traverse. We sampled places roughly every 8 seconds (about 100 meters), resulting in 450 places.

The precision at 100% recall (P@100R) is the percentage of correct matches when the system is forced to match each query image to one of the reference images. The recall at $N$ (R@N) metric is the percentage of correct matches if at least one of the top $N$ predicted place labels is correctly matched.

We consider a query image to be correctly matched only if it is matched exactly to the correct place – our ground truth tolerance is zero, noting that the distance between the sampled places within the datasets is relatively small.

We compare the performance of our method against three conventional VPR approaches: Firstly, the Sum-of-Absolute-Differences (SAD) [30] which computes the pixel-wise difference between each query image and all reference images. For a fair comparison, we applied the same resizing and patch-normalizing steps as in our approach (see Section IV-A). Secondly, DenseVLAD [29] which uses densely sampled SIFT image descriptors. Lastly, NetVLAD [28] which generalizes across different datasets and is robust to viewpoint and appearance changes. For NetVLAD and DenseVLAD, we used the original input image size of $640\times 360$ pixels for Nordland and resized the input images to $640\times 480$ pixels for Oxford RobotCar, potentially giving them an advantage over the low-dimensional input images in our proposed method.

We also compare against a non-ensemble SNN [14], which in [14] was limited to just 100 places because of a relatively small network size. To compare against [14] on our large datasets, we increase the network size of their approach to contain $K_{E}=|\mathcal{R}|$ output neurons (i.e. one output neuron per place). We note that increasing the number of neurons in their SNN results in significantly longer training and inference times (Figure 3), so we trained their network for only 26 epochs. In Section V-C, we additionally compare our method to [14] in a small-scale environment (for which [14] was designed).

We first compare our ensemble SNN to conventional VPR techniques, with the aim of merely demonstrating the potential of SNN-based approaches, as opposed to outperforming these VPR techniques. The results are summarized in Table I.

For the Nordland dataset, our ensemble SNN model obtains a R@1 of 52.6%, outperforming SAD (R@1: 45.1%), NetVLAD (R@1: 35.1%) and DenseVLAD (R@1: 37.9%). We note that NetVLAD is known to perform relatively poorly on the rural Nordland dataset, as it was trained on urban data. For the Oxford RobotCar dataset, the R@1 of our ensemble SNN model is 40.5%, while the SAD approach has a similar R@1 of 41.3%, the NetVLAD and DenseVLAD methods obtain a higher R@1 of 44.8% and 53.1% respectively. The performance of our ensemble SNN achieves a similar R@25 to NetVLAD and DenseVLAD, demonstrating the performance capability of our method.

Table I also presents the performance of the previous non-ensemble SNN model [14], which is our main competitor. [14] catastrophically failed to perform place recognition at large-scale, with a precision at 100% recall of just 0.3% on the Nordland and 4.0% on the Oxford RobotCar datasets. We note that [14] specialized in place recognition on small datasets. In addition to poor performance, the large number of output neurons within a single network for [14] results in significantly increased inference times. This is opposed to our modular approach, where the neuronal dynamics of independent and compact ensemble members are cheaper to compute. We highlight the computational advantages and better scalability of our method in Figure 3. The inference times of our method is still slower compared to conventional VPR methods such as NetVLAD. However, deploying our method on neuromorphic hardware can significantly decrease the inference times via use of hardware parallelism.

This section evaluates that it is crucial to introduce global regularization by detecting and ignoring hyperactive neurons. As shown in Table I and Figure 2b, our ensemble SNN model where hyperactive neurons are ignored improves the precision at 100% recall compared to the base ensemble SNN model (that includes hyperactive neurons) on both Nordland (absolute increase of 16.9%) and Oxford RobotCar (absolute increase of 10.4%). Figure 4 compares the neuron precision of hyperactive and non-hyperactive neurons trained on the Nordland dataset and highlights that the precision at recognizing correct places of non-hyperactive neurons is significantly higher than that of hyperactive neurons.

We further evaluate the sensitivity of our ensemble SNN with respect to the threshold value $\theta$ (see Section III-D). Specifically, we evaluate the precision at 100% recall at different threshold values ($0<\theta<\theta_{max}=200$). Note that $\theta=0$ corresponds to the baseline performance where both hyperactive and non-hyperactive neurons are used. For both the Nordland and Oxford RobotCar datasets, Figure 5 shows that our method is not sensitive to particular values of $\theta$, with a wide range of high-performing settings. Importantly, any $\theta>0$ improves performance compared to the baseline model where hyperactive neurons are included ($\theta=0$).

This ablation presents a like-for-like comparison of our proposed ensemble SNN and a non-ensemble SNN model from [14] which facilitates direct comparison to the previous state-of-the-art VPR system using SNNs. As [14] was designed for small-scale datasets, we do so by considering a much smaller dataset limited to 100 places; the previous section has already shown that [14] fails catastrophically in large environments. Specifically, we trained $N=5$ ensemble members, each containing $K_{E}=400$ excitatory neurons. We used the first $\kappa_{cal}=25$ places to calibrate $\tau_{gi}$ and $\theta$.

The P@100R of our proposed ensemble SNN model at 91.0% is considerably higher than that of prior work on non-ensemble SNNs [14] (79.0%), which in conjunction with the results in Section V-A demonstrates that our ensemble method is both scalable and provides improved performance in both small and large scale environments. The PR curve for these experiments is shown in Figure 6.