Gluing Neural Networks Symbolically Through Hyperdimensional Computing ^††thanks:

This work is an extension of a series of works. First, [4, 5, 6, 7, 8] describe some methods of encoding arbitrary data into functional hypervectors. Namely, ideas presented in [7] and [8] were used to facilitate the process of converting output signals from networks into hypervectors. This work builds on Hyperdimensional Active Perception [2] and the notion of forming a memory unit from the class space much like in (1) and probing this with (2) to discover the best classification. More specifically, a followup paper shows this concept applied to the standard problem of classifying images, where neural network outputs were being directly encoded into memories [3]. Here, the authors use neural hashing networks to convert images into short, rankable, similarity-based binary vectors. These vectors were then projected into hyperdimensional spaces, and then used similarly to what is described in section IV to fuse architecturally different hashing network outputs into a more powerful, consensus model. Our work is most similar to this, but with the novel additions of extremely broadening the scope of networks that can be fused, directly encoding output signals and not the output, and presenting life-long learning strategies that can increase the performance.

As stated before, our work is an advancement of [3]. However, there are other works that attempt to either fuse data with hypervectors or encode neural networks in hypervectors. For example, in [9], hyperdimensional data is fused together to improve classification accuracy by incorporating many modalities of data. Effectively, we use a similar idea in our work, except the data is the direct encoding of neural output signals. A followup work to this uses a neural network to aid the consensus classification from hypervectors as a second stage of processing [10]. In theory, this is something that can be used in place of the method of determining the correct class presented in section IV. Furthermore, the idea of using superpositions to compress aspects of networks into one is explored in [11], where the idea is applied to network parameters to fuse them into one. We focus on the output signals of only a part of the network. Finally, the problem of classification using hypervectors has been explored in previous works, such as [6] and [12], in which the basics of record-based classification ala (1) and (2) were explored, but do not attempt data fusion, consensus or encoding of network signals, but focus on the task of classifying itself.

Finally, it is necessary to mention prior work that does not directly use HD techniques or hypervectors, but still enables ensembling or consensus among different models. These are mentioned mainly to show how our work differs from these. Ensemble computing/learning generally makes use of multiple learning algorithms to improve the overall predictive performance that can be achieved by any one model alone. A survey of popular techniques can be found in [13, 14, 15, 16]. The primary difference between our work here and work in ensembling is our goal to “normalize” the space on which our model’s predictions are aggregated, by using the same-size, binary hypervectors. In consensus learning, such as some of the techniques in [16], the ensembling occurs through consensus from all models. Thus, all models participate in “voting” on the final prediction. Our work can be viewed as ensemble consensus, but it should be noted that binary hypervectors and how they correlate spatially can have non-linear relationships. It’s entirely possible to aggregate multiple models, but only receive a statistically significant response, given some input, for a few models, or perhaps only one. This arises due to the “noisiness”, present in hypervectors.

The key cornerstone to being able to combine neural networks and hypervectors is the observation that a method of translating neural network outputs to hypervectors must be designed. In previous work, such as [17, 18, 2, 3, 8], the direct output of the network was used to do this. In other works, symbolic representations of output classes or values were used. However, these techniques are not necessarily extendable to all networks, or they do not sufficiently capture what the network has learned from the data, apart from broad classes. We inevitably reach the conclusion that parts of the network must be directly encoded. For the purpose of demonstration in this paper, we assume a neural network consists of multiple layers, where the final layer’s output signals are combined to determine the output class. Suppose these signals are available; we can encode the neural output signals directly into hypervectors to get a more informed hypervector encoding that conveys “why” the network made that classification choice.

To achieve conversion from neural signals to hypervector encodings, we further assume the signals are simple, real values. As such, we organize them into a real vector, sometimes referred to as an embedding. Embeddings are high level spaces that are projected into low level dimensions, thus capturing semantics of the high level input space by moving related objects in that space closer together. To convert a real vector embedding into a hypervector encoding, we recognize that the real vector itself is an ordered sequence of real values. We have existing hypervector encoding techniques to encode ordered sequences of numbers, such as [17], but encoding real numbers themselves is more challenging. Fortunately, neural output signals can be normalized to a range of $[-1,1]$ using the $\tanh$ function, which allows us to encode signals by using a sufficiently fine binning of values, and assigning each bin a hypervector. The techniques described in [17, 18, 3, 8] and, namely, in [2], can be used to create meaningful bin hypervectors by slowly changing the hypervector for -1 into the hypervector for 1 with each additional bin, by randomly changing a constant number of unchanged bits. Then, (1) indicates that a real vector can be represented as a bundling of component positions (their symbolic hypervectors forming a basis), each one a binding of the signal value hypervector to the component ID (a randomly selected vector). The pipeline from input to hypervector encoding is shown in Fig. 2.

Suppose we only have one neural network to start with. Training examples are presented to the network, which prompt activations in the final layer that are used to form the prediction for the network, outputting signals for each neuron. Each output is encoded by quantizing it as the nearest bin hypervector in the range $[-1,1]$. Then, a consistent list of hypervectors representing component positions binds the signal hypervector to the appropriate bin. The aggregation of these hypervectors across all components through consensus summation forms the final hypervector representation of the real vector of signals. This can now be stored as a “memory”. For each new training example for a particular class, memories are aggregated by class from the gold standard labels - these are consensus summed once every training example for the class has been seen. Consensus sum forces an “average” vector to be created that is forced to throw out the unpopular vote in every component and choose the most commonly represented bit across the terms. This facilitates “learning”. Thus, the bundled memories for each class are a learned representation.

Classes are represented by random hypervectors themselves as ID hypervectors. Once training is complete, each learned hypervector corresponding to a class is bound to the corresponding hypervector representation. Finally, these terms are themselves bundled, forming a final classification hypervector. When a new example is provided at testing time, the neural network creates a new real vector of signals, which is quantized to a hypervector encoding - to identify the correct class we XOR this with the model and probe for the correct class as shown in (2). Once again, this process is visualized in Fig. 1. If training is resumed, hypervector models can easily incorporate new training examples into the aggregates, further learning. With a fine enough bin, virtually no information is lost from real vectors to hypervectors. As we will see in the results in section VI, this transformation does not impact the performance negatively with sufficiently hyperdimensional hypervector lengths.

Since hypervectors exist as discrete entities, they can be used either as a form of memory, or actual models that solve problems. As such, there is no reason multiple models cannot be used. In this context, given a neural network, one can have many hypervectors that see a subset of the training data, then predict the result at testing time in tandem. Consensus can be achieved by repeating the same records strategy in (1). Each hypervector model can be assigned its own random ID hypervector, which is bound to the model. These bound terms are then bundled with consensus summation. Given a new example, we can probe the singular model to determine which class has the most agreement across models.

This strategy can be used in other interesting ways. Suppose we train a hypervector model from a neural network to solve a problem. If it doesn’t achieve 100% accuracy, we can collect the examples that were not correctly classified, and then form a new hypervector model to address these partially. If we still do not achieve 100% accuracy, simply repeat the process until no improvement can be made. The weighted consensus summation in (3) can be used to account for each subsequent model observing less and less training examples, where the weights are the proportion of the training examples used for that model. In Fig. 3, this process is demonstrated with semi-realistic numbers. The model $A$ is trained, but only achieves 76% accuracy, thus it contributes 0.76 to the final model. Then, model $B$ handles the remaining 24%, but only achieves 70% accuracy there, thus it contributes $0.24\times 0.70=0.168$ to the final model. Updating $A$ to the weighted average of $A$ and $B$ improves its performance. A fleet of such error corrected models can be used in consensus to effectively reach 100% accuracy, if desired. In an online learning setting, we can grow more and more models to handle errors. If we have too many models, simply consensus the worst performing models into a single hypervector, and throw away the individual models. This allows life long learning and improvement.

Now that we’ve established the training process for a single neural network’s output signals, we can address doing this for many neural networks to fuse together their output signals symbolically into a single consensus model. We note that subsections IV-A, IV-B, and IV-C set up the pre-processing steps up until this point. To fuse the models, we repeat the same strategy in subsection IV-C. A record is created using (1), where random ID hypervectors are generated for each neural network and bound to the hypervector models. These terms are then consensus summed. The result is a single predictive hypervector model that takes into account all neural networks and lets the “vote” on the correct class when given a new testing example. Thus, “gluing” the models together through a symbolic hyperdimensional representation, a process that we will refer to a HD-Glue for short.

Interestingly, as is the case with all record based formulations, you do not need to incorporate every network in the voting process. It suffices that one network votes, collapsing the contributions of others into noise. However, at that point, it would be better to use the network’s learned hypervector model. This is more useful for when some networks aren’t available as a resource. Furthermore, multiple HD-Glue models can be formed, in a process like the one described in subsection IV-C, to ensure that the consensus of networks achieves 100% accuracy on the training set.

Here we describe our process for testing HD-Glue on the MNIST data-set [19]. The MNIST data set contains 60k examples of hand-written digits 0-9 that are compressed into 40-by-40 pixel greyscale images. It is a common benchmark for new learning techniques.

Here we describe testing HD-Glue on the CIFAR-10 data-set [20]. CIFAR-10 contains 10 different classes of objects/animals, with 60k 32-by-32 pixel color images.

The CIFAR-100 data-set [20] differs only in number of training classes, ten times that of CIFAR-10, and has much fewer training examples per class, making it a more difficult problem to solve. This is a very rigorous benchmark of the HD-Glue method, with the most room to improve consensus performance over constituent models. Our methodology here is the same as CIFAR-10 in subsection V-B. We differ only in testing the benchmarks across different architectures, and then in a separate experiment testing differing embedding lengths, to demonstrate that HD-Glue is agnostic to the architectures of the neural networks. Pre-existing models used were VGG11, VGG13, VGG16, and VGG19 from the VGG family [21], and ResNet18 and ResNet34 from the ResNet family [22].

Our results are shown in Table I. In this experiment, 5 encoders were created, used to discriminate between 2 of the MNIST digits, each. Partitioning classifiers like this simulates what would happen in an online setting where new classes and new networks simultaneously come into existence. Each one is steadily presented more and more examples, tracking the accuracy from few-shot settings to large numbers of examples. By keeping the output layers of encoder networks the same number of neurons, other benchmarking algorithms can be used to compare to HD-Glue.

Furthermore, the results for our online learning experiment on MNIST are shown in Table II. The results here do not use any error correction nor multiple consensus models per class. The performance is broken down on a class-by-class basis, tracking the performance as each class is presented new examples in an online fashion. Once every existing class has seen 100 new examples, 2 new classes are added by adding a new model, and 100 examples are presented again for each class. This simulates online learning across an unbalanced dataset. By the end, 3000 examples have been presented, the final performance being the same as if training on all 3000 examples at once. We can see that HD-Glue gracefully handles new classes and quickly learns them. Despite digits 2 and 3 seeing 400 examples, and digits 8 and 9 seeing only 100, the latter outperforms the former. This likely reflects the difficulty of the task, moreso than the amount of data. Existing classes gracefully decay in performance as new classes are introduced, as the single hypervector predictive model is forced to discriminate between more and more classes.

Results for the CIFAR-10 data-set are shown in Table III. Once again, we use simple image encoders to learn representations, which then predict the class using a final layer. We use the signal values of this output layer to either encode hypervector representations, or use the real valued vector of signals in benchmark algorithms. We keep the output layer the same number of components across all 20 encoders generated, each encoder with a different initialization. Results are split into 10 model consensus and 20, to show the improvement of HD-Glue with more models. We also use different dimension sizes for the hypervectors to show how HD-Glue improves with higher hyperdimensionality. In all but two cases, HD-Glue outperforms all benchmarks with 10 models, but outperforms all other models at 20, showing this improvement.

Additionally, we include results for using stronger networks on CIFAR-10, namely VGG11 [21], in Table IV. Here, we use high performing networks trained only on CIFAR-10 to assess how well consensus through HD-Glue works in this scenario. We have dropped poorly performing benchmarks here for simplicity. When the networks are too similar, the benefits of HD-Glue are not as apparent. HD-Glue prefers diverse networks in its consensus. Still, the results are competitive, while maintaining the benefits of HD representations. It should be noted that Logistic Regression likely performs so well in this experiment because of how similar the models are. However, Linear-SVM catches up to this with enough examples.

In the case of CIFAR-100, we have an opportunity to challenge HD-Glue in a harder setting. The 100 classes of CIFAR-100 are easily handled as hypervectors can perfectly recall hundreds of aggregated vectors [2]. For obtuse numbers of classes, multiple hypervectors can be used to handle subsets of classes. In Table V, we show the results of our benchmarks using a diverse set of image classification models, trained on CIFAR-100. Namely, we use VGG11, VGG13, VGG16, and VGG19 from [21], and ResNet18 and ResNet34 from [22]. Once again, HD-Glue outperforms all other benchmarks. Compared to the results in Table IV, this implies diverse networks are preferable in consensus as opposed to the same network with different initializations.

Moreover, we also show the results of CIFAR-100 in the case of networks with differing embedding lengths in Table VI. We use the same networks, however ResNet18 and ResNet34 have been retrained with a lower number of output signals, by half. This lowers their performance a little. We can see that for reasonable hypervector lengths, HD-Glue consensus will outperform ResNet34, the best performing individual model, after just 500 training examples.

It seems clear that HD-Glue best performs when very different networks are combined together, supporting the results in [3]. One key observation from Table IV’s results, is that a limitation of our hyperdimensional consensus manifests for too similar networks. A simple question arises: can these results be improved? Consider the intersection of all correct answers each of the individual 3 VGG networks provide: a performance of 94.6% can be achieved if the correct result is chosen from the 3 networks. When best 2 out of 3 consensus is considered, the accuracy falls to 90.6%, comparable to our results in Table IV for HD-Glue. Of that discrepancy, HD-Glue itself correctly classifies 0.58% of examples that our intersected models cannot. That leaves an improvement of 4.5% that is still within the realms of possibility - meaning HD-Glue can be further improved. However, it’s clear that a more complicated HD structure than the HIL is necessary to lessen this gap.

Furthermore, as the networks are trained, HD-Glue very quickly approaches a higher performance, bootstrapping the consensus - this was also observed in [3]. However, HD-Glue has the benefit of opening the door to many other neural network architectures that can be glued together, due to using the output signals of the network directly. Moreover, it is competitive to other methods that can be used for quick consensus prediction, without needing end-to-end learning. In fact, with diverse networks in consensus, HD-Glue can outperform these methods. Given how fast and cheap computations in HD-Glue are, these properties are well-worth the effort, especially since similar real-valued vector methods would require consistent vector lengths, a restriction we do not have with hypervectors.

Suppose the modalities of glued models in HD-Glue go beyond that of image domain. In theory, one can combine classical image models with spectral models, log-polar models, etc. In fact, there is nothing stopping the creation of a super-model that incorporates every pre-trained image model in existence; the only limitation here is how many neural models can be loaded in parallel. In a distributed setting, this idea becomes far more applicable. Furthermore, what if entire fields are bridged with gluing? Indeed, the idea of fusing video, audio, and text based models together poses an interesting idea; can such a consensus model decide on what sort of data is relevant to a task? These are open questions to future work.

The implication that more diverse networks increase the performance of glued models indicates a new potential way to tackle the problem of machine learning: targeting specific aspects of the task to solve with specifically constructed models that use resources in a multifaceted approach, with less resources used, that are glued together. An interesting side effect of such an approach, is that transfer learning is much easier, since we could simply use the hypervectors as input to our new task, and constrain learning to that domain.

One big limitation to HD-gluing is the fact that neural network models are needed to begin with. It is generally infeasible, outside of distributed settings, to have dozens of large neural networks operating in tandem at testing time. There is a pressing need for a more advanced learning strategy than compressing many neural models to a single hypervector. While what HD-Glue can achieve through a single hypervector model is impressive, there is clearly a need to use many hypervectors together, in order to extract more knowledge from the neural network to the hypervector.

In such a construct, transfer learning into a purely hyperdimensional “network” of hypervectors can omit the need to keep the neural network models around at all. If the student hypervector model accepts some kind of encoding of the input directly, then it can transfer learn what to do with that input to classify from the teacher neural network. After training, the teacher network can be thrown away and a purely hyperdimensional model remains. This would provide a hypercompression of neural models, which can be put directly into autonomous vehicles, using minimal space or resources.

Another big avenue of future work lies in the life-long-learning aspects of HD-gluing. In our results, we performed learning until 100% performance on the training set was achieved, through the error handling method described in section IV-C. But the testing set itself could also be incorporated into this strategy. In an online learning setting, HD-gluing permits forming models that can achieve 100% accuracy on all examples seen so far, and form compact memories of previous examples that it can store to aid this process. Additionally, learned hypervector models that have been glued together can be compressed together into a further compact memory, to free up room for new models. This process can continue indefinitely. New classes could come into the picture, or new neural networks. What are the limitations of this as a life-long-learning strategy, and how does it compare to the state of the art? This is a question outside the scope of this paper.