Collective Intelligence for Deep Learning: A Survey of Recent Developments

Implicit relationships and recurring patterns in nature (such as texture and scenery) can benefit from employing approaches from cellular automata in learning alternative representations of natural images. Like CeNNs, the Neural Cellular Automata (neural CA) model proposed by Mordvintsev et al. Mordvintsev et al. (2020) treated each individual pixel of an image as a single neural network cell. The networks are trained to predict its color based on the states of its immediate neighbors, thereby developing a model of morphogenesis for image generation. They demonstrated that it was possible to train neural networks to reconstruct entire images this way, even when each cell lacks information about its location and rely only on local information from its neighbors. This approach enabled the generation algorithm to be resistant to noise, and moreover, allowed images to regenerate when damaged. An extension of neural CA Randazzo et al. (2020) enabled individual cell to perform image classification tasks, such as handwritten digit classification (MNIST) by only examining the contents of a single pixel, and passing a message on to the cell’s immediate neighbors (See Figure 5). Over time, a consensus will be formed as to which digit is the most likely pixel, but interestingly, disagreements may result depending on the location of the pixel, especially if the image is intentionally drawn to represent different digits.

The regeneration with neural CA has been explored beyond 2D images. In a later work, Zhang et al. Zhang et al. (2021) employed a similar approach to 3D voxel generation. This is particularly useful for high-resolution 3D scanning, where 3D shape data is often described with sparse and incomplete points. Using generative cellular automata they can recover full 3D shapes from only a partial set of points. This approach is also applicable outside of pure generative domains, and can also be applied to the construction of artificial agents in active environments such as Minecraft. Sudhakaran et al. Sudhakaran et al. (2021) trained neural CAs to grow complex entities from Minecraft such as castles, apartment blocks, and trees, some of which are composed of thousands of blocks. Aside from regeneration, their system is able to regrow parts of simple functional machines (such as a virtual creature in the game), and they demonstrate a morphogenetic creature grow into two distinct creatures when cut in half in the virtual world (See Figure 6).

Cellular Automata is also naturally applicable to provide visual interpretation for images. Qin at al. Qin et al. (2018) examined the use of a Hierarchical CA model for visual saliency, to identify items in an image that stand out. By getting a CA to operate on visual features extracted from a deep neural network, they were able to iteratively construct multi-scale saliency maps of the image, with the final image being close to the target items. Sandler et al. Sandler et al. (2020) later investigated the use of CA for the task of image segmentation, an area where deep learning enjoys tremendous success. They demonstrated the viability of performing complex segmentation tasks using CAs with relatively simple rules (with as little as 10K neural network parameters), with the advantage of the approach being able to scale up to incredibly large image sizes, a challenge for traditional deep learning models with millions or even billions of model parameters, which are bounded by GPU memory.

The rise in deep learning gave birth to the use of deep neural networks for reinforcement learning, or deep reinforcement learning (Deep RL), equipping reinforcement learning agents with modern neural networks architectures that can address more complex problems, such as high dimensional continuous control or vision-based tasks from pixel observations. While Deep RL shares successful characteristics with deep learning, in that employing sufficient computation resources will generally lead to the solution of a target training task to be found. But like deep learning, Deep RL has its share of limitations. Agents trained to perform a particular task often fail when the task is slightly altered. Furthermore, neural network solutions generally only work for a specific morphology with well-defined input and output mappings. For instance, a locomotion policy trained for a 4-legged ant might not work for a 6-legged one, and a controller that expects to receive 10 inputs won’t work if you give it 5, or 20 inputs.

The evolutionary computation community started approaching some of these challenges earlier on, by incorporating modularity Schilling (2000); Schilling and Steensma (2001) in the evolutionary process that govern the design of artificial agents. Having agents that are composed of identical but independent modules foster self-organization via local interactions between the modules, enabling systems that are robust to changes in the agent’s morphology, an essential requirement in evolutionary systems. These ideas have been presented in the literature of work on soft-bodied robotics Cheney et al. (2014), where robots consist of a grid of voxel cells–each controlled by an independent neural network with local sensory function that can produce a localized action. Through message passing, the group of cells that make up the robot are able to self-organize and perform a range of locomotion tasks (See Figure 7). Later work Joachimczak et al. (2016) even proposes incorporating metamorphosis in the evolution of the placement of the cells to produce configurations robust to a range of environments.

Recently, soft-bodied robots have even been combined with the neural CA approach discussed earlier to enable these robots to regenerate themselves. Horibe et al. (2021) To bridge the gap between policy optimization (where the goal is to find the best parameters of the policy neural network) usually done in the Deep RL community and the type of morphology-policy co-evolution (where both the morphology and the policy neural network is optimized together) work done in the soft-bodied literature, Bhatia et al. Bhatia et al. (2021) has recently developed an OpenAI Gym-like Brockman et al. (2016) environment called Evolution Gym, a benchmark for developing and comparing algorithms for co-optimizing design and control, which provided an efficient soft-bodied robot simulator written in C++ with a Python interface.

Modular, decentralized self-organizing controllers have also started to be explored in the Deep RL community. Wang et al. Wang et al. (2018) and Huang et al. Huang et al. (2020) explored the use of modular neural networks to control each individual actuator of a simulated robot for continuous control. They expressed a global locomotion policy as a collection of modular neural networks (in the case of Huang et al. Huang et al. (2020), identical networks) that correspond to each of the agent’s actuators, and trained the system using RL. Like soft-bodied robots, every module is only responsible for controlling its corresponding actuator and receives information from only its local sensors (See Figure 8). Messages are passed between neighboring modules, propagating information between distant modules. They show that a single modular policy can generate locomotion behaviors for several distinct robot morphologies, and show that the policies generalize to variations of the morphologies not seen during training, such as creatures with extra legs. As in the case of soft-bodied robots, these results also demonstrate the emergence of centralized coordination via message passing between decentralized modules that are collectively optimizing for a shared reward.

The aforementioned work hints at the power of embodied cognition, which emphasizes the role of the agent’s body in generating behavior. Although the focus of much of the work in Deep RL is in learning neural network policies for an agent with a fixed design (e.g., a bipedal robot, humanoid, or robot arm), embodied intelligence is an area that is gathering interest in the sub-field Ha (2018); Pathak et al. (2019). Inspired by previous work on self-configuring modular robots Stoy et al. (2010); Rubenstein et al. (2014); Hamann (2018), Pathak et al. Pathak et al. (2019) investigates a collection of primitive agents that learn to self-assemble into a complex body while also learning a local policy to control the body without an explicit centralized control unit. Each primitive agent (which consists of a limb and a motor) can link up with nearby agents, allowing for complex morphologies to emerge. Their results show that these dynamic and modular agents are robust to changes in conditions and the policies can generalize to not only unseen environments, but also to unseen morphologies consisting of a greater number of modules. We note that these ideas can be used to allow general DL systems (not confined to RL) to have more flexible architectures that can even learn machine learning algorithms, and we will discuss this later on in the Meta-Learning section.

Aside from adapting to changing morphologies and environments, self-organizing systems can also adapt to changes in their sensory inputs. Sensory substitution refers to the brain’s ability to use one sensory modality (e.g., touch) to supply environmental information normally gathered by another sense (e.g., vision). However, most neural networks are not able to adapt to sensory substitutions. For instance, most RL agents require their inputs to be in an exact, pre-specified rigid format, otherwise they will fail. In a recent work, Tang and Ha Tang and Ha (2021) explored permutation invariant neural network agents that require each of their sensory neurons (receptors that receive sensory inputs from the environment) to deduce the meaning and context of its input signal, rather than explicitly assume a fixed meaning. They demonstrate that these sensory networks can be trained to integrate information received locally, and through communication between them using an attention mechanism, can collectively produce a globally coherent policy. Moreover, the system can still perform its task even if the ordering of its sensory inputs (represented as real-valued numbers) is randomly permuted several times during an episode. Their experiments show that such agents are robust to observations that contain many additional redundant or noisy information, or observations that are corrupt and incomplete.

Collective intelligence can be viewed at several different scales. The brain can be viewed as a network of individual neurons functioning collectively. Each organ can be viewed as a collection of cells performing a collective function. Individual animals can be viewed as a collection of organs working together. As we zoom out further, we can also look at human intelligence beyond biology and see human civilization as a collective intelligence solving (and producing) problems that are beyond the capabilities of a single person. As such, while in the previous section, we discussed several works that leverage the power of collective intelligence to essentially decompose a single RL agent into a collection of smaller RL agents working together towards a collective goal, resembling a model of collective intelligence at the biological level, we can also view multi-agent problems as a model of collective intelligence at the societal level.

A major focus of the collective intelligence field is to study the group intelligence and behaviors emerged from a large collection of individuals, whether in humans citetapscott2008wikinomics, animals Sumpter (2010) insects Dorigo et al. (2000); Seeley (2010), or artificial swarm robots Hamann (2018); Rubenstein et al. (2014). This focus has clearly been missing in the Deep RL field. While multi-agent reinforcement learning (MARL) is a well-established branch of Deep RL, most learning algorithms and environments proposed have targeted a relatively small number of agents Foerster et al. (2016); OroojlooyJadid and Hajinezhad (2019), and thus not sufficient to study the emergent properties from large populations. In the most common MARL environments Resnick et al. (2018); Baker et al. (2019); Jaderberg et al. (2019); Terry et al. (2020), “multi-agent” simply means 2 or 4 agent trained to perform a task by means of self-play  Bansal et al. (2017); Liu et al. (2019); Ha (2020). Collective intelligence observed in nature or in society, however, relies on a much larger number of individuals than typically studied in MARL, involving population sizes from thousands to million. In this section, we will discuss recent works from the MARL sub-field of Deep RL that had been inspired by collective intelligence (as their authors have even noted in their publications). Unlike most MARL works, these work started to employ large population of agents (each enabled by a neural network), from thousands to millions, in order to truly study their emergent properties at the macro level (1000+ agents), rather than at the micro-level (2-4 agents).

Recent advances in Deep RL have demonstrated the capabilities of simulating thousands of agents in complex 3D simulation environments using only a single GPU Heiden et al. (2021); Rudin et al. (2021). A key challenge is in approaching the problem of multi-agent learning at a much larger scale, leveraging such advances in parallel computing hardware and distributed computation, with the goal of training millions of agents. In this section, we will example recent attempts at training a massive number of agents that interact in a collective setting.

Rather than focusing on realistic physics or environment realism, Zheng. et al. Zheng et al. (2018) developed a platform called MAgent, a simple grid-world environment that can allows millions of neural network agents. Their focus is on scalability, and they demonstrate that MAgent can host up to a million agents on a single GPU (in 2017). Their platform supports interactions among the population of agents, and facilitates not only the study of learning algorithms for policy optimization, but more critically, enables the study of social phenomena emerging from the millions of agents in an AI society, including the emergence of languages and societal hierarchy structures that may have emerged. Environments can be built using scripting, and they have provided examples such as predator-prey simulations, battlefields, adversarial pursuit, supporting different species of distinct agents that may exhibit different behaviors.

MAgent inspired many recent applications, including multi-agent driving Peng et al. (2021), which looks at emergent behavior of entire populations of driving agents to optimize the driving policies which not only affect a single car, but aim to improve the safety of the population as a whole. These directions are good examples that demonstrate the difference between problems framed for deep learning (finding a driving policy for a single car) versus problems in collective intelligence (finding a driving policy for the entire population).

Inspired by the game genre of MMORPGs (Massively Multiplayer Online Role-Playing Games, aka MMOs), Neural MMO Suarez et al. (2021) is an AI research environment that supports a large number of artificial agents that have to compete for finite resources in order to survive. As such, their environment enables large-scale simulation of multi-agent interactions that requires agents to learn combat and navigation policies alongside other agents in a large population all attempting to do the same. Unlike most MARL environments, each agent is allowed to have their own distinct set of neural network weights, which has been a technical challenge in terms of memory consumption. Preliminary experimental results in early versions of the platform Suarez et al. (2019) demonstrated agents with distinct neural network weight parameters developed skills to fill different niches in order to avoid competition within a large population of agents.

As of writing, this project is in active development in the NeurIPS machine learning community Suarez et al. (2021) to work towards studies of large agent populations, long time horizons, open-ended tasks, and modular game systems. The developers provide active support and documentation, and also develop additional training, logging, and visualization tools to enable this line of large-scale multi-agent research. This work is still in its early stages, and only time will tell if platforms that enable the study of large populations such as Neural MMO or MAgent gain further traction within the Deep RL communities.

In the previous sections, we described works that express the solution to problems in terms of a collection of independent neural network agents acting together to achieve a common goal. These parameters of these neural network models are optimized for the collective performance of the population. While these systems have been shown to be robust and adapt to changes in its environment, they are ultimately hardwired to perform a certain task, and cannot perform another task unless retrained from scratch.

Meta-learning is an active area of research within deep learning where the goal is to train the system to learn. It is a large sub-field of ML, including areas such as simple transfer learning from one training set to another. For our purposes, we follow the line of work from Schmidhuber Schmidhuber (2020), where he views meta learning as the problem of ML algorithms that can learn better ML algorithms, which he believes is required to build truly self-improving AI systems.

So unlike traditionally training a neural network to perform one task, where the weight parameters of neural networks are traditionally optimized with a gradient descent algorithm, or with evolution strategies Tang et al. (2022), the goal of meta-learning is to train a meta-learner (which can be another neural network-based system) to learn a learning algorithm. This is a particularly challenging task, with a long history see Schmidhuber Schmidhuber (2020) for a review). In this section, we will highlight recent promising works that make use of collective agents that can learn to learn, rather than learn to perform only a particular task (which we have covered in the previous section).

Concepts from self-organization can be naturally applied to train neural networks to meta-learn by extending the basic building blocks that compose artificial neural networks. As we know, artificial neural networks consist of identical neurons which are modeled as non-linear activation functions. These neurons are connected in a network by synaposes which are weight parameters which are normally trained with a learning algorithm such as gradient descent. But one can imagine extending the abstraction of neurons and synapses beyond static activation functions and floating point parameters. Indeed, recent work Ohsawa et al. (2018); Ott (2020) have explored modeling each neuron of a neural network as an individual reinforcement learning agent. Using the terminology of RL, each neuron’s observations are its current state which change as information is transmitted through the network, and each neuron’s actions enable each neuron to modify its connections with other neurons in the system, hence the problem of learning to learn is treated as a multi-agent RL problem where each agent is part of the collection of neurons in a neural network. While this approach is elegant, the aforementioned works are only capable of learning to solve toy problems and are not yet competitive with existing learning algorithms.

Recent methods have gone beyond using simple scalar weights to transmit scalar signals between neurons. Sandler et al. Sandler et al. (2021) introduce a new type of generalized artificial neural network where both neurons and synapses have multiple states. Traditional artificial neural networks can be viewed as a special case of their framework with two-states where one is used for activations, the other is used for gradients produced using the backpropagation learning rule. In the general framework, they do not require the backpropagation procedure to compute any gradients, and instead rely on a shared local learning rule for updating the states of the synapses and neurons. This Hebbian-style bi-directional local update rule would only require that each synapse and neuron only requires state information from their neighboring synapse and neurons, similar to cellular automata. The rule is parameterized as a low-dimensional genome vector, and is consistent across the system. They employed both evolution strategies, or conventional optimization techniques to meta-learn this genome vector, and their main result is that the update rules meta-learned on the training tasks generalize to unseen novel test tasks. Furthermore, the update rules perform faster than gradient-descent based learning algorithms for several standard classification tasks.

A similar direction has been taken by Kirsch et al. Kirsch and Schmidhuber (2020), where the neurons and synapses of a neural network are also generalized to higher dimension message-passing systems, but in their case each synapse is replaced by an recurrent neural network (RNN) with the same shared parameters. These RNN synapses are bi-directional and govern the flow of information across the network. Like Sandler et al. Sandler et al. (2021), the bi-directional property allows for the network to be used for both inference and learning at the same time by running the system in forward-pass mode. The weights of this system are essentially stored in the hidden states of the RNNs so by simply running the system, they can train themselves using the error signals as feedback. Since RNNs are general-purpose computers, they were able to demonstrate that the system can encode the gradient-based backpropagation algorithm by training the system to simply emulate backpropagation, rather than explicitly calculating gradients via hand-engineering. Of course, their system is much more general than backpropagation, and thus capable of learning new learning algorithms that are much more efficient than backpropagation (See Figure 13).

The previous two works mentioned in this section are only recently published at the time of writing, and we believe that these decentralized local meta-learning approaches have the potential to revolutionize the way neural networks are used in the future in a way that challenges the current paradigm that separates model training and model deployment. There is still much work to be done in demonstrating that these approaches can scale to larger datasets, due to inherently much larger memory requirements (due to much larger internal states of the system). Furthermore, while the algorithms are able to produce learning algorithms that are vastly more sample efficient compared to gradient descent, this efficiency is only apparent in the early stages of learning, and performance tends to peak very early on. Gradient descent, while less efficient, is less biased towards few-shot learning, and can continue to run for many more cycles to refine the weight parameters that will ultimately produce networks that achieve higher performance.