Executive Function: A Contrastive Value Policy for Resampling and Relabeling Perceptions via Hindsight Summarization?
Research Note

Raw perception data is dense. However, in order to use it to predict, we would like to transform it into sparse²²2Sparsity is defined as density of zeros in a matrix representations. Sparse representations are useful in that they can be converted into tokenized representations (vocabularies), which allow for compositionality and causal reasoning.

We propose that it is particularly important to learn world models which have distributions of activations which are compositional in time and compositional in abstractive depth.

We define composition in time as mapping an event $A$ into an equivalent sequence of events $f(A)=a_{0},a_{1},...,a_{n}$ where $a_{i+1}$ occurs after $a_{i}$, as well as the inverse mapping $g(a_{0},a_{1},...,a_{n})=A$. This allows for the sequencing of events and long-horizon planning.

Composition in abstraction - that is, the ability to represent a concept either as an abstract idea (eg. “x = y”) or a series of concrete examples (eg. “when x = 1, y = 1, when x = 2, y = 2, when x = 3, y = 3”) - allows for the communication of concrete representations to other neural networks. Mathematically, we would like to learn a mapping from an abstract concept $A$ to a series of concrete examples: $f(A)=a_{0},a_{1},...,a_{n}$, and an inverse mapping $g(a_{0},a_{1},...,a_{n})=A$. Note that compositionality in abstraction is a softer variant of hierarchical representation as multiple mappings may exist.

Composition may also be recognizable as communication. For example:

This is the process of two trained neural networks (human brains) communicating abstractions between each other’s networks using words. In the case of more shared parameters (common culture, familiarity), fewer words need to be said to transmit the same abstract idea between the two humans, and small changes in words can induce substantially different generalizations for humans (Luntz, 2007). As Chomsky has noted, the only indispensable operation of human grammar is that of Merge, which is not unlike compositionality in time and abstraction mentioned above (Chomsky, 1995).

Communication, however, does not only occur externally to other humans using language. It also occurs to ourselves (via our inner voice) and between layers in neural networks using non-word representations. After all, outputs from a layer $L$ are latent variables which serve as inputs to layer $L+1$. If you come from a machine learning background, you may notice that the diagram is simply the autoencoder setup shifted right by 1.

Sparsity, compositionality in time and compositionality in abstract representation are necessary properties of causal world models.

Although specifically how to learn causal world models is still an open research question, imparting inductive biases for sparse representations has been useful for improving neural network performance - key-value attention allows sparsity of communication in time (Vaswani et al., 2017), while skip connections allow sparsity of communication in abstractive depth (He et al., 2015), since it allows lower layer world models to communicate directly with higher layers of a network.

However, only having a sparse and compositional world model is not sufficient for it to be causal. For a world model to be causal it must be able to accurately model interventions and counterfactuals (Pearl, 2009). For example, “does sunlight catalyze the production of ATP in plants?” requires understanding how the intervention (sunlight) affects your world model of plant energy generation. Counterfactuals are a special type of intervention which is non-factual, for example, “without ATP synthase, would sunlight affect the production of ATP in plants?” If a model is able to answer a large number of these questions over a diversity of interesting interventions, we might consider that it has learned a causal model of the topic of sunlight and ATP.

Note that, with an existing world model of part of the input (say “sunlight”), it becomes easier to understand the question “does sunlight catalyze the production of ATP in plants?” If you didn’t know what sunlight means, on the other hand, your first instinct might be ask “what is sunlight?” to be able to systematically collect new training data about the part of the world model that was unknown. Similar contrastive prediction-loss motivated resampling has been shown to impart generalization with systematicity (Akyürek et al., 2021), especially when inputs are collected for contrastive interventions and outputs have contrastive predictions induced by the same contextual data (see Figure 2).

Although there are many works describing useful inductive biases for neural network architectures trained on static training data, learning a policy for data labeling and data resampling over compositional representations needed for continual few-shot learning has not been well-addressed. To motivate this, we draw on observations of human learning, as humans are arguably experts in embodied continual few-shot learning that have behavior policies which have been pretrained by millenia of evolution.

Perceptions are subtly biased towards confirming predictions. Humans are prone to confirmation bias, whereby they create summaries which confirm our predictions without actually learning to control or predict accurately (Garrison and Hoskisson, 1989). At the subconscious level, we experience gestalt, subconsciously filling in missing parts of a whole (Wertheimer, 1923).

However, when prediction loss is high enough, we experience this as an uncomfortable feeling of cognitive dissonance (Kaaronen, 2018), and will take actions to reduce prediction loss by changing $y$ or $\hat{y}$. Anticipated prediction error can be reduced by taking actions within the external environment to change $y$ to match $\hat{y}$, while realized prediction error can be changed by modifying network weights so that $\hat{y}$ becomes closer to $y$ (“learning”), a behavior which is aided by using executive function to resample or relabel perceptions (changing $y$ and $\hat{y}$) (“thinking”).

Although reducing prediction loss is a key drive, if we experience too little prediction loss, we also strive to increase it via curiosity-driven exploration, thus staying balanced within a “golden mean” of error (Plato, 375 BC) formed by a Markov blanket over our predictions.³³3In regimes with high prediction loss, there is chaos, fear and uncomfortable uncertainty. In regimes with low prediction loss, boredom. In the middle is happiness. As in folk wisdom, “the journey is the reward.”

When a prediction involves a ego-perception (ie. something that we will do in the future), in order to minimize expected predictive loss, we attempt to control our environment through action, a process called active inference (Friston et al., 2006) (Figure 3). This can be consciously shaped by summarization and visualization - in sports, positive self-talk and visualization is a technique which is correlated with improved performance by creating prediction loss with respect to target performance (Raalte and Vincent, 2017).

Both conscious and unconscious predictions motivate action, regardless of their origin - ideas that “pop into your head” tend to have connections to past events. For example, when you have just read the sentence “does sunlight catalyze the production of ATP in plants?”, you may be more likely to eat a salad tonight or go outside (Luntz, 2007).

The most straightforward way to minimize realized prediction loss is by updating weights or growing neural connections such that they directly change the world model so that the predictions are closer to perceptions. In artificial neural networks this occurs via backpropagation (Rumelhart et al., 1986) or many other optimization methods, and in the human brain, via Hebbian learning (Hebb, 1949).

To assist with learning, as with most animals, we can resample data via physical movement which results in new contrastive perceptions (Friston and Kiebel, 2009). For example, at a cocktail party, you may move your head to look at a speaker’s lips to be able to better distinguish the utterances of the speaker from the background noise. We can also employ longer-term learning loops to resample data, for example, calling a friend for a second opinion on how to approach a situation. In this case, you acquire two summary labels for the situation (one from yourself, one from your friend). This contrastive data makes it easier to learn accurate causal world models.

Although taking actions is an effective way to gather contrastive data, it is also quite expensive and impractical in many settings. Shifting attention to novel internal perceptions allows us to sample new perceptions and predictions without motor control. Humans have learned to hack this process for pleasure and control. Attending to the sensations of your body which are easily predictable via meditation may be a method to consciously lower prediction loss by changing the contents of the prediction stack (Jamieson, 2015; Pagnoni, 2019). Mindfulness lowers reward prediction error and activates the putamen in practicioners (Kirk and Montague, 2015). Consciousness presence has been hypothesized to be due to top-down predictions successfully suppressing informative interoceptive signals (Seth et al., 2012).

The Complementary Learning Systems theory (McClelland et al., 1995) proposes that the hippocampus encodes “fast” learning policies resulting in direct storage of incoming perceptions, while the neocortex replays these perceptions slowly (interspersed with other perceptions) to achieve a slow learning of abstractions and subconscious behaviors.

We propose that fast learning may be a behavior that uses the tools of summarization, active inference and multiple stateful passes through a pre-trained network (which activate latent abstractions) such that the resulting output data forms a batch which has sparse and compositional representations such that it can be more easily learned, a phenomenon we call learning a communicative mapping to a grammar of learning, similar to an online prompt engineering problem. Specifically, for any network $\theta$ which has been pre-trained on dataset $D$, there exists a data distribution $D^{\prime}$ such that an input $d^{\prime}\in D^{\prime}$ will produce a sparse, compositional distribution of outputs which can be resampled and relabeled contrastively, thus inducing efficient learning. We call this $D^{\prime}$ a grammar of learning for the network and the original grammar of learning without pre-training data a universal grammar of learning ⁴⁴4This is largely just an inductive bias for sparse and compositional input data. Given a novel input $e$, if the network can find a series of transformations to map $e$ into a latent representation $d^{\prime}$, then $e$ can be more easily learned by association with $d^{\prime}$.

The average human reaction time is  250ms and the human eye has a resolution of  576 megapixels, resulting in a data stream of at least  7GB/second. To have the compositional and sparse representations required for learning of causality, we must significantly reduce the dimensionality of this firehose of data. This done by hindsight summarization (Figure 4). Hindsight summarization has access to a memory and a grammar of learning to enable rapid non-linear data transformations via existing world models. Creating a hindsight summary changes the composition of the perception stack attended by executive function, creating new prediction losses which can be used directly as training data (Figure 5) or indirectly to produce further summaries, forming a stream of consciousness which quickly changes the composition of the entire perception stack.

Streams of consciousness like beneficial self-talk (Raalte and Vincent, 2017) and cognitive behavioral therapy (Manjaly and Iglesias, 2020) may be learned behaviors to modify experienced prediction loss using hindsight summarization.

These can be complex and diverse. Adopting a growth mindset by framing unpredictable perceptions as a hypothesis test (so that any outcome is now expected with a certain probability) is correlated with reduced cognitive dissonance during experimentation for the authors. Making explicit predictions before reading (a form of data relabeling via increasing attention to prediction loss) has been shown to improve human learning (Thomas-Fair, 2005; Brod, 2021), and may be an example of consciously increasing prediction loss to focus learning mechanisms. They may also occur unconsciously - humans make long-range predictions at multiple levels of latent depth when reading (Caucheteux et al., 2021).⁵⁵5Eric Hayot, a comparative literature professor, once told one of the authors to “kick the ladder” - that is, after finishing a draft, rewrite it with the benefit of hindsight so that you weave anticipation of the summary into the piece, and remove the confusing scratch writings you used to bootstrap the summary for yourself as an author. In this way, the reader’s executive function naturally anticipates the conclusion with less cognitive dissonance.

Regions of the brain associated with executive function are larger in humans than other primates (Donahue et al., 2018), and humans tend to be better at few-shot learning than other animals, which may be related to the ability to set conscious plans (Fuster, 2017).

Analogues to simple patterns of hindsight summarization for non-embodied agents can also be seen in SOTA approaches to self-supervised learning, such as data2vec (Baevski et al., 2022), where summaries can be considered to be the target latent representations of the teacher, and the future masking of Figure 5 is analogous to the input masking in the student model in Baevski et al. (2022). However, data2vec does not use multiple steps of conditional computation, as humans do.

Hindsight summarization can also be compared to other hindsight schemes such as HER (Andrychowicz et al., 2018), however summarization is a learned path function over the past trajectories rather than a deterministic function of the last state, as in HER. Unlike generalized hindsight (Li et al., 2020), hindsight summarization encodes information about a trajectory of observations, not just rewards, and runs on a much faster timescale, possibly summarizing a sequence of perceptions which span intervening training epochs.

Sequential steps of hindsight summarization with intervening training and counterfactual sampling may be recognizable as the Scientific Method, a well-known meta-learning algorithm for scientific discovery. See Figure 6 for how we might model this stream of consciousness and use insights to inform future behavior.