The Need for a Big World Simulator:
A Scientific Challenge for Continual Learning

The real world is an unfathomably complex system, both to humanity and to the agents that it designs. The small agent, big world frame captures this perspective by positing that to an agent with bounded computational resources, or capacity, the world appears complex and non-stationary (Dong et al., 2022). As such, in order for said bounded capacity agent to continually fruitfully engage with the world, it must continuously ingest new knowledge while selectively retaining previously acquired knowledge. We refer to this process as continual learning.

Thus far, the study of continual learning has revolved around the discovery of phenomena which plague the naive application of algorithms designed in traditional machine learning settings. Such phenomena include the likes of “catastrophic forgetting” which describes an agent’s tendency to forget information from the past (Kirkpatrick et al., 2017; Zenke et al., 2017), and “plasticity loss” which describes an agent’s inability to absorb new information with increased time (Dohare et al., 2021; 2023; Kumar et al., 2023b; Lyle et al., 2023). These studies have brought about the design of synthetic environments which produce artificially generated and controllable data streams. These environments are designed to stress test an agent’s resilience to these phenomena.

However, if we frame the goal of continual learning to design agents which continue to engage fruitfully with the world, existing benchmarks do not capture the essence of the world. For instance, on current benchmarks, evaluation metrics for catastrophic forgetting measure an agent’s ability to remember everything in the past, which is unlikely to be necessary to fulfill a definition of fruitful engagement with the world. Current benchmarks also suffer from a limited variation in data since they draw from datasets designed for non-continual learning settings, like MNIST, CIFAR, and ImageNet. Further, they exhibit unnatural distribution shifts, such as the abrupt change in pixel permutation seen in Permuted MNIST and the random reassignment of labels to images in Random Label CIFAR (Goodfellow et al., 2013). Such constraints do not aptly capture the gradual and often subtle evolution of data encountered in real-world settings.

Moreover, these benchmarks do not embody the small agent, big world view. In particular, for natural models of an optimal capacity-constrained agent operating in a vastly more complex world, the agent is always able to substantially improve its performance if granted greater capacity. For example, performance shortfall might shrink by a factor of at least some fixed $A$ each time capacity is scaled by a factor $B$. In existing continual learning benchmarks, there are diminishing returns to increasing an agent’s capacity. In particular, at sufficient scale, each successive factor $B$ increase in capacity reduces performance shortfall by an exponentially smaller rather than fixed factor.

In this paper, we emphasize the need for a “big world simulator.” By this we mean a new type of synthetic environment that more accurately reflects the objectives and complexity of continual learning in practical settings. Such an environment should simultaneously retain the property of existing benchmarks that rapid prototyping of algorithms at small scale is possible. Furthermore, we echo Kumar et al. (2023a) by suggesting a concrete objective – that of minimizing average error over an infinite horizon subject to a capacity constraint – which reflects an agent’s ability to fruitfully engage with the world. By highlighting environment criteria and a clear objective, we aim to guide future research toward developing algorithms that can be evaluated at small scale while maintaining practical relevance in real-world applications.

We view this workshop as an opportunity to engage with the continual learning community in exploring the development of an environment simulator that can facilitate scientific advances in the area. This paper serves as an important part of this communication, laying out principles and concepts that can help frame the discussion. The paper is organized as follows. In Section 2, we briefly review a common form of synthetic continual learning benchmark that is currently used for evaluating agents. In Section 3, we formalize the notion of an environment and an agent as they pertain to continual learning. In Section 4, we propose a set of criteria for the design of future synthetic environments. In Section 5, we discuss the implications of these desiderata on evaluating forgetting and implasticity. Finally, in Section 6, we propose an illustrative example of a big world simulator that satisfies the two criteria.

Synthetic benchmarks are useful for rapid prototyping, enabling a researcher to evaluate an agent’s capabilities with respect to an array of identified challenges (Osband et al., 2020; 2022). Despite three decades of active research on continual learning (Ring, 1994; Thrun, 1998), there remains comparative difficulty in i) identifying the challenges unique to continual learning and (ii) measuring progress made on previously identified challenges. For example, recent work highlights that the challenge of catastrophic forgetting associated with existing continual learning benchmarks has been addressed by computationally inefficient solutions which are infeasible in most real-world scenarios (Prabhu et al., 2020). In this section, we briefly discuss a commonly used recipe for creating synthetic continual learning benchmarks and its fundamental limitations.

Most synthetic continual learning benchmarks use the following recipe: take an existing non-continual dataset and create a continual learning problem by incorporating some form of non-stationarity. This data stream is often delineated into a sequence of tasks, where the non-stationary is applied at specific points in time to create a new task by transforming the observation distribution or the target function. Within each task, the environment faced by the agent is stationary. For example, in Permuted MNIST (Srivastava et al., 2013; Goodfellow et al., 2013; Kirkpatrick et al., 2017), each task involves a fixed, distinct permutation of the pixels of all images in the MNIST dataset, and the challenge is to classify the digit of each image with these new inputs (See Appendix A).

This recipe for creating a continual learning benchmark has limitations stemming from its dependence on applying non-stationarity to a previously collected dataset. First, the dataset is limited in its complexity to observations and classes that have been largely collected for the purposes of training an agent to convergence. Second, the non-stationarities used are limited to artificial, unnatural and abrupt changes to the experience stream unlike the kind of structured non-stationarity of the world. These limitations of the benchmark can result in an agent that either (i) is able to outscale the challenge of learning continually (Prabhu et al., 2020), or, (ii) can no longer benefit from additional capacity. There are certainly advantages to using the recipe: several datasets already exist, along with baselines and performance levels for comparison. However, addressing these limitations can lead to a continual learning benchmark that provides a controllable, open-ended, and challenging experience stream from which an agent must learn continually.

Before outlining the criteria for a big world simulator, we first propose information-theoretic characterizations of an environment and an agent. These characterizations will help formalize the notion that the world appears complex and non-stationary to a bounded capacity agent, necessitating a trade-off between ingesting and ejecting information.

Equipped with formal characterizations of an environment, an agent, and error incurred by an agent interacting with the environment, we now propose two criteria for a big world simulator.

The forgetting that an agent experiences at time $t+1$ is the information about the future sequence $X_{T:t+2}$ which was contained in the previous agent state $U_{t}$, but not in the current agent state $U_{t+1}$ nor the most recent observation $X_{t+1}$. Since agent state $U_{t+1}=\pi(U_{t},X_{t+1})$, this mutual information exactly captures which relevant information was decanted from the agent state incorporate information from $X_{t+1}$. Note that forgetting is forward looking as it measures mutual information between the previous agent state and the future sequence as opposed to the past sequence as done in the literature. Forgetting about the past is not “catastrophic” insofar as what is forgotten is uninformative about the future.

Meanwhile, the implasticity that an agent experiences at time $t$ is the information about the future sequence $X_{T:t+1}$ which was contained in $X_{t}$, which we failed to include in $U_{t}$. This directly reflects what is referred to in the literature as “loss of plasticity”. An agent loses plasticity ,or incurs loss from implasticity, if it fails to digest information about $X_{t}$ which is relevant for making predictions about the future $X_{T:t+1}$.

Theorem 1 suggests that if our environment is $k$-complex, then an effective continual learning agent will effectively reduce $\mathcal{L}_{\pi}$ by improving upon forgetting and/or implasticity with this additional compute. Perhaps with this additional capacity, the agent will be able to hold on to more relevant information throughout its experience and incur lower error due to not forgetting. Or perhaps, the agent will elect to extract relevant information out of the data at each time step and suffer less error from implasticity. However, it’s evident that a capacity-constrained agent in a $k$-complex environment will inevitably experience error due to forgetting and implasticity.

We now draw the connection to the criterion that a finite capacity agent should never stop learning. Consider an agent $\pi$ which stops learning i.e. for some $T$, $U_{t}=U_{T}$ for all $t>T$. If the $f$ is non-stationary w.r.t $\pi$, then

where $(a)$ follows from the definition of non-stationarity and $(b)$ follows from the fact that $\pi$ stops learning. Therefore, if $f$ appears non-stationary to all agents $\pi$, then an agent which stops learning will necessarily incur error due to implasticity. To mitigate this, an effective capacity-constrained agent ought to never stop learning.

In this section, we propose a Turing-complete machine as an illustrative example of a big world simulator. A Turing-complete machine can be considered the biggest possible world because it is capable of executing any computable program, world, or environment (Turing, 1936). The specific machine that we use to simulate the big world is the cellular automaton Rule 110, which is the only elementary cellular automaton proven to be Turing-complete (Cook et al., 2004). Figure 4 in Appendix C illustrates how Rule 110 updates each cell.

The two desiderata of a big world are satisfied in theory by the the problem of predicting future observations from Rule 110. This is because Rule 110 defines a transition over an infinite state. An agent with finite capacity tasked with predicting a future state can only observe a finite region of the state and can only predict up to some finite horizon. Critically, the region observed by the agent at each time step depends on both the observed and unobserved regions from previous time steps. Increasing the agent’s capacity can improve its predictions, but the agent must never stop learning because of the infinite unobserved region.²²2To simulate the infinite state, we use a periodic boundary condition to connect the boundaries of the observed regions to the boundaries of the unobserved regions. While an agent with unbounded capacity could observe the infinite state, in our finite simulation, an agent with unbounded capacity could observe its entire history. See Figure 3 for different example simulations.

We now demonstrate that the Turing-complete prediction environment empirically satisfies the two desiderata of a big world simulator. We trained a neural network with SGD to predict the observation 16 steps into the future given the current observation. In Figure 3, we observe that increasing the agent’s capacity (in the form of depth) leads to make increasingly more accurate predictions of the future observation. This relationship approximates $k$-complexity, and we anticipate a stronger relationship for the capacity of a history-encoding agent. Lastly, because of the unobserved region which simulates the infinite state, the agent must not stop learning. In Appendix C, we observed that some agent algorithms did stop learning due to loss of plasticity.

In this paper, we have highlighted the need for a big world simulator that accurately mirrors the challenges and complexities inherent in continual learning. We propose two properties which a big world simulator should exhibit: (1) there are no diminishing returns to increasing an agent’s capacity, and (2) an optimal finite capacity agent should never stop learning. Furthermore, we reiterate the concrete objective of minimizing average error over an infinite horizon subject to a capacity constraint and provide a decomposition of this objective into two terms which closely resemble the notions of “forgetting” and “plasticity loss” from the literature. Finally, we present a Turing-complete prediction environment as an illustration of a simulator with desired big world properties. We hope that this work stimulates discussion around next steps for designing a simulator that facilitates the discovery of effective continual learning agent design and evaluation metrics which better reflect the objective.