Self-Programming Artificial Intelligence Using Code-Generating Language Models

Artificial intelligence technology has sought to emulate human cognitive capabilities within a computational substrate. Machine learning, and more recently deep learning, has enabled computers to derive programs from relevant data in order to address novel tasks for which they were not explicitly programmed.

Advanced artificial intelligence is theorized to require self-modifying programs that continuously imbue themselves with extended capabilities (Schmidhuber, 2006, 1987), surpassing their pre-programmed functionalities to achieve increased computational and predictive effectiveness. Meta-learning approaches have made significant progress along this avenue, from self-referential evolutionary algorithms (Schmidhuber, 1987) to RNN-based learners (Santoro et al., 2016) and modern gradient-based meta-learners (Finn et al., 2017; Khodak et al., 2019; Li et al., 2017). Adjacent work in neural architecture search (Real et al., 2018; Stanley et al., 2009; Zoph and Le, 2016) has led to the development of various algorithms for automatically optimizing the architectural parameters of neural networks.

In recent years, AI-based computer code generation has seen dramatic progress (Chen et al., 2021; Le* et al., 2022; Li et al., 2022; Wang et al., 2021) stemming from breakthroughs in large-scale language processing models (Brown et al., 2020; Devlin et al., 2018; Raffel et al., 2020; Vaswani et al., 2017). Modern language models (LMs) trained on computer programming tasks can not only generate syntactically correct computer code, but can also formulate logically correct computer programs from human-language prompts and understand pre-written programs to derive human-interpretable information.

At the intersection of these paradigms, we set out to explore a completely new avenue of research by combining self-modification approaches with code-generation approaches to achieve self-programming AI. Although theoretical formulations of self-programming AI (Schmidhuber, 2006) have been posed in the past, no practical implementation of a fully self-programming AI has been achieved to date under real-world computational constraints.

In this paper, we implement a LM-based code generation model with the ability to rewrite and improve its own source code, thereby achieving the first practical implementation of a self-programming AI system. With free-form source code modification, it is possible for our model to change its own model architecture, computational capacity, and learning dynamics.

Since this system is designed for programming deep learning models, it is also capable of generating code for models other than itself. Such models can be seen as sub-models through which the main model indirectly performs auxiliary tasks. We explore this functionality in depth, showing that our model can easily be adapted to generate the source code of other neural networks to perform various computer vision tasks. We illustrate our system’s ability to fluidly program other deep learning models, which can be extended to support model development in various other fields of machine learning.

Meta-learning. Meta-learning is the subfield of machine learning that attempts to "learn how to learn". To be precise, let $\omega$ denote an assumption on "how to learn" and $p(\mathcal{T})$ be a task distribution, where a "task" $\mathcal{T}=\{\mathcal{D},\mathcal{L}\}$ for some dataset $\mathcal{D}$ and loss function $\mathcal{L}$ (Hospedales et al., 2022). Then, the goal of meta-learning can be viewed as computing

Approaches in this subfield are broadly classified into RNN-based learners (Santoro et al., 2016), gradient-based meta-learners (Finn et al., 2017; Khodak et al., 2019; Li et al., 2017), and self-referential evolutionary algorithms (Schmidhuber, 1987).

A select body of prior work within meta-learning establishes the conceptual groundwork for theoretical self-programming AI systems. Notably, Gödel machines (Schmidhuber, 2006) pose a theoretical formulation wherein an intelligent system can continuously rewrite itself in a globally optimal way. Schmidhuber (1987) discusses some of the evolutionary concepts that might underpin such systems.

Automated Code Generation. Vaswani et al. (2017) introduced the transformer, an attention-based architecture that alleviated many of the previous issues with prior sequence-to-sequence models. The success of this architecture and subsequent innovations in Natural Language Processing (Brown et al., 2020; Devlin et al., 2018; Raffel et al., 2020) have led to breakthroughs in AI-driven code generation (Chen et al., 2021; Drori et al., 2022; Le* et al., 2022; Li et al., 2022; Wang et al., 2021).

Chen et al. (2021) revolutionized the application of language models to code generation. Le* et al. (2022) and Wang et al. (2021) extended the transfer learning concepts introduced in Raffel et al. (2020) to create systems capable of efficiently executing multiple tasks, including code generation and understanding. Li et al. (2022) created a system that was able to achieve top-half performances at various code competitions. Drori et al. (2022) showed that a pretrained language model fine-tuned on code was capable of solving, generating, and explaining university level math problems, as well as problems from challenging high-school math competitions. Although these systems are capable of writing programs for diverse and complex tasks, they are not capable of modifying and improving their own source code.

AutoML. AutoML is a nascent and incredibly complex field that aims to automatically generate machine learning algorithms to solve problems without human tuning. Part of the difficulty lies in the fact that intricate, well-crafted neural networks are often necessary to obtain high performance on even mildly nontrivial problems such as CIFAR-100. Thus, the subfield of Neural Architecture Search(NAS) formed with the goal of creating algorithms capable of efficiently generating suitable architectures for a given problem. There are three approaches typically used for NAS: reinforcement learning-based NAS (Jaafra et al., 2019; Zoph and Le, 2016), gradient-based NAS (Shi et al., 2021), and evolutionary NAS (ENAS) (Liu et al., 2021).

Evolutionary algorithms are based upon the principles of Darwinian evolution and natural selection: a population of algorithms are created and evaluated, the "fittest" (best-performing) survive and are used as "parents" for a future generation of algorithms that are then evaluated, and so on (Liu et al., 2021). Some evolutionary approaches include regularized evolution search (Real et al., 2018, 2020), particle swarm optimization (Haidar et al., 2021), and genetic algorithms (Whitley, 2021). While AutoML techniques are usually restricted to one type of network (such as convolutional neural networks), our system is capable of editing and improving all sorts of neural network source code.

Our Contribution. We extend the methodologies of automated code generation as well as some simple evolutionary ideas from the field of AutoML to create the first functional self-programming AI system. In particular, we use a code-generating transformer model based on Raffel et al. (2020) and a simple genetic algorithm (Whitley, 2021).

Our self-programming results verify that our model is capable of editing its own source code to extend its capabilities.

Furthermore, on all three tasks for programming other models, our system is able to autonomously design networks that achieve effective performance. Our approach is both flexible by way of its free-form programming capabilities and simple in its search and selection procedures. Other methods to search for optimal networks, such as reinforcement learning, may be unstable (Kumar et al., 2020) when applied to different domains, and therefore may require specialized tuning in order to be effective.

Accuracy on MNIST is around 1.5% below SoTA and EMNIST ByClass is about 10% below SoTA, but performance on CIFAR-10 lags significantly behind the current SoTA (Vahidian et al., 2021). This is likely due to the fact that competitive results on CIFAR-10 typically require specialized model designs, which are unlikely to be achieved since our approach uses a simple class of models with greedy search over random code modifications. Performance on these three problems could possibly be improved to by using more sophisticated source code data and search algorithms specialized for computer vision model development.

As shown in Figure 4a, the average performance of generated models on CIFAR-10 tends to increase with the total number of candidates (the product of the number of code search iterations multiplied by the number of candidates per iteration) up to a certain point. Figure 4b shows that the average performance of generated models increases significantly with the number of candidates tried per epoch, and is not as sensitive to the number of code search iterations.

Figure 5 plots the average model size (number of parameters) of generated models over 10 consecutive reprogramming steps. The most significant increases were found to typically occur in the number of convolutional layers. Other significant increases occurred in the size of the hidden layers. The size of the convolutional layers did not increase significantly in any of the three tasks (it was found to fluctuate for CIFAR-10 and MNIST), and the number of hidden layers was rarely found to be increased.

In the three tasks involving generating other models, the code-generating model consistently chose to increase convolutional layer counts, which was found to produce significant increases in performance. These deeper convolutional network source codes were naturally reused by the algorithm as input in subsequent code generation queries. In the initial training loop, most networks had no convolutional layers and a small minority had a single convolutional layer. Within four loops, every network had at least 2 convolutional layers and often more. The number of hidden layers did not significantly increase, indicating that deeper feed-forward architectures may not be necessary to achieve increased performance on these problems.

As can be seen in 6, the number of trainable parameters in the best-performing network increased significantly over time. This agrees with our current understanding of neural networks that larger models with more parameters are more expressive and capable of effectively addressing more complex tasks (Hestness et al., 2017; Kaplan et al., 2020).

All in all, our results demonstrate that our system is able to effectively rewrite the code for a variety of types of neural networks. From a self-programming perspective, our self-programming AI system is capable of extending itself with additional functionalities by way of programming sub-models that can perform auxiliary tasks. From a more practical perspective, our neural network code generation approach can be seen as a kind of free-form AutoML solution that can be used to support model development in various task domains.

We propose and experimentally validate the first functional self-reprogramming AI system. In our experiments, we successfully implement the system and show it can freely modify its own neural network design. Over several iterations of the algorithm, reprogrammed models tend to achieve increasingly favorable performance on a simple supervised text-to-text code modification task.

Furthermore, the code-generating model used in our self-programming system can easily be adapted to program other models, in order to perform various other machine learning tasks.