Towards Automated Machine Learning Research

1. 1.

   This work does not make any assumptions about the inherent capabilities of LLMs to reason or have a deep understanding of ML topics. Even a random string generator can yield a meaningful hypothesis given unlimited attempts, akin to the infinite monkey theorem, which suggests that a monkey hitting keys at random on a typewriter for an infinite amount of time will almost surely type a given text, such as the complete works of Shakespeare. Our assumptions on the state of LLMs and ML are as follows.

   1. (a)

      LLMs are good enough at generating feasible outputs, thus narrowing down our search space meaningfully from a set of random outputs.

   2. (b)

      LLMs (and ML models in general) are good enough in pattern recognition. Therefore, training a reward model on performance would allow the model to identify common patterns among successful hypotheses.

   In short, we solely explore if LLMs can identify what would ”look like” a good hypothesis based on the patterns that it has seen in previous examples.

2. 2.

   In this work, we solely lay the foundation and do not claim that our automated research necessarily would yield state-of-the-art (SOTA) results in machine learning research. We explore the feasibility of operationalizing steps involved in ML research to a level where the current state of LLMs can generate feasible hypotheses efficiently.

3. 3.

   This work was conducted using the authors’ personal time and resources, limiting the scope to a small set of datasets and experiments. We hope that larger-scale experiments conducted at research labs interested in exploring this topic could further solidify this framework.

This work assumes an existing baseline solution and explores incremental innovations in its components, focusing on one component at a time. For example, in a neural network, the component of interest might be an activation function. We generate a set of viable alternatives (hypotheses) and evaluate their performance against baseline components.

The framework includes a generator for hypothesis creation, a validator to ensure basic validity, and an evaluator to measure success metrics such as validation loss. A reward model is trained to prioritize hypotheses that perform well compared to baselines, reducing computational burden while maintaining a high probability of discovering valuable new hypotheses.

While we manually verified and adjusted the validator and evaluator functions, the generation of hypotheses was not manually reviewed, highlighting the potential for fully automated research. This framework aims to accelerate innovation and make advanced machine learning techniques more accessible, with potential applications in various scientific discovery tasks.

The generator is the mechanism through which a feasible hypothesis $h_{i}$ is reached and sent for validation and evaluation. Here, it is a language model prompted by natural language, optionally followed by a reward model. In the activation function case study, these hypotheses take the form of novel activation functions. LLMs are trained on a comprehensive corpus of existing activation functions and related mathematical formulations, enabling them to propose viable functions.

We experiment with two types of base prompts. The first type encourages the model to discover incremental proposed blocks, to which we refer to as Incrementality Encouraging Prompting (IEP for short). The following is an example of such a prompt, used for generating activation function blocks.

”define a python class that inherits from pytorch nn.Module. I should be able to use it as an activation function. Make sure if it has any parameters, all of them are set to default values so I can initialize without specifying any parameters. Try to come up with something that combines characteristics of Sigmoid/Tanh, ReLU, and ELU.”

The second type of base prompt aims to reduce the likelihood of trivial incrementality. We refer to this as Novelty Encouraged Prompting (NEP) prompting. An example of such a prompt for activation function is as follows.

”define a python class that inherits from pytorch nn.Module. I should be able to use it as an activation function. Make sure if it has any params, all of them are set to default values so I can initialize without specifying any params. This function should not resemble common activation functions like ReLU, ELU, Sigmoid, or Tanh, and should explore unusual mathematical operations, combinations, or transformations. The expression can involve basic arithmetic, trigonometric functions, exponentials, or other non-linear operations, but avoid straightforward or commonly used forms in neural networks.”

The generator then involves a few wrappers around this base prompt, to request the implementation code for the proposed hypothesis in a parsable way.

Code 1 is an example of an activate function generated using the incrementality-encouraging base prompt. As prompted, the model clearly borrows characteristics from two commonly used activation functions of ReLU and Sigmoid.

Code 2 shows another activation function generated using the novelty-encouraged prompt. The model avoids directly using existing activation functions, adhering meaningfully to the instructions. While further exploration in prompt engineering is beyond this work’s scope, we believe it could significantly reduce LLMs’ inductive bias towards state-of-the-art methods, minimize borrowing from existing literature, and encourage the proposal of less explored functions.

Figure 2 visualizes the shape of the proposed activation functions.

For more examples, please refer to the Appendix.

Each proposed hypothesis is first passed through a validator function, denoted $v$. This function checks the validity of the hypothesis, ensuring it meets necessary criteria before further evaluation. For a proposed activation function $h_{i}$, the validator function $v(h_{i})$ returns a binary value indicating whether $h_{i}$ is a valid activation function (that is, $v(h_{i})\in\{0,1\}$).

For our case study on activation functions, we implement the validator manually and as a unittest¹¹1Fully automating this function is very feasible, but out of the scope of this effort.. The validator checks if $h_{i}$ inherits from nn.Module, has the required initialization and forward functions, all its parameters have default values, and can pass a few basic test cases. We provide code snippets of this validator in the Appendix section Code 9.

Valid hypotheses are fully evaluated using an evaluation function, denoted $e$. The goal of this function is to replace a baseline component $b_{j}$ with an alternative hypothesis $h_{i}$ and measure the performance of the model in the task(s) of interest. For our case, this translates to integrating the hypothesis into a machine learning model and measuring its performance, specifically its loss on the validation split of the dataset (val-loss). To streamline the process, we perform a single iteration of forward and backward passes to obtain a preliminary loss value, rather than conducting a full training and evaluation cycle. Formally, for a model $m$ and an activation function $h_{i}$, the val loss is calculated as $e(m(h_{i}))$.

In our experiments, we hard-coded the architecture to a 2-layer MLP, and used cross-entropy and MSE loss for a set of classification and regression tasks. We also define the problem as solving the classification and regression instances in a one-pass learning setup (only one epoch) to reduce the computation required for each $h_{i}$ and enable more extensive exploration across a larger set of hypotheses.

Passing the set of hypotheses $\mathcal{H}$ to the evaluator function results in the collection of pairs of generated hypotheses (activation functions) and their corresponding validation losses, in the form of $(h_{i},l(h_{i}))$. We define the reward as the win rate of the proposed hypothesis $h_{i}$ over the baselines. Specifically, we measure two metrics:

Baseline Win Rate (B-WR): This metric measures the percentage of times $h_{i}$ outperforms any given baseline $b_{j}$ across different tasks and over different runs. Formally, it measures $P(l(h_{i})<l(b_{j}))$.

Baseline State-of-the-Art Win Rate (BSOTA-WR): This metric measures the win rate of the proposed hypothesis over the best runs of the entire baseline set $\mathcal{B}$ in each task / dataset. Formally, it is defined as $P(l(h_{i})<\min(l(b_{j}))|_{j=1,\ldots,|\mathcal{B}|})$. Please note that the minimum operation is done on the average loss of different runs for each baseline, thus the best baseline run still contains a distribution of losses (resulting from several runs / random initialization), allowing for calculating a probabilistic win rate.

The reward model is then trained as a ranking model mapping the content of the hypothesis $h_{i}$ to its downstream performance (i.e. loss). In other words, this model is aimed to looking at the content of a proposed component (i.e. code of an activation function), and be able to predict how well it is likely to perform in terms of winning over the baseline set. Intuitively, the reward model learns patterns in the content of the proposed activation functions, leading to better performance.

In the initial round of hypothesis generation, every hypothesis is evaluated using a brute-force approach, where each one is passed through the validator and evaluator in a fully exploratory iteration. This process allows for comprehensive data collection, yielding the success metrics B-WR and BSOTA-WR for each validated and evaluated hypothesis. We employ three LLMs to generate a dataset of 2000 validated and evaluated hypotheses for each component type.

Once this initial data is collected, the second iteration leverages a trained reward model to streamline the process. The reward model is used to prune the newly generated hypotheses, filtering them to select the top-$k$ candidates based on their predicted performance. These top-$k$ hypotheses, expected to be the most promising, are the only ones that proceed to the full evaluation phase, which involves more intensive computational resources.

Our goal is to generate viable and high-performing proposed components, efficiently. In the following, we provide details on how we measure success in these aspects.

As mentioned in section Quantitative Results and Analysis we present a scatter plot visualization to compare the two success metrics for the hypotheses generated across different block types. Figure 10 illustrates the scatter plots for the pre-processor, activation, and regularization block types, respectively.

Here we provide efficiency curves for the preprocessor and regularize blocks in figure 11 and 13 respectively. It can be observed that similar to activation functions, the reward ranking model trained on the preprocessor blocks can effectively speed up the discovery of the most promising proposed components. However, when it comes to the regularizers, the trained reward models, especially the ones trained on the Gemini-pro dataset, fail to generalize to other datasets. We also provide the metrics for the rewards models in tables 3 and 4 respectively.

The Sigmoid-ELU (SigELU): An Activation Function is a hybrid activation function that combines the Sigmoid function for non-negative inputs and the Exponential Linear Unit (ELU) function for negative inputs.

The activation function is defined as:

where $\text{sigmoid}(x)=\frac{1}{1+\exp(-x)}$ and $\alpha$ is a hyperparameter that controls the scaling for negative inputs.

The name Sigmoid-ELU Activation (SigELU) reflects the combination of the Sigmoid function for non-negative inputs and the ELU function for negative inputs:

• •

  Sigmoid for Non-Negative Inputs: The Sigmoid function is well-known for its smooth, bounded output between 0 and 1. It is particularly useful for squashing input values to a manageable range, which can help stabilize the training process and make the model’s output more interpretable in certain contexts.

• •

  ELU for Negative Inputs: ELU (Exponential Linear Unit) is effective for handling negative inputs. It produces non-zero outputs for negative values, which helps to alleviate the vanishing gradient problem often encountered with ReLU in deep networks. The parameter $\alpha$ allows for controlling the steepness of the negative part, adding flexibility to the function.

• •

  Smooth Transition: The activation function ensures a smooth transition between the positive and negative parts, which can contribute to better gradient flow and more stable training.

• •

  Sigmoid Function for Non-Negative Inputs: The Sigmoid function, defined as $\text{sigmoid}(x)=\frac{1}{1+e^{-x}}$, is a smooth and differentiable function for all real numbers. Its derivative is given by:

  $$\frac{d}{dx}\text{sigmoid}(x)=\text{sigmoid}(x)\cdot(1-\text{sigmoid}(x))$$

• •

  ELU-like Function for Negative Inputs: The ELU-like function defined as $\alpha(\exp(x)-1)$ is also smooth and differentiable for all real numbers. Its derivative is:

  $$\frac{d}{dx}\left(\alpha(\exp(x)-1)\right)=\alpha\exp(x)$$

• •

  Combination of Both Functions: The combination of these functions using a piecewise definition ensures that the function is differentiable. Since both components are differentiable, and the transition between them occurs at $x=0$, the overall function is differentiable at $x=0$.

• •

  Continuity at $x=0$: At $x=0$, both functions yield the same value if we choose $\alpha=1$:

  $$\text{sigmoid}(0)=\frac{1}{1+e^{0}}=\frac{1}{2}$$

  $$\alpha(\exp(0)-1)=\alpha(1-1)=0$$

  Therefore, if $\alpha=\frac{1}{2}$, the function value is continuous at $x=0$.

• •

  Smooth Transition: The derivative at $x=0$ for both functions should also match for smooth transition:

  $$\frac{d}{dx}\text{sigmoid}(0)=\text{sigmoid}(0)\cdot(1-\text{sigmoid}(0))=\frac{1}{2}\cdot\frac{1}{2}=\frac{1}{4}$$

  $$\frac{d}{dx}(\alpha(\exp(0)-1))=\alpha\exp(0)=\alpha=\frac{1}{2}$$

  Therefore, the function transitions smoothly if we ensure the parameters are set appropriately.

Given these properties, the Sigmoid-ELU activation function is fully differentiable and suitable for use in neural networks.

The ScaledSinusoidalDecay activation function is defined as follows:

Given an input $x$, the output $y$ of the activation function is calculated as:

where:

• •

  scale is a parameter that controls the amplitude of the sinusoidal component.

• •

  $\sin(x)$ introduces a periodic, oscillatory behavior to the activation function.

• •

  $\exp(-|x|)$ is an exponential decay function that diminishes the output as the magnitude of the input increases.

• •

  shift is a parameter that shifts the output, providing additional flexibility in the function’s range.

The ScaledSinusoidalDecay activation function offers several advantages that make it a strong candidate for deep learning applications:

• •

  Controlled Non-linearity: The sine component introduces periodic non-linearity, which can be beneficial for learning complex patterns that are not purely linear. This is particularly useful in applications where the relationship between input features and the output is oscillatory or involves repeated cycles.

• •

  Attenuation of Large Inputs: The exponential decay term $\exp(-|x|)$ serves to attenuate the influence of large input values, preventing them from dominating the output. This can lead to better stability during training, especially in scenarios where the input data contains large outliers.

• •

  Parameter Flexibility: The inclusion of the scale and shift parameters allows for fine-tuning the function’s behavior to suit specific tasks. For instance, adjusting the scale can amplify or reduce the overall impact of the activation, while the shift can move the activation range to better align with the desired output.

• •

  Smooth Gradients: The combination of sine and exponential functions ensures that the gradients of the ScaledSinusoidalDecay activation function are smooth and continuous. This is advantageous for optimization algorithms like gradient descent, as it helps in avoiding issues related to vanishing or exploding gradients.

• •

  Regularization Effect: The exponential decay can act as a regularizer by suppressing the influence of extreme values. This can lead to more robust models that generalize better to unseen data, particularly in deep networks where overfitting is a concern.

The ScaledSinusoidalDecay activation function is a versatile and powerful tool in the design of neural networks. By combining sinusoidal non-linearity with exponential decay, and allowing for adjustable scaling and shifting, this function offers a unique blend of flexibility and control. It is particularly well-suited for tasks that require the learning of complex, cyclical patterns, or where the attenuation of large inputs is beneficial. Its smooth gradients and regularization properties further enhance its utility, making it a strong candidate for a wide range of deep learning applications.

The ‘NormalizedPCA‘ function is designed to preprocess data by first normalizing the features and then applying PCA for dimensionality reduction. This two-step process ensures that the data is appropriately scaled and transformed, allowing for more effective analysis and modeling.

Given a dataset $\mathbf{X}\in\mathbb{R}^{n\times d}$, where $n$ is the number of samples and $d$ is the number of features, the ‘NormalizedPCA‘ function performs the following operations:

1. **Feature Normalization:** The feature normalization step involves standardizing the features to have zero mean and unit variance. This is achieved using the StandardScaler:

where $\tilde{x}_{ij}$ is the normalized feature value, $x_{ij}$ is the original feature value, $\mu_{j}$ is the mean of the $j$-th feature, and $\sigma_{j}$ is the standard deviation of the $j$-th feature.

2. **Dimensionality Reduction with PCA:** After normalization, PCA is applied to reduce the dimensionality while preserving the maximum variance. PCA transforms the data $\mathbf{X}_{\text{scaled}}$ to a lower-dimensional space:

where $\mathbf{W}_{\text{pca}}$ contains the principal components (eigenvectors) corresponding to the largest eigenvalues of the covariance matrix of $\mathbf{X}_{\text{scaled}}$.

1. **Effective Scaling:** Normalizing features ensures that all features contribute equally to the PCA, avoiding bias towards features with larger magnitudes. This scaling step is crucial because PCA is sensitive to the scale of the input features.

2. **Improved Dimensionality Reduction:** By applying PCA after normalization, the function effectively reduces the dimensionality while retaining the most significant variance. This results in a lower-dimensional representation that captures the essential structure of the data.

3. **Enhanced Model Performance:** Proper normalization and dimensionality reduction improve the performance of machine learning models by reducing overfitting and speeding up convergence. Normalized data allows PCA to perform a more accurate reduction, leading to better generalization.

4. **Consistency and Interpretation:** The combination of scaling and PCA provides a consistent and interpretable transformation of the data. Normalized features ensure that PCA components represent the true variance, making the results more meaningful and actionable.

Let $\mathbf{X}\in\mathbb{R}^{n\times d}$ be the input data matrix. The preprocessing steps are as follows:

1. 1.

   **Normalize the Data:**

   $$\mathbf{X}_{\text{scaled}}=\text{StandardScaler}(\mathbf{X})$$

   where each feature is scaled to have zero mean and unit variance.

2. 2.

   **Apply PCA:**

   $$\mathbf{X}_{\text{pca}}=\text{PCA}(\mathbf{X}_{\text{scaled}})$$

   where PCA reduces the dimensionality based on the specified number of components or variance threshold.

In summary, the ‘NormalizedPCA‘ function provides a comprehensive preprocessing solution by combining scaling and PCA. This approach ensures that the data are properly prepared for subsequent analysis, improving the effectiveness of dimensionality reduction and enhancing overall model performance.

The SineSquaredDecay transformation is applied to each feature in the input data and can be formalized by the following equations:

Given an input feature matrix $X$, the transformation for each feature $x_{i}$ in the training set train_X and validation set val_X is defined as:

where:

• •

  $\sin^{2}(x_{i})$ applies a non-linear, periodic transformation to the input feature.

• •

  $\exp(-|x_{i}|)$ introduces an exponential decay, which diminishes the influence of large feature values, ensuring that no single feature dominates the input space.

• •

  $\sigma$ represents the noise_scale parameter, which controls the magnitude of the added Gaussian noise.

• •

  $\epsilon_{i}$ is a Gaussian noise term drawn from a normal distribution $\epsilon_{i}\sim\mathcal{N}(0,1)$, added to the transformed features to enhance feature diversity and regularization.

The SineSquaredDecay transformation offers several advantages for preprocessing, particularly in scenarios where non-linear relationships and feature regularization are critical:

• •

  Capturing Complex Patterns: The use of the sine function, squared, introduces non-linear and periodic behavior into the features, which can help capture complex underlying patterns in the data. This is particularly useful in situations where the relationship between features and the target variable is not purely linear.

• •

  Feature Scaling and Regularization: The exponential decay term $\exp(-|x_{i}|)$ ensures that the transformed features do not become excessively large, which can help in preventing certain features from overpowering others. This acts as an inherent regularization mechanism, making the feature set more balanced.

• •

  Noise Augmentation: The addition of Gaussian noise controlled by the noise_scale parameter serves as a regularizer by slightly perturbing the input data. This prevents the model from overfitting to specific patterns in the training set, thereby improving generalization to unseen data.

• •

  Diverse Feature Representations: The combined effect of non-linear transformation and noise addition results in a rich and diverse feature set. This diversity can be particularly advantageous in ensemble models or in scenarios where the model benefits from a wide variety of input features.

The SineSquaredDecay transformation is a powerful tool for preprocessing in machine learning pipelines. Its ability to introduce complex non-linearities, combined with an effective regularization mechanism through noise, makes it a robust choice for enhancing model performance. By using this transformation, practitioners can create a feature space that is both rich in diversity and resilient to overfitting, ultimately leading to more effective and generalizable models.

The DropWeightL2 regularization function integrates two distinct regularization techniques: dropout-like regularization applied directly to weights and L2 weight decay. By applying these methods simultaneously, DropWeightL2 seeks to improve model generalization and stability during training.

The DropWeightL2 function is defined as follows:

1. **Enhanced Model Robustness:** - **Dropout-Like Regularization:** Although dropout is typically applied to activations, applying a similar dropout-like effect to weights introduces noise into the weight parameters. This encourages the network to be less reliant on specific weights, promoting robustness, and reducing the risk of overfitting. - **Effect:** This technique helps in regularizing the model by preventing it from fitting too closely to the training data and improving generalization.

2. ** Effective weight decline: ** - ** L2 penalty: ** The term L2 weight penalty discourages large weights by adding a quadratic penalty to the loss function. This helps in controlling the complexity of the model and reducing overfitting. - **Effect:** Regularizing weights through L2 penalty improves model performance by constraining weight magnitudes, thereby simplifying the model and improving its generalization ability.

3. **Combination of Techniques:** - **Dual Regularization:** Combining dropout-like behavior with L2 regularization leverages the strengths of both methods. Dropout-like regularization introduces stochasticity into the weights, while L2 regularization ensures that weight magnitudes are kept in check. - **Effect:** This combination can lead to better generalization by balancing the benefits of both techniques.

Let $\mathbf{W}\in\mathbb{R}^{d\times k}$ represent the weight matrix of a layer, where $d$ is the number of input features and $k$ is the number of output features. The regularization loss introduced by the DropWeightL2 function is formulated as:

where:

• •

  $\lambda$ is the weight penalty coefficient (L2 regularization strength).

• •

  $w_{ij}$ represents the weight value on the $i$ -th row and $j$ -th column.

• •

  $\tilde{w}_{ij}$ represents the weight value after applying a dropout-like mechanism. Mathematically, it can be modeled as:

  $$\tilde{w}_{ij}=\begin{cases}w_{ij}&\text{with probability }(1-p)\\ 0&\text{with probability }p\end{cases}$$

  where $p$ is the dropout rate.

The total loss of regularization is accumulated in all layers of the model and the resulting value is added to the primary loss function during training.

where:

• •

  $\lambda$ is the weight penalty coefficient (L2 regularization strength).

• •

  $w_{ij}$ represents the weight value on the $i$ -th row and $j$ -th column.

• •

  $\text{Dropout}(w_{ij})$ is the dropout-like effect applied to the weight $w_{ij}$, introducing noise during regularization.

The total loss of regularization is accumulated in all layers of the model and the resulting value is added to the primary loss function during training.

The DropWeightL2 regularization function offers a unique approach by integrating dropout-like regularization with L2 weight penalty. This dual regularization strategy improves the robustness of the model, prevents overfitting, and improves generalization. By applying both methods simultaneously, ‘DropWeightL2‘ provides a comprehensive regularization solution that balances weight control with stochastic noise.

The SineDecay Regularizer is applied to the parameters of a neural network model. The regularization loss, reg_loss, is computed by summing the sine-transformed and exponentially decayed values of the model’s parameters. Formally, the regularization loss is defined as follows:

where:

• •

  $\theta_{ij}$ represents the $j$-th parameter of the $i$-th layer in the model.

• •

  $N$ is the number of layers in the model.

• •

  $M_{i}$ is the number of parameters in the $i$-th layer.

• •

  scale is a hyperparameter that controls the amplitude of the sine transformation.

• •

  decay is a hyperparameter that determines the rate of exponential decay, attenuating the influence of large parameters.

The SineDecay Regularizer offers several advantages that make it a valuable tool for enhancing the performance and robustness of neural network models:

• •

  Encouraging Smoothness and Periodicity: The sine transformation encourages the parameters to adopt smoother, periodic distributions. This can be particularly beneficial for models dealing with data that has inherent periodicity or cyclical patterns.

• •

  Attenuation of Large Parameters: The exponential decay component reduces the impact of large parameter values on the regularization loss. This helps in preventing overfitting by discouraging the development of overly large weights, which can dominate the model’s output.

• •

  Parameter Diversity: By combining sine and exponential decay, the regularizer introduces diversity in the parameter values, which can lead to a more robust and generalizable model. This is especially useful in complex models where standard regularizers like L1 or L2 might not be sufficient.

• •

  Flexibility with Hyperparameters: The scale and decay hyperparameters offer flexibility in tuning the regularizer’s effect. This allows practitioners to adjust the regularization strength to suit the specific needs of their model and dataset.

The following is the implementation of the SineDecay Regularizer in Python using PyTorch:

The SineDecay Regularizer is a powerful and flexible tool for regularizing neural network models. Using the periodic nature of the sine function and the attenuating effect of exponential decay, this regularizer provides a unique approach to controlling model complexity and improving generalization. Its ability to encourage smooth, diverse parameter values while mitigating the risk of overfitting makes it a valuable addition to the regularization techniques available in deep learning.