La-MAML: Look-ahead Meta Learning for Continual Learning, ML Reproducibility Challenge 2020

The Continual Learning (CL) problem involves performing well on a sequence of tasks under limited compute. Current algorithms in the domain are either slow, offline or sensitive to hyper-parameters. La-MAML, an optimization-based meta-learning algorithm claims to be better than other replay-based, prior-based and meta-learning based approaches.

In continual learning, the main task is to learn an unknown number of sequentially arriving tasks $T_{1},\cdots,T_{K}$ in a supervised setting. In standard supervised learning, gradient descent does a good job of finding the minimum of one task. However, in continual learning, there is an issue of catastrophic forgetting, where a model forgets previous tasks that it has learned.

In particular, if a classifier trained on $T_{1}$ is then trained on the training data for task $T_{2}$, the catastrophic forgetting phenomenon demonstrates that its accuracy on $T_{1}$ will drop. In this way, the model unlearns what it has learned on task $T_{1}$.

Catastrophic forgetting results because the gradient of the model on the data for task $T_{1}$ may not necessarily be aligned with the data for task $T_{2}$. In the extreme case, where the gradients are perfectly unaligned, the model would be expected to forget everything it had previously learned.

Given infinite memory, this might be a trivial problem, as one could simply store all the training data for previous tasks. However, the challenge posed in continual learning is that the amount of memory given is limited. Various solutions have been developed to solve this problem, such as using a replay buffer to store previous samples, reserving a subset of neurons for each task and slowly growing the network as new tasks are seen, and storing previous gradient directions and making sure new updates are approximately aligned with them.

La-MAML is a meta-learning approach to continual learning that uses mechanisms from all three solutions. La-MAML was shown to perform better than Meta Experience Replay (MER) algorithm (previous state-of-the-art) on standard continual learning metrics, retained accuracy (RA) and backward-transfer and interference (BTI).

La-MAML consists of an inner level and an outer level of optimization. In the inner level, we start at some meta-parameters $\theta_{0}$ and try to minimize the task-specific loss function $l_{t}$ via $k$-steps of SGD, arriving at $\theta_{k}$. This can roughly be expressed as follows:

In the outer level, the goal is to find a good initialization $\theta_{0}$, doing so by measuring the loss of $\theta_{k}$ on the task-general loss function $L_{t}=\sum_{i=1}^{t}l_{i}$, roughly

where $U_{k}$ denotes $k$ steps of gradient descent on the loss function $L_{t}$.

In this section, we introduce a few terms and definitions that are used in this paper.

Retained Accuracy (RA): average accuracy of the model across all the tasks at the end of training

Backward Transfer and Interference (BTI): For each task, we calculate the difference in accuracy on that task right after that task was just learned and at the end of the entire training procedure. The BTI is the average of these values.

Now, we introduce some of the algorithms, both from previous papers and the current paper:

Experience Replay (ER): ER is an algorithm that stores a small replay buffer to store old data using reservoir sampling, ensuring that each sample has an equal probability of being in the buffer after it is seen. The buffer is then used to remember old tasks in future training.

La-ER: La-ER is an algorithm used in ablation study where parameter updates are identical to those used in experience replay (using the replay buffer), but the learning rate is varying throughout training and is updated in the same way that La-MAML is. By comparing La-ER to ER, we can see the benefits of tuning the learning rate according to the gradients during training.

C-MAML: C-MAML is the base algorithm used by La-MAML where the learning rate of both the inner and outer loops are fixed. The difference between C-MAML and La-MAML is that in La-MAML, the step size $\alpha$ is updated during each meta-iteration and is used for both inner and outer loops.

Sync: Sync is a hybrid between C-MAML and La-MAML where the learning rate for the outer loop is fixed, but the learning rate for the inner loop is updated as in La-MAML. THe difference between this and La-MAML is that in La-MAML, the outer loop and inner loop step sizes are both updated via gradient descent.

Further the experiments are conducted on two types of setups, distinguished based on how the incoming task datasets can be used by the model during training:

Single Pass: In this setup, once processed, data samples are not accessible anymore unless they were added to a replay memory.

Multiple Pass: In this setup, we can re-train the model using the already seen data over and over many times. This means that it is easier to get higher performance in this setup, compared to the Single Pass setup where data can only be used once.

We describe some terms associated with La-MAML:

Asynchronous Updates: It refers to the idea that the same inner learning rate (which gets modulated during each outer loop) is used for the outer loop meta-update also. This is in contrast to the synchronous updates of algorithms such as Meta-SGD, where the outer learning rate is fixed, evenwhen the inner learning rate is getting modulated.

Sample Efficient Objective: La-MAML’s objective function (as described in the original paper) is different from that of MER. La-MAML’s objective function tries to align the gradients of the current task with the average of old tasks, while MER’s objective function tries to align the gradient of the current task pairwise with gradients of all the old tasks. La-MAML makes use of the new research that their technique is enough to obtain good results. This makes MER much slower in training time.

Main Claims of the paper:

1. 1.

   La-MAML achieves better RA/BTI values than MER and other baselines, on real-world visual classification benchmarks.

2. 2.

   La-MAML is faster than MER.

3. 3.

   Learnable LRs and the asynchronous update of La-MAML are especially responsible for the increase in performance.

Additional Claims:

1. 1.

   The performance increase obtained for La-MAML increases as we move from toy datasets such as MNIST to real world datasets such as CIFAR and Imagenet.

2. 2.

   La-MAML is more robust to hyperparameter tuning than its variants.

We used the author’s code for most of the experiments. The hyperparameters were given in their paper. We only had to reoptimize hyperparameters for a few experiments.

All of our experiments have been run on free compute that we could get access to. For bug fixes and software developments involving less than an hour, we used Google Colab Notebooks (Tesla T4 GPU). For most of the experiments in our reproduced results, we used Kaggle Notebooks (Tesla P100 GPU). For a few experiments which take more than 9 hours (maximum limit of Kaggle Notebooks), we used the GPUs provided by our college (Tesla P4 GPU Cluster). And finally for experiments that require more than 20GBs of RAM such as the ones on the MNIST datasets, we used the 10 hour GPUs provided for free by Codeocean (Tesla K80 GPU).

Since most of the results were very close to that of the original paper, we only needed to reoptimize hyperparameters for a few experiments. We also verify the hyperparameter robustness patterns for the La-MAML variants in different settings than given in the original paper.

We used Weights and Biases to keep track of our experiments. It is very easy to incorporate and provides free cloud storage to make the results of our experiments publicly available.

Our results confirm the main claims of the paper. We were able to reproduce metrics close to 6% of the original paper.

From our results, we make the following observations:

• •

  La-MAML is a scalable, fast and robust algorithm outperforming previous baselines in the continual learning arena. It is scalable meaning that it achieves higher margins of improvement on real world datasets.

• •

  La-MAML has two mechanisms: (1) modulated per-parameter learning rates & (2) asynchronous meta-updates, which are individually responsible for its high performance. This is verified by the ablation studies on La-ER, Sync and C-MAML in sections 5.1.1 and 5.1.2. Thirdly, La-MAML’s sample efficient objective is responsible for its shorter running time.

• •

  Among the La-MAML variants, La-MAML achieves high accuracy while requiring less hyperparameter tuning.

In general, we notice a trend of having high variance in results, between the different seeds of the same experiment, in all the various continual learning algorithms. The ablation experiments show that the individual mechanisms introduced in the paper can be used individually in combination with other algorithms in related fields of meta-learning and multi-task learning.