Projected Subnetworks Scale Adaptation

The pre-trained model was trained on WebImageText (WIT) (Radford et al., 2021), a dataset specially gathered with 400M (image, text) pairs, in contrast to 100K in MSCOCO (Lin et al., 2014). From the pre-trained initialization, we train on the first task of MSCOCO for 50 epochs, until the accuracy is high on MSCOCO and low for all the unseen datasets that the pre-trained model performed well on. We reuse 10 datasets used in CLIP’s evaluation including ImageNet (Deng et al., 2009), CIFAR100 (Krizhevsky, 2009), STL-10 (Coates et al., 2011), Caltech-101 (Fei-Fei et al., 2004), Stanford Cars (Krause et al., 2013), Oxford Flowers 102 (Nilsback & Zisserman, 2008), GTSRB (Stallkamp et al., 2012), Food-101 (Bossard et al., 2014), EuroSAT (Helber et al., 2017), and FGVC Aircraft (Maji et al., 2013). While MSCOCO contains (image, caption) pairs, the other datasets contain (image, label) pairs. Hence, with the prompt templates provided by Radford et al. (2021) for each dataset (e.g. “a photo of a CLASS”, we can convert labels to captions.

While the pre-trained initialization performs well across the 11 datasets, the MSCOCO-finetuned model loses many transferable representations from WIT, such that the average zero/few-shot accuracy is low. MSCOCO is on a much smaller scale than WIT, in terms of number of images (100K vs 400M), range of classes (e.g. ImageNet labels such as stingray, tarantula, mousetrap are not found in MSCOCO), and less diverse images (e.g. MSCOCO contains natural and day-to-day scenes, while WIT contains natural scenes, sketches, blurry images, low-res images, texts, websites). The lifelong goal is to gradually increase the average zero/few-shot accuracy across all tasks.

Given the range of datasets, we evaluate zero/few-shot transferability between them, such that learning one dataset will yield high zero/few-shot performance on another dataset. First, we fine-tuned CLIP on each dataset from a MSCOCO-finetuned initialization.

In Table 0(a), we measured the cosine distance between each pair of models fine-tuned on two different datasets. This infers the spatial distance in the parameter space, which parameters are closer to each other, and which parameters require further gradient updates from the initialization. We are able to identify 3 sets of datasets, grouped by distance: (i) $\leq 0.1$, (ii) $0.1-0.3$, (iii) $\geq 0.3$. In Table 0(b), we evaluate the functional distance of each fine-tuned model by computing the zero/few-shot performance per model on each dataset.

Based on the relational analysis between seen and unseen tasks, given the distance between models fine-tuned on each dataset (relative cosine distance in the parameter space), and given the zero/few-shot performance of each fine-tuned model, some clear task groupings can be identified. The order for the seen (unseen) tasks is: MSCOCO (FGVCAircraft, EuroSAT, Food101) → ImageNet (STL10, StanfordCars) → Caltech101 (CIFAR100, GTSRB, Flowers102).

Meta learners are trained to develop zero/few-shot capabilities. Given meta parameters, some meta learners map input distributions to base parameters. The base parameters are task-specific representations, and dynamically generated with respect to the task. In the case of gradient-based meta learners, the meta parameters and base parameters reside in the same parameter space. A gradient is computed with respect to the new task’s input distribution and applied to the meta parameters, and this returns the base parameters. As a result, we can project the task-specific representations and subnetworks within the meta parameters to the base parameter space, and use a gradient-based meta learner to retain the same model architecture and output space as training a model without a meta learning training procedure. We train CLIP with MAML’s (Finn et al., 2017a) training procedure for 10,000 epochs, meta learning rate and base learning rate of 0.0005. To retain the same scale of model and data, we use the same CLIP architecture and capacity, and retain the same dataset size by training MAML(CLIP) on Split-MSCOCO (Chiaro et al., 2020) (which organizes labelled MSCOCO into 5 tasks: transport, animals, sports, food, interior). We train MAML(CLIP) until it attains similar zero/few-shot performance as CLIP (Table 2).

When updating the meta learning parameters $\theta$ on a new task, $\theta$ may drift by a certain distance to $\theta^{{}^{\prime}}$. Given that the base parameters $\{\theta_{\textnormal{base},t}\}^{t\in T}$ and $\theta$ reside in the same parameter space, we evaluate how far $\theta$ can drift while being able to compute the same (or within bounded error $\varepsilon$) base parameters $\texttt{dist}(\theta_{\textnormal{base},t},\theta^{{}^{\prime}}_{\textnormal{base},t})\leq\varepsilon$.

From Table 0(d), we first measure the Euclidean distance between the original meta parameters and their MAML-computed base parameters. This informs the approximate radius of the parameter subspace. In Table 0(c), we test whether drifting the meta parameter to each task’s base parameter $\theta=\theta_{\textnormal{base},t}$ will still be able to compute the same task base parameters. Relative to the subspace radius, we find that the base parameters can indeed be re-located if the meta parameter is drifted to the end-points of the subspace. Given that the base parameters can be located if the drift is within the bounds of the subspace, we next evaluate whether the base parameters can be located if the drift exceeds the bounds of the subspace. In Table 0(e), we evaluate $S=1000$ random parameters, and interpolate between this random parameter $\theta_{\textnormal{rand},s}$ and the meta parameter $\theta$ to return an interpolated meta parameter $\theta_{\textnormal{int}}=(1-r)\theta+r\theta_{\textnormal{rand},s}$ with interpolation coefficient $r$. We find that for varying interpolation coefficients (and thus varying Euclidean distances), once the Euclidean distance increases substantially, then the computed base parameters drift in a similar fashion from the original base parameters. As a result, we are interested in maximizing the radius of the parameter subspace in which the meta parameter can drift, while returning the same base parameters (within bounded error).

Model. We evaluate with CLIP (Radford et al., 2021), the standard vision-language model, specifically the pre-trained initialization of the ResNet-50 variant. For training/fine-tuning on a new task, we retain the Adam optimizer, decoupled weight decay regularization, temperature clipping, and a batch size of 32. From the pre-trained initialization, we train CLIP on MSCOCO for 50 epochs until both loss convergence and verification that zero/few-shot performance across the datasets is weakened. We fine-tune for 10 epochs and also validate loss convergence.

Adaptation baselines. Fine-tuning (Single Task Learning) is the default baseline in the online/continual learning setting, where the model sequentially learns each incoming task without any forgetting mitigation. Joint Training (Multi Task Learning) is a baseline where the model can train on all future tasks at once. We do not include the base task (MSCOCO), and evaluate when there are at least 2 tasks (Task 3).

We evaluate against 5 baseline adaptation methods, including 3 general-purpose continual learning strategies, and 2 large-model-specific adaptation strategies (that have also been evaluated on CLIP). Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) uses weight regularization to retain weights from previous tasks. The regularization strength for weight penalty $\lambda$ is $1,000$. Gradient Projection Memory (GPM) (Saha et al., 2021) learns new tasks by performing gradient steps in the orthogonal direction to the gradient subspaces that are important to the past tasks. We use a 0.01 learning rate. BatchEnsemble (Wen et al., 2020) uses a base network (slow weights) and stores separate parameters (fast weights) to compute the parameters per ensemble, thus $N$ ensembles do not require $N$ sets of parameters. Each ensemble member is responsible for one task. We retain -0.5 random sign init for fast weights and 0.5 fast weights learning rate multiplier.

CLIP-Adapter (Gao et al., 2021) fine-tunes with feature adapters, specifically an additional bottleneck layer to learn new features and perform residual-style feature blending with the original pre-trained features. In-line with the implementation for CLIP, we fine tune the visual adapter. PAINT (Ilharco et al., 2022) is another vision-language model adaptation technique. It is a patching method that interpolates between parameters before fine-tuning and parameters after fine-tuning on a task to be patched. We implement sequential patching, where we iteratively repeat the patching procedure on each new task, and pick the mixing coefficient that optimizes average accuracy on the held-out validation sets from the supported and patching tasks. All baselines are trained with the CLIP model.

Zero-shot. Unlike traditional visual systems trained on a fixed set of discrete labels, Radford et al. (2021) shared a new paradigm in learning to align images with raw texts in an open-vocabulary setting. For zero-shot classification, all class labels are first converted to sentences using their prompt templates, and the model computes their text embeddings. To classify an image, the model computes the image embedding, and then an image-text similarity score (cosine similarity) is computed between the image embedding and each class’ text embeddings to find strongest alignment. The calculated similarity of these embeddings per class, scaled by a temperature parameter, is then normalized into a probability distribution via softmax. The class having the highest similarity with the image is the predicted one. We perform zero-shot evaluation with the altered CLIP-Adapter model, and perform zero-shot evaluation with the task-indexed BatchEnsemble model. For all other methods, albeit differences in the training procedure, the model architecture and parameters are available for directly applying this zero-shot evaluation procedure.

Few-shot. For a given task, an N-way-K-shot support set (N classes, K samples per class) is provided to assist the method to make inferences. Then evaluation is performed on the query set. For meta learners, the meta learner computes the base parameters with respect to the support set. Specific to gradient-based meta learners, including MAML and SNP(++), we compute the gradient of the support set with respect to model parameters, update the model parameters, then evaluate on the query set.

For standard CLIP and the other methods, we use nearest-neighbor classification. We first compute the mean image features per class in the support set, then measure the (cosine) distance between them and the image features for a given query set image. Based on the nearest class mean, the closest class’ mean image features is the predicted class.

We tabulate our proposed method in comparison to baselines in Table 4, and proposed method in comparison to different configurations in Table 5.

Transferability between tasks plays an important role. From task 1 to 3, we find that positive backward transfer exists across all baselines, and that some datasets have improved zero/few-shot performance with task shift. Furthermore, we find that the gradual removal of subnetworks with SNP++ may worsen performance. The removal of subnetworks is motivated by alleviating interference between task-specific representations between two tasks; in this case, it appears that the attempt at removal overwrites the transferable representations. Further sub-procedures that identify the optimal subnetworks to remove based on a transfer-interference trade-off can improve the utility of subnetwork removal, especially in a setting with many tasks.

Adaptation techniques specialized for large models (CLIP in particular) outperform general-purpose continual learning methods. For large models, regularization-based methods that do not require task indexing or separate context vectors can perform competitively to non-regularization-based methods.

Our proposed adaptation method, SNP and SNP++, outperforms existing baselines. It can consistently retain low negative backward transfer, fulfilling the objective of minimizing loss of zero/few-shot performance with task shift. It performs comparably in maximizing positive backward transfer. In terms of balancing between positive and negative backward transfer, SNP and SNP++ strikes the optimal balance, attaining the highest average accuracy.

We find that our proposed method works better with the memory buffer that regularizes the base parameter drift. Though we are not storing trajectories or large replay buffers (only storing one support set instance), pure-regularization SNP can also perform in a stable fashion. Different hyperparameters of $\beta_{\textnormal{base}}$ and $\beta_{\textnormal{meta}}$ tend to retain similar performances, and no major loss in accuracy is observed. We do note that setting the $\beta_{\textnormal{meta}}$ too low can worsen performance, particularly in comparison to non-SNP baselines. This may occur where the drift of the meta parameters is under-regulated, and regulating base parameter drift is insufficient and acts as a second-order regularizer of the meta parameter drift.

We find that combining subnetworks and interpolating between them underperforms SNP and additive SNP++. Subnetwork addition/removal would manipulate the number of base parameters, but would result in first-order interpolation between the unchanged meta parameters and new meta parameters with the modified subnetwork set. Thus, interpolating between subnetworks results in second-order interpolation, and the error with respect to each task accumulates when the meta parameters interpolate. Given a very large number of tasks and lower model capacity, second-order interpolation offers an efficient subnetwork manipulation.

Subnetwork projection improves the maintainance of zero/few-shot capabilities in an online setting. Moreover, subnetwork projection supports the manipulation of task-specific representations for many more applications.