Diverse Distributions of Self-Supervised Tasks
for Meta-Learning in NLP

In this work, we focus on Model Agnostic Meta-Learning (MAML) Finn et al. (2017), which is an optimization-based meta-learning method. To efficiently adapt to a task training data, MAML jointly optimizes the initial point of a neural network model and a gradient-descent based optimizer. This is framed as a bi-level optimization consisting of an inner loop for task-specific learning and outer loop for fast adaptation across tasks:

where $\theta$ are the parameters of the model, $\alpha$ is the (learnable) inner loop learning rate, $\Theta:=\{\theta,\alpha\}$, $\mathcal{L}_{i}$ is the loss for task $T_{i}$, $(\mathcal{D}^{tr},\mathcal{D}^{val})\sim T_{i}$ are support and validation data for the task $T_{i}$, and $\beta$ is the outer loop learning rate which is a hyper-parameter. Typically, multiple steps of gradient descent are performed in the inner loop. Training such methods proceeds in an episodic framework Vinyals et al. (2016), where in each episode a mini-batch of tasks are sampled along with their support and validation sets, and the model parameters are optimized as above.

Meta-learning assumes access to a distribution $\mathcal{P(T)}$ over tasks. The goal is to utilize tasks $T_{i}\sim\mathcal{P(T)}$ sampled from this distribution to train a learning procedure that generalizes to unseen tasks $T^{\prime}\sim\mathcal{P(T)}$ from the distribution. Supervised meta-learning often utilizes a fixed task dataset to create $\mathcal{P(T)}$ by sub-sampling from all class labels Vinyals et al. (2016). Bansal et al. (2020b) sought to provide an unsupervised approach that proposes tasks from unlabelled data. The resulting Subset Masked Language Modeling Tasks (SMLMT) approach proposes self-supervised tasks to enable meta-learning and improves few-shot learning across a diverse set of classification tasks.

Sampling an $N$-way task from SMLMT requires first sampling a size-$N$ subset of the vocabulary, which are subsequently mapped to consecutive integer ids and serve as labels for the task. Then to sample examples for each label, sentences containing that word are sampled and the occurrences of the word are masked out. Note that a task in SMLMT is a sentence classification task where each input sentence consists of exactly one word type that is masked throughout the sentence and the label for the sentence is the underlying word type that was masked. This enables sampling combinatorially many classification tasks for meta-learning.

The methods discussed so far use a static task distribution for learning with tasks sampled i.i.d from this distribution for training. Curriculum learning Bengio et al. (2009); Graves et al. (2017) instead posits that choosing the order in which instances are presented, with gradually increasing complexity, can enable faster learning and better generalization. We explore whether a curriculum in task sampling is beneficial for meta-learning by proposing a method to sample increasingly difficult tasks during training. To enable this we need a method to propose difficult tasks based on the current state of the model during training.

Since words act as labels in SMLMT, words that are closer in the representational space of the neural model will be more difficult to distinguish, leading to more difficult tasks. On the other hand, nearest-neighbors can be too difficult to induce effective learning for a model. This is related to findings in negative sampling in metric learning literature Schroff et al. (2015); Suh et al. (2019) where using “too hard” negatives typically hurts performance.

To alleviate this problem, we cluster representations computed from the model and uniformly sample words within the same cluster to create difficult but not impossible tasks (similar to the "static" clustering approach). Secondly, we adopt an easy-to-hard curriculum by controlling the ratio between the harder tasks from the dynamic distribution $\mathcal{D}_{t}$ and the easier ones from the static distribution $\mathcal{S}$, consisting of tasks sampled i.i.d from uniform random word sampling or fixed word-clustering. At step $t$, let $\lambda_{t}$ be the probability of sampling a task from $\mathcal{D}_{t}$ and $1-\lambda_{t}$ from $\mathcal{S}$. Then the dynamic curriculum is defined by sampling tasks from the following mixture distribution with $\lambda_{t}$ linearly annealed over the training epochs from $0$ to $1$:

To construct $\mathcal{D}_{t}$, we consider the following word (i.e. label) representation for clustering, obtained by the average representation under the model of the masked sentences corresponding to a word:

where $S(w_{i})$ is the set of all sentences containing the word $w_{i}$ with the word $w_{i}$ masked out (as defined in SMLMT), $f_{\theta_{t}}(.)$ is the representation from the neural model for instance $x$ that is fed into the softmax classification layer, and $\theta_{t}$ are the model parameters at step $t$.

To make the computation of $\hat{w}^{(t)}_{i}$ tractable, we first approximate the quantity by the expectation over a subset of $S(w_{i})$. Moreover, since computing the representations $\{\hat{w}^{(t)}_{i}\}$ for all vocabulary words and clustering at every step $t$ of the training will be computationally infeasible, we consider doing this after $m$ steps of meta-training. This also allows the model to train on the current distribution for sometime before abruptly changing the distribution. Finally, while the model is being updated between time step $t$ and $t+m$, we use the model snapshot at $t$ to create the word clusters asynchronously for the model at $t+m$, which allows the task generation to run in parallel to the model training.

SMLMT uses a data-augmentation strategy to automatically assign labels to unlabelled sentences by consistently masking out the same word type in a set of sentences. The masked out word then serves as the label for the sentences. This cloze-style approach to creating tasks was inspired by the success of masked language modeling Devlin et al. (2018) in learning useful representations. While this leads to significant improvements in sentence-level classification on a range of real downstream tasks Bansal et al. (2020b), it is unclear whether a word masking approach is the most efficient to learning useful sentence representations. To probe this question further, we explore an alternative to SMLMT that directly assigns semantic labels to sentences without any augmentation.

Specifically, we consider pre-trained sentence representations for proposing tasks, which have been proven useful for improving semi-supervised learning Du et al. (2020). We use a pre-trained sentence embedding model Du et al. (2020); Wenzek et al. (2020) to embed all sentences in a corpus and cluster them. To propose an $N$-way task, we first randomly sample $N$ cluster-ids and remap them to random consecutive integers $\{1,\ldots,N\}$. Then examples for each label are sampled from the corresponding cluster, creating a classification task for classifying the sentences into their underlying cluster labels. Note that the step of remapping the cluster-ids ensures that the model cannot memorize the sentence to cluster mapping, which would lead to meta over-fitting Hsu et al. (2019).

SMLMT proposes sentence-level tasks and thus lacks cross-sentence reasoning. This is confirmed by the poor downstream few-shot performance of models trained on SMLMT (see Sec. 4.3). Since models trained on SMLMT have never seen pairs of sentences as input, it leads to a train-test mismatch for sentence-pair classification tasks. To remedy this, we introduce a simple but effective contrastive learning task over sentence-pairs that bridges this gap. Contrastive learning has been used to learn effective sentence representations Logeswaran and Lee (2018). Next sentence prediction, a sentence-pair task, was used in the training of BERT Devlin et al. (2018) which was later found to be not effective Liu et al. (2019b). BERT considered segments instead of full sentences, however the downstream tasks often require reasoning over complete sentences. Thus, we consider classifying whether two sentences come from the same document as opposed to different documents, as a sentence-pair task to enable cross-sentence reasoning. This simple objective was found to be quite effective in our experiments. Note that during meta-training, this can be treated as an additional task in the task distribution. Since the SMLMT task distribution consists of an exponential number of tasks, we sample the sentence-pair task in an episode with a fixed probability $\alpha$, which is hyper-parameter.

We consider the challenging few-shot setting where models are trained on unlabelled corpora and then evaluated on target tasks with only $k$ examples per label ($k\leq 32$) to allow fine-tuning of the models on the target task. Since our focus is on unsupervised meta-learning, we closely follow the experimental setup of Bansal et al. (2020b).

We start by a quantitative exploration of the various self-supervised task proposals without resorting to full fine-tuning on downstream tasks. Our goal here is to understand properties of these task distributions and how they relate to each other. To do this, we consider models meta-trained on a specific type of task proposal (rows in Table 1) and evaluate their performance in the few-shot setting on tasks sampled from all of the other task proposal methods (columns therein). We use r$i$ (or c$j$) below to refer to row $i$ (or column $j$) in the table.

We consider the following task proposal methods: Frequency (FREQ, c1): using the frequency-based word sampling in SMLMT; Inter-Cluster (X-C, c2): using the word-clustering approach explained in sec 3.1 but sampling all labels of task from different clusters; Intra-Cluster (I-C, c3&4): using the word-clustering approach explained in sec 3.1 which samples all labels of task from the same cluster; Sentence Cluster (S-C, c5): this is the sentence clustering approach to task proposal presented in sec 3.3. For evaluation, we consider 4-way tasks sampled from the above methods and evaluate average accuracy over 5000 tasks. We consider a BERT model (r1) which is not trained on the SMLMT distribution but is trained on the related masked language modeling (MLM) task. To enable evaluation of this model, we use it as a prototypical network model Snell et al. (2017). We also consider meta-trained models trained on the SMLMT distribution with uniform sampling Bansal et al. (2020b) (r2), frequency-based sampling (r3), and intra-cluster sampling (r4). Note that all models are trained on Wikipedia corpus.

Results are in Table 1. First, since BERT wasn’t trained on any of the task distributions, we find low accuracy on all these tasks on r1, indicating that they contain information different than what is learned from MLM. Moreover, the highest accuracy of this model is on Sentence Cluster tasks (r1c5; random baseline is 25%), even though the domain of this task is quite different than the training data of BERT. Next, lets consider the vanilla SMLMT model which uses uniformly random word sampling to create the meta-learning task distribution. Interestingly, we find that it gives high accuracy on frequency-sampled tasks (r2c1). Similarly, accuracy is high on the inter-cluster tasks (r2c2), even though the model wasn’t meta-trained directly on this distribution. More importantly, performance drops significantly ($\approx 18\%$) on the tasks sampled using the intra-cluster approach (r2c3). This performance drops even further ($\approx 10\%$; r2c4) when the tasks are sampled from a different domain (common crawl) than the training domain of the model (Wiki). Accuracy on Sentence Cluster is also very high (r2c5), without training on this distribution. Models trained on frequency-based sampling perform similarly (r3). We also show the performance of a model trained on tasks sampled using the intra-cluster approach. Note that this model was trained on Wikipedia corpus, and even though it was trained on intra-cluster tasks, we still see a significant performance drop on intra-cluster tasks on a different domain (r4c4 vs r4c3). Finally, consider models trained on the sentence clustering tasks. These perform poorly on all of the tasks proposed by SMLMT (r5c1–4), indicating that this task distribution does not contain the same amount of information as SMLMT.

In summary, these results indicate that: (1) the intra-cluster tasks are more difficult than frequency-based sampling, and inter-cluster tasks are as easy as uniform-sampling (r2c2) (2) sentence cluster tasks are the easiest among all task proposals (c5), and training on this task distribution leads to poor performance on the SMLMT distributions (r5c1–4; but not vice versa), indicating lack of information in this distribution as compared to SMLMT. From this analysis we expect intra-cluster task distribution to be richer as compared to the other alternatives and models meta-trained on these should improve downstream performance over the others. As we will see in the next section, the downstream performance improvements are highly correlated with these unsupervised evaluations.

FewRel Han et al. (2018); Gao et al. (2019) is a common benchmark for few-shot learning in NLP, which consists of many few-shot relation classification tasks created by sub-sampling from a pool of relation labels. The resemblance to the popular few-shot benchmarks like MiniImageNet Vinyals et al. (2016) makes FewRel one of the few widely used datasets for training and evaluating NLP meta-learning methods.

Before submitting to the competition site for test set results, we first use the validation set to select the best model(s), where we observed that the Cluster approaches performs better than the other task proposals (see validation results in Supplementary). We then compare their test set performance with previously published results: the BERT-Pair, Proto-Adversarial (CNN), and Proto-Adversarial (BERT) are supervised meta-learning models trained on FewRel training data and using BERT or CNN as the text encoder. See Gao et al. (2019) for details. Interestingly, our unsupervised meta-learned models that do not use any FewRel training data outperform the supervised baselines in the 5-shot setting. Performance is lower than BERT-Pair on 1-shot tasks, potentially because our models have not been trained for 1-shot tasks like BERT-Pair. Finally, fine-tuning our best model on the FewRel training data leads to the best overall performance.