EXnet: Efficient In-context Learning for Data-less Text classification

Large pre-trained language models (LMs) have revolutionized Natural Language Processing (NLP) and demonstrated state-of-the-art results over a varied set of tasks Min et al. (2021). Despite the success, most pre-trained models need to be fine-tuned over large corpus of manually-annotated training data, especially when the origin of the data is from a different domain. Annotation of such large-scale task-specific data for training each downstream model is extremely expensive, and a time-consuming process. To solve this issue, recent research focus on zero-shot learning with PLMs that involves solving downstream tasks without any further gradient-based training (parameter updates) of the models. For instance, GPT-3 (Brown et al., 2020) & other GPT-family of models such as GPT-Neo showed an interesting ability of In-context learning in which the model could learn unseen tasks on-the-fly using a few demonstration examples, without any parameter updates. Zhong et al.Zhong et al. (2021) demonstrated that it is possible to convert text classification tasks to Yes/No and a train a prompt-based model to perform zero-shot learning. Unfortunately, pure zero-shot models provide low task-specific accuracy and are not useful for many downstream applications. In-context learning that takes additional context in terms of both prompt and examples outperforms classic Zero-shot models and likely to work better for many real-world applications. We argue that adding a small number of demonstration examples is a small overhead but can help provide a significant accuracy improvements. Popular PLMs do support In-context learning but have major limitations. Apart from a huge training overhead, they have a significant inference costs and make them inaccessible to a large number of practitioners. Further, they provide poor accuracy on domain-specific tasks, support only limited number of examples, and also, sensitive to the order of examples (Zhao et al., 2021).

In this paper, we propose EXnet, a light-weight model architecture that enables us to utilize In-context support examples for few-shot text classification. Our model doesn’t have a limit on the number of input task examples, and there is no visible impact on the ordering of the examples. Specifically, this paper makes the following contributions:

• •

  We present EXnet, a simple and scalable model architecture trained specifically to learn classification tasks with few-shot examples, without any parameter updates.

• •

  We evaluate EXnet on a collection of 9 unseen datasets (during training), which test both the cross-domain and cross-task understanding. Even the smallest variant of EXnet with 15 million parameters and only two in-context examples, shows performance gains of up to 48% compared to GPT-Neo (1.3 billion parameters) which is 86$\times$ the size of our model.

This section provides a comprehensive overview of the previous works related to EXnet. The overall landscape of few-shot learning can be grouped into two categories, few-shot learning with or without parameter updates. Generally, few-shot learning with parameter updates is termed as few-shot fine-tuning because it involves utilizing a small set of $K$ labeled samples to fine-tune a model. Gupta et al.Gupta et al. (2020) explored transfer learning with BERT for few-shot fine-tuning, where they use text-label pairs as input, and predict if the label belongs to the text. Their results show that this simple approach outperforms all previous baselines over the ARSC dataset, suggesting that utilizing both text-label pairs as input has the potential to demonstrate cross-domain understanding. Wei et al.Wei et al. (2021) presented a data augmentation strategy powered by curriculum learning and BERT-based triplet networks for few-shot fine-tuning, where they show a slight 3% improvement in performance over previous approaches. Ohashi et al.Ohashi et al. (2021) presented an approach where they utilize semantic knowledge from labels to learn distinct representations specific to each label, which in turn enables them to get improved model performance in few-shot fine-tuning.

Recently, prompt learning has also been explored for few-shot learning where PLMs are provided a task instruction/prefix which helps in utilizing their pre-trained knowledge to understand the task instruction and thus improving the few-shot performance. Schick et al.Schick and Schütze (2020) presented PET, a prompt-based semi-supervised method that shows strong few-shot performance. PET utilizes cloze-style input patterns similar to Masked-Language modeling (MLM). Utama et al.Utama et al. (2021) demonstrate through their experiments that few-shot fine-tuning performs poorly compared to zero-shot counterpart, which proves that few-shot fine-tuning degrades PLMs performance, particularly in the case of these sentence-pair classification tasks. A possible explanation for this phenomenon is that the PLMs lose their pre-trained knowledge by catastrophic forgetting during the few-shot gradient updates, and learns lexical shortcuts instead of meaningful patterns. An extreme case of few-shot learning without parameter updates is zero-shot learning, where no examples of the task are required. Yin et al.Yin et al. (2019) made the first attempt at unifying datasets and evaluating the zero-shot performance of language models over them which were only fine-tuned on natural language inference (NLI) tasks. However, Ma et al.Ma et al. (2021) demonstrated some major drawbacks in NLI-based zero-shot text classification. They show that there are negligible gains on some tasks with NLI-based fine-tuning compared to standard BERT pre-trained on next-sentence prediction task. Moreover, they show that these NLI-based models only rely on shallow lexical patterns, along with high variance and a lack of stability. Zhong et al.Zhong et al. (2021) explored the idea of utilizing prompts in question-answering format which involve the task of predicting if the label description belongs to the text or not. They collect a large set of 43 datasets comprising 441 labels, which they categorize into groups of similar datasets and only evaluate on those groups that are unseen during training. Their experiments show that larger PLMs like T5-Large show much better performance compared to the smaller ones. In-context learning is the most promising variation of few-shot learning without gradient updates, which involves giving a set of label examples in the input to PLMs, along with a query text that must be predicted based on the understanding of the task from those examples. Brown et al.Brown et al. (2020) presented GPT-3, which was the first PLM to demonstrate this ability of In-context learning, mostly on text generation tasks. One of the major disadvantages of GPT-3 has been it’s huge size (175 billion parameters), and it’s range of biases that have been demonstrated by Zhao et al.Zhao et al. (2021).

None of the prior work explored building a light-weight in-context model that can support a unlimited number of examples.

EXnet utilizes a Support set $S$ and a Query $Q$ as input which is a text in a question format with a fixed template $T$ as $T(x,l)$ = "question: is the text about <$l$>? [SEP] text: <$x$>". The support set $S$ consists of $K$ examples of text which belongs to the class $l$ mentioned in $Q$. Each support example also follows the format of $T$. For training, we use the Binary Cross-entropy loss function with the AdamW Loshchilov and Hutter (2018) optimizer having a learning rate of 0.00002. The architecture of EXnet has been schematically shown in Figure 1. The idea of using a single encoder to encode both the query and the support set, and then concatenating their embeddings for few-shot learning, was inspired by metric learning-based few-shot methods in Computer Vision, such as Relation Networks Sung et al. (2018) and Matching Networks Vinyals et al. (2016). EXnet consists of the following key components:

• •

  Encoder module: We use a single encoder $E$ to generate embeddings for both the query $Q$ and support set $S$. Thus, $E$ generates $K$ embeddings $e_{1}...e_{k}$ for $K$ examples in $S$ and one embedding $e_{q}$ for $Q$. The $K$ support set embeddings are then concatenated together into a single tensor $e_{S}$. Here, $E$ can be any encoder-only transformer model such as BERT Devlin et al. (2018). For simplicity, we experiment with three different variants of BERT based on size/number of parameters.

• •

  Projector blocks: Two projector blocks $P1$ and $P2$ are used to convert $e_{S}$ and $e_{Q}$ respectively into equal dimension representations. $P_{1}$ consists of three GeLU activated feed-forward layers with a 30% dropout in between the second and third layer. $P_{2}$ consists of only a single GeLU activated feed-forward layer. We observe that keeping this asymmetry in projections enables the model to show slightly better performance.

• •

  Cross attention module: The encoder $E$ only attends either query $Q$ or the support set $S$ at a time, without being able to utilize both of them at the same time. Thus, we use this cross-attention module to attend to both $Q$ and $S$ at the same time, which would enable EXnet to make use of the knowledge from the in-context examples in an effective manner to predict the answer to the query $Q$. The projected vectors are passed into this cross attention module $C$ which utilizes the first projected vector as Key and Value, and the second projected vector as Query in the Multi-Head Attention mechanism like the original TransformerVaswani et al. (2017).

• •

  Feed-Forward network: The output representation by the cross attention module is passed on to a Feed-Forward network (FFN) consisting of two GeLU activated Feed-Forward layers, followed by a final Sigmoid activated Feed-Forward layer which predicts the probability of the presence of class $l$ mentioned in $Q$.

We use our evaluation datasets as discussed in Section 5.2, and measure EXnet’s performance over each of them using the most commonly used classification metric, i.e, F1-score. We tend to use F1-score instead of accuracy because some of the evaluation datasets are imbalanced. We compare three sizes of EXnet which we call EXnet-S, EXnet-M, and EXnet-L based on the increasing order of model size. EXnet-S is a model with 15 million parameters which consists of Small BERT Turc et al. (2019) as the Encoder which has 6 layers and an embedding dimension of 256. EXnet-M is a 112 million parameter model which comprises of BERT-base as the encoder which has 12 layers and an embedding dimension of 768. EXnet-L is a 338 million parameter model that consists of BERT-large as the encoder which has 24 layers and an embedding dimension of 1024. As a baseline, we use the 1.3 billion parameter GPT-Neo model in an In-context setting (without fine-tuning) for comparing with EXnet. Unlike EXnet which requires only $K$ positive examples, GPT-Neo required both $K$ positives and $K$ negative examples as showing only positives leads it to predict only "yes" for every query, previous work have shown that this is because GPT-Neo is trained on next word prediction. Furthermore, we also compare EXnet’s performance with the meta-tuned model from Zhong et al.Zhong et al. (2021).*Note: We report results of Zhong et al.from their original meta-tuned zero-shot (K=0) model. The results of our experiments are detailed in Table 1 and Table 2. It can be observed that EXnet is able to use the knowledge from the $K$ support examples, as increasing $K$ shows improved performance over most of the datasets. EXnet-M, and EXnet-L outperforms GPT-Neo over all the datasets in both cross-domain and cross-task evaluation by a large margin (up to 48% gains in terms of F1 score). EXnet-S also outperforms GPT-Neo on all the datasets, except for SUBJ where GPT-Neo shows a slightly higher performance by a margin of 0.01 F1 score, when $K=2$. In comparison to Zhong et al., EXnet with larger $K$ is able to outperform their Meta-tuned model (770M parameters) over most of the evaluation datasets. Based on our results, we make following observations -

• •

  Benefits of increasing $K$ are dependent on task complexity: In our cross-task evaluation, it can be seen from Table 2 that the complex tasks like SUBJ and ETHICS which involve very subjective human concepts, EXnet has a huge improvement when K is increased from 2 to 8 examples. This demonstrates that In-context learning has more utility whenever the concept required to understand the task is complex and subjective.

• •

  In-context learning is more useful in improving cross-task performance than cross-domain: As seen in 1, all the three variants of EXnet have the ability to understand cross-domain tasks, but increasing $K$ doesn’t provide reasonable boost in performance. A possible reason for this is the fact that cross-domain evaluation involves tasks which were similar to training tasks, due to which EXnet was able to understand the task well with limited examples ($K=2$).

Zero-shot models do not provide sufficient accuracy for most real-world applications. We argue that In-context learning without gradient updates is better suited for most real-world applications. For example, to detect new set emotions in product reviews, one can take a model trained on similar task on publicly available datasets. Existing in-context learning models are huge and does not provide great task accuracy.

In this paper, we presented EXnet, a lightweight model architecture specifically designed for In-context learning, that doesn’t have limitation over number of examples. We evaluated EXnet using 9 unseen datasets and demonstrated the strong cross-domain and cross-task understanding ability of EXnet, which even outperforms models that are 86$\times$ it’s size. In the future, an interesting area to explore can be pre-training the encoder in EXnet with different strategies to observe the trends in performance.