LabelPrompt: Effective Prompt-based Learning for Relation Classification

Pre-trained language models are trained in large-scale unlabelled text data to learn robust general-purpose features, such as lexical, syntactic, semantic, and word representation (Liu et al., 2022). Many big models, such as GPT (Liu et al., 2021b), BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), T5 (Raffel et al., 2020), have been proposed, greatly promoting the development of the research in natural language processing. Most NLP tasks benefit from the knowledge contained in the pre-training language model, and the increase in the size of a pre-training model as well as data facilitates the task. By fine-tuning the parameters of PLMs with additional specific modules and task-specific objective functions, the models can be adapted to most downstream tasks. Generally, the pre-training and fine-tuning paradigm become the foundation for many NLP tasks, such as named entity recognition (Wu et al., 2021), relation classification (Lyu and Chen, 2021), and question answering (Zhong et al., 2022), etc.

Although fine-tuning has achieved significant success in many fields, there remains a large gap between the objectives of pre-training and fine-tuning tasks (Liu et al., 2021a). Typically, during pre-training, a PLM is trained using self-supervised corpus texts with Masked Language Modelling (MLM) and Next Sentence Prediction (NSP) tasks (Devlin et al., 2019). This differs from other NLP tasks, such as classification and sequence tagging tasks. As a result, recent studies have sought to explore how to effectively use pre-trained language models and address this gap (Wang et al., 2021; Moiseev et al., 2022).

Prompt-based learning is a different approach that aims to reduce the gap between pre-training and fine-tuning. Rather than fine-tuning, prompt-based learning reformulates a downstream task to the original PLM training task. For example, a prompt template might transfer a machine translation instance as “English: I love this film. French: __”. A large PLM could then fill in the blank with a French translation of the English text. Essentially, prompt-based learning enables PLMs to perform downstream tasks by providing additional information in the form of a prompt (Lester et al., 2021; Gao et al., 2021).

Many studies have shown that prompt templates containing semantics can effectively activate knowledge in the model. Building on this idea, Petroni et al. (2019) considered language models as knowledge bases and proposed the LAMA (LAnguage Model Analysis) probe dataset. The LAMA probe provides manually created completion templates for exploring the knowledge in PLMs. By using hand-crafted query sentences, it extracts the knowledge base from PLMs and performs comparably to using PLMs directly. However, the manual design of templates as discrete prompts is time-consuming, expensive (Shin et al., 2021), and often requires a wealth of expert knowledge.

To extend prompt-based learning to a wider range of tasks and reduce the overhead of prompt text construction, Shin et al. (2020b) automatically generated prompts for different task sets based on a gradient search approach. Some researchers have also added a small set of task-specific prefixes to replace manual templates as continuous prompts, which are learnable by backpropagation (Li and Liang, 2021; Lester et al., 2021). Additionally, Zhong et al. (2021) suggests initialising continuous prompts based on the discrete prompts already explored by prompt search methods.

Another important component of prompt-based learning is the answer-mapping approach. Since the masked output does not always match the label, the aim of answer mapping methods is to map the model output to the answer space when the answers cannot be considered as task outputs. For example, sentiment analysis tasks require mapping the output of the model template “{ good, fantastic, unhappy, average }” to “{ positive, negative, neutral }”. The answer-mapping method is also known as a verbaliser. A verbaliser is a projection method that constructs a project between a natural word space and a task label space (Hu et al., 2022). Just like the prompt template, a verbaliser also has “manual design” (Yin et al., 2019; Cui et al., 2021), “discrete answer search” (Jiang et al., 2020b; Xia et al., 2022) and “continuous answer search” strategies (Hambardzumyan et al., 2021). Equally, the quality of a verbaliser can have an impressive impact on model performance.

In conclusion, prompt-based learning is widely considered as a highly effective method for utilising pre-trained language models, and it is a subject worthy of further study.

Relation Classification (RC) is a sub-task of Information Extraction (IE). The goal of RC is to identify the semantic relation between two entities in a given text. As shown in Table 1, given the sentence “Mark Fisher writes for the Dayton Daily News.”, the relation between the subject “Mark Fisher” and the object “Dayton Daily News” is “per:employee_of”. RC is a crucial task for many natural language understanding applications, including question answering, knowledge base construction, and text summarization (Hendrickx et al., 2010).

Early approaches for RC relied heavily on prior knowledge to extract additional features from the text (Califf and Mooney, 1997; Zeng et al., 2015). The emergence of large-scale PLMs, such as BERT (Devlin et al., 2019), has provided universal language representations that are useful for RC tasks (Vaswani et al., 2017; Cohen et al., 2020; Yamada et al., 2020). However, these methods often require significant amounts of labelled data and complex, task-specific neural modules, and tend to underperform in the few-shot scenario.

Recently, there have been studies exploring the use of prompt-based learning to leverage the knowledge contained in PLMs for relation classification. The LAMA dataset is a probe for analysing factual and commonsense knowledge in PLMs, which consists of a set of knowledge sources, each comprising a set of facts. For example, one template could be ”__ is the capital of France.” Hu et al. (2022) proposed a method that uses external knowledge to construct a verbaliser for obtaining more accurate label words for text classification. Han et al. (2021) applied logic rules to construct prompts with several sub-prompts, effectively using prior knowledge from relation classification. Chen et al. (2022) proposed a prompt-based learning approach called KnowPrompt that incorporates abundant semantics and prior knowledge in relation labels into relation classification.

All these methods aimed to effectively use prompt-based learning to narrow the gap between relation classification and pre-training tasks, and demonstrated promising results in both few-shot and full-data scenarios. However, none of these methods addressed the issue of inconsistency between prompt output and relation label texts. Inspired by KnowPrompt, we propose a new prompt-based learning approach, namely LabelPrompt, that explicitly constructs prompt templates using label tokens and adds supervised information corresponding to the labels at the model’s output.

The objective of the relation-align module is to align the label tokens with the deep decision-making layer of the model. In the context of relation extraction models, the label token on the input side may lose its inherent features in the subsequent encoder layers, which reduces its effectiveness in aiding the prediction of relations in the deep decision-making layer.

To tackle this problem, we propose the relation-align module, which is integrated at the end of our model. Furthermore, the verbaliser $V_{\phi}$ is incorporated behind the hidden vector $h_{c_{i}}$ of each label token $c_{i}$ in $x_{prompt}$ to reinforce its semantic meaning, just like the mask loss. To accomplish this, the probability score $p(c_{i}|\mathbf{x},c_{i})=p\left(\tilde{c_{i}}=\mathcal{C}|x_{prompt}\right)$ is formulated for each label token, which can be classified into itself by the verbaliser, where $\tilde{c_{i}}$ denotes $V_{\phi}(h_{c_{i}})$.

The cross-entropy loss for each label token is computed based on the probability distribution, and then the losses are averaged to obtain the label loss $\mathcal{L}_{label}$:

In this section, we introduce an entity-aware module aimed at addressing the issue of a PLM failing to consider given entities when predicting the relation between them.

Through analysis of numerous error cases, we observed that a PLM can detect a relation in a sentence, but it may not exist between the two given entities. In other words, the model does not take the given entities into account when predicting the relationship.

Drawing from TransE (Bordes et al., 2013), which suggests a correlation, $s+r=o$, between the features of entities ($s,o$) and the relation $r$, we propose the entity-aware module. The module employs the distance metric $d(s,r,o)=\lVert s+r-o\rVert_{2}$ to measure the distance between entities and relations. Furthermore, the dimensions of the encoder outputs for entities and relations are reduced to eliminate redundant features:

where $\phi_{sub}$, $\phi_{obj}$ and $\phi_{rel}\in\mathbb{R}^{p\times d}$ are trainable parameters, $p$ is the length of the hidden vector after dimension reduction.

In Equ. (9), the tuple ($s,r,o$) is considered a positive example while negative examples ($s^{\prime},r,o^{\prime}$) are randomly sampled for the entity-aware loss, where $s^{\prime}$ and $o^{\prime}$ denote two arbitrary spans from the sentence. The entity-aware loss, denoted by $\mathcal{L}_{entity}$, is formulated as follows:

where $\sigma$ is the sigmoid function and $\gamma$ is the margin that is set to 0.3.

Different from previous approaches that use simple negative samples, we use more challenging negative samples and concentrate on the contrastive loss between current sentences during the training phase. Our experiments reveal that other samples in the same batch can hinder the convergence of this portion of the loss, thus we disregard them.

As shown in Equ. (3), the input sequence comprises two distinct categories of tokens, namely prompt tokens and sentence tokens. However, the inclusion of additional unknown tokens can significantly impact the semantic meaning of the sentence and subsequently, the efficacy of relation extraction.

To mitigate the adverse impact of prompt tokens on sentence semantics, we employ an attention query strategy during the PLM encoding process. This approach utilises distinct query matrices for different token pairs in the self-attention layer to minimise the influence of prompt tokens on sentence semantics and enhance the quality of relation extraction outcomes. Further details are provided in Fig. 1.

The self-attention layer in the transformer encoder can be formulated as follow:

where $E$ represents the input and $Q\in\mathbb{R}^{d\times d}$, $K\in\mathbb{R}^{d\times d}$, $V\in\mathbb{R}^{d\times d}$ denote the mapping matrices. In our method, we select different query mapping matrices depending on which pair of tokens are interacting: $Q_{pp}$, $Q_{ps}$, $Q_{sp}$, and $Q_{ss}$. The effective initialisation of pre-training parameters enables the method to maintain strong few-shot performance even when additional query matrices is incorporated.

We propose three loss functions to optimise the model parameters: mask loss for relation prediction, label loss for semantic consistency of label prompt tokens, and entity loss for improving entity awareness.

Our objective is to minimise the final loss $\mathcal{L}$ which is the weighted sum of those losses:

where $\alpha_{1},\alpha_{2}$ are the weights for the label loss and entity loss, respectively. In this paper, we set them to 1 and 0.04 based on our empirical experimental results.

To better verify the effectiveness of our idea, we use the most popular relation classification datasets, including TACRED (Zhang et al., 2017), TACREV (Alt et al., 2020), ReTACRED (Stoica et al., 2021), and SemEval (Hendrickx et al., 2010).

TACRED is one of the most enormous and widely used crowd-sourced datasets in relation extraction. It is created by combining available human annotations from the TAC KBP challenges and crowd-sourcing, and it contains 42 relation types (including “no_relation”).

TACREV is a modified version of TACRED. It corrects the label errors in the validation and test sets while leaving the training set unchanged.

ReTACRED is a new completely re-annotated version of the TACRED dataset. The dataset fixes many labelling errors in the TACRED dataset and refactors the training, validation, and test sets.

SemEval is a traditional dataset in relation classification. It contains 9 symmetric relations and one special relation “Other”.

Annotation quality of the datasets. In the field of natural language processing, an essential component of dataset preparation is the quality of the annotation. A comparative analysis of the annotation quality of four datasets. The results indicate that the ReTACRED dataset has the highest annotation quality compared to the other three datasets. Although the TACREV and TACRED datasets share the same training data, Alt et al. (2020) improved the quality of annotations in the TACREV test set. The statistical details of each dataset have been tabulated in Table 2.

In this paper, we compare the proposed LabelPrompt method with several representative approaches for relation classification. LabelPrompt is a novel method that leverages label prompt tokens to enhance pre-trained language models for this task. We select baselines that cover different aspects of relation classification, such as fine-tuning, prompt-based learning, and knowledge integration.

For fine-tuning pre-trained models, RoBERTa is a very popular language model that improved BERT’s training strategy to overcome its under-training problem, and it also served as the text encoder of this paper. We also revise some other pre-trained models that incorporated different types of knowledge into their representations. For example, KnowBERT embeds multiple Knowledge Bases (KBs) into PLM to leverage structured, human-curated knowledge. MTB builds task-agnostic relation representations from entity-linked text and fine-tuned them on supervised relation extraction datasets. SpanBERT focuses on improving span representations by defining a masked span prediction task for representing and predicting spans of text with varying lengths. LUKE predicts randomly masked words and entities in a large entity-annotated corpus retrieved from Wikipedia.

For prompt-based learning models, we select two representative works of prompt-based learning models. PTR (Han et al., 2021) uses logic rules to create prompts with sub-prompts that encode the prior knowledge of each class. KnowPrompt (Chen et al., 2022) injects latent knowledge in relation labels into the prompts, thus injecting the knowledge into relation labels.

We use $\textrm{RoBERTa}_{large}$ (Liu et al., 2019) as the pre-trained language model for all our experiments and follow the same setting as the previous methods. We compare the performance of our proposed method with other methods in two scenarios: full-data and few-shot. In the full-data scenario, we use all the training samples in the dataset for network tuning. In the few-shot scenario, we use the $k$-shot approach, where $k\in[8,16,32]$, to evaluate our model’s performance. The $k$-shot approach means that we sample $k$ instances from each class in the training/validating set as the training/validating data while using all the test instances to evaluate the model’s performance.

We conducted our experiments on NVIDIA GeForce RTX 2080 Ti. We set the batch size to 16 and the learning rate to 4e-5 and 4e-6 for the few-shot and full-data scenarios, respectively.

We first evaluate and analyse the proposed LabelPrompt method by comparing it with both fine-tuning and prompt-based methods. The experimental results are reported in Table 3 and Table 4.

Few-Shot. In Table 3, we compare our approach with the traditional fine-tuning and prompt-based methods in the few-shot scenario. Our approach outperforms the other methods in most cases. Furthermore, on the TACREV and ReTACRED datasets, which have more accurate and consistent annotations, our method outperforms all the other methods in all the few-shot settings.

As compared with the fine-tuning methods, our results show that prompt-based methods perform better in the few-shot scenarios for the NLU tasks with low resources. We attribute this to the fact that prompt-based learning effectively uses the knowledge stored in PLMs to fit the task. In contrast, fine-tuning methods rely on complex decoding modules and training strategies based on PLMs, which require sufficient data.

As compared with the previous best prompt-based methods, LabelPrompt achieves 1.5%-6.2% performance gains on both the ReTACRED and TACREV datasets, which have high labelling quality. It also achieves 1.9%-2.7% improvements on the TACRED and SemEval datasets in the 8-shot setting.

These two aspects demonstrate that the explicit label tokens can aid the pre-trained model in the few-shot scenario. We believe that these additional tokens can reduce the search space for target tokens during the early stage of model training. To further validate our model’s few-shot performance, we design an additional 4-shot setting, as shown in Fig. 2. The experimental results demonstrate that, even with only 4 instances per class, the proposed method achieves the same performance as other methods in the 8-shot setting.

Full-Data. In the full-data scenario, as shown in Table 4, the proposed method also beat all the other fine-tuning and prompt-based approaches.

In comparison with KnowPrompt, LabelPrompt improves the performance by 0.7% in TACRED and by 1.1% in SemEval. This demonstrates that the prompt-learning method can be applied to both few-shot and full-data tasks. In fact, LabelPrompt leverages the knowledge embedded in PLMs to obtain remarkable results in downstream tasks with a simple design.

So far, we’ve discussed the performance of the proposed method, and it shows that LabelPrompt is superior to the other methods in both few-shot and full-data scenarios.

In this section, we evaluate the performance of each component of LabelPrompt in both few-shot and full-data settings. The results are presented in Table 5 and Fig. 3.