A Few-shot Approach to Resume Information Extraction via Prompts

With the introduction of the Transformer architecture [19], large-scale language models pre-trained on unsupervised tasks have consistently achieved state-of-the-art results on a wide range of NLP tasks. The most prominent of these models include the Encoder-based BERT [1], a task-based MLM pre-trained model primarily used for classification tasks, and its modified version, RoBERTa [11]. Another example is the T5 model [13], which uses the Seq2Seq MLM pre-training method. The pre-training and fine-tuning paradigm was widely applied to various NLP downstream tasks until 2020, when a novel pre-training and prompt paradigm was proposed.

The first prompt-based approach for text classification tasks, which utilizes models to answer cloze questions and predict labels, was proposed for sentiment classification [16]. This approach was shown to achieve higher accuracy than the traditional fine-tuning paradigm, even with limited training data. The underlying principle of prompt learning involves using manually designed templates to wrap sentences, which are then masked to provide the target label relation words. This masked text is then input into the model, which predicts the corresponding label relation words. The development of manual prompt templates led to the exploration of automatic prompt generation methods. These methods are broadly categorized into two groups: discrete prompts and continuous prompts [9].

As previously noted, prompt methods have demonstrated exceptional performance across various benchmark datasets. However, their efficacy in unique practical scenarios, such as information extraction from resumes, remains a question. This application is of great significance to businesses, given the daily influx of resumes. Though deep learning has facilitated automated resume screening, the resource demand for annotating resume texts for Pretrained Language Model (PLM) fine-tuning can be prohibitive. Prompt methods, requiring only a few labeled resume texts, present an economical solution, particularly for small businesses or niche industries. In this study, we leverage a seven-category sentence classification of English resume data [3]. This approach transforms the resume extraction task into a sentence classification task, aligns with the current research focus in prompt learning, and offers practical value, especially for organizations with limited resources. Moreover, the recurring specific words in resumes are ideal for constructing a Knowledgeable Verbalizer (KV), a technique known for its state-of-the-art (SOTA) results [6]. Thus, we focus on information extraction from resumes in this study.

As depicted in Figure 1, the sentence to be classified, denoted as X, is inputted into the wrap class. Following this, an indicative template sentence is appended. Subsequently, a mask token is integrated into the prompted sentence, representing the label relation word to be forecasted, such as “$\{$X$\}$ this sentence is about $\{$mask$\}$”. Consequently, a sentence classification problem is reframed as a fill-in-the-blanks (mask token) task. The Masked Language Model (MLM) explicitly addresses this issue. As a result, prompting enables the downstream task to align more closely with the pre-training task, facilitating rapid model adaptation for small datasets. The Knowledgeable Verbalizer (KV) extends a single label relation word to cover multiple associated terms. Initially, the output probability of each relevant word for the mask token is calculated. These probabilities are then consolidated, and the class with the highest probability of associated words is selected as the sentence’s predicted outcome, i.e., the mask token word. In conclusion, KV with a manually designed template has achieved superior scores across numerous datasets compared to other prompt methods’ baseline scores. Therefore, our initial hypothesis in this study was that the amalgamation of manually designed templates and KV is the most effective among current mainstream prompt methods. We conducted subsequent comparison experiments to validate this hypothesis. To assess the impact of different templates on the results, we devised several alternative templates for comparison experiments. The results suggest that the efficacy of prompt-learning is significantly impacted by template design. A baseline KV was created for resume text classification using the original Knowledgeable Prompt-learning method [6] and compared with the KV designed based on our refined rules. We term the KV constructed following our proposed rule as Manual Knowledgeable Verbalizer (MKV). On the 25-shot and 50-shot tasks, performance increased significantly from 54.96$\%$ to 63.65$\%$ and 59.72$\%$ to 76.53$\%$ respectively. These findings indicate our proposed MKV is more suited for the resume extraction task. We conducted comparative experiments on two models using 25/50/10-shot tasks and two distinct training techniques to compare the encoder structure of BERT and the Seq2Seq structure of the T5 model in the context of prompt-learning. In summary, our primary contributions include:

1. Development of a comprehensive set of prompt-learning techniques for the few-shot resume information extraction task.

2. Proposal of KV construction rules, based on the original Knowledgeable Prompt-tuning (KPT), that are more compatible with resume text. This provides insights for subsequent researchers to develop KVs for practical application scenarios.

3. Comparison of manual template construction with automatic template generation for the prompt method. We have demonstrated that, in the current state of prompt-learning for resume text, a method employing manually crafted prompts surpasses one using automatically generated prompts.

The research related to this paper is divided into two main groups. Prompt templates and constructs of prompt verbalizer.

Prompt Template. Since PET(Pattern-Exploiting Training)[16] was proposed, a surge of prompt research has been started. A PET follow-up study mentioned that smaller size models, such as BERT-base, are also capable of few-shot learning[17]. There are also some other studies of discrete prompts[4][14][18]. The idea behind a discrete prompt is that the words in the template are real. The reason for this is because in coordinate space, the words are in a discrete state. Another hand, as all token-level prompts have been automatically generated, some research has gone a step further by replacing the token-level prompt template in the token representation with continuous vectors directly and training these prompts instead of the fine-tune model(A.K.A Continuous Prompt)[8][7][12][5][10]. Using continuous vectors, we can find the best set of vectors to replace the discrete prompt template.

Prompt Verbalizer. The automatic construction method of verbalizer in few-shot learning is explored[15]. There are also corresponding verbalizers for some of the prompt methods mentioned in the previous paragraph, such as manual, soft and automatic verbalizers. Finally, the KV in this study also has better performance than other verbalizers[6].

In this work, we select the resume information sentence classification dataset as the experimental object. Hence, this study also considers the task setting from a practical application scenario. Suppose an IT company needs to fine-tune its company resume information extraction model. First, a training dataset consisting of several hundred resumes needs to be constructed. This results in the need to annotate tens of thousands of sentences. This is a non-negligible cost for a small company. Also, some small start-up companies may not have hundreds of resumes to create a training dataset. Thus, it is essential to minimize the investment of company human resource in producing training datasets for company.

In summary, we set the task to be in the case of annotating only one or two resumes. The resume dataset was extracted from 15,000 original resumes and 1000 of them were used as tagged objects [3] ¹¹1https://www.kaggle.com/datasets/oo7kartik/resume-text-batch. In addition, the total number of sentence samples in this dataset is 78786, and the total number of annotated resumes is 1000. So it is calculated that each resume contains about 79 sentences on average.

In this section, we firstly designed a series of prompt templates for resumes. Secondly, we presented a different KV construction method from the original KPT depending on the textual characteristics of the resume(Figure 1).

This study aims to investigate the impact of various factors on the outcomes of limited opportunity resume learning, as well as to assess the efficacy of our proposed MKV construction rules for Prompt Learning of resume material. We performed multiple comparative tests between the prompt’s template and verbalizer, evaluated the performance of several prompting methods against MKV, and examined the effectiveness of two structurally distinct PLM models in few-shot resume learning. These experiments utilized a specially created resume dataset.

For efficient iteration of different experiments, we used Openprompt⁵⁵5https://github.com/thunlp/OpenPrompt, an Open-Source Framework for Prompt-learning[2]. We used the F1-micro as the evaluation metric for all experiments. Given that few-shot learning typically allows only a limited number of training samples per class, we employed a random seed to extract the training set from our unbalanced dataset of resumes. This approach maintained the original sample distribution, thereby preserving the inherent imbalance of the resume samples. Distinct random seeds were used for the 25/50/100-shot experiments.

Initially, we conducted two comparison experiments to determine the most efficient template and Knowledge Verbalizer (KV). One experiment compared different manual templates, while the other compared the performance of our proposed MKV construction method with the original KV construction method. After establishing the performance of Manual Template (MT) and Manual Knowledge Verbalizer (MKV), they were compared with three other types of templates and verbalizers.

The methods we considered include: 1) The Automatic method, where the model generates discrete prompt template words automatically, 2) Soft prompt method, where by optimizing the vectors of the embedding layer, soft prompts are utilized, 3) P-tuning method, where token-level templates are replaced with dense vectors, which are trained to predict the masked words, and 4) Prefix prompt approach, where task-specific continuous sequence prefixes are trained instead of the entire transformer model.

Finally, to investigate the fine-tuning and prompt-tuning performance of RoBERTalarge and T5large, we conducted an additional experiment. The scores from the first three experiments were obtained after 4 training epochs, while the PLM comparison experiment used scores adjusted to the best epoch based on test set scores. To evaluate the performance of KV and MKV in the context of sample imbalance, we created and analyzed the confusion matrix of the 50-shot test dataset for both methods.

In this study, we use the prompt technique on the resume dataset for few-shot learning. We created templates informed by resume sentence structures and assessed their utility. Additionally, we refined the construction of a knowledgeable verbalizer, relying on Knowledgeable Prompt Tuning (KPT). For this, we devised construction rules for the MKV, tailored to the textual features of resumes. Experimental evaluations demonstrate our MKV’s effectiveness and robustness. While the final outcomes were satisfactory, they also elucidated the constraints inherent in the utilized prompt methodology. It is anticipated that future endeavors will develop a more universal prompt approach, capable of addressing a variety of industries and accommodating diverse resume formats.