Low-Resource Multi-Granularity Academic Function Recognition Based on Multiple Prompt Knowledge

In this paper, we mainly focus on three levels of academic function recognition tasks, which are the citation function recognition, the sentence function recognition and the keyword function recognition.

Citation function: Numerous previous studies have introduced citation classification schemes (Teufel et al., 2006; Jurgens et al., 2016; Cohan et al., 2019; Pride and Knoth, 2020) as a way to identify the meaning or purpose of a specific citation. Note that for terminological consistency, we refer to the “citation classification” and the “citation intent classification” as the “citation function recognition”. Jurgens et al. (2016) and Cohan et al. (2019) adopt ML based and DL based methods to recognition the citation function, respectively. Furthermore, Yu et al. (2020) propose a interactive hierarchical attention model with aggregating the heterogeneous contexts to recognize the intention of citing behaviors and retweeting behaviors. Similarly, Beltagy et al. (2019) also fine-tune the SciBERT to recognize the citation function achieve SOTA performance.

Structure function: Structure function recognition tasks for scientific publications mainly concentrate on the abstract sentence function recognition and body sentence structural function recognition. For abstract sentence function, previous studies are mainly based on machine learning (ML) methods (Ruch et al., 2007; Hassanzadeh et al., 2014; Liu et al., 2013) and deep learning (DL) methods (Lui, 2012; Dernoncourt et al., 2016). Recently, SciBERT (Beltagy et al., 2019) is also adopted to directly classify the sentence function and achieve competitive performance. In terms of body sentence structural function recognition, there are also a series of works based on ML (Huang et al., 2016a, b; Lu et al., 2018) and DL (Qin and Zhang, 2020; Wang et al., 2020) based methods. Moreover, scholars also consider the context semantics and structural relation of surrounding sentences (Jin and Szolovits, 2018; Wang et al., 2019) to boost the sentence function recognition performance.

Keyword function: Kondo et al. (2009) were first to conduct research on automatic recognition of lexical semantic functions. They utilize conditional random filed (CRF) to divide the words in the academic title into four labels of “domain”, “problem”, “method”, and “others”. Nanba et al. (2010) adopt support vector machine (SVM) to recognize the “technology” and “effect” word in academic and patent literature. Cheng et al. (2021) use recurrent neural network (RNN) based sequence-to-sequence model to obtain the problem and method words in academic texts. Lu et al. (2020) and Zhang et al. (2021) utilize BERT-based model to classify the problem and method semantic function carried by the keywords in academic literature.

However, above mentioned ML, DL, and even PLMs based methods require a large number of annotated data to achieve competitive performance.

Since obtaining the training datasets for these tasks are challenging and expensive, it is important to develop systems that perform decent in low-resource settings, where few labeled examples are available. Intuitively, we can adopt NLP techniques to increase the amount of training data, i.e., data augmentation, including synonym replacement, random insertion, random swap, random deletion (Wei and Zou, 2019), paraphrasing formulation, and back translation (Chen et al., 2020a, b; Xie et al., 2020). Another typical way, widely adopted in computer vision and general NLP tasks, is to design meta-learning paradigm that can learn and adapt to new environments rapidly with a few training examples (Bao et al., 2019; Yao et al., 2021; Sun et al., 2021; Zhang et al., 2022a). These methods usually train a meta-learner that extracts knowledge from various related sub-tasks during meta-training and leverages the knowledge to learn new tasks during meta-testing quickly. Considering that the function of academic publication is already relatively well-defined, there is no need to identify new function category. In this paper, we mainly adopt the data augmentation based methods as target baselines.

Fine-tuning the PLMs is a conventional approach to leverage the rich knowledge during pre-training and has achieved satisfying results on supervised tasks (Devlin et al., 2019; Han et al., 2021a). However, tuning the extra classifier requires adequate training examples to achieve decent performance, it is still challenging to apply fine-tuning in low-resource settings, including few-shot and zero-shot learning scenarios (Brown et al., 2020; Yin et al., 2019). Recently, a series of studies (Petroni et al., 2019; Schick and Schütze, 2021; Liu et al., 2021b; Jiang et al., 2022) using prompts to bridge the gap of objective forms in pre-training and fine-tuning and make full use of PLMs. Manual prompts have achieved promising performance in low-resource sentiment classification and natural language inference (Schick et al., 2020; Han et al., 2021b). A typical prompt consists of two parts: a template and a set of label words, i.e., verbalizer. Zhou et al. (2022) propose a data augmentation method that combines prompting method with generating label-flipped data. To ease the manual effort of suitable prompt design, automatic prompt search has been extensively explored. Shin et al. (2020) explore gradient-guided search to generate both templates and label words. Gao et al. (2021) utilize sequence-to-sequence models to generate prompt candidates. These auto-generated hard prompts cannot achieve competitive performance compared with manual prompts (Han et al., 2021b). Thus, a series of research works on soft prompts have been proposed, which directly use learnable continuous embeddings as prompt templates and work well on those large-scale PLMs (Li and Liang, 2021; Qin and Eisner, 2021; Lester et al., 2021). Since the function of academic literature is relatively well-defined and does not require diversification, verbalizer mining methods (Hu et al., 2021; Schick et al., 2020; Cui et al., 2022) are not considered in this paper. We manual adopt the function labels and their related words as the verbalizer of prompts. This process does not require much human effort. Moreover, we propose to combine the manual prompt with automatically learned continuous prompt which provides multi-perspective representations and takes full advantage of knowledge in the PLM.

Our research objective is to use the text data of a scientific publication to recognize multi-granularity academic functions, e.g., the academic function of a citation, a abstract sentence, or a keyword. Formally, an academic function recognition training dataset can be denoted as $\mathcal{D}=\{\mathcal{X},\mathcal{Y}\}$, where $\mathcal{X}$ is the instance set and $\mathcal{Y}$ is the academic function label set. Each instance $x\in\mathcal{X}$ consists of several tokens ${x}=(w_{0},w_{1},\dots,w_{|{x}|})$ along with a class label $y\in\mathcal{Y}$. The common approach is to train a model on the dataset $\mathcal{D}$. Whereas in real-world scenarios, annotated data for scientific NLP tasks are usually scarce. For instance, each class has only a few dozen or even about 10 labeled instances. Suppose there is a set of unlabeled instances $\mathcal{N}$, $\mathcal{|N|}$ is typically much larger than the number of training instances $\mathcal{|D|}$. In this study, we explore an effective method to recognize multi-granularity academic function in low-resource settings.

To illustrate the effectiveness of MPT, we conduct extensive experiments on the following academic function recognition tasks of different granularities: (1) Citation Function Recognition; (2) Abstract Sentence Function Recognition; (3) Keyword Function Recognition.

Citation Function Recognition. We carry out our experiment on a citation function recognition dataset, the SciCite (Cohan et al., 2019), which is more than five times larger and covers multiple scientific domains compared with the ACL-ARC (Jurgens et al., 2016) dataset. Specifically, there are 11,020 instances in the dataset extracted from 6,627 papers in the Computer Science domain and Medicine. Since they utilize a concise annotation scheme, only three types of function labels, i.e., “Background”, “Method”, and “Result comparison”, are assigned.

Structure Function Recognition. We evaluate our method on the medical scientific abstracts benchmark dataset PubMed RCT 20k (Dernoncourt and Lee, 2017) for abstract sentence function recognition, where each sentence of the abstract is annotated with one label associated with the rhetorical structural (“Background”, “Objective”, “Method”, “Result”, and “Conclusion”).

Keyword Function Recognition. We conduct experiments of keyword function recognition task on PMO-kw dataset, which is a Chinese dataset proposed by Zhang et al. (2021) in computer science domain. PMO-kw contains 310,214 keywords extracted from 100,025 papers. Each keyword is annotated with one label associated with the domain-independent keyword semantic functions (“Problem”, “Method”, and “Others”).

A summary of statistics of these tasks and datasets are shown in Table 1.

Before introducing our proposed Mix Prompt Tuning, we first give some essential preliminaries about prompt tuning for academic function recognition tasks.

Fine-tuning. Let $\mathcal{M}$ be a language model (PLM) pre-trained on large scale corpora. Previous works adopting the PLM to recognize the academic function mainly utilized the further pre-training and fine-tuning paradigms. Gururangan et al. (2020) find that the PLM could perform better by applying the self-supervised pre-training on the target domain corpus. For the fine-tuning paradigm, the instance ${x}$ is firstly converted to the sequence $($[CLS]$,x_{0},x_{1},\dots,x_{|{x}|},$ [SEP]$)$ by adding special tokens, [CLS] and [SEP], into it. Then $\mathcal{M}$ is used to encode the converted sequence into contextualized vectors $(\bm{h}_{[\text{{CLS}}]},\bm{h}_{0},\bm{h}_{1},\cdot,\bm{h}_{|{x}|},\bm{h}_{[\text{{SEP}}]})$. The mainstream methods all adopt the contextualized hidden state vector $\bm{h}_{[\text{{CLS}}]}$ of $x_{[\text{{CLS}}]}$ as the representation of the whole input sequence. Then a dense layer with learnable parameters $\bm{W}$ and $\bm{b}$ is utilized to estimate the probability distribution of the instance ${x}$ with a softmax function:

, where ${\bm{p}}\in\mathbb{R}^{|\mathcal{Y}|}$. The parameters of $\mathcal{M}$, $\bm{W}$, and $\bm{b}$ are tuned to minimize the loss $-\frac{1}{|\mathcal{X}|}\sum_{{x}\in\mathcal{X}}log({\bm{p}}(y|x))$.

Prompt-tuning. As mentioned above, fine-tuning requires learning extra classifiers on top of the PLM under different classification objectives, which needs more annotated data and makes the model hard to generalize well. Recently, a series of works (Radford et al., 2019; Petroni et al., 2019; Brown et al., 2020) in general text classification tasks use prompt learning to bridge the gap between pre-training and downstream tasks, which make full use of PLMs by reformulating tasks as cloze-style objectives.

Formally, a prompt $\mathcal{P}$ consists of a template $\mathcal{T}$, label words $\mathcal{V}$, and a verbalizer $\phi$. For each instance $x$, the template is leveraged to map $x$ to the prompt input $x_{prompt}=\mathcal{T}(x)$, which is named as template wrapping. The template $\mathcal{T}(\cdot)$ defines the location and number of added additional tokens. Taking the abstract sentence function recognition task as an example, we set the template $\mathcal{T}(\cdot)=$ “$\cdot$ This sentence describes the [MASK] of the paper.”, and map $x$ to $x_{prompt}=$ “$x$ This sentence describes the [MASK] of the paper.”. At least one [MASK] is inserted into $x_{prompt}$ for the PLM $\mathcal{M}$ to fill the label words, while keeping the original tokens of $x$.

After the template wrapping, we can obtain the hidden vector $\bm{h}_{[\text{{MASK}}]}$ of the [MASK] from $\mathcal{M}$ by encoding $x_{prompt}$. Then we can utilize $\bm{h}_{[\text{{MASK}}]}$ to produce a probability distribution $\bm{p}([\text{{MASK}}]|x_{prompt})$ that reflects which tokens of $\mathcal{V}$ are suitable for replacing the [MASK] token:

where $\bm{e}$ is the embedding of the token $v$ in $\mathcal{M}$.

In prompt learning, there is a verbalizer as an injective mapping function $\phi:\mathcal{Y}\to\mathcal{V}$ that maps task labels to label words $\mathcal{V}$. For instance, we set $\phi(y=\text{``background''})\to\text{``literature''}$ and $\phi(y=\text{``method''})\to\text{``uses''}$. According to whether $\mathcal{M}$ predicts “literature” or “uses”, we can know the function of the instance $x$ is “background” or “method”. Note that for academic function recognition tasks, there may be a set of label token $\phi(y)=\mathcal{V}_{y}\subset\mathcal{V}$ that correspond to a particular label $y$. For instance, “tool”, “approach”, and “method” all indicate the “method” function in the citation function recognition task. Thus, the probability distribution of the [MASK] over $\mathcal{Y}$ can obtained by $\bm{p}_{\mathcal{T}}(y|x)=\sum_{v\in\mathcal{V}_{y}}\bm{p}_{\mathcal{T}}([\text{{MASK}}]=v|\mathcal{T}(x))$. The parameters of $\mathcal{M}$ are tuned to minimize the loss $\mathcal{L}$:

So far, we have shown how to reformulate an academic function recognition task as a language modeling task using prompts. Here, we propose the Mix Prompt Tuning (MPT) base on iPET (Schick and Schütze, 2021) framework. The proposed semi-supervised method can not only combine the expert knowledge to design the appropriate manual prompt templates of academic function, but also adopt automatically learned soft continuous prompt and manually designed hard prompt templates. In this way, manually designed templates can improve the performance lower bound and maintain the performance stability. Moreover, automatically learned soft continuous prompt might elicit more knowledge using “model’s language” to improve the performance upper bound. As for the verbalizer $\phi$, since each class in an academic function recognition task is clearly defined, in this paper, we utilize the same one verbalizer for all of the templates in one task.

Suppose we have a set of unlabeled instances $\mathcal{N}$, which is typically larger than the instance number of training set $\mathcal{D}$. First, we define a set $\Gamma$ of templates, which contains learnable soft continuous prompt templates, i.e., Prompt 3 and Prompt 4 in Figure 2, and manual designed fixed hard prompt templates, i.e., Prompt 1 and Prompt 2 in Figure 2. Then we fine-tune the PLM $\mathcal{M}$ based on one prompt template $\mathcal{T}\in\Gamma$ and the fixed verbalizer $\phi$ in the prompt tuning manner. With the original training instances $\mathcal{X}$ and the unlabeled instances $\mathcal{N}$, multiple PLMs tuned on prompt set can produce pseudo labeled training set by:

where $Z=\sum_{\mathcal{T}\in{\Gamma}}\omega(\mathcal{T})$ and $\omega(\mathcal{T})$ denotes the weight of template $\mathcal{T}$. The pseudo labeled training set is finally utilized to train a classifier.

Since there is high probability that PLMs tuned on some prompt templates perform worse than PLMs tuned on other prompt templates, to force them learn from each other, we further train several generations of models using unlabeled dataset and knowledge distillation strategy. Specifically, we formally denote the first generation PLMs tuned on $\mathcal{D}$ as $\mathcal{M}^{0}=\{\mathcal{M}_{1}^{0},\dots,\mathcal{M}_{i}^{0},\dots,\mathcal{M}_{|\Gamma|}^{0}\}$, where $\mathcal{M}_{i}^{0}$ is tuned with template $\mathcal{T}_{i}$ and verbalizer $\phi$. After $k$ generations training, we can get models $\mathcal{M}^{1},\dots,\mathcal{M}^{j},\dots,\mathcal{M}^{k}$, where $\mathcal{M}^{j}=\{\mathcal{M}_{1}^{j},\dots,\mathcal{M}_{i}^{j},\dots,\mathcal{M}_{|\Gamma|}^{j}\}$ and $\mathcal{M}_{i}^{j}$ is trained with $\mathcal{T}_{i}$ on training dataset $\mathcal{D}_{i}^{j}$. Note that we need to keep the proportion of classes in the dataset constant to avoid learning a biased distribution with iteration. Thus, we expand the size of the labeled dataset by a constant factor $d\in\mathbb{N}$, i.e., $c_{j}(y)=d\cdot c_{j-1}(y)$, where $c_{j}(y)$ denotes the count of instances with label $y$ in generation $j$ using template set $\Gamma$.

To obtain the training dataset $\mathcal{D}_{i}^{j}$, we randomly sample $\lambda\cdot(|\Gamma|-1)$ models $\mathcal{S}$ from previous generation $j-1$ tuned PLMs $\mathcal{M}^{j-1}$ except $\mathcal{M}_{i}^{j-1}$, where $\lambda\in(0,1]$. With the subset models $\mathcal{S}$, the unlabeled data can be annotated by formula 4 to construct a labeled dataset $\mathcal{D}_{\mathcal{S}}$:

Inspired by the findings (Guo et al., 2017) that instances predicted with high confidence are typically more likely to be classified correctly. As a result, we sample the instances with high probability score of the labels to avoid training next generation model on mislabeled data and improve the performance lower bound. Formally, for class $y\in\mathcal{Y}$, $\mathcal{D}_{\mathcal{S}}(y)\subset\mathcal{D}_{\mathcal{S}}$ is constructed by choosing the top $c_{j}(y)-c_{0}(y)$ scores of label $y$ instances from $\mathcal{D}_{\mathcal{S}}$. , the training dataset $\mathcal{D}_{i}^{j}$ is composed of the initial training set $\mathcal{D}$ and even sampled pseudo labeled training set $\bigcup_{y\in\mathcal{Y}}\mathcal{D}_{\mathcal{S}}(y)$, i.e., $\mathcal{D}_{i}^{j}=\mathcal{D}\cup\bigcup_{y\in\mathcal{Y}}\mathcal{D}_{\mathcal{S}}(y)$. The final enlarged dataset $\mathcal{D}^{\prime}$, annotated by the last generation models $\mathcal{M}^{k}$, is adopted to train a classifier.

As $\mathcal{D}_{\mathcal{S}}$ may not contain enough instances for a class $y$ under the extremely unbalanced label distribution, we obtain all $\mathcal{D}_{\mathcal{S}}(y)$ by choosing the highest $\bm{p}_{\mathcal{S}}(y|x)$ instances.

We now describe the templates and verbalizers adopted for each task. For all tasks, we add two or three soft learnable continuous tokens between the instance content and the [MASK] as soft prompt templates. Moreover, we manually design four hard templates for each task by combing the domain knowledge. Two of these manually designed hard templates are added with the task description. Specifically, for citation function recognition, we adopt “Citation function identifies the meaning or purpose behind a particular citation.” (Pride and Knoth, 2020) as the description and add it at the beginning of $\mathcal{T}_{1}$ and $\mathcal{T}_{2}$. For structure function recognition, we adopt “An abstract is divided into semantic headings such as background, objective, method, result, and conclusion.” (Dernoncourt et al., 2016). For keyword function recognition, we adopt “The functions carried by the keywords in the literature are problem, method and others (in Chinese).” (Lu et al., 2020). Since each class in an academic function recognition is clearly defined and does not require diversification, we construct the verbalizer of labels with respect to a class according to the class definition or description that reflect the expert knowledge. This process does not require much human effort. More details of prompt templates and verbalizers for all tasks are shown in Table 2.

To test the effectiveness of our method, we compare it with several types of recent models:

Supervised baselines. We compare several fine-tuning and prompt-tuning baselines in supervised manner, which have been proved the effectiveness on general text classification tasks in low-resource setting.

Fine-tuning. As PLMs have achieved promising results on various NLP tasks, a lot of efforts have been devoted to fine-tuning PLMs for text classification as well. In this paper, we select (1) BERT (Devlin et al., 2019), (2) RoBERTa (Liu et al., 2019), and (3) SciBERT (Beltagy et al., 2019) as the representative fine-tuning baselines. For fair comparison, we choose the base version of these models, e.g., BERT-base-uncased.

Prompt-tuning. We apply the regular prompt-tuning paradigm (described in subsection 4.1) with the hard and soft prompt templates (shown in Table 2) to form the (4) PT-hard and (5) PT-soft models, respectively. Since there are multiple templates, we tune the PLM with different prompt templates and report the best performance.

Semi-supervised and data augmentation baselines. Semi-supervised learning has been widely used in different NLP tasks combining with massive amount of unlabeled data to improve the model performance, as unlabeled data is often plentiful compared to labeled data. In this paper, we adopt five classic or previous SOTA methods for semi-supervised learning in NLP that rely on data augmentation: (6) Unsupervised Data Augmentation (UDA) (Xie et al., 2020) connects data augmentation with semi-supervised learning and outperforms previous SOTA. It employs back translation and TF-IDF to generate diverse and realistic noise and enforces the model to be consistent with respect to these noise. (7) TMix (Chen et al., 2020b) proposes to create virtual training samples by apply linear interpolations within hidden space, produced by PLMs, as a data augmentation method for text. (8) MixText (Chen et al., 2020b) leverages TMix both on labeled and unlabeled data for semi-supervised learning. The authors propose the label guessing method to generate labels for the unlabeled data in the training process. (9) PET (Schick and Schütze, 2021) is a semi-supervised training procedure that reformulates input examples as cloze-style phrases by prompt templates and verbalizers to help leverage the knowledge contained in PLMs. Along with PLMs, these well-designed templates are used to assign soft labels to unlabeled examples. A standard classifier is trained based on the original labeled data and soft labeled data. (10) iPET (Schick and Schütze, 2021) is the iterative variant of PET. It trains several generations of models using PET manner on datasets of increasing size so that classifiers can learn from each other. We adjust the open-sourced implementations of UDA ²²2https://github.com/SanghunYun/UDA_pytorch, TMix, MixText³³3https://github.com/GT-SALT/MixText, PET, and iPET⁴⁴4https://github.com/timoschick/pet to conduct academic function recognition tasks on corresponding datasets.

Since label distributions of academic function recognition datasets are imbalanced, we conduct comparative experiments on balanced and imbalanced label distributions under the low-resource setting. Previous baselines, such as MixText and PET, only conduct the experiments on balanced sampled training set. Moreover, taking the unlabeled dataset of MixText as an example, it is unrealistic and impractical that the unlabeled dataset is carefully selected and balanced sampled. Thus, to conduct the experiments on balanced label distribution, we randomly sample $K=\{4,8,16,32,64,128\}$ instances in each class from the training set and test the model on the entire test set. Whereas, for the experiments on imbalanced label distribution, i.e., original label distribution, we randomly sample the same number of training instances ($K*|\mathcal{Y}|$) with the experiments on balanced label distribution by keeping the original label distribution. For all datasets, we use the macro F1 score and the Accuracy.

All our models and baselines are implemented with PyTorch framework (Paszke et al., 2019) and Huggingface transformers (Wolf et al., 2020). All of the models related to prompt learning are also implemented with the Open-Prompt toolkit (Ding et al., 2021). We fine-tune the PLMs with the AdamW optimizer (Loshchilov and Hutter, 2018). Previous study (Gao et al., 2021) find that, with a large validation set, a model could learn more knowledge and hyperparameters could also be optimized. Different from the experimental settings of Xie et al. (2020) and Chen et al. (2020b) that adopt large validation sets to optimize the hyperparameters, to keep the initial goal of learning from limited data, we assume that there is no access to a large validation set and the size of validation set is the same as training set. For citation function recognition task and structure function recognition task, we use SciBERT (Beltagy et al., 2019) as our PLM backbone. Whereas for keyword function recognition task, since there is only Chinese dataset, we use bert-base-multilingual-cased⁵⁵5https://huggingface.co/bert-base-multilingual-cased as our PLM backbone. We use a learning rate of $1\cdot 10^{-5}$, a batch size of 16, a maximum sequence length of 128 for citation function recognition and structure function recognition, and a maximum sequence length of 256 for keyword function recognition. Same with the PET framework, we set $\lambda=0.25$ and $d=5$. For the comparative experiment of semi-supervised methods, the number of unlabeled instances is selected from $\{600,800,1000\}$. We tune the entire model for 6 epochs under 3 different random seeds and report the best test performance.

In this subsection, we introduce the specific comparative results and provide possible insights of our proposed MPT under two settings, i.e., balanced sample $K$ instances ($K$-shot) in each class and randomly sample $K*|\mathcal{Y}|$ instances from the original training set.

We also conduct experiments, which are shown in Figure 3, to test our model performances with 16 instances per class and different amount of unlabeled data (from 200 to 1000) on Rct-20k, SciCite, and PMO-kw datasets. We can observe that, although for different datasets, models vary in the choice of optimal unlabeled instance size. Overall, with more unlabeled data, the overall performance of MPT becomes much higher. It validates the effectiveness of our proposed MPT in making full use of unlabeled data.

In this paper, we focus on adopting SciBERT as the backbone of MPT in the main experiments. Can we further extend MPT to other PLMs like BERT and RoBERTa with different pre-train corpus and strategy? To achieve this, we replace the SciBERT or Multilingual BERT in MPT into BERT and RoBERTa. Meanwhile, we keep the same prompt templates and verbalizers with the SciBERT based MPT. As shown in Figure 4, it depicts that MPT performs significantly better than fine-tuning when data is extremely scarce. As the volume of data grows, BERT fine-tuning shows comparative performance with BERT based MPT. Moreover, part of the performance improvement of RoBERT or SciBERT based MPT over BERT based MPT may be due to the contribution of pre-train knowledge. We will further discuss it in the next subsection.

Compared to supervised baselines, our proposed MPT and other semi-supervised baselines utilize SciBERT or multilingual BERT. As a result, part of the performance improvement might come from the additional in-domain pre-train corpus. Thus, we compare BERT and SciBERT/Multilingual-BERT based MPT with BERT and SciBERT fine-tuning to test the effect of in-domain pre-training. SciBERT utilizes the same structure as BERT but is pre-trained on scientific domain data, as in-domain pre-train is a common way to improve the model performance (Gururangan et al., 2020; Sung et al., 2019). As shown in Figure 5 shows results of fine-tuning and MPT both using BERT and SciBERT. We can observe that, compared with BERT based MPT, although SciBERT fine-tuning achieves significant better performance when data is extremely scarce and comparative performance with the volume of data grows, it still performs worse than MPT. This observation indicates that MPT not only makes full use of the unlabeled data but also stimulates the pre-train knowledge effectively.

The rapid development of large language models (LLMs), exemplified by ChatGPT, has attracted extensive attention. These massively PLMs, which undergo instruction fine-tuning and align with human intent, have demonstrated excellent performance not only in generative tasks, such as question answering and machine translation, but also in diverse tasks, such as information extraction, text classification, and reasoning when guided by prompts. Therefore, we further compared the 4-shot performance of our proposed MPT with the best 3 to 5 shot best performance of mainstream LLMs, e.g., GPT-4, GPT-3.5-turbo, ERNIE-bot, and ChatGLM2, in the few-shot scenario. Consistent with previous works (Zhang et al., 2022b; Dong et al., 2022; Xu and Zhang, 2024; Liu et al., 2024), we evaluate the performance in an In-Context Learning manner. As shown in Table 7, MPT significantly outperforms these models.

Our proposed MPT can achieve superior performance for citation function recognition, structure function recognition, and keyword function recognition in different low-resource settings. Previous methods on academic function recognition tasks can mainly be summarized in three categories: (1) a machine learning method based on hand-crafted features (Jurgens et al., 2016; Lu et al., 2018; Nanba et al., 2010); (2) a deep learning model trained from scratch (Cohan et al., 2019; Qin and Zhang, 2020; Wang et al., 2020; Cheng et al., 2021); (3) a PLM fine-tuning method (Beltagy et al., 2019; Huang et al., 2022; Lu et al., 2020; Zhang et al., 2021). Those methods require a large number of annotated data to achieve competitive performance. Our proposed method is based on prompt learning, which is a new paradigm of utilizing PLMs (Liu et al., 2021a) and has great potential in low-resource scenarios. It utilize prompts to bridge the gap of objective forms in pre-training and fine-tuning, which leads to more effective utilization of pre-train knowledge than the standard fine-tuning. Moreover, we introduce prompt learning with mixing multiple types of prompt templates. Whereas previous s studies in other domain tasks are solely based on manual hard templates or automatically learned soft templates. Our comparative experiment results between fine-tuning and prompt learning in Section 5.3 further validate its effectiveness for low-resource academic function recognition tasks.

Our proposed MPT is a semi-supervised method that utilizes multiple prompt templates to annotate unlabeled data. The semi-supervised framework is similar to iPET (Schick and Schütze, 2021), one of the strong semi-supervised baselines. Differently, compared with this method that only adopts the manual prompt, we combine the manual prompt with automatically learned continuous prompt, which can provide multi-perspective representations and take full advantage of knowledge in the PLM and unlabeled data.

Moreover, existing works lack attention to scientific classification tasks in low-resource settings. Since obtaining the annotated data for scientific NLP tasks is still challenging and expensive, it is common in real-world scenarios that there is usually no annotated data or only a small number of annotated instances. To alleviate the dependence on annotated data for scientific classification tasks, we propose the MPT, which combines multiple prompts with a semi-supervised framework. Extensive experiments on a series of academic function recognition tasks at different granularities prove the feasibility of MPT.

This study has the following implications. First, in the practical scientific NLP scenario, there is a contradiction between the massive unlabeled scientific publications and the scarcity of annotated data. To this end, we propose MPT, a semi-supervised and prompt learning based solution coping with practical low-resource academic function recognition scenarios. Second, the prompt learning paradigm is promising for low-resource scientific NLP tasks. Moreover, MPT is a semi-supervised solution and fuses the manual prompt with automatically learned continuous prompt. It provides multi-perspective representations and takes full advantage of knowledge in the PLM and unlabeled data resources. Third, our proposed MPT has the ability to perform multi-granularity academic function recognition. Moreover, the MPT presented in this study is a general approach that can be easily deployed in other scientific NLP tasks with minor adjustment to the prompt templates and verbalizer. Finally, MPT is a method for low-resource scenario which can be considered a type of Green AI approach, as they aim to develop and use models in a way that is more resource-efficient. By using the method, it is possible to build and train models with less data and compute power, which can reduce the environmental impact of the AI system.