Adaptive Prompt Learning-based Few-Shot Sentiment Analysis

Sentiment analysis originates from the analysis of subjectivity in sentencesbib1 . Due to the emergence of a large number of network resources, the research of sentiment analysis has become an active field since 2000bib2 . Early sentiment analysis mainly focused on building an sentiment dictionary for text classification. It was constructed manually by summarizing words containing sentiment tendencies, and labeling the sentiment polarity and intensity of these words to varying degrees. Therefore, it is necessary to build a high-quality sentiment dictionarybib3 . Due to the flexibility and non-standard of language, it is difficult to construct a general and efficient rule applicable to all contexts. Machine learning based sentiment analysis mainly relies on NLP researchers or engineers to use their domain knowledge to define and extract significant features from the original data, such as n-gram features, and then use traditional machine learning classifiers such as support vector machine, naive Bayes and maximum entropy for supervised learningbib4 .Li, Gbib5 builds a model with the prior knowledge of the categorization information in order to extract meaningful features from the unstructured texts by using TF-IDF, short for term frequency-inverse document frequency.

In recent years, with the development of deep learning theory, neural network has gradually matured in the field of sentiment analysis. Deep neural network can effectively capture the high-level semantic information of text without complex feature engineering, and the expression ability index of the model is times better than that of the shallow model. Among them, convolutional neural network and recurrent neural network are the most widely usedbib6 .Li, Dbib7 proposed the BLSTM and CNN Stacking Architecture (BCSA) to enhance the ability to recognition emotions. Besides, Chenbib8 propose HUSN which have a novel sentiment classification algorithm that utilizes user’s review habits to enhance hierarchical neural networks.Sadr, Hbib9 proposed model employs recursive neural network due to its tree structure as a substitute of pooling layer in the convolutional network with the aim of capturing long-term dependencies and reducing the loss of local information.

The purpose of pre-trained language models (PLMs) is to use a large number of texts that have appeared in people’s life to train the model, so that the model can learn the probability distribution of each word in these texts, so as to model the model that conforms to these text distributions.

Traditional PLMs technology aims to to learn word embedding. Because downstream tasks no longer need to use these models, they are usually very low in computational efficiency, such as skip grambib10 and glovebib11 . Although these pre-trained word vectors can capture the semantic meaning of words, they are context independent and can not capture the high-level concepts of text, such as grammar and semantics. Elmobib12 proposed a context sensitive text representation method, which constructs the text representation through the deep bidirectional language model, which effectively solves the problem of polysemy. In 2018, Devlin et albib13 . proposed BERT (bidirectional encoder representations from transformers) pre-trained language model. The model trains massive corpus through bidirectional transformer encoder and uses masked language model (MLM) to generate in-depth bidirectional language representation. After pre-training, you only need to add an additional output layer for fine-tuning to achieve the performance of state of the art in a variety of downstream tasks. In this process, there is no need to make task specific structural modifications to Bert.

Bert has opened a new era, and a large number of pre-trained language models have emerged since then. For example, Robertabib14 retains the original Bert architecture with longer training time, larger batch, longer sequence and more data. At the same time, delete the prediction of the next sentence and use dynamic masking. Albertbib15 solves the problems of higher memory consumption and slow Bert training speed. ERNIEbib16 introduced the knowledge mask strategy, including entity level mask and phrase level mask, to replace the random mask in Bert. In addition,Bert also can be used in other languages. Farahanibib17 proposed a monolingual Bert for the Persian language which is lighter than the original multilingual Bert model. There is Bert-wwmbib18 for Chinese which is not only a continuous mask of entity words and phrases, but also a continuous mask of all words that can form Chinese words.

With the increasing volume of pre-trained language model, the hardware requirements, data requirements and actual cost of fine-tune are also rising. In addition, the rich and diverse downstream tasks also make the design of pre-training and fine-tuning stage cumbersome and complex. Therefore, researchers hope to explore smaller, lighter, more universal and efficient methods. Prompt method is an attempt in this direction which include hand-crafted prompt method and automated prompt. Prompt method is a technology that adds additional text to the input segment in order to better use the knowledge of the pre-trained language model. Schick Tbib19 et al. designed Pattern Exploiting Training(PET), which is a semi-supervised training task. The input example is redefined as the phrase of cloze to help the language model understand the given task. Jiang et al.bib20 proposed a mining based method, which can automatically find a given set of training input and output templates. This method finds the intermediate word or dependency path between input and output in a large text corpus containing input and output strings, and uses the frequently occurring intermediate word or dependency path as a template.

Davison et al.bib21 designed an input (head relation tail) template using LM by studying the tasks related to the knowledge base. Liu x et al.bib22 proposed a method called P-tuning, abandoned the conventional requirement that “the template is composed of natural language", used the token never seen in the model to form the template, transformed the construction of the template into a continuous parameter optimization problem, and realized the automatic construction of the template.

In the hand-crafted prompt method, the accuracy of the model depends very much on the quality of the constructed template, and the effects of different templates may vary greatly. For some tasks, it is not so easy to discover an optimal prompt manually. In the automated prompt method, the prompt is constructed by the model and does not rely on manual work. But both of those method can not use the semantic information of input texts in prompt construction process.

To solve the above problems, this paper proposes a template construction method by introducing seq2seq-attention structure which can dynamically generate matching template vectors and makes full use of the original text information. At the same time, with the idea of pre-trained model, we design a template construction strategy based on pre-training, which can make full use of the public sentiment analysis datasets of high resources fields and apply it to the in other field of low resources.

The main contributions of this paper include three aspects:

1. [1.]

2. 1.

   We propose an adaptive prompt method by introducing seq2seq-attention structure. This method has the advantages of both hand-crafted prompt and automated prompt and can make full use of semantic information of input text.

3. 2.

   The experimental results on the FewCLUE datasets show that the proposed method is effective in the sentiment analysis few-shot task.

4. 3.

   We proposed to pre-train the adaptive prompt module in high resources tasks, and migrate to or fine-tune in low resource tasks which can effectively play a significant effect in low resource tasks.

Our work is based on adaptive prompt learning model, which improve traditional hand-crafted prompt learning method (HPL). The HPL model includes input layer, hand-crafted prompt, pre-trained language model and output layer.

Given a pre-trained model M, vocabulary V, a input sequence X of length n:{x₁,x₂…x_n}, verbalizer W:{w₁,w₂…w_l}, a hand-crafted prompt P:{p₁,p₂…p_i,[MASK],p_i+1…p_m}, where value of [MASK] comes form W. Firstly, The sequence input X, and manual prompt P form a template t. Then, the template t will be mapped into e(t): {e(p₁),e(p₂)…e(p_i),e([MASK]),e(p${}_{i+1})$…e(p_m),e(x₁),e(x₂)…e(x_n)},by pre-trained model embedding layer e(where each token p₁ will be mapped into e(p₁)).After that, the pre-trained model M is used to calculate the probability values of [MASK] in e([MASK]) to select the best word in W which has the maximum probability. For example, for the sentiment calculation of “The weather is very good", hand-crafted prompt “it is [MASK] ", verbalizer {“good”, “bad”}, and then the traditional prompt model will construct the template “It is [MASK], The weather is very good. " Finally, M will return the predictive value. As shown in Figure 1(a).

While the strategy of traditional hand-crafted prompt learning is intuitive and work well in some sentiment tasks, there are also two issues with this approach. 1) It is hard for human to discover optimal prompts in all tasks. 2)Even if in the same task, the optimal prompts of different input sequences are different. Usually, it is hard for algorithms to dynamically found “best” prompt for a special input sequences in task. We consider automatic adaptive prompt design to solve the defects of manual prompt design. As shown in figure 1(b), we use adaptive prompt layer to generate prompt instead of manual craft prompt. In order to strengthen the relationship between the prompt and the input sequence X, we consider to use the text information of the input sequence X to automatically generate the corresponding prompt, that is, generate an adaptive prompt. (in traditional automated method, there is no direct correlation between prompt and input X, we trained the adaptive prompt using the context information of input X). In this work, we use seq2seq-attention structure as adaptive prompt layer to generate adaptive prompt. The seq2seq structure can generate target sequence (prompt sequence) by a specific method according to a source sequence(words embedding vectors of input X) which are suitable for generating prompt sequence. Meanwhile,an attention structure is introduced to better capture the semantic details of input X as shown in Figure 2.

In general, seq2seq-attention structure as adaptive prompt layer to generate a vector sequence h with a length of s consist the adaptive prompt from input X. And each vector dimension of the sequence is consistent with the output vector dimension of the embedding layer of the pre-trained model. Meanwhile, the adaptive prompt’s embedding vectors are continuous which enables us to find a better continuous prompt beyond the original vocabulary could expressbib22 . In addition, adaptive prompt layer can capture text information of input x by attention structure(the yellow part in the figure 2), which can make the generated prompt fitter with the input text.

Although automated prompt has various advantages in most tasks, such as wide application range, strong generalization ability, stable and balanced performance, the algorithm may falls into local optimal solution in many cases. The hand-crafted prompt has excellent performance in some cases, but this method requires experienced expert participation and is unstable. In order to combine the advantages of the two methods, we design a Hybrid prompt composed of a hand-crafted part and automated part as shown in Figure 3.

The hybrid prompt model combines the word vector generated by hand-crafted prompt and the prompt layer. Through the hybrid prompt embedding layer, the template can be represented a triple $<$ X, P, h $>$= {e(p₁),e(p₂)…e(p_i),e([MASK]),e(p${}_{i+1})$…e(p_m), h₁,h₂…h_s, e(x₁),e(x₂)…e(x_n)}

where P is the hand-crafted prompt, h is the adaptive prompt embedding vector sequence , X is the input text sequence. And the final model prediction result set $y=\{p(i),i\in W\}$ is calculated by pre-trained model.

Both adaptive prompt and hand-crafted prompt effect the final result y. The result showed that(4. 4. 1 for details), the model can learn to adjust the weights of P and h to generate better output results. Therefore, theoretically, the model has the advantages of both hand-crafted prompt and automatic prompt. When a “good” hand-crafted prompt can be found, the model effect can be further improved. Even if a “good” hand-crafted cannot be found, an excellent prompt can be generated by the adaptive prompt part.

We evaluate AP mainly on the FewCLUE public datasets EPRSTMT(E-commerce Product Review Dataset for Sentiment Analysis) task which is labelled as Positive or Negative and collect by ICIP Lab of Beijing Normal University. The datasets used in migration experiment include social media public sentiment datasets(more than 100000 data with emotional labels on Sina Weibo, and about 50000 positive and negative comments respectively), hotel comment data(more than 7000 hotel review data, more than 5000 positive and 2000 negative reviews), user comments data by a take-out platform( 4000 positive and 8000 negative user comments collected by a takeout platform), online shopping data which have 7 categories (books, fruits, shampoo, water heater,milk, clothes and hotels) and more than 60000 comments in total, with about 30000 positive and negative comments respectively. The data in English field include 7000 movie datasets with about 3500 positive and 3500 negative data respectively.

In order to fully obtain all the information of the sentence, this paper sets the maximum length of the sentence to twice the length of the digits in the data set. In the experiment, the pre-trained model adopts Roberta-wwm-ext as pre-trained language model. The batch size value is set to 5 and output length of adaptive prompt layer is set to 2 in Chinese and 4 in English, The model adopts Adam optimizer and adopts different learning rates for different optimization methods.

The model consists of two trainable parts, one is the pre-trained model parameters, the other is the seq2seq-attention parameters, that is, the adaptive prompt layer. Based on this, our optimization methods can be divided into two categories: one is to fine-tune all parameters (prompt+LM tuning).In this setting, there are prompt-relevant parameters, which can be fine-tuned together with the all or some of the parameters of the pre-trained modelsbib23 . And the other is to fine-tune only the seq2seq-attention part (fixed LM prompt tuning). In the scenario where additional prompt-relevant parameters are introduced besides parameters of the pre-trained model, fixed-LM prompt tuning updates only the prompts’ parameters using the supervision signal obtained from the downstream training samples, while keeping the entire pre-trained LM unchangedbib23 .