Comparison Study Between Token Classification and Sequence Classification In Text Classification

Text classification engines has been used as an alternative and complement to human annotation for many years. Essay tests are an example of a text responses where students are given a particular prompt topic to write about and elaborate on. These texts are evaluated for their writing quality in areas such as grammar, fluency of elaboration and how well they are organized. However, evaluating texts in large amount is impractical and classifying it is mostly very complicated and difficult [1].

Text classifications engines can’t genuinely read and comprehend texts the way humans or raters can. Text classification engines do not use direct qualities of a text such as how fluent it is or how grammatically correct. But human raters may explicitly evaluate numerous of those intrinsic variables to produce a prediction. This gives a clear distinction between machine scoring and human labeling. In [2] they studied whether human scoring or machine scoring classification models are better at classifying scientific research abstracts according to a fixed set of discipline groups. Text engines are mostly trained by collecting generally a few couple of thousand sample text which were previously assessed by human raters. There has always been skepticism over machine scoring over time mostly in regard to the fact that computer models does not understand the written text as they human does [3]. However, by looking at labeling between human raters and machine models and create a an statical metric like Cohen’s Quadratic Weighted Kappa score we could measure the quality of scoring models (human vs machine) as long as we prove the sample data is statistically rules significant.

There are two major categories of text classification engines available in the field natural language processing: traditional and neural network-based models. Traditional text classification engines examine a text using carefully crafted features. The effectiveness of such models is closely correlated with the caliber of the underlying components. The most important element of such engines is manually designing the features and rules, which is time-consuming. A feature free method for learning the relationship between a text assigned label in [4] investigates the use of recurrent neural network model as a modern methods. For the purpose of text classification engine, various neural network models are used in the field [5, 6, 7, 8].

In comparison to statistical models with handmade characteristics, neural network models have recently been utilized for text classifiers [9, 10, 4] have produced better results. Word representations in particular are utilized as the input to a neural network model which produces a single dense vector for each word representation in the whole text. Based on a non-linear transformation in each layer on the representation, a label is given. It has been demonstrated that across several domains, neural network models are more reliable than statistical models without the use of handmade features.

Text classifiers engines have been using both convolutional neural networks [11] and recurrent neural networks [12] modles. A single-layer LSTM also has been used over the word sequence to model the text, and they separately model sentences and documents using a two-level hierarchical CNN structure. It is well known that LSTMs excel at modeling temporal and ordered data while CNNs excel at capturing local information. The effectiveness of LSTMs and CNNs using the identical text classification settings has been discussed [10].

Pre-trained language models have recently been used to significantly increase the performance of phrase prediction tasks by combining representations from many layers, creating an auxiliary sentence, using multitask learning, etc.

Large pre-trained language models have recently demonstrated amazing abilities in representation and generalization, like GPT [13], BERT [14], XLNet [15], etc. These models now perform a variety of downstream tasks including text summarization, generation, classification, and regression with enhanced accuracy. There are numerous innovative methods for improving already-trained language models. Auxiliary sentence construction was suggested by [16] to improve sentiment classification problems. To address sequential sentence classification tasks, Cohan [17] introduced additional distinct tokens to obtain representations of each sentence. A work in [16] on several fine-tuning techniques, such as combining text representations from various layers and leveraging multi-task learning has been done.

It makes sense to use BERT type models or generally transformer-based models to learn text representations given the enormous success of pre-trained language models like BERT in learning text representations with deep semantics. The BERT approach, which relies heavily on self-attention, can capture interactions between any two words throughout the entire body of articles (long texts). Previous work on [16] demonstrates that merging text representations from various layers does not significantly increase performance. The length of texts for the text classification job is close to the maximum allowed by the BERT model, making it challenging to create an auxiliary phrase.

In this article, we suggest using two type classification heads for AES engine to our base model to learn text representations that fully reflect deep semantics. These two models are Token Classifier and Sequence Classier models. A token classifier uses all the states of model (features that are transferred over many layers of pre-trained models) and maps all the numerical representation into a label, however the Sequence Classifier utilize a slice of information from all the states and maps the representations into a label.

In summary, our contributions are:

• •

  Investigate the use two different classifier from different in nature in text classification engines on a standard dataset which allows us to compare results.

• •

  Explore the advantages and disadvantages of each method.

• •

  To label long texts which are longer than 256 token using token classification method.

Modern natural language processing systems learn effective models with the help of annotated data as supervision. These models can only be utilized within that language because they are typically trained on data that is only available in that language (typically English). Since it is impractical to collect data in every language, cross lingual language understanding (XLU) and low-resource cross-language transfer have gained popularity. The Cross-lingual Natural Language Inference (XNLI) corpus is the extension of the Multi-Genre NLI (MultiNLI). There are 122k train, 2490 validation, and 5010 test samples in the dataset [18] in all 15 languages. The model is trained on the training set and the test set is used to find the best hyper-parameter and model. The validation set is used is an unseen set which is used later for testing the performance of the classifiers.

Table 1 shows basic statistics about XNLI dataset on just English data such as the number of samples in each set, number overall tokens in all samples, number of texts whose number of tokens is greater than 256 and finally the distribution of all labels. XNLI dataset is used as sample set of data for testing the hypothesis of scoring regime on two different methods. The dataset is chosen since the label distribution and mean of labels are perfectly balanced.

Table 2 shows a sample set of text data which we used for classification purpose. Premise and Hypothesis text are combined and construct our sample text. The labels are integer encoded numbers which zero represents entailment, one represents neutral and 2 represents contradiction. This sample would be suitable for comparison study about the different nature of transformer classifiers.

The development of neural network-based natural language processing has had a significant impact on the current possibilities for text classification engines. A few preliminary corresponding comments on the meaning of some of the terms used as well as some additional methodological background will be provided beforehand because the terminology used in a machine learning context is frequently different from that used in psychometrics and is also occasionally ambiguous. Following that, more background information is provided along with two significant NLP techniques based on neural networks and their accompanying AES successes.

Powerful neural network architectures known as Transformers [24, 25, 26] have achieved state of art status in a number of machine learning domains, including Natural Language Processing (NLP), Neural Machine Translation (NMT) [27] and text generation summarization. Unfortunately, a conventional Transformer scales quadratically with the number of tokens, making it unaffordable for a sequence larger than 512 tokens. For this problem, several solutions have been proposed in [28, 29, 30, 31]. Most methods limit the attention mechanism to focus on nearby neighborhoods or include structural priors on attention such as sparsity, pooling-based compression, clustering, or convolution techniques.

A dataset must first be preprocessed into an input format before it can be used to train a model. Your data must be transformed and put together into batches of texts. This preprocessing method is called tokenization, we use a tokenizer to break text into a series of tokens, then turn those tokens into numbers and put them together into certain ID numbers. Transformer architectures are trained on the large set of data with a restriction on the size of tokenized input. Most of transformers have a limit of 512 input tokens.

In this work we are not changing the model architecture to address the issues of long texts by increasing the limit size, however we are proposing to use the 256 limit size architecture using a method using overlapping segments. In the Token Classier model, we are introducing an stride hyper parameter that is the amount the text which we are overlapping. A single long text will be chunked and as $N$ number of texts of 256 token size, in such way that we track the overflow tokens.

In Figure 1, the text containing 532 tokens, was passed through a tokenizer that returned tokenized inputs of length 512. Since the number of tokens in the text was more than the max length, the text was truncated to length 512, and the remaining tokens were put in the next entry. The last 20 token ids of the first entry were also present at the beginning of the second entry due to the use of the stride parameter. Since the second entry falls short of the max length of 512, padding was added to the second entry.

The results of scoring using Sequence Classification and Token Classification are compared. First the pre-trained Electra model [36] is fine-tuned on training set. A grid search technique is used to find a best model on our metric. QWK metric is our main metric to evaluate the performance of the model. The hyper parameter that we are optimizing our models are:

• •

  Learning Rate

• •

  Batch Size

• •

  Stride

In this work, the performance between the Sequence Classifiers and Token Classifiers are studied. The token classification model has better performance on text classification and labeling texts. The use of all of the prediction sequences that models’ outputs help to extract more information for predicting labels. The sequence classifier is the most common used model for scoring but its performance is not as well as the token classification method, because sequence classifier just uses a first sequence prediction token which ignores all the other sequences information and then maps the texts into its label. Certainly, the use all the sequences that are extracted from the features provided better performance because the base model are exactly the same for the this study . For future work, different type of aggregation in token classification can be explored instead of using a mean of label for every single token. Also, we could verify the performance on the Kaggle essay data set.