Reed at SemEval-2020 Task 9: Fine-Tuning and Bag-of-Words Approaches to Code-Mixed Sentiment Analysis

The Internet today has a vast collection of data in various forms, including text and images. A big part of this data comes from various social media platforms, which enable users to share thoughts about their daily experiences. The millions of tweets, updates, location check-ins, likes, and dislikes that users share every day on different platforms form a large bank of opinionated data. Extracting sentiment from this data, though immensely useful, is also challenging, giving rise to the NLP task known as sentiment analysis. Several models have been proposed to perform this task over the years, such as those built on top of the Recursive Neural Tensor Network [Socher et al., 2013] or the more recent BERT model [Devlin et al., 2018]. Although most of these language technologies are built for the English language, a lot of Internet data comes from multilingual speakers who combine English with other languages when they use social media. Thus, it is also important to study sentiment analysis for this so-called ‘code-mixed’ social media text. In this paper, we explore sentiment analysis on code-mixed Hinglish (Hindi-English) tweets, specifically as a participant in Task 9 of SemEval-2020 [Patwa et al., 2020].

We had two main strategies. We began by fine-tuning pre-trained BERT models to our target task. During this initial phase, we observed that the accuracies yielded by bert-base, bert-multilingual and bert-chinese on the validation data were approximately the same. This made us hypothesize that the pre-trained weights were mostly unhelpful for the task. To test this, we attempted to recreate the BERT fine-tuning results only using feedforward neural networks trained on a bag-of-words (BoW) representation.

We found that the the results of fine-tuning bert-large (24 layers) could be approximated by a 2-layer BoW feedforward neural network, as our best-performing fine-tuned model had an accuracy of 63.9% and our best-performing bag-of-words model had an accuracy of 60.0% on the validation corpus. In the evaluation phase of the competition, our fine-tuned model had an F-score of 69.9% without bagging [Breiman, 1996] and 71.3% with bagging. Once the official system rankings were published on April 6, 2020¹¹1see https://competitions.codalab.org/competitions/20654#learn_the_details-results, our best submission (CodaLab username: gopalanvinay) placed 4th out of 62 entries. All code needed to recreate our results can be found here²²2see https://github.com/gopalanvinay/thesis-vinay-gopalan.

The organizers of SentiMix simultaneously organized a Hinglish and an analogous Spanglish (code-mixed Spanish-English) task [Patwa et al., 2020]. We participated in the Hinglish track.

The task was to predict the sentiment of a given code-mixed tweet. Entrants were provided with training and validation data. These datasets were comprised of Hinglish tweets and their corresponding sentiment labels: positive, negative, or neutral. Besides the sentiment labels, the organizers also provided language labels at the word level. Each word is tagged as English, Hindi, or universal (e.g. symbols, mentions, hashtags). Systems were evaluated in terms of precision, recall and $F_{1}$-measure.

All data was provided in tokenized CoNLL format. In this format, the first line (prefaced with the term “meta”) provides a unique ID and the sentiment (positive, negative or neutral) of the tweet. Each tweet is then segmented into word/character tokens and each token is given a language ID, which is either HIN for Hindi, ENG for English and O if the token is in neither language. Below is an example for a positive-sentiment tweet with id 173:

It means sidhi sadhi ladki best couple (positive)

For the experimental results reported in this paper (except for the final system results, which use the official competition test set), we trained our systems using the provided training set of 14K tweets and report results on the provided validation set of 3K tweets. Since the task is a relatively balanced three-way classification task, we used simple accuracy as our hillclimbing metric.

We used PyTorch v1.2.0 ³³3see https://pytorch.org/docs/1.2.0/ and Python 3.7.1⁴⁴4see https://docs.python.org/3/. All additional details needed to replicate our results are provided in the README of our project’s code repository ⁵⁵5see https://github.com/gopalanvinay/thesis-vinay-gopalan.

For our baseline experiments, we used the PyTorch-based implementation of BERT [Devlin et al., 2018] provided by Huggingface’s transformers package [Wolf et al., 2019]. We adapted the SST-2 task of the run_glue.py script. The original task was configured for the binary classification of sentences from the Stanford Sentiment Treebank [Socher et al., 2013], as administered by the General Language Understanding Evaluation (GLUE) benchmark [Wang et al., 2018]. We adapted the task for three-way (positive, negative, neutral) classification.

The fine-tuning results suggested two possibilities:

1. 1.

   Even though the pre-training makes little difference, the complex Transformer model still learns deep and interesting patterns to achieve an accuracy in the low sixties.

2. 2.

   The Transformer model is only learning simple heuristics that could be just as easily learned by simpler models.

To create a BoW system, we iterated through the training data and stored the frequency of each word in the corpus. We then created a vocabulary $\textbf{V}=\{v_{1},v_{2},...,v_{n}\}$, which was the set of words that appeared with frequency at least $K$ in the training data, for some positive frequency threshold $K\in\mathbb{N}$. After removing stop words from the vocabulary, we used this vocabulary to transform each tweet into an $n$-length vector whose $j^{th}$ element equaled 1 if word $v_{j}$ $\in\textbf{V}$ appeared in the tweet (and 0 otherwise).

We then trained a simple classifier using these BoW vector representations of the tweets. For our classifier, we used a standard feedforward neural network with $M$ layers and hidden size $H$. We built and trained the NNs using PyTorch [Paszke et al., 2019]. In order to classify tweets into positive, negative and neutral, the final layer applies the softmax function to the 3-dimensional output of the NN, which normalizes the output into a probability distribution over the three possible sentiments of the input tweet. We used a cross-entropy loss function for training.

Because we obtained slightly better performance with the fine-tuned BERT model, we used this as the basis for our competition system. To improve its performance, we experimented⁷⁷7We also experimented with the roberta-base model [Liu et al., 2019], but were not able to improve of BERT’s results. with various hyperparameter settings, including learning rate, weight decay, max_grad_norm and Adam epsilon. Our best validation result of 63.8% was yielded by bert-large-cased using a learning rate of $2\times 10^{-5}$, a weight decay of 0, a max_grad_norm of 1, and an Adam epsilon parameter of $1\times 10^{-8}$. This system became our first submitted system, achieving an F-score of 69.9% according to the official scoring script.

For our second submission, we used bagging [Breiman, 1996] to make our BERT classifier more robust. We created 10 bootstrap samples from the given training data. We then trained 10 instances of the bert-large-cased model on these bootstrap samples, and then combined the results of all the models by voting, i.e., if a plurality of the 10 models predicted a particular tweet as positive (negative/neutral), then the tweet’s label is deemed as positive (negative/neutral). Our bagged system obtained an F-score of 71.3%, placing us $4^{th}$ overall in the competition out of 62 entrants.

In this paper we investigated sentiment analysis on code-mixed Hinglish (Hindi-English) tweets as participants of Task 9 of the SemEval 2020 competition. We implemented two main approaches: 1) applying transfer learning by fine-tuning pre-trained models like BERT and 2) training feedforward neural networks on bag-of-words representations. We found that the results of fine-tuning bert-large-cased (24 layers) could be approximated by a 2-layer BoW feedforward NN. During the evaluation phase of the competition, our top system obtained an F-score of 71.3%, placing $4^{th}$ out of 62 entries in the official system rankings.

The fact that we managed fourth place with a system that was little better than a bag-of-words classifier suggests that existing pre-trained vector representations are not particularly good models of code-mixed language. One obvious reason is the difference between the pre-training domain (English Wikipedia) and the target domain (code-mixed tweets). While we were initially hopeful that the multilingual BERT model released by Google might be more effective than the English-only models, it was not. It remains unresolved whether this is because of the formality differences between the two domains, because of the lack of romanized Hindi in Google’s training data, or because language models simultaneously pre-trained on multiple languages do not generalize sufficiently to properly understand code-mixed data.