To BERT or Not to BERT: Comparing Task-specific and Task-agnostic Semi-Supervised Approaches for Sequence Tagging

An integral component of fine-grained sentiment analysis is the ability to identify segments of text towards which opinions are expressed. These segments are referred to as Opinion Target Expressions or OTEs. An example of this task is provided in Figure 1. The commonly used labeled datasets for Opinion Target Expression (OTE) detection are those released as part of SemEval Aspect-based Sentiment shared tasks: SemEval-2014 Laptops Pontiki et al. (2014) (SE14-L) and SemEval-2016 Restaurants Pontiki et al. (2016) (SE16-R). These consist of reviews from the laptop and restaurant domains, respectively, with OTEs annotated for each sentence of a review. We use the provided train-test splits but further split the training data randomly into 90% training and 10% validation sets. As unlabeled data that is similar to the domain and task, we extract restaurant reviews from the Yelp²²2https://www.yelp.com/dataset dataset (Yelp-R) and reviews of electronics products from Amazon Product Reviews dataset³³3http://jmcauley.ucsd.edu/data/amazon/ (Amazon-E) (see Table 1).

NER is the task of identifying and categorizing named entities from unstructured text into pre-defined categories such as Person (PER), Location (LOC), Organization (ORG) etc. Figure 1 contains an example of this task. CONLL-2003 Tjong Kim Sang and De Meulder (2003) and CONLL-2012 (OntoNotes v5.0) Pradhan et al. (2012) are the commonly used labeled datasets to build and evaluate performance for Named Entity Recognition models (Lample et al., 2016; Ma and Hovy, 2016; Akbik et al., 2018, inter-alia). We focus on the English parts of these datasets. CONLL-2003 contains annotations for Reuters news for 4 entity types (Person, Location, Organization, and Miscellaneous). CONLL-2012 dataset contains 18 entity types, consisting of various genres (weblogs, news, talk shows, etc.) with newswire being majority. We use the provided train, validation and test splits for these datasets. As newswire is the predominant genre in these datasets, as we use stories from the CNN and Daily Mail datasets⁴⁴4https://github.com/abisee/cnn-dailymail (CNN-DM) as an unlabeled dataset from the news genre (see Table 1).

Slot-labeling is a key component of Natural Language Understanding (NLU) in dialogue systems, which involves labeling words of an utterance with pre-defined attributes - slots. For this task, we use the widely-used MIT-Movie dataset⁵⁵5https://groups.csail.mit.edu/sls/downloads/movie/ as labeled data which contains queries related to movie information, with 12 slot labels such as Plot, Actor, Director, etc.. An example from this dataset is demonstrated in Figure 1. We use the default train-test split, and create a validation set by randomly selecting 10% of the training samples. IMDb Movie review dataset (IMDb) is used as in-domain unlabeled data Maas et al. (2011) (see Table 1).

CVT is a semi-supervised approach proposed by Clark et al. (2018) that leverages unlabeled data in a task-specific manner. The underlying model is a two-layer CNN-BiLSTM sentence encoder followed by a linear layer and a softmax per prediction module. There are two kinds of prediction modules - primary and auxiliary. CVT alternates between learning from labeled and unlabeled data during training. The key idea is that the primary prediction module, which has an unrestricted view of the data, trains on the task using labeled examples, and makes task-specific predictions on unlabeled data. The auxiliary modules, with different restricted views of the unlabeled data, attempt to mimic the predictions of the primary module. Standard cross-entropy loss is minimized when learning from labeled examples, while for unlabeled examples, KL-Divergence Kullback and Leibler (1951) between the predicted primary and auxiliary probability distributions is minimized (see Clark et al. (2018) for more details). We demonstrate the training strategy in Figure 2. Thus, the model is trained to produce consistent results despite seeing partial views of the input - thereby improving underlying representations.

We use Glove 840B.300d embeddings Pennington et al. (2014) instead of Glove 6B.300d embeddings used by the authors for a larger vocabulary coverage. For each of the labeled datasets (Section 2), we use the corresponding domain/task-relevant unlabeled data to train a sequence tagging model for 400K steps, with early stopping enabled using validation set convergence.

BERT Devlin et al. (2019) has achieved state-of-the-art results on many NLP tasks. The key innovation lies in the use of bi-directional Transformers as well as the Masked Language Model (MLM) and Next Sentence Prediction (NSP) objectives used during training. Learning happens in two steps: 1) training the model on a very large generic dataset (using the two objectives above); 2) fine-tuning the learned representations on a downstream task in a supervised fashion. For our experiments we use BERT_Base, which has 12 layers, 768 hidden dimensions per token and 12 attention heads and is pre-trained on the cased English Wikipedia and Books Corpus data (Wiki+Books). In order to fine-tune on the downstream sequence tagging task, the model we use consists of the BERT_Base encoder, followed by a dropout layer and a classification layer that classifies each token into B-label, I-label, O, where $label\in\{label_{i},label_{i+1},...,label_{n}\}$. Cross-entropy loss is the loss function used.

In this setup, we use BERT_Base architecture and pre-train it from scratch on the domain/task relevant unlabeled data. Each training step trains on a batch of size 256. A validation set is created from each unlabeled dataset by random sampling (details in Appendix A). The convergence criteria is set to be validation MLM accuracy improvement $\geq$ 0.05 when evaluated every 30K steps. We then perform the second step of fine-tuning on the downstream task data, as in the regular BERT setup.

Here, we start with BERT_Base trained on the generic unlabeled dataset (English Wikipedia and Book Corpus) and continue pretraining on the corresponding domain/task-relevant unlabeled data (Section 2). Inspired by the nomenclature in Gururangan et al. (2020), we refer to this model as Adaptively Pretrained BERT_Base. Further, we perform fine-tuning on the downstream task data, as with the previous BERT models.

We report mean F1 (with standard deviation) on the labeled test splits for each task over 5 randomized runs, and compare the models using statistical significance tests over these runs. Further, we report the approximate number of unlabeled sentences seen by each model. Table 2 shows the results for the OTE detection task. Here, out of the 3 BERT-based variations, the best result is achieved by the APBERT_Base model across both SemEval datasets. For SemEval2016 Restaurants, we find the mean F1 from the APBERT_Base model to be comparable to that of CVT (p-value 0.26). Both models outperform the SOTA baseline. For SemEval2014 Laptops, APBERT_Base is found to have a statistically significant (p-value 0.04) higher F1 than CVT, and both models outperform SOTA.

In Tables 3 and 4, we present F1 results on NER and Slot-labeling task, respectively. For all 3 datasets, we find CVT to outperform all BERT models (statistically significant for CONLL-2003 and MIT Movies dataset, at p-values 0.0086 and 0.0085, respectively). For these tasks, BERT_base outperforms APBERT_Base models. Furthermore, CVT outperforms SOTA for Slot-labeling task.

These results show that the CVT model, using unlabeled data in a task-specific manner, is more robust across different tasks and types of unlabeled data. For OTE detection, the unlabeled data is closely related to both domain and task, while for NER and Slot-labeling, the unlabeled data is related to genre (newswire) and domain (movies), but not necessarily to the specific tasks. In line with the findings of Gururangan et al. (2020), Adaptive Pre-training shows best results when using unlabeled data that is domain and task relevant (superior results for the OTE task). It is also worth noting that CVT requires significantly smaller amount of unlabeled data than the BERT models (Tables 2, 3 and 4).

Table 5 shows computational cost and environmental impact by means of estimated CO₂ emissions occurring during training. We use the procedure described by Strubell et al. (2019). Tesla V100 GPUs are used for training. For computational cost, we refer to the average cost per hour for the training instances used.⁶⁶6https://aws.amazon.com/ec2/instance-types/p3/ To compute energy consumed, we query the NVIDIA System Management Interface ⁷⁷7https://web.archive.org/web/20190504134329/https:/developer.nvidia.com/nvidia-system-management-interface multiple times during training, to note the average GPU power consumption. For CPU and DRAM power usage, we use Linux’s turbostat package.⁸⁸8http://manpages.ubuntu.com/manpages/xenial/man8/turbostat.8.html The models trained using Yelp Restaurants unlabeled data are used as an example in Table 5, but the same computations hold for other models. Note that we do not perform initial pretraining of BERT_base nor pretrain the Glove 840B.300d embeddings used in CVT, but these come at a one-time cost that we consider constant. Worth noting though, that BERT_base pretraining is more expensive than Glove pretraining. As is evident, training the CVT model incurs much less financial cost than the corresponding BERT models ($\sim$11x lower than APBERT_base), while also emitting lesser CO₂ emissions ($\sim$18x lower than APBERT_base).⁹⁹9 If we consider just fine-tuning BERT_Base on the supervised data of the downstream task (OTE detection on SemEval2016 Restaurants Data) the numbers corresponding to Table 5 are: HW: 1/8, Hours: 0.283, Cost: 0.87, Power: 0.094, CO₂: 0.09. Although fine-tuning BERT on the downstream task (using supervised data) is relatively cheap, and one could amortize the cost of pre-training BERT over a large number of such tasks, this requires an understanding of what the number and type of such tasks are.