Reproducibility, Replicability and Beyond: Assessing Production Readiness of Aspect Based Sentiment Analysis in the Wild

In recent times, online marketplaces of goods and services have witnessed an exponential growth in terms of consumers and producers, and have proliferated in a wide spectrum of market segments, such as e-commerce, food delivery, healthcare, ride sharing, travel and hospitality, to name a few. The Indian e-commerce market segment alone is projected to grow to $300-350$M consumers and $$100-120$B revenue by $2025$ ¹¹1How India Shops Online – Flipkart and Bain & Company.. In the face of ever-expanding choices, purchase decision-making is guided by the reviews and ratings: Watson et al. [29] estimates that the average product rating is the most important factor in making purchase decisions for $60\%$ of consumers. Similarly, the academic research on Aspect Based Sentiment Analysis (ABSA) has come a long way since its humble beginning in the SemEval-$2014$ ²²2SemEval-$2014$ Task $4$.. Over the past $6$ years, the accuracy on a benchmark dataset for aspect term polarity has grown by at least $11.4\%$. We ask, is this progress enough to support the burgeoning online marketplaces?

We argue on the contrary. On one hand, industrial-strength systems need to demonstrate several traits for smooth operation and delightful consumer experience. Breck et al. [1] articulates several essential traits and presents a rubric of evaluation. Notable traits include: (a) “All hyperparameters have been tuned”; (b) “A simpler model is not better”; (c) “Training is reproducible”; and (d) “Model quality is sufficient on important data slices”. On the other hand, recent academic research in several fields has faced criticisms from within the community on similar grounds: Dhillon et al. [6] points out the inadequacy of benchmark dataset and protocol for few-shot image classification; Dacrema et al. [4] criticises the recent trend in recommendation systems research on the ground of lack of reproducibility and violations of (a)–(c) above; Li et al. [14] criticises the recent trend in information retrieval research on similar grounds. A careful examination of the recent research we conduct in this work reveals that the field of ABSA is not free from these follies.

To this end, it is instructive to turn our attention to classic software engineering with the hope of borrowing from its proven safe development practises. Notably, Kang et al. [10] advocates the use of model assertions – an abstraction to monitor and improve model performance during the development phase. Along similar lines, Ribeiro et al. [20] presents a methodology of large-scale comprehensive testing for NLP, and notes its effectiveness in identifying bugs in several (commercial) NLP libraries, that would not have been discovered had we been relying solely on test set accuracy. In this work, in addition to the current practice of reporting test set accuracies, we report performance on two challenging data slices – e.g., hard set [31], and, contrast set [7] – to further bolster the comprehensiveness of empirical evaluation.

For widespread adoption, data efficiency is an important consideration in real-world deployment scenarios. As an example, a large e-commerce marketplace in India operates in tens of thousands of categories, and a typical annotation cost is $3$¢ per review. In this work, we introduce and open-source two additional large-scale datasets curated from product reviews in lifestyle and appliance categories to aid replicability of research and study of transfer across domains and locales (text with similar social/linguistic characteristics). In particular, we note that just a small fraction of the in-domain training dataset, mixed with existing in-locale cross-domain training datasets, guarantees comparable test set accuracies.

In summary, we make the following notable contributions:

• •

  Perform a thorough reproducibility study of models sampled from a public leaderboard ³³3Papers With Code: ABSA on SemEval 2014 Task 4 Sub Task 2. that reveals a consistent $4-5\%$ drop in reported test set accuracies, which is often larger than the gap in performance between the winner and the runner-up.

• •

  Consistent with the practices developed in software engineering, we bolster the empirical evaluation rigour by introducing two challenging data slices that demonstrates an average $12-55\%$ drop in test set accuracies.

• •

  We study the models from the perspective of data efficiency and note that as little as $10-25\%$ of the domain-specific training dataset, when used in conjunction with existing cross-domain datasets from within the same locale, largely closes the gap in terms of test set accuracies between complete cross-domain training and using $100\%$ of the domain-specific training instances. This observation has immense implications towards reduction of annotation cost and widespread adoption of models.

• •

  We curate two additional datasets from product reviews in lifestyle and appliances categories sampled from a large e-commerce marketplace in India, and make them publicly accessible to enable the study of replicability.

Reproducibility and replicability have been considered the gold-standard in academic research and has witnessed a recent resurgence in emphasis across scientific disciplines: see for e.g., McArthur et al. [18] in the context of biological sciences and Stevens et al. [23] in the context of psychology. We follow the nomenclature established in [23] and define reproducibility as the ability to obtain same experimental results when a different analyst uses an identical experimental setup. On the other hand, replicability, is achieved when the same experimental setup is used on a different dataset to similar effect. While necessary, these two traits are far from sufficient for widespread deployment in production.

Breck et al. [1] lists a total of $28$ traits spanning the entire development and deployment life cycle. Since our goal is only to assess the production readiness of a class of models. we decide to forego all $14$ data-, feature- and monitoring-related traits. We borrow $1$ (“Training is reproducible”) and $2$ (“All hyperparameters have been tuned” and “Model quality is sufficient on important data slices”) traits from the infrastructure- and modeling-related rubrics, respectively.

Further, we note that the ability to transfer across domains/locales is a desirable trait, given the variety of market segments and the geographic span of online marketplaces. In other words, this expresses data efficiency and has implications towards lowering the annotation cost and associated deployment hurdles. Given the desiderata, we articulate our production readiness rubric as follows:

• •

  Reproducibility. A sound experimental protocol that minimises variability across runs and avoids common pitfalls (e.g., hyperparameter-tuning on the test dataset itself) should reproduce the reported test set accuracy within a reasonable tolerance, not exceeding the reported performance gap between the winner and the runner-up in a leaderboard. §6 articulates the proposed experimental protocol and §7 summarises the ensuing observations.

• •

  Replicability. The aforementioned experimental protocol, when applied to a different dataset, should not dramatically alter the conclusions drawn from the original experiment; specifically, it should not alter the relative positions within the leaderboard. §4 details two new datasets we contribute in order to aid the study of replicability, whereas §7 contains the ensuing observations.

• •

  Performance. Besides overall test-set accuracy, an algorithm should excel at challenging data slices such as hard- [31] and contrast sets [7]. §7 summarises our findings when this checklist is adopted as a standard reporting practice.

• •

  Transferability. An algorithm must transfer gracefully across domains within the same locale, i.e. textual data with similar social/linguistic characteristics. We measure it by varying the percentage of in-domain training instances from $0\%$ to $100\%$ and locating the inflection point in test set accuracies. See §7 for additional details.

Note that apart from the “The model is debuggable” and “A simpler model is not better” traits, the remaining traits as defined by Breck et al.[1] are independent of the choice of the algorithm and is solely a property of the underlying system that embodies it, which is beyond the scope of the present study. Unlike [1], we refrain from developing a numerical scoring system.

First popularised in the SemEval-$2014$ Task $4$ [19], ABSA has enjoyed immense attention from both academic and industrial research communities. Over the past $6$ years, according to the cited literature on a public leaderboard ⁴⁴4Papers With Code: ABSA on SemEval 2014 Task 4 Sub Task 2., the performance for the subtask of Aspect Term Polarity has increased from $70.48\%$ in Pontiki et al. [19], corresponding to the winning entry, to $82.29\%$ in Yang et al. [32] on the laptop review corpus. The restaurant review corpus has witnessed a similar boost in performance: from $80.95\%$ in [19] to $90.18\%$ in [32].

Not surprisingly, the field has witnessed a phase change in terms of the methodology: custom feature engineering and ensembles that frequented earlier [19] gave way to neural networks of ever-increasing complexity. Apart from this macro-trend, we notice several micro-trends in the literature: the year $2015$ witnessed a proliferation of LSTM and its variants [24]; years $2016$ and $2017$ respectively witnessed the introduction [25] and proliferation [26, 16, 3, 2] of memory networks and associated attention mechanisms; in $2018$ research focused on CNN [31], transfer learning [13] and transformers [12], while memory networks and attention mechanisms remained in spotlight [11, 27, 9, 15]; transformer and BERT-based models prevailed in $2019$ [30, 33], while attention mechanisms continued to remain mainstream [22].

While these developments appear to have pushed the envelope of performance, the field has been fraught with “winner’s curse” [21]. In addition to the replicability and reproducibility crises [18, 23], criticisms around inadequacy of baseline and unjustified complexity [4, 6, 14] applies to this field as well. The practice of reporting performance in challenging data slices [31] has not been adopted uniformly, despite its importance to production readiness assessment [1]. Similarly, the study of transferability and replicability has only been sporadically performed: e.g., Hu et al. [8] uses a dataset curated from Twitter along with the ones introduced in Pontiki et al. [19] for studying cross-domain transferability.

For the Reproducibility rubric, we consider the datasets released as part of the SemEval 2014 Task 4 - Aspect Based Sentiment Analysis ⁵⁵5SemEval 2014: Task 4 http://alt.qcri.org/semeval2014/task4/ for our experiments, specifically the Subtask 2 - Aspect term Polarity. The datasets come from two domains – Laptop and Restaurant. We use their versions made available in this Github ⁶⁶6https://github.com/songyouwei/ABSA-PyTorch repository which forms the basis of our experimental setup.

The guidelines used for annotating the datasets were released as part of the challenge. For the Replicability rubric, we tagged two new datasets from the e-commerce domain viz., Men’s T-shirt and Television, using similar guidelines.

The statistics for these four datasets are presented in Table 1. As we can observe, the sizes of the Men’s T-shirt and Television datasets are comparable to the laptop and restaurant datasets, respectively.

For the Performance rubric, we evaluate and compare the models on two challenging subsets viz., hard as defined by Xue et al. [31] and contrast as defined by Gardner et al. [7]. We describe below the process to obtain these datasets:

• •

  Hard data slice: Hard examples have been defined in Xue et al. [31] as the subset of review sentences containing multiple aspects with different corresponding sentiment polarities. The number of such hard examples from each of the datasets are listed in Table 2.

• •

  Contrast data slice: In order to create additional test examples, Gardner et al. [7] adds perturbations to the test set, by modifying only a couple of words to flip the sentiment corresponding to the aspect under consideration. For e.g., consider the review sentence: “I was happy with their service and food”. If we change the word “happy” with “dissatisfied”, the sentiment corresponding to the aspect “food” changes from positive to negative. We take a random sample of $30$ examples from each of the datasets and add similar perturbations as above to create $30$ additional examples. These $60$ examples for each of the four datasets thus serve as our contrast test sets.

As part of our evaluation, we focus on two families of models which cover the major trends in the ABSA research community: (i) memory network based, and (ii) BERT based. Among the initial set of models for the SemEval 14 challenge, memory network based models had much fewer parameters compared to LSTM based approaches and performed comparatively better. With the introduction of BERT [5], work in NLP has focused on leveraging BERT based architectures for a wide spectrum of tasks. In the ABSA literature, the leaderboard ⁷⁷7https://paperswithcode.com/sota/aspect-based-sentiment-analysis-on-semeval has been dominated by BERT based models, which have orders of magnitude more parameters than memory network based models. However, due to pre-training on large corpora, BERT models are still very data efficient in terms of number of labelled examples required. We chose three representative models from each family for our experiments and briefly describe them below:

• •

  ATAE-LSTM [28] represents aspects using target embeddings and models the context words using an LSTM. The context word representations and target embeddings are concatenated and combined using an attention layer.

• •

  Recurrent Attention on Memory (RAM) [2] represents the input review sentence using a memory network, and the memory cells are weighted using the distance from the target word. The aspect representation is then used to compute attention scores on the input memory, and the attention weighted memory is refined iteratively using a GRU (recurrent) network.

• •

  Interactive Attention Networks (IAN) [17] uses separate components for computing representations for both the target (aspect) and the context words. The representations are pooled and then used to compute an attention score on each other. Finally the individual attention weighted representations are concatenated to obtain the final representation for the 3-way classification task, with positive, negative, and neutral being the three classes.

• •

  BERT-SPC [5] is a baseline BERT model that uses “[CLS] + context + [SEP] + target + [SEP]” as input for the sentence pair classification task, where ‘[CLS]’ and ‘[SEP]’ represent the tokens corresponding to classification and separator symbols respectively, as defined in Devlin et al. [5] .

• •

  BERT-AEN [22] uses an attentional encoder network to model the semantic interaction between the context and the target words. Its loss function uses a label smoothing regularization to avoid overfitting.

• •

  The Local Context Focus (LCF-BERT) [33] is based on Multi-head Self-Attention (MHSA). It uses Context features Dynamic Mask (CDM) and Context features Dynamic Weighted (CDW) layers to focus more on the local context words. A BERT-shared layer is adopted to LCF design to capture internal long-term dependencies of local and global context.

We present an extensive evaluation of the aforementioned models across the four datasets: Laptops, Restaurants, Men’s T-shirt and Television, as per the production readiness rubrics defined in §2. While trying to reproduce the reported results for the models, we faced two major issues; (i) the official implementations were not readily available, and (ii) the exact hyperparameter configurations were not always specified in the corresponding paper(s). In order to address the first, our experimental setup is based on a community designed implementation of recent papers available on GitHub ⁸⁸8https://github.com/songyouwei/ABSA-PyTorch. Our choice for this public repository is guided by its thoroughness and ease of experimentation. As an additional social validation, the repository had 1.1k stars and 351 forks on GitHub at the time of writing. For addressing the second concern, we consider the following options; (a) use commonly accepted default parameters (for e.g., using a learning rate of $1e^{-4}$ for Adam optimizer). (b) use the public implementations to guide the choice of hyperparameters. The exact hyperparameter settings used in our experiments are documented and made available with our supporting code repository ⁹⁹9https://github.com/rajdeep345/ABSA-Reproducibility for further reproducibility and replicability of results.

From the corresponding experimental protocols described in the original paper(s), we were not sure if the final numbers reported were based on the training epoch that gave the best performance on the test set, or whether the hyperparameters were tuned on a separate held-out set. Therefore, we use the following two configurations; (i) the test set is itself used as the held out set, and the model used for reporting the results is chosen corresponding to the training epoch with best performance on the test set; and (ii) 15% of the training data is set aside as a held out set for tuning the hyperparameters and the optimal training epoch is decided corresponding to the best performance on the held out set. Finally the model is re-trained, this time with all the training data (including 15% held out set), for the optimal no. of epochs before evaluating the test set. For both the cases, we report mean scores over $5$ runs of our experiments.

Despite the limitations, the present study takes an important stride towards closing the gap between empirical academic research and its widespread adoption and deployment in production. In addition to further strengthening the rubric and judging a broader cross-section of published ABSA models in its light, we envision to replicate such study in other important NLP tasks. We hope the two contributed datasets, along with the open-source evaluation framework, shall fuel further rigorous empirical research in ABSA. We make all the codes and datasets publicly available ¹⁰¹⁰10https://github.com/rajdeep345/ABSA-Reproducibility.