OATS: Opinion Aspect Target Sentiment Quadruple Extraction Dataset for Aspect-Based Sentiment Analysis

In this section, we present the results of different baseline systems on the selected tasks, highlighting the benefits and challenges of current methods on the OATS dataset and the specific task.

ASQP Table 4 (a) reports the performance on the ASQP task for the quadruple extraction using five different baseline systems on the three datasets. On a high level, the overall performance of the hotels domain is better compared to the other two domains in all the baseline systems.

For the Amazon_FF domain, even though Template-ILO gave the best performance, all the other methods are not far from it. Similarly, for the Coursera domain, GAS-T5 performed better than the Paraphrase and GEN-SCL-NAT, but the Template method is close to its performance. Paraphrase outperformed all the other methods in the hotels domain with a 34.51% F1 score.

Prior works have shown that the Template method significantly outperformed Paraphrase and GAS in the restaurant domain Template-Quad. That trend only translated to the Amazon_FF domain, leaving out Coursera and Hotels. Similarly, GEN-SCL-NAT, which outperformed the Paraphrase method on the restaurants and laptops domain, failed to carry it to any of the three domains.

ASTE In Table 4 (b), we present the performance of five baseline systems on the ASTE task, which focuses on extracting the triplet of aspect term, opinion term, and sentiment polarity across three datasets. Notably, the BMRC method emerges as the standout performer, surpassing other baselines across all domains for this task, while the GEN-SCL-NAT method consistently lags behind. The success of the BMRC method can be attributed to the approach taken by BMRC. Instead of solely considering the relationship from target to opinion phrases, they also incorporate the potential backward relation from opinion phrases to target expressions. This deviates from other methods like seq2path and Opinion-Tree-Generation, which deploy two distinct paths in a tree structure for targets and opinion phrases to identify the quadruples.

R-ASD Table 3.2 (b) shows the aspect sentiment joint detection task results at the review-level (R-ASD) and sentence-level (S-ASD). BERT-based methods significantly outperform the generative approaches. One primary factor driving this difference is the augmentation strategy BERT-pair methods use, which expands the dataset based on the number of categories multiplied by the polarities in that dataset. For example, with 27 aspect categories and 4 different polarities in the Coursera domain for the ASQP task, this results in a total of 894024 instances ($8278*27*4$). Additionally, these methods transform the multi-label, multi-class classification of the ASD task into a simpler binary classification, making the task inherently easier than the challenge generative models face when extracting categories and polarities directly from a review sentence.

TASD We present the results of the TASD task experiments on all the domains in Table 3.2 (a). The generative-based models performed better than the BERT-based approaches on all the domain datasets. The T5-ABSA method performed best for Amazon_FF and Coursera domains, while the GAS method outperformed the hotels domain. All the generative-based approaches are not far from each other in the hotels domain, while the BERT-based methods have a significant difference in the Amazon_FF and hotels domain when compared to the generative approaches.

Explicit and Implicit Targets and Opinions We observe distinct patterns of explicit and implicit targets and opinions across the OATS domains from the statistics presented in Table 6 (b). Amazon_FF, for instance, showcases a predominance of implicit targets, suggesting users often leave their main subject of discussion implied. Conversely, Coursera leans more towards explicit target mentions, possibly indicative of the direct nature of feedback in educational settings. Notably, regardless of the domain, explicit opinions outnumber their implicit counterparts. This trend underlines that while users might leave their subjects (targets) implied, they tend to be direct about their sentiments or opinions on them. The Hotels domain stands out with the highest counts of explicit mentions for both targets and opinions, hinting at the straightforward nature of feedback in this domain. It is important to recognize these domain-specific trends as they aid in tailoring models appropriately, potentially improving their accuracy and adaptability.

Challenging combination of elements in OATS We tested the Paraphrase-T5 method ASQP on all the OATS datasets for six different ABSA tasks to see how hard it is to detect various ABSA elements. The results are in Table 6 (a). The TSD task did the best on all the domains, but TOWE was the hardest. This shows that connecting an opinion phrase to its target is especially tough in OATS. Interestingly, even though TOWE is kind of a part of ASTE, it does not do as well. This might be because having sentiment information in ASTE helps pick out opinion phrases better, giving it an edge.

Furthermore, integrating aspect categories into TSD creating the TASD task, saw a noticeable performance drop. This highlights the challenge of pinpointing aspect categories for specific targets. This challenge could also explain the lower scores of the ASD and ASQP tasks in the Amazon_FF and Coursera domains compared to TSD. Interestingly, in the hotels domain, ASD emerged as the top-performing task, which contributed to a superior TASD and ASQP performance relative to other domains. This success in the hotels domain might also explain the lower scores of the ASD and ASQP tasks in the Amazon_FF and Coursera domains compared to TSD. The complex nature of ASQP, which demands establishing relationships between categories, opinion phrases, and targets, might account for its sub-optimal performance stemming from the above two reasons.

The main differences between the current ASQP datasets and OATS are: (a) we annotated the data from scratch with all the elements together instead of aligning from several sources, (b) we include all the implicit aspect terms and opinion phrases, unlike ASQP, which excluded implicit opinion terms, (c) we provide both the review-level and sentence-level annotations facilitating the analysis and experiments for the review/text-level aspect-based sentiment analysis that differentiates MEMD-ABSA from our dataset, (d) there are $\approx 5.8K$ opinion quadruples in total for the ASQP corpus, which includes two restaurant datasets that are just 80% of opinion quadruples of the Coursera domain itself, which is the smallest in the OATS corpus. (e) OATS has a total of $\approx 27.5K$ opinion quadruples, almost five times larger than the ASQP corpus. Also, if we include the review-level annotations, we have nearly $44.5K$ opinions in OATS.

The OATS corpus will help analyze and understand the inherent relationships among all the elements of ABSA and exploit them to solve the ABSA task holistically. The comprehensive nature of the OATS corpus not only allows the evaluation of the ASQP task but also unlocks the potential analysis and exploration of all the subtasks of ABSA, including TASD, ASTE, TOWE, TAD, and many more. As pointed out by ASQP, the characteristic of tackling various ABSA tasks in a unified framework enables the knowledge to be easily transferred across related subtasks, which is especially beneficial under low-resource settings. It also allows cross-task transfer for subtasks that underperform when trained using only a task-specific dataset. The provided review-level annotations will help many researchers who like to explore ABSA for whole reviews rather than individual sentences or single-sentence reviews.

In this section, we describe the tasks of ABSA and their related datasets, followed by triplet and quadruple extraction task-specific methods.

ABSA still presents a lot of difficulties in the rapidly developing field of generative artificial intelligence. We present a thoroughly curated ABSA dataset including quadruples, both explicit and implicit attributes, and views, spanning three unique domains. In addition, the dataset incorporates review-level sentiment polarity for each aspect category, providing a comprehensive perspective of the sentiments expressed in the reviews. Our annotations outnumber those found in previous datasets. We do an in-depth study, give comprehensive annotation guidelines, and provide dataset statistics. We also enable evaluations of generative and non-generative benchmarks on a range of common ABSA tasks.

We identified the following limitations of this work:

1. 1.

   While the OATS dataset spans multiple domains, including the relatively unexplored domains of Amazon fine foods and Coursera course reviews, it exclusively contains English-language reviews. The dataset currently lacks reviews in low-resource languages. We plan to address this in future works by releasing multi-lingual datasets, inclusive of the low-resource setting.

2. 2.

   The experimental results showcased in this paper are purely in-domain; models were fine-tuned and tested within the same domain. In subsequent works, we aim to design experiments to tackle cross-domain and out-of-domain ABSA tasks.

3. 3.

   Our dataset incorporates review-level ABSA tuples focusing only on aspect categories and sentiment polarities. It currently omits review-level target expressions and opinion phrase annotations, which would render the dataset more comprehensive for ABSA. We earmark this as an avenue for future exploration.

We point out the following ethical considerations while building and using the OATS dataset:

1. 1.

   The data collected from platforms such as Amazon Finefoods, Coursera, and TripAdvisor Hotels come from public reviews provided by users. We ensured that all identifiable information, including usernames, avatars, and any other potentially identifying details, were removed to preserve the anonymity of the reviewers.

2. 2.

   Our data processing methodology focused on extracting and analyzing the content of the reviews without altering the original sentiment or meaning. We handled this data with the utmost care to ensure that the sentiments and opinions of the original reviewers were not misrepresented.

3. 3.

   We recognize that reviews from online platforms might not represent the complete spectrum of users’ opinions, as they may be influenced by various factors like platform algorithms, user demographics, and more. We urge users of this dataset to be aware of potential biases and always consider the data in the proper context.

4. 4.

   While the reviews were publicly available, the original authors might not have expected their content to be used in research. We’ve taken measures to respect their privacy, but future users of this dataset should also be aware of this consideration.

5. 5.

   While the reviews are publicly accessible, we acknowledge the platforms (Amazon Finefoods, Coursera, TripAdvisor Hotels) as the source of this data. We have only used this data for research and academic purposes, ensuring that we respect the terms of use for each platform.

This research was supported by Adobe Gift funding to the University of Houston collaboration and by the National Science Foundation (NSF) under grant #1910192. The authors acknowledge the use of the Sabine Cluster and the advanced support from the Research Community Data Core at the University of Houston to carry out the research presented here. We also want to acknowledge the Viet Lai (NLP researcher from University of Oregon), for his constant support to the BRAT Annotation tool maintenance, and helping with all the issues related to the tool.