Bidirectional Machine Reading Comprehension for
Aspect Sentiment Triplet Extraction

Fine-grained opinion mining consists of various tasks, including aspect term extraction (ATE) (Wang et al. 2016; He et al. 2017; Li et al. 2018; Xu et al. 2018; Li and Lam 2017), opinion term extraction (OTE) (Liu, Joty, and Meng 2015; Poria, Cambria, and Gelbukh 2016; Xu et al. 2018; Wu et al. 2020b), aspect-level sentiment classification (ASC) (Dong et al. 2014; Tang, Qin, and Liu 2016; Li et al. 2019c; He et al. 2018; Hazarika et al. 2018; Nguyen and Shirai 2015; Wang et al. 2018), etc. The studies solve these tasks individually and ignore the dependency between them.

To explore the interactions between different tasks, recent studies gradually focus on the joint tasks such as aspect term-polarity co-extraction (He et al. 2019; Mitchell et al. 2013; Li and Lu 2017; Li et al. 2019a), aspect and opinion terms co-extraction (Liu, Xu, and Zhao 2015; Wang et al. 2016, 2017; Dai and Song 2019), aspect category and sentiment classification (Hu et al. 2019), and aspect-opinion pair extraction (Chen et al. 2020; Zhao et al. 2020). Besides, there are also a lot of studies (Chen and Qian 2020; He et al. 2019) solving multiple tasks with a multi-task learning network. However, none of these studies could identify aspects, opinion expressions and sentiments in a unified framework. To deal with this issue, Peng et al. (2020) proposed a two-stage framework to solve aspect sentiment triplet extraction (ASTE) task, which aims to extract triplets of aspects, opinion expressions and sentiments. However, the model suffers from error propagation due to its two-stage framework. Besides, separating the extraction and pairing of opinion entities means that the associations between different tasks are still not adequately considered.

Machine reading comprehension (MRC) aims to answer specific queries based on a given context. Recent researches have proposed various effective architectures for MRC, which adequately learn the interaction between the query and context. For example, BiDAF (Seo et al. 2017) employs a RNN-based sequential framework to encode queries and passages, while QANet (Yu et al. 2018) employs both convolution and self-attention. Several MRC systems (Peters et al. 2018; Radford et al. 2019) adopt context-aware embedding as well and obtain comparable results, especially BERT-based MRC model (Devlin et al. 2019).

Recently, there is a tendency to apply MRC on many NLP tasks, including named entity recognition (Li et al. 2020), entity relation extraction (Li et al. 2019b; Levy et al. 2017), and summarization (McCann et al. 2018), etc. Due to the advantages of MRC framework, we naturally transform ASTE into a multi-turn MRC task to better construct the associations among aspects, opinions, aspect-opinion relations and sentiments through well-designed queries. Different from the existing methods (Li et al. 2020, 2019b), we innovatively propose a bidirectional framework, which can identity triplets more comprehensively by making the two directions complement each other. This framework can be further extended to other tasks such as entity relation extraction.

To deal with ASTE task, we propose a bidirectional machine reading comprehension (BMRC) framework. The overall framework is illustrated in Figure 2. Concretely, we first design non-restrictive extraction queries and restrictive extraction queries to extract aspect-opinion pairs. Considering that each pair can be triggered by an aspect or an opinion expression, we further construct a bidirectional structure. In one direction, the aspects are first extracted via non-restrictive extraction queries, and then the corresponding opinion expressions for each aspect are identified via restrictive extraction queries. We define the above process as A$\rightarrow$O direction. Similarly, in O$\rightarrow$A direction, the framework recognizes opinion expressions and their corresponding aspects in a reversed order. After that, we design sentiment classification queries to predict the sentiment polarity for each aspect. Furthermore, our model jointly learns to answer the above queries to make them mutually beneficial. Besides, we adopt BERT as the encoding layer for richer semantics representations. During inference, the model fuses the answers to different queries and forms the triplets.

In BMRC, we adopt a template-based approach to construct queries. Specifically, we first design the non-restrictive extraction query and the restrictive extraction query in the A$\rightarrow$O direction as follows:

• •

  A$\rightarrow$O non-restrictive extraction query $q_{A\rightarrow O}^{\mathcal{N}}$: We design query ‘What aspects?’ to extract the collection of aspects $A=\left\{a_{i}\right\}_{i=1}^{\left|A\right|}$ from the given review sentence $X$.

• •

  A$\rightarrow$O restrictive extraction query $q_{A\rightarrow O}^{\mathcal{R}}$: We design query ‘What opinions given the aspect $a_{i}$?’ to extract the corresponding opinions $O_{a_{i}}=\left\{o_{a_{i},j}\right\}_{j=1}^{\left|O_{a_{i}}\right|}$ for each aspect $a_{i}$ and form aspect-opinion pairs.

Reversely, the O$\rightarrow$A direction extraction queries are constructed as follows:

• •

  O$\rightarrow$A non-restrictive extraction query $q_{O\rightarrow A}^{\mathcal{N}}$: We use query ‘What opinions?’ to extract the collection of opinion expressions $O=\left\{o_{i}\right\}_{i=1}^{\left|O\right|}$.

• •

  O$\rightarrow$A restrictive extraction query $q_{O\rightarrow A}^{\mathcal{R}}$: We design query ‘What aspect does the opinion $o_{i}$ describe?’ to recognize the corresponding aspects $A_{o_{i}}=\left\{a_{o_{i},j}\right\}_{j=1}^{\left|A_{o_{i}}\right|}$ for each opinion expression $o_{i}$.

With the above queries, opinion entity extraction and relation detection are naturally fused, and the dependency between them is gracefully learned via the restrictive extraction queries. Then, we devise sentiment classification queries to classify the aspect-oriented sentiments as follows:

• •

  Sentiment Classification query $q^{\mathcal{S}}$: We design query ‘What sentiment given the aspect $a_{i}$ and the opinion $o_{a_{i},1}/.../o_{a_{i},\left|O_{a_{i}}\right|}$?’ to predict sentiment polarity $s_{a_{i}}$ for each aspect $a_{i}$.

With sentiment classification queries, the semantics of aspects and their corresponding opinion expressions can be adequately considered during sentiment prediction.

Given the review sentence $X=\left\{x_{1},x_{2},...,x_{N}\right\}$ with $N$ tokens and each query $q_{i}=\{q_{i,1},q_{i,2},...,q_{i,\left|q_{i}\right|}\}$ with $\left|q_{i}\right|$ tokens, the encoding layer learns the context representation for each token. Inspired by the successful practice on many NLP tasks, we adopt BERT as the encoder. Formally, we first concatenate the query $q_{i}$ and the review sentence $X$ to obtain the combined input $I=\{[{\rm CLS}],q_{i,1},q_{i,2},...,q_{i,\left|q_{i}\right|},[{\rm SEP}],x_{1},x_{2},...,x_{N}\}$, where $[{\rm CLS}]$ and $[{\rm SEP}]$ are the beginning token and the segment token. The initial representation $\mathbf{e}_{i}$ for each token is constructed by summing its word embedding $\mathbf{e}_{i}^{w}$, position embedding $\mathbf{e}_{i}^{p}$, and segment embedding $\mathbf{e}_{i}^{g}$. Then, BERT is used to encode the initial representation sequence $E=\left\{\mathbf{e}_{1},\mathbf{e}_{2},...,\mathbf{e}_{\left|q_{i}\right|+N+2}\right\}$ as the hidden representation sequence $H=\left\{\mathbf{h}_{1},\mathbf{h}_{2},...,\mathbf{h}_{\left|q_{i}\right|+N+2}\right\}$ with the stacked Transformer blocks.

To jointly learn the subtasks in ASTE and make them mutually beneficial, we fuse the loss functions of different queries. For non-restrictive extraction queries in both two directions, we minimize the cross-entropy loss as follows:

where $p\left(*\right)$ represents the gold distribution, and $\hat{p}\left(*\right)$ denotes the predicted distribution.

Similarly, the loss of restrictive extraction queries in both two directions is calculated as follows:

For the sentiment classification queries, we minimize the cross-entropy loss function as follows:

Then, we combine the above loss functions to form the loss objective of the entire model:

The optimization problem in Eq.(7) can be solved by any gradient descent approach. In this paper, we adopt the AdamW (Loshchilov and Hutter 2017) approach.

During inference, we fuse the answers to different queries to obtain triplets. Specifically, in the A$\rightarrow$O direction, the non-restrictive extraction query $q_{A\rightarrow O}^{\mathcal{N}}$ first identifies the aspect collection $A=\left\{a_{1},a_{2},...,a_{\left|A\right|}\right\}$ with $\left|A\right|$ aspects. For each predicted aspect $a_{i}$, the A$\rightarrow$O restrictive query $q_{A\rightarrow O,i}^{\mathcal{R}}$ recognizes the corresponding opinion expression collection and obtains the set of predicted aspect-opinion pairs $V_{A\rightarrow O}=\left[\left(a_{k},o_{k}\right)\right]_{k=1}^{K}$ in the A$\rightarrow$O direction. Similarly, in the O$\rightarrow$A direction, the model identifies the set of aspect-opinion pairs $V_{O\rightarrow A}=\left[\left(a_{l},o_{l}\right)\right]_{l=1}^{L}$ in a reversed order. Then, we combine $V_{A\rightarrow O}$ and $V_{O\rightarrow A}$ as follows:

where ${V}^{\prime}$ and ${V}^{\prime\prime}$ denote the intersection and difference set of $V_{A\rightarrow O}$ and $V_{O\rightarrow A}$, respectively. Each aspect-opinion pair in ${V}^{\prime\prime}$ is valid only if its probability $p\left(a,o\right)$ is higher than the given threshold $\delta$. The probability of each opinion entity is calculated by multiplying the probabilities of its start and end positions.

Finally, we construct sentiment classification query $q_{i}^{\mathcal{S}}$ to predict the sentiment $s_{a_{i}}$ of each aspect $a_{i}$. Based on these, the triplet collection $T=\left[\left(a_{i},o_{i},s_{i}\right)\right]_{i=1}^{\left|T\right|}$ can be obtained.

To verify the effectiveness of our proposed approach, we conduct experiments on four benchmark datasets⁵⁵5https://github.com/xuuuluuu/SemEval-Triplet-data from the SemEval ABSA Challenges (Pontiki et al. 2014, 2015, 2016) and list the statistics of these datasets in Table 1. Specifically, the golden annotations for opinion expressions and relations are derived from Fan et al. (2019). And we split the datasets as Peng et al. (2020) did.

For the encoding layer, we adopt the BERT-base (Devlin et al. 2019) model with 12 attention heads, 12 hidden layers and the hidden size of 768, resulting into 110M pretrained parameters. During training, we use AdamW (Loshchilov and Hutter 2017) for optimization with weight decay 0.01 and warmup rate 0.1. The learning rate for training classifiers and the fine-tuning rate for BERT are set to 1e-3 and 1e-5 respectively. Meanwhile, we set batch size to 4 and dropout rate to 0.1. According to the triplet extraction F₁-score on the development sets, the threshold $\delta$ is manually tuned to 0.8 in bound $[0,1)$ with step size set to 0.1. We run our model on a Tesla V100 GPU and train our model for 40 epochs in about 1.5h.

To comprehensively measure the performances of our model and the baselines, we use Precision, Recall, and F₁-score to evaluate the results on four subtasks, including aspect term and sentiment co-extraction, opinion term extraction, aspect-opinion pair extraction, and triplet extraction. For reproducibility, we report the testing results averaged over 5 runs with different random seeds. At each run, we select the testing results when the model achieves the best performance on the development set.

To demonstrate the effectiveness of BMRC, we compare our model with the following baselines:

• •

  TSF (Peng et al. 2020) is a two-stage pipeline model for ASTE. In the first stage, TSF extracts both aspect-sentiment pairs and opinion expressions. In the second stage, TSF pairs up the extraction results into triplets via an relation classifier.

• •

  RINANTE+ adopts RINANTE (Dai and Song 2019) with additional sentiment tags as the first stage model to joint extract aspects, opinion expressions, and sentiments. Then, it adopts the second stage of TSF to detect the corresponding relations between opinion entities.

• •

  Li-unified-R+ jointly identifies aspects and their sentiments with Li-unified (Li et al. 2019a). Meanwhile, it predicts opinion expressions with an opinion-enhanced component at the first stage. Then, it also uses the second stage of TSF to predicts relations.

• •

  RACL+R first adopts RACL (Chen and Qian 2020) to identify the aspects, opinion expressions, and sentiments. Then, we construct the query ’Matched the aspect $a_{i}$ and the opinion expression $o_{j}$?’ to detect the relations. Note that RACL is also based on BERT.

The experimental results are shown in Table 2. According to the results, our model achieves state-of-the-art performances on all datasets. Although the improvements on aspect term and sentiment co-extraction and opinion term extraction are slight, our model significantly surpasses the baselines by an average of 5.14% F₁-score on aspect-opinion pair extraction and an average of 9.58% F₁-score on triplet extraction. The results indicate that extracting opinion entities and relations in pipeline will lead to severe error accumulation. By utilizing the BMRC framework, our model effectively fuses and simplifies the tasks of ATE, OTE, and relation detection, and avoids the above issue. It is worth noting that the increase in precision contributes most to the boost of F1-score, which shows that the predictions of our model own higher reliability than those baselines. Besides, RACL+R outperforms than other baselines because BERT can learn richer context semantics. TSF and Li-unified-R+ achieve better performances than RINANTE+ because TSF and Li-unified-R+ introduce complex mechanisms to solve the issue of sentiment contradiction brought by the unified tagging schema. Different from those approaches, our model gracefully solves this issue by transforming ASTE into a multi-turn MRC task.

Considering that the datasets released by Peng et al. (2020) remove the cases that one opinion expression corresponds to multiple aspects, we also conduct experiments on AFOE datasets ⁶⁶6https://github.com/NJUNLP/GTS (Wu et al. 2020a) and report the results in Table 3. The AFOE datasets, which retains the above cases, are also constructed based on the datasets of Fan et al. (2019) and the original SemEval ABSA Challenges. And we further compare our model with two baselines, including IMN+IOG and GTS⁷⁷7It worth noting that this paper has not been published when we submit our paper to AAAI 2021. (Wu et al. 2020a) . Specifically, IMN+IOG is a pipeline model which utilizes the interactive multi-task learning network (IMN) (He et al. 2019) as the first stage model to identity the aspects and their sentiments. Then, IMN+IOG use the Inward-Outward LSTM (Fan et al. 2019) as the second stage model to extract the aspect-oriented opinion expressions. And GTS is a latest model which proposes a grid tagging schema to identify the aspect sentiment triplets in an end-to-end way. Particularly, GTS also utilizes BERT as the encoder and designs an inference strategy to exploit mutual indication between different opinion factors. According to the results, our model and GTS significantly outperform IMN+IOG because the joint methods can solve the error propagation problem. Compared with GTS, our model still achieves competitive performances, which verify the effectiveness of our model.

We first validate whether the restrictive extraction query could effectively capture and exploit the dependency between opinion entity extraction and relation detection for better performance. Accordingly, we construct a two-stage model similar to TSF, called ‘Ours w/o REQ’. In the first stage, we remove the restrictive extraction query $Q^{\mathcal{R}}$ from BMRC for only the opinion entity extraction and sentiment classification. The stage-2 model, which is responsible for relation detection, is also based on MRC with the input query ‘Matched the aspect $a_{i}$ and the opinion expression $o_{j}$?’. Experimental results are shown in Figure 3. Although the performances on aspect extraction and opinion extraction are comparable, the performances of ‘Ours w/o REQ’ on triplet extraction and aspect-opinion pair extraction are evidently inferior than BMRC. The reason is that with the removal of the restrictive extraction query, the opinion entity extraction and relation detection are separated and no dependency would be captured by ‘Ours w/o REQ’. This indicates the effectiveness of the restrictive query at capturing the dependency.

To explore the effect of bidirectional MRC structure, we compare our model with two unidirectional models, including ‘Ours w/o AO’ and ‘Ours w/o OA’. Concretely, ‘Ours w/o AO’ extracts triplets only through O$\rightarrow$A direction, and ‘Ours w/o OA’ extracts triplets through A$\rightarrow$O direction. As shown in Figure 3, ‘Ours w/o OA’ shows inferior performance on opinion term extraction without O$\rightarrow$A direction MRC, while ‘Ours w/o AO’ shows worse performance on aspect term extraction. This further harms the performances on aspect-opinion pair extraction and triplet extraction. The reason is that both aspects and opinion expressions can initiate aspect-opinion pairs, and the model will be biased when relations are forced to be detected by either aspects or opinions only. By introducing the bidirectional design, the two direction MRCs can complement each other and further improve the performance of aspect-opinion pair extraction and triplet extraction.

In order to examine the benefit that the relations between aspects and opinion expressions provide for the sentiment classification, we compare the performances of our model and ‘Ours w/o REQ’. Experimental results on aspect term extraction and aspect term and sentiment co-extraction are shown in Table 4. Since ‘Ours w/o REQ’ separate relation detection and sentiment classification in two stages, the detected relations cannot directly provide assistance to sentiment classification. According to the results, although removing relation detection from joint learning does not harm the performance of aspect term extraction seriously, the performances of aspect term and sentiment co-extraction are all significantly weakened. This clearly indicates that the relations between aspects and opinion expressions can effectively boost the performance of sentiment classification.

We analyze the effect of BERT and our contributions from two perspectives. First, we construct our model based on BiDAF, which is a typical reading comprehension model without BERT, and refer it to ‘Ours w/o BERT’. According to the results shown in Table 5, it significantly surpasses TSF by an average of 8.15% F1-score on triplet extraction, which shows that our model can achieve SOTA performance without BERT. Besides, compared with ‘Ours w/o BERT’, our model further improves 7.69% F1-score, which is brought by BERT. Second, we compare our model with ’Ours w/o REQ’ and TSF. The ablation model ’Ours w/o REQ’ can be regarded as an implementation version of TSF based on BERT and MRC framework. By comparing it with TSF, the results show that BERT based MRC model can bring an average of 5.41% F1-score improvement on triplet extraction against the counterpart model without BERT. By further introducing the bidirectional MRC structure and three types of queries, our model further outperforms ’Ours w/o REQ’ by 10.4% F1-score. These two-fold analyses indicate that our contributions play a greater role in improving performances than BERT.