Opinion Transmission Network for Jointly Improving Aspect-oriented Opinion Words Extraction and Sentiment Classification

ALSC aims to classify the sentiment of a given aspect in a sentence into one set of pre-defined sentiment categories. Specifically, given a sentence containing $n$ words $s=\{w_{1},w_{2},...,w_{n}\}$ and an aspect $w_{a}$ in $s$ (we notate an aspect as one word $w_{a}$ for simplicity, and $a$ is the index of the aspect in the sentence), the task is to assign a label $y^{ALSC}\in C$ to an input pair $<s,w_{a}>$, where $C$ is the set of pre-defined sentiment categories (i.e., positive, negative and neutral).

AOWE aims at extracting the corresponding opinion words towards a given aspect from a sentence. Different from ALSC, it is formalized as an aspect-oriented sequence labeling task [3]. Given an input pair $<s,w_{a}>$, the task is to assign a label $y_{i}^{AOWE}\in\{B,I,O\}$ for each word $w_{i}$ in the sentence $s$. The three labels B, I and O refer to the beginning, inside and outside of an aspect, respectively, and they follow the standard BIO notation used in sequence labeling. The spans composed by the tags $B$ and $I$ represent the corresponding opinion words of the aspect $w_{a}$. It is obvious that a sentence may have different labeling results for different aspects. An example is shown in Table 1.

Figure 2 shows the overall architecture of Opinion Transmission Network (OTN). It consists of a base ALSC module and a base AOWE module, as well as bidirectional opinion transmission mechanisms, respectively from AOWE to ALSC (AOWE2ALSC) and ALSC to AOWE (ALSC2AOWE). Following most state-of-the-art works [15, 8, 4], we employ a typical attention-based BiLSTM network as our base ALSC module in this work. In terms of AOWE, we adopt CNN as the base module for two considerations. The first reason is that CNN is widely used in various sequence labeling tasks and achieve state-of-the-art results, such name entity recognition [20], Chinese words segmentation [14], and aspect extraction [12]. Secondly, it can work in parallel and has more fast computation efficiency. Additionally, we enhance the ALSC module and the AOWE module with position embeddings [4] to incorporate aspect information.

For the ALSC task, distinguishing the aspect-related opinion words is helpful to predict the sentiment polarity of the aspect. Thus, we design the opinion transmission mechanism AOWE2ALSC to integrate opinion words information from AOWE into ALSC. Specifically, we transform the prediction results of the AOWE module into the form of auxiliary attention, as additional sentiment evidence for the ALSC module.

The ALSC2AOWE mechanism aims to exploit implicit opinion clues of ALSC to improve the AOWE task. In the ALSC module, the attention weights over words can indicate the aspect-related opinion words, while it is low-dimension and easily ignored when incorporated into the AOWE module. Therefore, we step backward and leverage the intermediate features in the attention layer of the ALSC module as latent opinions to improve the AOWE task.

The base ALSC module is an attention-based BiLSTM network enhanced with position embedding technique. Given a sentence $s=\{w_{1},w_{2},\cdots,w_{n}\}$ and an aspect $w_{a}$ in $s$, we first concatenate the word embedding and position embedding of each word $w_{i}$ as the word representation $\mathbf{e}_{i}$, i.e., $\mathbf{e}_{i}=[\mathbf{E}_{word}(w_{i});\mathbf{E}_{pos}(l_{i})]$. The $l_{i}$ indicates the relative distance of the word $w_{i}$ to the aspect $w_{a}$ and is calculated as $l_{i}=|i-a|$. The $\mathbf{E}_{word}$ and $\mathbf{E}_{pos}$ respectively represent the word embedding table and position embedding table.

With the enhanced word representations $\{\mathbf{e}_{1},\mathbf{e}_{2},\cdots,\mathbf{e}_{n}\}$, a BiLSTM network is applied to encode them and generate the corresponding hidden states $\{\mathbf{h}_{1},\mathbf{h}_{2},\cdots,\mathbf{h}_{n}\}$. Then we use the aspect representation $\mathbf{e}_{a}$ as query and employ the attention mechanism to capture potential opinion clues for the ALSC task. The attention weight $\alpha_{i}$ of the word $w_{i}$ is defined as:

where $\mathbf{W}_{e}$ denotes the weight matrix, $\mathbf{v}_{u}$ represents the weight vector, and $\mathbf{b}_{u}$ is the bias.

Finally, the aspect-related sentence representation $\mathbf{r}_{a}$ is a weighted sum of context representations $\mathbf{H}=\{\mathbf{h}_{1},\mathbf{h}_{2},\cdots,\mathbf{h}_{n}\}$, i.e., $\mathbf{r}_{a}=\mathbf{H}\boldsymbol{\alpha}$. In the base ALSC module, the representation $\mathbf{r}_{a}$ is fed into a linear layer and a softmax layer to predict the sentiment polarity of the aspect $a$ in the sentence $s$.

Similarily, the word representation $\mathbf{e}_{i}$ in the base AOWE is obtained by concatenating the word embedding and position embedding of each word $w_{i}$. We then employ a CNN encoder to capture context information in the sequence $\{\mathbf{e}_{1},\mathbf{e}_{2},\cdots,\mathbf{e}_{n}\}$ and obtain the corresponding feature vector $\mathbf{c}_{i}$ of the word $w_{i}$:

where $\theta_{\text{CNN}}$ represents the parameters of the CNN encoder.

The CNN encoder consists of 5 CNN layers. Each layer has a set of convolution filters, and each filter can map representations of k continuous words to single feature scalar, where k is the kernel size. ReLU activation is applied to each feature vector. We will present the details of the hyperparameters of the base AOWE module in the experiment settings.

To further incorporate the aspect information, we concatenate the CNN feature vector $\mathbf{c}_{i}$ with the word embedding of the given aspect $a$ as the final representation of each word $w_{i}$:

Finally, the sequence representations $\{\mathbf{r}_{1}^{o},\mathbf{r}_{2}^{o},\cdots,\mathbf{r}_{n}^{o}\}$ is fed into a two-layer perceptron and a softmax layer to predict the tag probability distribution for each word in the sentence $s$:

where $\mathbf{W}_{o1}$ and $\mathbf{W}_{o2}$ are the weight matrices, $\boldsymbol{b}_{o}$ denotes the bias.

As we have mentioned, aspect-oriented opinion words in a sentence can provide powerful evidence to infer the corresponding sentiment of the aspect. Therefore, we propose the opinion transmission mechanism AOWE2ALSC to leverage predictions from the AOWE module to help the ALSC module focus on aspect-oriented opinion words, thereby make more comprehensive sentiment predictions.

Specifically, we map the predicted probabilities of BIO tags over each word in the AOWE module to a probability distribution of each word being aspect-related opinion word as follows:

where $\mathbf{W}_{trans}\in\mathbb{R}^{3\times 1}$ is a weight matrix and maps probabilities of each word being tagged with $B,I,O$ to a single score.

Since the probability distribution $\mathbf{p}$ can be regarded as an additional attention knowledge from the AOWE moduel, we also merge the context representations $\mathbf{H}$ in the ALSC module with $\mathbf{p}$:

Finally, we concatenate the opinion representation $\mathbf{r}_{opinion}$ with the original representation $\mathbf{r}_{a}$ in the ALSC module to predict the aspect-level sentiment:

where $\mathbf{W}_{a}$ is the weight matrix and $\mathbf{b}_{a}$ denotes the bias.

The base ALSC module can capture some latent aspect-related opinion words through the attention mechanism. However, the attention weight $\alpha_{i}$ is the 1-dimension scalar in the ALSC module and easily neglected when we use it to enhance the AOWE module. Therefore, we exploit the attention feature $\mathbf{h}_{i}^{a}$ in Equation 1 to enrich the context representations of the ALSC module as follows:

Finally, the enriched context representation $\mathbf{r}_{i}^{o^{\prime}}$ is used to the predict tag of the word $w_{i}$ for the AOWE task:

For the ALSC task, we use cross-entropy loss between predicted sentiment label and the gold sentiment label as the task loss, which is defined as follows:

where $D$ indicates all data sample, $C$ denotes the sentiment label set, and $\hat{y}_{i}^{ALSC}$ is the predicted probability of the input sample belonging to the $i$-th sentiment.

In terms of the AOWE task, we define the cross-entropy loss as follows:

here the tags $\{O,B,I\}$ are correspondingly converted into labels $\{0,1,2\}$, and $\hat{y}_{i,j}^{AOWE}$ denotes the probability that the $i$-th word is predicted as the label $j$.

Because OTN is a joint model for both tasks of ALSC and AOWE, we minimize the losses $L^{ALSC}$ and $L^{AOWE}$ iteratively to optimize the OTN model.

As aforementioned, the OTN model is a joint model without requiring strict annotations on the same data for the ALSC task and the AOWE task. To verify this, we respectively use the datasets 14res for ALSC and 16res for AOWE. They are respectively derived from SemEval Challenge 2014 task 4 [11] and SemEval Challenge 2016 task 5 [10]. The original SemEval datasets do not provide the annotations of the corresponding opinion words for each aspect. Therefore, [3] annotate aspect-related opinion words for each sample and remove the samples without containing opinion words. Table 2 shows the statistics of the two datasets, the “Opinion” and “Pair” respectively denote the number of opinion words and pairs of aspects and opinion words in Table 2.

We adopt widely-used evaluation metrics for the two tasks. For ALSC, we use accuracy and macro-F1 score as evaluation metrics [1, 5]. For AOWE, we follow the previous work [3] and use precision, recall, and F1-score to measure the performance of different methods. An opinion word/phrase is deemed to be correct on the condition that the starting and ending positions of the prediction are both the same as those of the golden word/phrase.

We use 300-dimension GloVe [9] word embeddings pre-trained from 840B tokens to initialize word vectors, which are fixed during the training stage. The position embeddings are 100-dimension vectors and randomly initialized by a uniform distribution $U(-0.01,0.01)$. The dimension of the LSTM cells is 400. Table 3 shows the hyperparameters of the CNNs in the AOWE module. We adopt dropout on the embedding layer and the output layer with probability 0.5. Adam optimizer [7] is applied to update model parameters. The initial learning rate is 1e-3 and the mini-batch size is 16. We randomly select 20% samples from training sets as the validation sets for tuning hyperparameters and early stopping. We report the average results of 5 repeated experiments for each model.

We compare our OTN model with the following methods for ALSC and AOWE.

The main experiment results of ALSC and AOWE are shown in Table 4.

In terms of the ALSC task, the performance of attention-based and memory-based methods are comparable, while the CNN-based method GCAE performs worst. This shows the importance of modeling long-term dependency for ALSC. Thus we adopt an attention-based BiLSTM as our base ALSC module. By employing position embedding to incorporate aspect information, the base module achieves very competitive performance. Compared to the base module, our joint model OTN achieves further improvements through leveraging the additional opinion information from the AOWE task.

As for the AOWE task, the methods Dependency-rule and BiLSTM both perform poorly. The former uses coarse-grain POS patterns and lacks robustness. The latter fails to consider aspect information and outputs the same results for different aspects in a sentence. In contrast, the aspect-dependent methods PE-BiLSTM and IOG obtain obvious improvements. With the help of CRF, IOG+CRF achieves minor improvements against IOG. Different from LSTM-based methods, our designed base module using CNN and incorporating aspect information produces very competitive results on the AOWE dataset. Nevertheless, our joint model OTN still outperforms it by 0.8% in F1-score.

Benefiting from the two tailor-made opinion transmission mechanisms, OTN performs better than the two base modules, which proves the existence of mutual indication between the ALSC and AOWE tasks. Besides, OTN consistently outperforms other compared methods in both tasks. The comparison validates the effectiveness of our model.

To investigate the effects of the two opinion transmission mechanisms on OTN, we also conduct ablation study. In the experiments of “-ALSC task” and “-AOWE task”, we keep the model architecture unchanged but respectively remove the ALSC data for “-ALSC task” and AOWE data for “-AOWE task’. The “-AOWE2ALSC” and “-ALSC2AOWE” indicate that we remove the AOWE2ALSC mechanism or ALSC2AOWE mechanism from OTN.

Table 5 shows the results of ablation study. We can observe the performance of the model drops when it is trained on a single task or without opinion transmission mechanisms. The observation proves that OTN exploits the connection between the ALSC and AOWE tasks successfully and achieves improvements through our proposed opinion transmission mechanisms.

Most recent ALSC research utilizes the attention-based networks to capture the latent sentiment clues from the sentence for the given aspect, such as ATAE-LSTM [15], IAN [8] and PBAN [4], etc. On this basis, [13] employs memory network to conduct multi-hop attention to obtain more powerful sentiment clues for detecting the sentiment polarity of the aspect. Following the idea, memory-based methods achieve competitive performance on the ALSC [13, 1, 18, 19]. In addition, CNN [17], capsule network [2], and additional document sentiment data [5] are also applied for this task.

AOWE is a relatively new ABSA subtask. [3] formalizes it as an aspect-oriented sequence labeling task, and designs a state-of-the-art sequence labeling model based on LSTMs. Before them, a few works focus on the pair of aspect and opinion words. [6] proposes a rule-mining method to extract aspect words and regards the nearest adjective of aspect as the corresponding opinion words. [21] uses dependency-tree templates to extract valid aspect-opinion pairs.