Attention Transfer Network for Aspect-level Sentiment Classification

ASC Formalization Formally, given a sample $<s,t>$ from the ASC dataset $\mathcal{A}$, $s=\{w_{1},w_{2},...,w_{n}\}$ is a review sentence consisting of $n$ words and $t=\{w_{l},w_{l+1},...,w_{r}\}$ is a given opinion target containing $|r-l|$ words. The opinion target $t$ is a continuous subsequence of $s$. The goal of ASC is to predict the sentiment polarity (i.e., positive, neutral and negative) of the opinion target $t$ in the sentence $s$.

Before transferring attention knowledge, we first pre-train a DSC module on the large-scale DSC dataset $\mathcal{D}$. In this work, we employ a conventional attention-based BiLSTM as our DSC module.

For a review document $d=\{w_{1}^{d},w_{2}^{d},...,w_{n}^{d}\}$, we map it into the corresponding word representations $\{\mathbf{w}_{1}^{d},\mathbf{w}_{2}^{d},...,\mathbf{w}_{n}^{d}\}$ by looking up an embedding table $\mathbf{E}_{emb}\in\mathbb{R}^{|V|\times d_{e}}$, where $|V|$ is the vocabulary size and $d_{e}$ denotes the word embedding dimension. Then a BiLSTM network is applied to capture the contextual information for each word and generate a sequence of hidden states $\{\mathbf{h}_{1}^{d},\mathbf{h}_{2}^{d},...,\mathbf{h}_{n}^{d}\}$. To obtain the document representation $\mathbf{r}_{d}$, we employ the attention mechanism to aggregate the sentiment words that are significant for sentiment classification as follows:

$$\mathbf{r}_{d}={\sum_{i=1}^{n}}\alpha_{i}{\bf h}_{i}^{d},$$

(1)

where $\alpha_{i}$ is the attention weight of $\mathbf{h}_{i}^{d}$ and defined as:

	$\displaystyle\alpha_{i}$	$\displaystyle=\frac{\text{exp}(f({\bf h}_{i}^{d},{\bf h}_{avg}^{d}))}{\sum_{j=1}^{n}\text{exp}(f({\bf h}_{j}^{d},{\bf h}_{avg}^{d}))},$		(2)
	$\displaystyle f({\bf h}_{i}^{d},{\bf h}_{avg}^{d})$	$\displaystyle={\bf h}_{i}^{d}\cdot{\bf W}_{d}\cdot{\bf h}_{avg}^{d}+{\bf b}_{d},$		(3)

where $\mathbf{h}_{avg}^{d}$ is the average of all the hidden states, i.e., $\mathbf{h}_{avg}^{d}=\sum_{i=1}^{n}\mathbf{h}_{i}^{d}/n$, ${\bf W}_{d}$ and ${\bf b}_{d}$ are respectively the weight matrix and bias.

Finally, the representation $\mathbf{r}_{d}$ is fed to a linear layer and a softmax layer to predict the sentiment label of the review document $d$. We pre-train the DSC module by minimizing the cross-entropy loss between the predicted sentiment distribution and the ground truth. After pre-training is finished, all parameters in the DSC module are fixed.

As shown in the left part of Figure 1, the base ASC module has a similar architecture to the DSC module. The difference is that the ASC task needs to model opinion target information. To obtain target-aware context representations, we additionally employ position embedding besides word embedding, which is an effective method of modeling position information [Lin et al. (2016, Gehring et al. (2016]. Therefore, the base ASC module is an attention-based BiLSTM network enhanced with position embedding.

Specifically, given a sentence $s=\{w_{1},w_{2},...,w_{n}\}$ and an opinion target $t=\{w_{l},w_{l+1},...,w_{r}\}$ in $s$, we first map each word $w_{i}$ into its word embedding representation $\mathbf{w}_{i}$ by using the word embedding table. To incorporate opinion target information with position embedding, we calculate the relative distance $l_{i}$ of each word $w_{i}$ to the opinion target $t$:

$$l_{i}=\begin{cases}l-i&\text{ if }i<l,\\ 0&\text{ if }l\leq i\leq r,\\ i-r&\text{ otherwise }.\end{cases}$$

(4)

The distance index $l_{i}$ is mapped into the positional representation $\mathbf{p}_{i}$ by looking up a position embedding table $\mathbf{E}_{pos}\in\mathbb{R}^{L\times d_{p}}$, where $L$ denotes the maximal position index and $d_{p}$ is the embedding dimension. Then we concatenate the word embedding representation $\mathbf{w}_{i}$ and position embedding representation $\mathbf{p}_{i}$ as the repsentation $\mathbf{e}_{i}$ of the word $w_{i}$, i.e., $\mathbf{e}_{i}=[\mathbf{w}_{i};\mathbf{p}_{i}]$, where $[\cdot;\cdot]$ denotes the vector concatenation operation. Similarly, we employ a BiLSTM to receive the word represenations $\{\mathbf{e}_{1},\mathbf{e}_{2},\cdots,\mathbf{e}_{n}\}$ as input and generate target-aware context representations $\{\mathbf{h}_{1},\mathbf{h}_{2},\cdots,\mathbf{h}_{n}\}$. Different from the attention part of the DSC module, we use the opinion target represenation $\mathbf{t}=\sum_{i=l}^{r}\mathbf{h}_{i}/(r-l)$ as query in the ASC task to extract target-dependent sentiment clues:

	$\displaystyle f({\bf h}_{i},{\bf t})$	$\displaystyle={\bf h}_{i}\cdot{\bf W}_{s}\cdot{\bf t}+{\bf b}_{s},$		(5)
	$\displaystyle\beta_{i}$	$\displaystyle=\frac{\text{exp}(f({\bf h}_{i},{\bf t}))}{\sum_{j=1}^{n}\text{exp}(f({\bf h}_{j},{\bf t}))},$		(6)
	$\displaystyle\mathbf{r}_{s}$	$\displaystyle={\sum_{i=1}^{n}}\beta_{i}{\bf h}_{i},$		(7)

where ${\bf W}_{s}$ and ${\bf b}_{s}$ are respectively the weight matrix and bias.

Finally, the target-dependent sentence representation $\mathbf{r}_{s}$ is used for detecting the sentiment polarity of the target $t$, and the base ASC module can optimized by minimizing the following cross-entropy loss:

	$\displaystyle\hat{y}_{i}={\rm softmax}({\bf W}_{o}{\bf r}_{s}+{\bf b}_{o}),$		(8)
	$\displaystyle\mathcal{L}_{o}=-\sum_{i\in\mathcal{A}}y_{i}log(\hat{y}_{i}),$		(9)

where ${\hat{y}}_{i}$ and $y_{i}$ respectively are the predictive class distribution and golden class distribution.

To leverage the attention knowledge of the DSC module, we simultaneously input the sentence $s$ into the base ASC module and the pre-trained DSC module when performing the ASC task, generating the attention weights $\beta_{i}$ in Equation 6 and $\alpha_{i}$ in Equation 2.

As mentioned before, the attention mechanism of the ASC module cannot reach full potential due to limited training data, which means that the attention weights $\beta_{i}$ may fail to capture target-relevant sentiment words. In contrast, sufficient DSC data enables the DSC module to extract sentiment words more accurately. Thus we propose the Attention Guidance approach to guide the learning of the attention weights $\beta_{i}$ with the help of $\alpha_{i}$. Nevertheless, there is a tiny gap between the attention weights $\alpha_{i}$ and $\beta_{i}$. Since the DSC task only detects the overall sentiment of a review, the sentiment words captured by $\alpha_{i}$ are global and target-irrelevant. To make up the gap, we use a heuristic method to transform target-irrelevant attention weight $\alpha_{i}$ into target-relevant weight $\delta_{i}$:

	$\displaystyle\alpha_{i}^{\prime}$	$\displaystyle=\frac{1}{2^{(l_{i}-1)}}\alpha_{i},$		(10)
	$\displaystyle\delta_{i}$	$\displaystyle=\frac{e^{\alpha_{i}^{\prime}}}{\sum_{i=1}^{n}e^{\alpha_{i}^{\prime}},}$		(11)

where $l_{i}$ denotes the relative distance between the word and the target as in Equation 4. We can see that a word nearer to the target receives a higher attention weight according to $\delta_{i}$, because the closer word has a bigger probability of modifier relation to the target.

Finally, we apply KL (Kullback–Leibler divergence) to describe the differences between attention distributions $\beta$ and $\delta$:

	$\displaystyle KL(\delta||\beta)$	$\displaystyle=\sum_{i=1}^{n}\delta_{i}log\frac{\delta_{i}}{\beta_{i}},$		(12)
		$\displaystyle=\sum_{i=1}^{n}(\delta_{i}log\delta_{i}-\delta_{i}log\beta_{i}).$		(13)

In the pre-trained DSC module, the above term $\sum_{i=1}^{n}\delta_{i}log\delta_{i}$ in Equation 13 is invariant for the given sentence $s$ and the opinion target $t$. Therefore, we can minimize the loss $\mathcal{L}_{a}=\sum_{i=1}^{n}-\delta_{i}log\beta_{i}$ to guide the ASC module to focus on target-relevant sentiment words. In the Attention Guidance approach, the final loss is defined as follows:

$$\mathcal{L}=\mathcal{L}_{o}+\lambda\mathcal{L}_{a}.$$

(14)

where $\lambda$ is the hyperparameter that controls the importance of $\mathcal{L}_{a}$ .

Attention Guidance learns the attention ability of the DSC module through an auxiliary supervision signal. However, it cannot leverage the attention weights from the DSC module during the testing stage and wastes the pre-trained knowledge. To make full use of the additional attention capacity, we further propose the Attention Fusion approach to incorporate them directly.

Specifically, we design a fusion gate $g$ to integrate the global attention weight $\alpha_{i}$ from the DSC module and the target-dependent attention weight $\beta_{i}$ from the ASC module, thereby generating more comprehensive and accurate attention weight $\gamma_{i}^{\prime}$:

	$\displaystyle g$	$\displaystyle=\sigma({\bf W}_{g}[\alpha_{i};\beta_{i}]),$		(15)
	$\displaystyle\gamma_{i}$	$\displaystyle=g\alpha_{i}+(1-g)\beta_{i},$		(16)
	$\displaystyle\gamma_{i}^{\prime}$	$\displaystyle=\frac{e^{\gamma_{i}}}{\sum_{i=1}^{n}e^{\gamma_{i}}},$		(17)

where $\sigma$ denotes sigmoid function and ${\bf W}_{g}$ is the weight matrix.

Finally, we replace $\beta_{i}$ in Equation 7 with the new attention weight $\gamma_{i}^{\prime}$ to obtain the target-dependent sentence representation $\mathbf{r}_{s}$ for sentiment prediction.

We evaluate our model on two ASC benchmark datasets from SemEval 2014 Task 4 [Pontiki et al. (2014]. They respectively contain reviews from Restaurant and Laptop domains. Following previous studies [Tang et al. (2016b, Chen et al. (2017, He et al. (2018], we remove samples with conflicting polarities in all datasets. The statistics of the ASC datasets are shown in Table 2.

To pre-train the DSC module, we employ two larget-scale DSC datasets, respectively Yelp Review and Amazon Review [Li et al. (2018a]. The DSC dataset Yelp Review is applied to transfer attention knowledge for the ASC dataset Restaurant. The Amazon Review is used for the dataset Laptop. Table 2 shows their statistics. In this work, we adopt Accuracy and Macro-F1 score as the metrics to evaluate the performance of different methods on the ASC task.

In our experiments, word embeddings are initialized by 300-dimension GloVe [Pennington et al. (2014]. After initialization, the word vectors are fixed and not fine-tuned during the training stage. All the weight matrices and biases are given the initial value by sampling from the uniform distribution $U(-0.1,0.1)$. The dimension of LSTM cell hidden states is set to 300. We employ stochastic gradient descent (SGD) with momentum [Qian (1999] to train models. The initial learning rate and momentum parameter are respectively set to 0.1 and 0.9. In addition, we apply dropout [Hinton et al. (2012] with probability 0.5 on embedding layer as a regularizer. The parameter $\lambda$ in Attention Guidance approach is set to 0.4. All hyper-parameters were tuned on 20% randomly held-out training data. Finally, we run each model five times and report the average result of them.

We divide compared methods into two groups according to whether using transferred knowledge.

Majority assigns each instance in the test set with the most frequent sentiment label in the training set.

Feature-based SVM [Kiritchenko et al. (2014] is the top system of SemEval 2014 Task 4. It uses n-gram features, parse features and lexicon features to train an SVM classifier.

TD-LSTM [Tang et al. (2016a] applies two LSTM networks to model the left context and right context of opinion target respectively, then concatenates their last hidden states for sentiment prediction.

ATAE-LSTM [Wang et al. (2016] concatenates the word embedding and target embedding as the input of LSTM, then employs the attention mechanism to capture target-dependent sentiment information.

IAN [Ma et al. (2017] proposes the interactive attention to interactively learn representations of the context and target. The two representations are then concatenated for prediction.

MemNet [Tang et al. (2016b] uses multi-hops attention on the word embeddings to generate the target-dependent sentence representation.

RAM [Chen et al. (2017] works similar to the method MemNet. It employs BiLSTM to build memory and applies GRU-based multi-hops attention.

IARM [Majumder et al. (2018] incoporates the neighboring targets-related information for ASC by using memory networks.

MGAN [Fan et al. (2018] proposes a fine-grained attention mechanism to capture the word-level interaction between target and context, then combines it with coarse-grained attention for ASC.

GCAE [Xue and Li (2018] uses a convolutional neural network (CNN) with gating mechanisms to perform the ASC task.

TNet [Li et al. (2018b] proposes target specific transformation component to integrate target information into the word representation.

PRET+MULT [He et al. (2018] shares shadow embedding and LSTM layers between the ASC model and the DSC model through multi-task learning.

TransCap [Chen and Qian (2019] employs capsule network to share the bottom features between the ASC task and the DSC task.

The main results are shown in Table 3. We classify the results into three groups: the first lists the classic methods for the ASC task, the second presents two existing transfer-based methods, and the last is our base ASC model and enhanced versions with transferring attention knowledge. We use ATN-AG and ATN-AF respectively to represent ATN using Attention Guidance and Attention Fusion.

The method Feature-SVM obtains competitive results on the restaurant dataset but performs poorly on the laptop dataset. This may be attributed to that the performance of simple feature-based methods heavily relies on the quality of hand-crafted features. IAN achieves better performance than TD-LSTM and ATAE-LSTM by using the interactive attention mechanism to learn the representations of context and opinion target. With combining of fine-grained and coarse-grained attention mechanisms, MGAN achieves the best performance among all pure attention-based models. Among the memory-based methods, it can be observed that RAM outperforms MemNet and IARM on the laptop dataset, which validates the effectiveness of multi-hops attention based on recurrent network. GCAE performs poorly compared with other neural methods, as CNN is not good at capturing the long-term dependencies between context words. TNet achieves state-of-the-art performance by designing target-specific transformation mechanism between LSTM and CNN.

PRET+MULT and Transcap transfer knowledge implicitly from large-scale DSC data to the ASC task through sharing parameters and multi-task learning. They show superiority compared to some methods without transferring knowledge. For example, the base model of PRET+MULT is an attention-based LSTM similar to ATAE-LSTM. We can observe that PRET+MULT outperforms ATAE-LSTM significantly, and achieves 2.78% and 5.44% accuracy improvements respectively on the restaurant and laptop datasets. Transcap obtains better results compared to PRET+MULT, which verifies the effectiveness of capsule network for capturing shared features.

Our base ASC model attention-based BiLSTM enhanced with position embedding performs better than some attention-based models, such as ATAE-LSTM and IAN. This result indicates that position embedding is beneficial for modeling target information in the ASC task. On this basis, our attention transfer models ATN-AG and ATN-AF respectively achieve about 1% and 2% improvements in accuracy on the restaurant dataset, and over 2.8% improvements on the laptop dataset. In addition, they surpass two existing methods that use transferred knowledge obviously, i.e., PRET+MULT and Transcap. These comparisons demonstrate the effectiveness of our proposal of explicitly transferring attention knowledge from resource-rich DSC data to the ASC task. Compared with ATN-AG, ATN-AF achieves better performance on the restaurant dataset. It is reasonable because ATN-AG cannot leverage the attention weights of the DSC module during the testing stage. Nevertheless, ATN-AG still obtains comparable results on the laptop dataset and has a faster inference speed than ATN-AF.

To investigate the effect of DSC data size on our approaches, we vary the percentage of DSC data from 0% to 100% to report the results of ATN-AG and ATN-AF. The critical values 0% and 100% respectively mean no DSC data and using the complete DSC dataset. The results are shown in Figure 3.

We can observe that our approaches ATN-AG and ATN-AF both achieve very stable improvements on the two datasets with the increase of DSC data size. This indicates that the ASC task indeed benefits from the transferred attention knowledge from the pre-trained DSC module. Consistent and stable improvements show the robustness of our approaches.

To analyze the effect of hyper-parameter $\lambda$ in Equation 14 on ATN-AG, we adjust it in [0, 1] to conduct experiments and the step is 0.1. Figure 3 shows the performance of ATN-AG with different $\lambda$ on the restaurant and laptop datasets.

We can see that the curves on two datasets have an overall upward trend when $\lambda$ $<$ 0.4, but become flat or downward once $\lambda$ $>$ 0.4. In the upward part, the attention knowledge from the DSC module is a useful guidance signal to help the ASC module to focus on sentiment words more accurately, thus improve the performance of ASC. Once the weight $\lambda$ exceeds 0.4, the transferred attention knowledge begins to dominate the attention process while the ASC module loses the mastership and perform worse. Therefore, we finally set $\lambda$ to be 0.4 on two datasets.

In the ATN model, we propose the approaches Attention Guidance and Attention Fusion to help the ASC module to capture sentiment clues more accurately. To verify this, we analyze some dozens of instances from the test set. Compared with the base ASC model, we find that our attention transfer methods can deal with low-frequency sentiment words and complex sentiment patterns such as negation. Table 4 shows the attention visualizations of two examples and the corresponding sentiment predictions under the base model, ATN-AG and ATN-AF. Note that the darker color means higher attention weight.

In the first example, the base ASC model mainly focuses on the adverb “mostly”, while fails to capture the critical sentiment clue “reliable”. According to the statistics, the word “reliable” only appears five times in the training set. This indicates that the base model is not good at catching low-frequency sentiment words, thus makes wrong sentiment predictions. In contrast, the enhanced models ATN-AG and ATN-AF with transferred attention knowledge both successfully capture the informative word “reliable”, and give the right predictions.

From the second example, we can see that the base ASC model mainly focuses on the word “enjoy” rather than the sentiment negator “not”. It is hard for the base model to learn the negation with the insufficient labeled dataset. With the help of the external attention knowledge, our approaches ATN-AG and ATN-AF pay more attention to the negator “not”, and make correct sentiment predictions.

The above observations show that our approaches indeed improve the low-resource task ASC with the transferred attention knowledge and retain good interpretability.

Early works adopt supervised learning and devote to designing effective features for the ASC task, such as n-gram features [Kiritchenko et al. (2014] and sentiment lexicons [Vo and Zhang (2015]. The performance of these methods heavily depends on labor-intensive feature engineering. With the development of deep learning, ?) use two Long Short-Term Memory (LSTM) [Hochreiter and Schmidhuber (1997] networks to respectively model the left context and right context of the given opinion target. However, it cannot capture the association between the context and opinion target. To address the issue, recent works employ the attention mechanism to catch target-dependent sentiment context and achieve very promising resutls [Wang et al. (2016, Ma et al. (2017, Fan et al. (2018]. Instead of single attention, some works propose multi-hops attention based on memory networks [Sukhbaatar et al. (2015] to detect more powerful sentiment clues [Tang et al. (2016b, Chen et al. (2017, Majumder et al. (2018].

Despite attention-based models showing the potential for ASC, they highly rely on data-driven attention mechanism. Unfortunately, public ASC datasets are all small-scale because of the complexity of annotation. Insufficient labeled data finally limits the effectiveness of attention mechanism for the ASC task. Different from the above methods, we improve the attention capacity of the ASC model in this work, by transferring substantial attention knowledge from the DSC model pre-trained with resource-rich document-level sentiment classification data.

Transfer learning aims to extract knowledge from one or more source tasks and then apply them to a target task. Neural transfer learning has proven effective for image recognition [Donahue et al. (2014] and natural language processing tasks [Mou et al. (2016, Dong and De Melo (2018, Wu et al. (2020]. ?) are the first to transfer knowledge from document-level review data to improve the ASC task through sharing embedding and LSTM layers. ?) employ capsule network to share bottom features between the ASC task and DSC task. In this work, we aim to transfer attention knowledge from the DSC model explicitly to improve the effectiveness of attention mechanism for the ASC task. In contrast to the two existing works, our proposed approaches show better performance and good interpretability.