Context-Guided BERT for Targeted Aspect-Based Sentiment Analysis

Self-attention networks, exemplified in the Transformer (Vaswani et al. 2017), have become the de facto go-to neural models for a variety of NLP tasks including machine translation (Vaswani et al. 2017), language modeling (Liu and Lapata 2018; Dai et al. 2019) and sentiment analysis (Shen et al. 2018; Wu et al. 2019). BERT, a popular and successful variant of the Transformer, has successfully been applied to various NLP tasks (Reddy, Chen, and Manning 2019; Devlin et al. 2019). In addition, previous studies have also shown some evidence that self-attention weights may learn syntactic (Hewitt and Manning 2019) and semantic (Wu, Nguyen, and Ong 2020) information.

Attention has been successfully applied to many NLP tasks. Various forms of attention are proposed including additive attention (Bahdanau, Cho, and Bengio 2015), dot-product attention (Luong, Pham, and Manning 2015) and scaled dot-product attention used in self-attention (Vaswani et al. 2017). Many of them rely on a softmax activation function to calculate attention weights for each position. As a result, the output vector is in the convex hull formed by all other hidden input vectors, preventing the attention gate from learning subtractive relations. Tay et al. (2019) recently proposed a way to overcome this limitation by allowing attention weights to be negative (“quasi” attention), which allows input vectors to add to ($+$1), not contribute to ($0$), and even subtract from ($-$1) the output vector.

Early research in ABSA introduced classic benchmark datasets and have proposed many baseline methods including lexicon-based and pre-neural classifiers (Pontiki et al. 2014; Kiritchenko et al. 2014; Pontiki et al. 2015, 2016). Since the debut of recurrent neural networks, various RNNs have been developed to generate aspect-aware sentence embeddings and sentiment labels (Tang et al. 2016; Chen et al. 2017; Li et al. 2018). Likewise, researchers have also adapted CNNs (Xue and Li 2018; Huang and Carley 2018), recursive neural networks (Nguyen and Shirai 2015), aspect-aware end-to-end memory networks (Tang, Qin, and Liu 2016) and cognitively inspired deep neural networks (Lei et al. 2019) to generate aspect-aware sentence embeddings.

Motivated by attention mechanisms in deep learning models, many recent ABSA papers have integrated attention into neural models such as RNNs (Wang et al. 2016; Chen et al. 2017; Liu and Zhang 2017; He et al. 2018), CNNs (Zhang, Li, and Song 2019), and memory networks (Ma et al. 2017; Majumder et al. 2018; Liu, Cohn, and Baldwin 2018) to learn different attention distributions for aspects and generate attention-based sentence embeddings. Most recently, self-attention-based models such as BERT have been applied to ABSA, by using BERT as the embedding layer (Song et al. 2019; Yu and Jiang 2019; Lin, Yang, and Lai 2019), or fine-tuning BERT-based models with an ABSA classification output layer (Xu et al. 2019). These papers show that using BERT brings significant performance gains in ABSA.

Building on ABSA, Saeidi et al. (2016) proposed a generalized TABSA task (with multiple potential targets) with a new benchmark dataset and LSTM-based baseline models. Various neural models have been proposed for TABSA such as a memory network with delayed context-aware updates (Liu, Cohn, and Baldwin 2018) and interaction-based embedding layers to generate context-aware embeddings (Liang et al. 2019). Researchers have also tried to integrate attention mechanism with LSTMs to predict sentiment for target-aspect pairs (Ma, Peng, and Cambria 2018). With the recent success of BERT-based models, various papers have used BERT to generate contextualized embeddings for input sentences, which are then used to classify sentiment for target-aspect pairs (Huang and Carley 2019; Hu et al. 2019). More recent papers have fine-tuned BERT for TABSA either by (i) constructing auxiliary sentences with different pairs of targets and aspects or (ii) modifying the top-most classification layer to also take in targets and aspects (Rietzler et al. 2020; Sun, Huang, and Qiu 2019; Li et al. 2019). To the best of our knowledge, no work has been published on modifying the BERT architecture for TABSA tasks. Instead of keeping BERT as a blackbox, we enable BERT to be context-aware by modifying its neural architecture to account for context in its attention distributions.

We formulate the Sentihood dataset as a TABSA task. Given a sentence $s$ with a sequence of words $\{w_{1},w_{2},...w_{n}\}$²²2We append a classifier token (i.e., [CLS]) in the beginning of each input sentence as in the BERT model (Devlin et al. 2019)., where some words are target pronouns $\{w_{i},...,w_{j}\}$ from a fixed set $T$ of $k$ predefined targets $\{t_{1},...,t_{k}\}$, the goal is to predict sentiment labels for each aspect associated with each unique target mentioned in the sentence. Following the setup in the original Sentihood paper (Saeidi et al. 2016), given a sentence $s$, a predefined target list $T$ and a predefined aspect list $A=${general, price, transit-location, safety}, the model predicts a sentiment label $y\in${none, negative, positive} for a given pair of $\{(t,a):(t\in T,a\in A)\}$. Note that the model predicts a single sentiment label for each unique target-aspect pair in a sentence. We show an example of TABSA in Fig. 1.

We use the SemEval-2014 Task 4 dataset (Pontiki et al. 2014) to formulate an ABSA task: Given a sentence, we predict a sentiment label $y\in${none, negative, neutral, positive, conflict} for each aspect $\{a:(a\in A)\}$ with a predefined aspect list $A=${price, anecdotes, food, ambience, service}.

Our Context-Guided BERT (CG-BERT) model is based on the context-aware Transformer model proposed by Yang et al. (2019), which we adapted to the (T)ABSA task. Multi-headed self-attention (Vaswani et al. 2017) is formulated as:

$\displaystyle\mathbf{A}_{\text{Self-Attn}}^{h}=\text{softmax}\left(\frac{\mathbf{Q}^{h}{\mathbf{K}^{h}}^{T}}{\sqrt{d_{h}}}\right)$

(1)

where $\mathbf{Q}^{h}\in\mathbb{R}^{n\times d}$ and $\mathbf{K}^{h}\in\mathbb{R}^{n\times d}$ are query and key matrices indexed by head $h$, and ${\sqrt{d_{h}}}$ is a scaling factor. We integrate context into BERT by modifying $\mathbf{Q}$ and $\mathbf{K}$ matrices of the original BERT model (Devlin et al. 2019):

$$\displaystyle\begin{bmatrix}\mathbf{\hat{Q}}^{h}\\ \mathbf{\hat{K}}^{h}\end{bmatrix}=\left(1-\begin{bmatrix}\lambda_{Q}^{h}\\ \lambda_{K}^{h}\end{bmatrix}\right)\begin{bmatrix}\mathbf{Q}^{h}\\ \mathbf{K}^{h}\end{bmatrix}+\begin{bmatrix}\lambda_{Q}^{h}\\ \lambda_{K}^{h}\end{bmatrix}\left(\mathbf{C}^{h}\begin{bmatrix}\mathbf{U}_{Q}\\ \mathbf{U}_{K}\end{bmatrix}\right)$$

(2)

where $\mathbf{C}^{h}\in\mathbb{R}^{n\times d_{c}}$ is the context matrix for each head and defined in Sec. 3.6, $\{\lambda_{Q}^{h},\lambda_{K}^{h}\}\in\mathbb{R}^{n\times 1}$ are learnt context weights, and $\{\mathbf{U}_{Q},\mathbf{U}_{K}\}\in\mathbb{R}^{d_{c}\times d_{h}}$ are weights of linear layers used to transform input context matrix $\mathbf{C}_{h}$. The modified $\mathbf{\hat{Q}}$ and $\mathbf{\hat{K}}$ are then used to calculate context-aware attention weights using the dot-product of both matrices. In contrast to the original implementation (Yang et al. 2019), here we allow both $\lambda_{Q}^{h}$ and $\lambda_{K}^{h}$ to differ across heads, which allows variance in how context is integrated in each head.

We use a zero-symmetric gating unit to learn context gating factors $\{\lambda_{Q},\lambda_{K}\}$:

$$\displaystyle\begin{bmatrix}\lambda_{Q}^{h}\\ \lambda_{K}^{h}\end{bmatrix}=\text{tanh}\left(\begin{bmatrix}\mathbf{Q}^{h}\\ \mathbf{K}^{h}\end{bmatrix}\begin{bmatrix}\mathbf{V}_{Q}^{h}\\ \mathbf{V}_{K}^{h}\end{bmatrix}+\mathbf{C}^{h}\begin{bmatrix}\mathbf{U}_{Q}\\ \mathbf{U}_{K}\end{bmatrix}\begin{bmatrix}\mathbf{V}_{Q}^{C}\\ \mathbf{V}_{K}^{C}\end{bmatrix}\right)$$

(3)

where $\{\mathbf{V}_{Q}^{h},\mathbf{V}_{K}^{h},\mathbf{V}_{Q}^{C},\mathbf{V}_{K}^{C}\}\in\mathbb{R}^{d_{h}\times 1}$ are weights of linear layers to transform corresponding matrices. We chose to use tanh as our activation function as this allows the context to contribute to $\mathbf{\hat{Q}}^{h},\mathbf{\hat{K}}^{h}$ both positively and negatively³³3The original implementation (Yang et al. 2019) used sigmoid.. This enriches the representation space of both matrices, and the resulting attention distribution. We note that tanh may increase the magnitude of $\mathbf{Q}$, $\mathbf{K}$, and large magnitude of $\mathbf{Q}$, $\mathbf{K}$ may push gradients to excessively small values, which may prevent model learning, as noted by Vaswani et al. (2017) and Britz et al. (2017). However, our results suggest that this did not negatively affect our model performance.

Our second neural network model (QACG-BERT) architecture proposes using a Quasi Attention function for (T)ABSA. The value of self-attention weights $\mathbf{A}_{\text{Self-Attn}}^{h}$ in a vanilla implementation (using softmax), is bounded between $[0,1]$. In other words, it only allows a convex weighted combination of hidden vectors at each position. This allows hidden states to contribute only additively, but not subtractively, to the attended vector. We include a quasi-attention calculation to enable learning of both additive as well as subtractive attention (Tay et al. 2019). Formally, we formulate our new attention matrix as a linear combination of a regular softmax-attention matrix and a quasi-attention matrix:

$\displaystyle\mathbf{\hat{A}}^{h}=\mathbf{A}_{\text{Self-Attn}}^{h}+\lambda_{A}^{h}\mathbf{A}_{\text{\emph{Quasi}-Attn}}^{h}$

(4)

where $\lambda_{A}^{h}$ is a scalar to represent the compositional factor to control the effect of context on attention calculation. $\mathbf{A}_{\text{Self-Attn}}^{h}$ is defined as in Eqn. 1. To derive the quasi-attention matrix, we first define two terms quasi-context query $\mathbf{C}_{Q}^{h}$ and quasi-context key $\mathbf{C}_{K}^{h}$:

$$\displaystyle\begin{bmatrix}\mathbf{C}_{Q}^{h}\\ \mathbf{C}_{K}^{h}\end{bmatrix}=\mathbf{C}^{h}\begin{bmatrix}\mathbf{Z}_{Q}\\ \mathbf{Z}_{K}\end{bmatrix}$$

(5)

where $\{\mathbf{Z}_{Q},\mathbf{Z}_{K}\}\in\mathbb{R}^{d_{e}\times d_{h}}$ are weights of linear layers to transform the raw context matrix, and $\mathbf{C}^{h}$ is the same context matrix in Eqn. 2 (Defined in Sec. 3.6). Next, we define the quasi-attention matrix as:

$\displaystyle\mathbf{A}_{\text{\emph{Quasi}-Attn}}^{h}=\alpha\cdot\text{sigmoid}\left(\frac{f_{\psi}(\mathbf{C}_{Q}^{h},\mathbf{C}_{K}^{h})}{\sqrt{d_{h}}}\right)$

(6)

where $\alpha$ is a scaling factor and $f_{\psi}(\cdot)$ is a similarity measurement to capture similarities between $\mathbf{C}_{Q}^{h}$ and $\mathbf{C}_{K}^{h}$. For simplicity, we use dot-product to parameterize $f_{\psi}$, and set $\alpha$ to be $1.0$. Other $f_{\psi}$ that have been used include negative $L$-1 distance (Tay et al. 2019). As a result, our $\mathbf{A}_{\text{Quasi-Attn}}$ is bounded between $[0,1]$. We then derive our bidirectional gating factor $\lambda_{A}$ as:

$$\displaystyle\begin{bmatrix}\lambda_{Q}^{h}\\ \lambda_{K}^{h}\end{bmatrix}=\text{sigmoid}\left(\begin{bmatrix}\mathbf{Q}^{h}\\ \mathbf{K}^{h}\end{bmatrix}\begin{bmatrix}\mathbf{V}_{Q}^{h}\\ \mathbf{V}_{K}^{h}\end{bmatrix}+\begin{bmatrix}\mathbf{C}_{Q}^{h}\\ \mathbf{C}_{K}^{h}\end{bmatrix}\begin{bmatrix}\mathbf{V}_{Q}^{C}\\ \mathbf{V}_{K}^{C}\end{bmatrix}\right)$$

(7)

$\displaystyle\lambda_{A}^{h}=1-(\beta\cdot\lambda_{Q}^{h}+\gamma\cdot\lambda_{K}^{h})$

(8)

where $\{\beta,\gamma\}$ are scalars that control the composition weightings. For simplicity, we set $\{\beta,\gamma\}$ = 1.0. We formulate the gating factor to be bidirectional, meaning it takes on both positive and negative values, and the output is bounded between $\{-1,1\}$. Our intuition is that the context-based quasi-attention may contribute either positively or negatively to the final attention weights. Consider Eqn. 4: as the first term $\mathbf{A}_{\text{Self-Attn}}^{h}$ is in $\{0,1\}$, and the second term is made up of a term ($\lambda_{A}$) that is in $[-1,1]$ and another ($\mathbf{A}_{\text{Quasi-Attn}}$) that is in $[0,1]$, hence the final attention $\mathbf{\hat{A}}$ lies in $[-1,2]$. That is to say, the final attention weights can take values representing compositional operations including subtraction ($-$1), deletion ($\times$0), inclusion/addition ($+$1/$+$2) among hidden vectors across positions. We hypothesize that the quasi-attention provides a richer method to integrate context into the calculation of attention.

We use the final hidden state (the output of the final layer of the BERT model) of the first classifier token (i.e., [CLS]) as the input to the final classification layer for a C-class classification. This is similar to previous studies (Sun, Huang, and Qiu 2019). For a given input sentence, we denote this vector as $e_{\text{CLS}}\in\mathbb{R}^{1\times d}$. Then, the probability of each sentiment class $y$ is given by $y=\text{softmax}(e_{\text{CLS}}\mathbf{W}_{\text{CLS}}^{T})$ where $\mathbf{W}_{\text{CLS}}\in\mathbb{R}^{C\times d}$ are the weights of the classification layer, and $y\in\mathbb{R}^{1\times C}$. The label with highest probability will be selected as the final prediction.

We use a single integer to represent a context associated with an aspect and a target in any (T)ABSA task, and only an aspect in the ABSA task. We transform these integers into embeddings via a trainable embedding layer, which derives $\mathbf{C}^{h}$ in CG-BERT, and $\mathbf{C}_{Q}^{h}$ and $\mathbf{C}_{K}^{h}$ in QACG-BERT. For example, given $|t|$ targets and $|a|$ aspects for any (T)ABSA task, the total number of possible embeddings is $|t|\cdot|a|$. We then concatenate the context embedding with the hidden vector for each position $\mathbf{E}\in\mathbb{R}^{n\times d}$, and pass them into a feed-forward linear layer with a residual connection to formulate the context matrix $\mathbf{C}^{h}=[\mathbf{E}_{c},\mathbf{E}]\mathbf{W}_{c}^{T}$ where $\mathbf{E}_{c}\in\mathbb{R}^{n\times d}$ is the context embedding and $\mathbf{W}_{c}\in\mathbb{R}^{d\times 2d}$ are the learnt weights for this feed-forward layer.

Previous studies show that fine-tuning pretrained BERT models increases performance significantly in many NLP tasks (Sun, Huang, and Qiu 2019; Rietzler et al. 2020). Since our models share most of the layers with a standard BERT model, we import weights from pretrained BERT models for these overlapping layers. The weights of the newly added layers are initialized to be small⁴⁴4We also tried initializing the weights in the newly added layers with larger variance, and found similar performance. random numbers drawn from a normal distribution $\mathcal{N}(0,\sigma^{2})$ with $\sigma=e^{-3}$. As a result, the gating factors in Eqn. 2 and Eqn. 1 start at values close to zero. This initialization enables the task-specific weights to start from the pretrained weights and slowly diverge during training.

We evaluate our models with two datasets in English. For the TABSA task, we used the Sentihood dataset ⁵⁵5https://github.com/uclnlp/jack/tree/master/data/sentihood which was built by questions and answers from Yahoo! with location names of London, UK. It consists of 5,215 sentences, with 3,862 sentences containing a single target and 1,353 sentences containing multiple targets. For each sentence, we predict sentiment label $y$ for each target-aspect pair $(t,a)$. For the ABSA task, we used the dataset from SemEval 2014, Task 4 ⁶⁶6http://alt.qcri.org/semeval2014/task4/, which contains 3,044 sentences from restaurant reviews. For each sentence, we predict the sentiment label $y$ for each aspect $a$. Each dataset is partitioned to train, development and test sets as in its original paper.

As in previous studies (Pontiki et al. 2014; Saeidi et al. 2016) we define two subtasks for each dataset: (1) aspect categorization and 2) aspect-based sentiment classification. For aspect categorization, the problem is to detect whether a aspect $a$ is mentioned (i.e., none means not mentioned) in the input sentence for a target $t$ if it is a TABSA task. For aspect-based sentiment classification, we give the model aspects that present (i.e., ignoring none’s) and have the model predict the valence of the sentiment (i.e., potential labels include negative and positive for Sentihood, and negative, neutral, positive, conflicting sentiment for Semeval Task 4).

As in the original BERT-base model (Devlin et al. 2019), our models consists of 12 heads and 12 layers, with hidden layer size 768. The total number of parameters for both of our models increased slightly due to the additional linear layers added comparing to previous BERT-based models for ABSA tasks (Sun, Huang, and Qiu 2019) which consists of about 110M parameters. The CG-BERT and QACG-BERT consists of about 124M parameters. We trained for 25 epochs with a dropout probability of 0.1. The initial learning rate is $2e^{-5}$ for all layers, with a batch size of 24. We used the pretrained weights from the uncased BERT-base model ⁷⁷7https://storage.googleapis.com/bert˙models/2020˙02˙20/uncased˙L-12˙H-768˙A-12.zip.

We used a single Standard NC6 instance on Microsoft Azure, which is equipped with a single NVIDIA Tesla K80 GPU with 12G Memory. Training both models across two datasets took approximately 11 hours.

For the TABSA task, we compared the performance of our models with previous models in Table 1. Following Ma, Peng, and Cambria (2018) and Sun, Huang, and Qiu (2019), for aspect categorization (is a given aspect present in the sentence? If aspect is not present, the label is by definition none), we report strict accuracy (model needs to correctly identify all aspects for a given target in the sentence to be counted as accurate), Macro-F1 (the harmonic mean of the Macro-precision and Macro-recall of all targets.) and AUC. For sentiment classification (given an aspect present in the sentence, is the valence negative or positive?), we report accuracy and AUC.

For the ABSA task, we compared our models with multiple best performing models for the SemEval-2014 Task 4 dataset in Table 2. Following Pontiki et al. (2016), for aspect categorization, we report Precision, Recall, and F1. For aspect-based sentiment classification, we report accuracies for three different evaluations: binary classification (i.e., negative or positive), 3-class classification (negative, neutral or positive) and 4-class classification ( negative, neutral, positive or conflict).

To visualization importance of input tokens, we conduct gradient sensitivity analysis (Li et al. 2016; Arras et al. 2017). In Fig. 3, we visualize the gradient scores of our QACG-BERT model over two input sentences for which our model predicts sentiment labels correctly. For the first example, words like rent and cheaper are most relevant to the price aspect while words like live, location, central are more relevant to the transit-location aspect. Thus, our model learns how to pick out words that are important under different contexts. The second example concerns two targets (i.e., locations) for the same aspect, price, where the sentiment label for LOC1 is negative while for LOC2 is positive. For LOC1, the model correctly identifies costs (which in context refers to LOC1) and posh⁸⁸8The BERT Tokenizer breaks up ‘poshest’ into po-sh-est. We note that the BERT vocabulary does not have ‘posh’ in it, but after fine-tuning, appears to learn that the word is price-relevant.. By contrast, the model identifies cheap when given LOC2 as context, and assigns a greater gradient value, compared to the same word for LOC1. As a result, the model is able to identify words corresponding to different targets and different aspects.

We inspected the quasi-attention parameters learnt by our models. Specifically, we took the QACG-BERT model trained on the Sentihood dataset and extracted values for the following four variables: the final attention weights matrix $\mathbf{\hat{A}}^{h}$, the bidirectional gating factor matrix $\lambda_{A}^{h}$, the vanilla self-attention matrix $\mathbf{A}_{\text{Self-Attn}}^{h}$ and the context-guided quasi-attention matrix $\mathbf{A}_{\text{\emph{Quasi}-Attn}}^{h}$. Fig. 4 illustrates the histogram of values drawn from 200 examples from the test set.

We made several key observations. First, the behaviour of $\lambda_{A}^{h}$ follows our intuition; it acts as a bidirectional control gate, with slightly more negative values, and determines whether context contributes to attention positively or negatively. Second, the learnt weights in $\mathbf{A}_{\text{\emph{Quasi}-Attn}}^{h}$ are not near zero, with the mass of the distribution between .25 and .50, thus, it does contribute to attention. Lastly, the non-zero weights in the final matrix $\mathbf{\hat{A}}^{h}$ are mainly positive, but some of the weights take on negative values due to the bidirectional gating factor. This is important as it enables the model to attend to and “de-attend from” different parts of the input.

Finally, we turn to visualizing the quasi-attention. In Fig. 5, we visualize the weights of $\mathbf{A}_{\text{Self-Attn}}^{h}$ and the product $\lambda_{A}^{h}\mathbf{A}_{\text{\emph{Quasi}-Attn}}^{h}$ extracted from one of the heads in the first self-attention layer, over an example input sentence from the test set. For this sentence, our model correctly predicted the sentiment label negative for the price aspect of LOC1. In this example, nice is a positive word for aspects like general. However, with respect to price, nice actually suggests a higher cost and thus is more negative. We see that while $\mathbf{A}_{\text{Self-Attn}}^{h}$ on nice is high, $\mathbf{A}_{\text{Quasi-Attn}}^{h}$ on nice is negative, which “subtracts” from the final attention. As a result, the sum derived by Eqn. 4 makes the final attention weight on the word nice less positive. We note that empirically, we find that the total attention $\mathbf{\hat{A}}^{h}$ is usually positive (Fig. 4(a)), i.e., quasi-attention, when negative, tends to be smaller in magnitude than self-attention.