Domain-level Pairwise Semantic Interaction for Aspect-Based Sentiment Classification

Before this sub-module, the semantic of these two sentences are extracted by backbone, and two $D$-dimensional semantic vectors i.e., $v_{1}$ and $v_{2}\in\mathbb{R}^{D}$ are generated, respectively. Then, in this sub-module, we learn a mutual vector $v_{m}\in\mathbb{R}^{D}$ from individual $v_{1}$ and $v_{2}$,

where [ ] is concatenation operation and $f_{m}(\cdot)$ is a mapping function of $[v_{1},v_{2}]$. Specifically, we use the multi-layer perceptron (MLP) as the mapping function. By summarizing the two feature vectors $v_{1}$ and $v_{2}$, the mutual vector $v_{m}$ is produced accordingly, which contains common high-level semantic information and discriminative semantic clues of two sentences.

After producing the mutual vector $v_{m}$, we can use $v_{m}$ to activate $v_{1}$ and $v_{2}$. In order to generate more discriminative information for later comparisons, the dot product of $v_{m}$ with two feature vectors $v_{1}$ and $v_{2}$ is carried out according to channels to locate the contrastive information in the two vectors. Then, the gate vectors, i.e., $g_{1}$ and $g_{2}$ are generated by a sigmoid function,

Therefore, $g_{i}$ can be used as an attention vector to highlight the important semantic representations belonging to individual $v_{i}$. For example, the previous example of S1-A1 and S1-A2 in Fig. 1, the gates will help to highlight two different key words “ambience” and “food”, respectively. This helps to distinguish different sentiment polarities belonging to different aspects.

When comparing two sentences, humans not only focus on the salient parts of one sentence, but also focus on the salient parts of the other one. Based on this, we introduce an adversarial interaction mechanism through residual attention. As shown in Fig. 2, two feature vectors $v_{1}$ and $v_{2}$ and two gate vectors $g_{1}$ and $g_{2}$ are combined in pairs, we could then get four attentive semantic vectors with

Intuitively, the semantic vector $v_{i}$($i\in\{1,2\}$), is guided by the attention of the gate vector $g_{j}$($j\in\{1,2\}$), strengthens or weakens certain semantic information, and then adds to itself to get the output. While $v_{i}^{self}\in\mathbb{R}^{D}$ reinforces the feature region belonging to its own gate vector, and $v_{i}^{other}\in\mathbb{R}^{D}$ reinforces the feature region belonging to another gate vector.

Then $v_{i}^{j}$(where $i\in\{1,2\}$, $j\in\{self,other\}$) are feed into the softmax classifier by $p_{i}^{j}=softmax(Wv_{i}^{j}+b)$, where $p_{i}^{j}\in\mathbb{R}^{C}$ represents the score vector of prediction, $C$ indicates the number of polarities, and $\left\{W,b\right\}$ is the parameter set of the softmax classifier. In order to effectively train the entire PSI module, we define the following loss function as

Among them, $J_{ce}$ is the cross-entropy loss, and $J_{rk}$ is the score ranking regularization loss with a coefficient of $\mu$. Specifically we choose the hinge loss function as the score ranking regularization $J_{rk}$,

where $p_{i}^{j}(y_{i})\in\mathbb{R}$ represents the score got in the predicted vector $p_{i}^{j}$, and $y_{i}$ denotes the index of the true polarity of sentence i, and $\varepsilon$ is the penalty term. The motivation of this design is that, $v_{i}^{self}$ is activated by its own gate vector. Hence, compared to $v_{i}^{other}$, it should be more discriminative to the corresponding label. That is, the score difference $p_{i}^{self}(y_{i})$-$p_{i}^{other}(y_{i})$ should be larger than a margin $\epsilon$, which means that $p_{i}^{self}(y_{i})$ should be larger than $p_{i}^{other}(y_{i})$ and must keep a distance with $p_{i}^{other}(y_{i})$. At the same time, when cross-entropy loss $J_{ce}$ is optimized, due to having the same label, $p_{i}^{self}(y_{i})$ and $p_{i}^{other}(y_{i})$ will tend to be closer. Therefore, $J_{rk}$ and $J_{ce}$ will be optimized adversarially. As a result, $v_{i}^{other}$ can learn the semantic information shared by the two sentences, and $v_{i}^{self}$ can learn its own unique information which will be more discriminative and reduce the noise of sentence pairs.

Next, we’ll provide an explanation on how to construct multiple sentence pairs in a batch for end-to-end training. Specifically, we randomly sample $N_{p}$ polarities in a batch (there are 3 polarities in total, i.e. positive, negative, neutral). For each polarity, we randomly sample $N_{s}$ training sentences. Consequently, there are $N_{p}\times N_{s}$ different sentences in each batch (we set the same sentence to express different aspects, belonging to different sentences). After getting a batch of sentences, we input these sentences into the backbone to generate their respective semantic vectors. For every sentence, we compare its semantic vector with the different sentences in the batch in accordance to Euclidean distance. We do not limit that two different sentences must have the same aspect, and encourage richer interactions between sentences(the following ablation study proves our point). Then, we can construct the inter/intra-pairs in a batch. The inter-pairs are the following sentence pairs which contains two situations. 1) The current sentence and itself (with different aspect and different polarity), e.g., S1-A1 & S1-A2 in Fig. 1; 2) The current sentence and the most similar sentence with different polarities from the current sentence, e.g., S1-A1 & S2(A1/A2). On the contrary, intra-pairs refer to the following sentence pairs. 1) The current sentence and itself (with different aspect and same polarity), e.g., S2-A1 & S2-A2; 2) The current sentence and the most similar sentence with the same polarity from the current sentence, e.g., S1-A2 & S2(A1/A2). This design permits the PSI to learn to distinguish between truly similar and highly overlapping pairs.

Because PSI is a practical plug-and-play module. In the training phase, the backbone and the PSI module can summarize the comparative clues from sentence pairs, and step by step improve the discriminant capacity of backbone representation for sentences. Therefore, in the testing phase, only the backbone model with updated parameters is used, but not the PSI module, so that the generalization ability of the model can be guaranteed without losing the performance of the model. To be specific, in testing phase, we input a sentence into the backbone, extract its semantic vector $X_{*}\in\mathbb{R}^{D}$, and then directly input $X_{*}$ into the softmax classifier. It is worth emphasizing that the softmax classifier are shared between the training phase and the testing phase. The score vector $P_{*}\in\mathbb{R}^{C}$ is applied to label prediction. Thus, our test scheme is the same as a regular backbone, which demonstrates the strong applicability of PSI.

We have carried out experiments on four datasets to verify the performance of our proposed model, PSI. Restaurant14 is from Semeval-2014 Task 4 [16], Laptop15 is from Semeval-2015 Task 12 [15], and the other two datasets (Restaurant16 and Laptop16) are from Semeval-2016 Task 5 [14]. The statistics for these datasets are shown in Table 1. And we use Accuracy (Acc.) and Macro-F1 (F1) as performance metrics.

Unless stated otherwise, we implement PSI as follows. For each aspect of a sentence, we concat the corresponding aspect at the end of the sentence. Then we adjust the length of each sentence to 85 (the maximum sentence length after the tokenizer tokenizes is 85). If it is not enough, fill it with zero, and then use it as the input of backbone. Firstly, we extract the semantic vector $v_{i}\in\mathbb{R}^{786}$ by BERT. Secondly, for all the datasets, we randomly sample 3 polarities, i.e., $N_{p}=3$. And for each polarity, we randomly sample 4 sentences to form a batch, i.e., $N_{s}=4$. For each sentence, we find its most similar sentence from its own polarity and the rest polarities, according to Euclidean distance between their semantic vectors. As a result, we obtain an intra-pair and an inter-pair for each sentence in the batch. For each pair, we concatenate $v_{1}$ and $v_{2}$ as input to a two-layer MLP, i.e., FC (1572→512), FC (512→786). Consequently, this operation generates the mutual vector $v_{m}\in\mathbb{R}^{786}$. All of our models are implemented by Pytorch with a single NVIDIA GTX 2080Ti GPU with 11G Memory. For all datasets, the coefficient $\mu$ in Eq. 4 is $1$, while the margin $\epsilon$ is 0.05 in score ranking regularization. Among them, BERT is optimized by Adam optimizer with $\beta_{1}$= 0.9, and the initial learning rate is 0.0001. For our PSI (backbone is BERT or BERT-Large) method, we use another Adam optimizer for training, and the initial learning rate is 0.00002 with $\beta_{1}$= 0.9. There are a total of 20 training epochs, and if the loss does not decrease for 5 consecutive epochs, it will invoke early-stop. In addition, we set a fixed seed when training the model to ensure the reproducibility of the results.

To fully evaluate the performance of our method, we apply PSI based on BERT or BERT-Large. We compare it with the state-of-the-art (SOTA) baselines including (1) ABSA models without BERT: TC-LSTM [18], ATAE-LSTM [22], RAM [3], IAN [12], Clause-LevelATT [21], LSTM+synATT+TarRep [7], kumaGCN [2], RepWalk [25] and IMN [8]. (2) BERT-based models for ABSA: BERT [5], BERT-QA [17], AC-MIMLLN [11] and CoGAN [4]. Table 2 shows the results of our experiments on four datasets.

From Table 2 we can come to the following conclusion. The performance of PSI (Based on BERT or BERT-Large) on Res14, Lap15 and Lap16 is better than those of all baselines. And in Res16, our PSI module approaches SOTA results. The experiments justify that PSI is a very powerful plug-and-play module, showing the effectiveness of our method.

As shown in Table 1, the mainstream datasets have the problem of class-imbalance. For example, in Res14, the “negative” comments is significantly more than the data of other polarities (“negative” accounts for 62%), while the “neutral” comments is far lower for other polarities (“neutral” accounted for 14%). In order to illustrate the advantages of the our method, we compared PSI (Based on BERT) with the standard BERT model on the Res14 dataset, for each polarity (positive, negative, and neutral). Table 7 shows the comparison results of the accuracy of different sentiment polarities.

It can be seen that the “positive” and “neutral” accuracy of our model (PSI) is better than that of BERT model. Specifically, PSI significantly improves the performance of “neutral” classification. And the overall accuracy of our model was greatly improved compared with BERT model. It indicates that the sentence pair interaction learning in PSI module can make the semantics between sentences complement each other and effectively learn the nuances between sentences. This can make the semantic representation of different polarities of sentences more distinguishable, and can effectively alleviate the problem of class-imbalance.

Our proposed method encourages richer interactions between sentences and do not limit that two different sentences must have the same aspect. In order to study the impact of different sample extraction methods (sentiment polarity and aspect) on ABSC, we conducted the following ablation experiments. We use BERT as semantic vector extractor to evaluate different sample extraction methods on Res14.

As mentioned above, our proposed sample extraction method is that Intra/inter pairs are constructed from the same/different sentiment polarities without limiting the range of aspect (Interacting Polarity, $I\_P$). And there are three other sample extraction methods, including 1) Interacting Aspect ($I\_A$). Intra/inter pairs are constructed from the same/different aspects without limiting the range of sentiment polarity. 2) Interacting Polarity and Limiting Aspect ($I\_P$ & $L\_A$). Intra/inter pairs are constructed from the same/different polarities by limiting the same aspect. 3) Interacting Aspect and Limiting Polarity ($I\_A$ & $L\_P$). Intra/inter pairs are constructed from the same/different aspects by limiting the same polarity.

From Table 4, in all the four datasets, the results of the other three ablation experiments are worse than $I\_P$ (Ours). For $I\_P$ & $L\_A$ and $I\_A$ & $L\_P$, results demonstrate that we should not limit aspects (in order to better distinguish different aspects of a sentence) and should also allow different polarities to interact in pairs (in order to better distinguish different sentiment polarities). For $I\_A$, due to class-imbalance, if we do not limit the range of sentiment polarity (by constructing the inter-pair of different sentiment polarities), there will be a lot of interactions between sentences belonging to the same majority class (“negative”), while the interaction between different sentiment polarities will be insufficient. Finally, for $I\_P$, we explicitly construct inter-pairs (belonging to different polarities) in each batch to ensure that the number of interactions between different polarities is sufficient. In this way, the nuances between different sentiment polarities can be better learned by the model. Therefore, we choose $I\_P$ as the sample extraction method for ABSC.

In addition, in order to investigate the properties of our proposed PSI, we use Bert as a semantic vector extractor to evaluate its key design on Res14. For fairness, when exploring different strategies for one design, we use the other designs as the base strategies described in the proposed approach and implementation details.

Mutual Vector Generation. To demonstrate the essentiality of $x_{m}$ in Eq. 1, we investigate different operations to generate it. 1) Individual Operation. The key of $x_{m}$ is to learn mutual information from both sentences in the pair. For comparison, we introduce a baseline without it. Specifically, we replace mutual learning in Eq. 1 by individual learning $\tilde{x}_{i}=f_{m}(x_{i})$, and use $\tilde{x}_{i}$ to generate the gate vector $g_{i}=sigmoid(\tilde{x}_{i})$ where $i\in\left\{1,2\right\}$ . 2) Elementwise Operations. We perform a number of widely-used elementwise operations to generate $x_{m}$, including Subtract Square: $x_{m}={(x_{1}-x_{2})}^{2}$, Sum: $x_{m}=(x_{1}+x_{2})$, and Product $x_{m}=(x_{1}\ast x_{2})$. 3) Interactive MLP. It is the mapping function described in the proposed approach. As shown in Table 5, the Individual operation (i.e., the setting without $x_{m}$) performs worst. Hence, it is necessary to learn mutual context by $x_{m}$. Based on experimental results, we choose the simple but effective Interactive MLP to generate the mutual vector in our experiments.

The Influence of the Number of Interactions. On the basis of sample extraction method $I\_P$, We further studied the influence of the number of sampling polarity $N_{p}$ and the number of corresponding sentences $N_{s}$ in each batch. The results are shown in Table 6. It can be seen that PSI is more sensitive to polarity-number than sentence-number. This is because more polarities often lead to richer diversity of sentence pairs. If 3 polarities are selected in a batch, sentence pairs must include the sentence belonging to minority category (neutral). So the sentences belonging to minority polarity (neutral) can learn semantic information from other majority polarity (positive or negative), thereby improving the generalization ability of the model. Therefore, we choose the best setting in our experiments, i.e., $N_{p}=3$ and $N_{s}=4$.

The Influence of Sentence Similarity. Furthermore, We also examine the influence of sentence similarity for intra-pairs and inter-pairs on the Res14 dataset based on $I\_P$. For 12($N_{p}\times N_{s}$) different sentences in each batch, we construct intra/inter-pairs for each sentence, respectively. Therefore, there are 24 sentence pair in a batch. The selection strategies including 1) Random. We randomly sampled 24 sentence pairs(intra/inter pairs) in a batch; 2) Sentence-Distance. We sampled 24 sentence pairs in accordance to the Euclidean distance between sentences in intra/inter pairs (i.e., Similar (S), Dissimilar (D)). Thus, 8 different Class-Polarity-Sentence settings were generated in Table 7.

From Table 7, we can see that most of results of Sentence-Distance are better than Random, which shows that when constructing sentence pairs, the polarity of the sentence and the similarity between them must be considered. Secondly, in the setting of Sentence-Distance, Whether inter or intra, the results of (S) is higher than (D) in most cases. This proves that similar sentence pairs can increase the training difficulty of the model, which can effectively help model to identify subtle semantic differences.

The Necessity of Ranking Regularization. In this section, we conduct experiments on the necessity of ranking regularization in the sub-module (Adversarial Interaction) of PSI. We compared the standard BERT, PSI without Ranking Regularization $J_{rk}$ (based on BERT) and PSI (based on BERT) in five datasets. It can be seen from the Table 8 that compared with the standard BERT, the performance of the BERT with PSI(with or without $J_{rk}$) is significantly improved, which proves the effectiveness of the PSI structure. Furthermore, the PSI with $J_{rk}$ is better than PSI without $J_{rk}$. This demonstrates that, without ranking regularization, the effectiveness of adversarial interaction will be greatly reduced. When without ranking regularization, the model is only optimized by cross-entropy loss $J_{ce}$, which will make the semantic vectors ($v_{i}^{self}$/$v_{i}^{other}$) activated by different gates lose their distinction. On the contrary, adding ranking regularization can keep semantic vectors activated by different gates at a distance. This allows the semantic vectors activated by the other’s gate to have the common semantic representation of the two sentences ($v_{i}^{other}$), and the semantic vectors activated by the own gate to have unique semantic representation ($v_{i}^{self}$), which is more discriminative. In summary, the PSI with ranking regularization helps model to find the similarities and differences between sentence pairs better, and reduce the noise of sentence pairs, which is effective and necessary.

In order to understand the discriminability of our method, we use UMAP [13] to visualize the polarity separability and compactness in the semantic features extracted from a standard BERT and the PSI (based on BERT) in Res14. In Fig. 3, it is evident that when using our PSI module, the clusters are farther apart and more compact, leading to a more clear distinction of various clusters representing different polarities. This also proves that adding PSI module can promote the model to learn better semantic representation of sentences and make them more distinguishable.