S²GSL: Incorporating Segment to Syntactic Enhanced Graph Structure Learning for Aspect-based Sentiment Analysis

Given a sentence of $n$ words $s$ = $\{w_{1},w_{2},\dots,$ $w_{\gamma+1}\dots,w_{\gamma+m}\dots,w_{n}\}$ , where the aspect $a$ = $\{w_{\gamma+1},\dots,w_{\gamma+m}\}$, we use the pre-trained language model BERT Devlin et al. (2019) as sentence encoder to extract contextual representations. For the BERT encoder, we follow BERT-SPC

Since aspects are vulnerable to irrelevant context, inspired by recent workNguyen et al. (2020); Shang et al. (2021), we use an end-to-end trainable soft masking dynamic local attention mechanism to construct a SeSG branch aiming to align each aspect and its corresponding opinion.

Attention Segment Masking Matrix  We first generate the word-level attention segments for each sentence by training left and right boundary soft masking matrices $\bar{\phi}_{l},\bar{\phi}_{r}\in\mathbb{R}^{n\times n}$, the formulations are calculated as below:

$$\bar{\phi}_{l}=\mathrm{Softmax}\left(\frac{QW_{L}^{Q}(KW_{L}^{K})^{T}}{\sqrt{d}}\odot\hat{M}\right)$$

(1)

$$\bar{\phi}_{r}=\mathrm{Softmax}\left(\frac{QW_{R}^{Q}(KW_{R}^{K})^{T}}{\sqrt{d}}\odot\hat{M}^{T}\right)$$

(2)

$$\left.\hat{M}_{ij}=\left\{\begin{matrix}1,&i\geq j\\ -\infty,&i<j\end{matrix}\right.\right.$$

(3)

where $Q$=$K$=$H^{c}$, $\bigodot$ is the element-wise product,and $W_{L}^{Q},W_{L}^{K},W_{R}^{Q},W_{R}^{K}\in\mathbb{R}^{d\times d}$ are trainable parameters. Notably, a mask matrix $\hat{M}$ is introduced to ensure that the left boundary position $lp$ and the right boundary position $rp$ generated at position $i$ satisfy $0$ $\leq$ $lp$ $\leq$ $i$ $\leq$ $rp$ $\leq$ $N$.

The attention segment masking matrix $M_{s}$ can be obtained by compositing the left and right boundary soft masking matrices $\bar{\phi}_{l}$ and $\bar{\phi}_{r}$ :

$$M_{s}=(\bar{\phi}_{l}L_{N})\odot(\bar{\phi}_{r}L_{N}^{T})$$

(4)

where $L_{N}\in\{0,1\}^{n\times n}$ refers to the upper-triangular matrix.

Then we combine the attention segment masking matrix $M_{s}$ with the multi-head attention matrices to enable the model to more focus on the semantically relevant contextual information around each word:

$$A^{SeS}=\mathrm{Softmax}\left(\frac{QW^{Q}(KW^{K})^{T}}{\sqrt{d}}\odot M_{s}\right)$$

(5)

where $W^{Q}$, $W^{K}$ are the trainable parameters, $A^{SeS}$ is a multi-head attention matrix with the number of $l$, where $l$ corresponds to the number of layers in the constituent tree.

Supervised Constraint  In the absence of supervised signal, dynamic local attention may not be able to effectively comprehend the semantically complete segment information around each word, so we further introduce segment-supervised signal to facilitate the learning of dynamic local attention. Specifically, we use the binary cross-entropy loss to represent the distinction between the attention matrix $A^{SeS}$ and the segment-supervised signal $Y^{seg}$:

$${\mathcal{L}}_{seg}=BCE(\sigma(A^{SeS}),Y^{seg})$$

(6)

$$\left.Y_{ij}^{seg}=\left\{\begin{matrix}1,&S_{ij}=1\\ 0,&else\end{matrix}\right.\right.$$

(7)

where $\sigma$ represents the sigmoid function, $S_{ij}$ = 1 indicates the $i$-th word and the $j$-th word belong to the same segment, $Y^{seg}$ refers to the segment-supervised signal at each layer of the constituent tree.

To effectively learn the representation of each word, we utilize graph convolutional network Kipf and Welling (2017) to extract the segment-aware semantic features $H^{SeS}$ = { $h_{1}^{SeS}$, $h_{2}^{SeS}$, …, $h_{n}^{SeS}$}, which is formulated as below:

$$H_{l}^{SeS}=\sigma(A^{SeS}W_{l}H_{l-1}^{SeS}+b_{l})$$

(8)

where $H_{l}^{SeS}$ represents the $l$-th GCN output, $H_{0}^{SeS}=H^{c}$ is the initial input, $\sigma$ denotes a nonlinear activation function, and $W_{l}$ and $b_{l}$ are the trainable parameters.

Sharing the same idea with past workTang et al. (2022), we also adopt syntactic dependency labels to enhance the latent tree construct. The difference from the past work is that we introduce an attention mechanism in the latent tree to effectively eliminate irrelevant dependency structures and construct a SyLG module.

Syntactically Enhanced Weight Matrix  To leverage dependency label information, we first use an off-the-shelf toolkit to obtain dependency information, then we utilize this information to generate a dependency type matrix $R=\left\{r_{i,j}\right\}_{n\times n}$, where $r_{i,j}$ represents the types of dependency between $x_{i}$ and $x_{j}$ Tian et al. (2021). Subsequently, we embed each dependency type $r_{i,j}$ into the vector $e_{ij}\in\mathbb{R}^{1\times dr}$, and finally obtain the relational adjacency matrix $M_{R}=\left\{e_{ij}|1\leq i\leq n,1\leq j\leq n\right\}$, where $e_{ij}$ refers to the embedding vector of dependency type between the $i$-th word and the $j$-th word. If $w_{i}$ and $w_{j}$ are not connected, we assign a "0" embedding vector to $e_{ij}$.

In order to induce a syntax-based latent tree, we need to generate a syntactically enhanced weight matrix. Specifically, we first use multi-head self-attention mechanism to compute a weight matrix $A_{a}$. Then, we transform the relation adjacency matrix $M_{R}$ into a syntactic relation weight matrix $A_{r}$ through a linear transformation, which has the same number of heads as $A_{a}$. Finally, the syntactically enhanced weight matrix $\bar{A}$ can be obtained by summing $A_{a}$ and $A_{r}$:

$$A_{a}^{\mathrm{k}}=\text{softmax}\left(\frac{QW^{Q}\times(KW^{K})^{\mathrm{T}}}{\sqrt{d}}\right)$$

(9)

$$A_{r}=(W_{R}^{T}M_{R}+b_{R})$$

(10)

$$\bar{A}=\mathrm{softmax}\left(A_{r}+A_{a}\right)$$

(11)

where $A_{a}^{k}$ is the attention score matrix of the $k$-th head, $W_{R}\in\mathbb{R}^{d_{r}\times N_{head}}$ is the weight matrix for the linear transformation.

Syntax-based Latent Tree Construction  Considering $\bar{A}$ as the initial weight matrix, we follow Zhou et al. (2021) to generate a syntax-based latent tree. We firstly define a variant of the Laplacian matrix for the syntax-based latent tree:

$$\bar{L}_{ij}=\begin{cases}\Phi_{i}+\sum_{i^{\prime}=1}^{n}\bar{A}_{i^{\prime}j}&\text{if}\ i=j\\ -\bar{A}_{ij}&\textit{otherwise}\end{cases}$$

(12)

where $\Phi_{i}=\exp({W}_{r}{h}_{i}^{c}+{b}_{r})$ is the non-normalized score that the $i$-th node is selected as the root node, $\bar{L}$ can be used to simplify the computation of weight sums. Therefore, the marginal probability $A_{ij}^{SyL}$ of the syntax-based latent tree can be computed using $\bar{L}_{ij}$:

	$\displaystyle{A}^{SyL}_{ij}$	$\displaystyle=\left(1-\delta_{1,j}\right)\bar{A}_{ij}\left[\bar{L}^{-1}\right]_{jj}$		(13)
		$\displaystyle-\left(1-\delta_{i,1}\right)\bar{A}_{ij}\left[\bar{L}^{-1}\right]_{ji}$

where $\delta$ is Kronecker delta, $A^{SyL}$ can be regarded as the adjacency matrix of the syntax-based latent tree.

We employ a root constraint strategy Zhou et al. (2021) to direct the root node towards the aspect word:

$$\mathcal{L}_{r}=-\sum_{i=1}^{N}(t_{i}\mathrm{log}\boldsymbol{P}_{i}^{r})+(1-t_{i})\mathrm{log}\left(1-\boldsymbol{P}_{i}^{r}\right)$$

(14)

where $\boldsymbol{P}_{i}^{r}=\Phi_{i}[\bar{L}^{-1}]_{i1}$ denotes the probability of the $i$-th word being the root node, and $t_{i}$ $\in$ {0, 1} indicates whether the $i$-th word is an aspect word.

Similar to the SeSG, we utilize graph convolutional network to extract the syntax-based latent Graph features $H^{SyL}$ = { $h_{1}^{SyL}$, $h_{2}^{SyL}$, …, $h_{n}^{SyL}$}, which is formulated as below:

$$H_{l}^{SyL}=\sigma(A^{SyL}W_{l}H_{l-1}^{SyL}+b_{l})$$

(15)

where $H_{l}^{SyL}$ denotes the $l$-th layer of GCN output, $H_{0}^{SyL}=H^{c}$ is the initial input, $W_{l}$ and $b_{l}$ are the trainable parameters.

Considering the complementarity between SeSG and SyLG, we design a self-adaptive aggregation

To avoid bias towards specific module information, we introduce an additional channel to balance the information between SeSG and SyLG. The specific approach is as follows:

$$H^{Com}=FFN(Concat([H^{SeS},H^{SyL}]))$$

(20)

where $H^{Com}\in R^{n\times d}$, $Concat(\cdot)$ represents the concatenation function.

Considering the different roles of the various module outputs, we assign different weights to these outputs to allow the model to more focus on the important module. Technically, given the input features $X=[X_{1},X_{2},X_{3}]$. The weight of each module is calculated by the following equation:

$$a_{i}=\mathrm{ReLU}(W^{T}X_{i}+b)$$

(21)

$$\alpha_{i}=\frac{exp(a_{i})}{\sum_{j=1}^{3}exp(a_{j})}$$

(22)

where $X_{1}=H^{SemG}$, $X_{2}=H^{SynG}$, $X_{3}=H^{Com}$, $W_{l}$ and $b_{l}$ are the trainable parameters.The final output feature $H^{F}$ is generated as follows:

$$H^{F}=Concat([\alpha_{1}H^{SemG},\alpha_{2}H^{SynG},\alpha_{3}H^{Com}])$$

(23)

where $H^{F}\in R^{n\times d_{3}}$, with $d_{3}$ = 3$d$.

We use average pooling at the final aspect nodes of $H^{F}$ to obtain the aspect representation $H^{a}$. Then, the sentiment probability distribution $y_{(s,a)}$ is calculated using a linear layer with a softmax function:

$$y_{(s,a)}=\mathrm{softmax}(W^{p}{H}^{a}+b^{p})$$

(24)

where ($s$,$a$) represents the sentence-aspect pair. Our training objective is to minimize the following objective function:

$$\mathcal{L}(\Theta)=\mathcal{L}_{C}+\lambda_{1}\mathcal{L}_{seg}+\lambda_{2}\mathcal{L}_{r}$$

(25)

where $\Theta$ denotes all trainable parameters of the model, $\lambda_{1}$ and $\lambda_{2}$ are hyper-parameters, and $\mathcal{L}_{C}$ is the standard cross-entropy loss function:

$$\mathcal{L}_{\mathrm{C}}=-\sum_{(s,a)\in D}\sum_{c\in C}\log y_{(s,a)}$$

(26)

where $D$ contains all sentence-aspect pairs and $C$ is the collection of different sentiment polarities.

We conduct experiments on four public datasets. The Restaurant and Laptop reviews are from SemEval 2014 Task 4 Pontiki et al. (2014). The Twitter dataset is a collection of tweets Dong et al. (2014). The MAMS dataset is consisted of sentences with multiple aspectsJiang et al. (2019). Each aspect in the sentence is labeled with one of the three sentiment polarities: positive, neutral, and negative. The statistics for the four datasets are shown in Table 1.

The Stanford parser²²2https://stanfordnlp.github.io/CoreNLP/ Manning et al. (2014) is used to obtain syntactic dependencies. Specifically, we use CRF constituency parser Zhang et al. (2020) to obtain the constituent tree. We use the bert-base-uncase³³3https://github.com/huggingface/transformers model as our context encoder. The model training is conducted using the Adam optimizer with a learning rate of $2\times 10^{-5}$ and L2 regularization of $10^{-5}$. The GCN layers of SeSG and SyLG are set to 3. Our model is trained in 20 epochs with a batch size of 16. The hyper-parameters $\lambda_{1}$ and $\lambda_{2}$ for the four datasets are (0.1,0.5), (0.1,0.45), (0.35,0.3) and (0.4,0.75). All experiments are conducted on an NVIDIA 3090 GPU. The model with the highest accuracy or F1 score among all evaluation results is selected as the final model.

We compare our S²GSL with some mainstream and lasted models in ABSA, including Attention-based methods: RAM Chen et al. (2017), MGAN Fan et al. (2018). Syntactic-based methods: R-GATWang et al. (2020),T-GCNTian et al. (2021). Latent-graph methods: ACLT Zhou et al. (2021), dotGCN Chen et al. (2022). Multi-graph combined methods: KumaGCN Chen et al. (2020), DualGCN Li et al. (2021), SSEGCNZhang et al. (2022) MGFN Tang et al. (2022). Other method: TF-BERT Zhang et al. (2023).

All baseline results on four datasets are shown in Table 2, we can find that our S²GSL outperforms all baselines on Laptop, Restaurant, and MAMS datasets. We got the second best on the Twitter dataset, reaching comparable with MGFN. We guess it is because the sentence structure of Twitter dataset is more complex than the other three datasets, basically samples containing only single aspect words, which cannot reflect the advantages of S²GSL. In contrast, the other three datasets, laptop, restaurant, and MAMS contain samples of multiple aspect words. The effect enhancement in Laptop, Restaurant, and MAMS effectively supports that S²GSL gets better sentiment recognition in the case of having multiple aspect words. Additionally, we conduct a parameter comparison between S²GSL and the GCN-based baseline methods. Notably, the number of parameters in S²GSL is comparable (detailed results can be found in A.2). These results demonstrate that S²GSL exhibits superior performance while incurring the same computational overhead.

We conduct ablation experiments to further investigate the effects of different modules, shown in Table 3. Excluding the syntax-based latent graph learning (w/o SyLG) results in a decrease in the model’s performance, which demonstrates the importance of the ability to adaptively capture syntactic relationships between words. We remove the segment-aware semantic graph learning module (w/o SeSG). Compared to the Twitter dataset, the performance of the Restaurant, Laptop and MAMS datasets decreases significantly, which is due to the fact that the sentence structures on these three datasets are more formal and each sentence can be constructed with multiple subordinate clauses. w/o Self-Adaptive Aggregation refers to the SyLG and SeSG modules cannot interact with each other, leading to a drop in performance on all four datasets. The study reveals that both SyLG and SeSG branches are crucial for handling complex sentences in all datasets, as removing either component leads to a noticeable drop in performance. However, compared to the other three datasets, the performance of the SeSG module on the Twitter dataset is not particularly significant, since each sentence on Twitter contains only one aspect word. This difference underscores the adaptability and effectiveness of S²GSL in varying complexity levels across datasets.

To demonstrate the effectiveness of supervised dynamic local attention, we visualize the attention weight of all aspects in a sentence, as shown in Figure 6. We can find that in the model without the constraints of segment-supervised signal (S²GSL w/o $\mathcal{L}_{seg}$), the aspect "food" incorrectly assigns a higher attention weight to the opinion "great" of "waitstaff". In contrast, attention weights of each

To investigate the validity of syntactic dependency label information, we analyzed the weight change of each aspect word assigned to the corresponding opinion word in the latent tree. Figure 7(b) shows the weight changes of the aspect words "appetizers" and "service" assigned to the corresponding opinion "ok" and "slow". As can be seen from the figure, their respective weights have increased by 0.061 and 0.055. This is because in most cases, each aspect is connected to its corresponding opinion through the same syntactic dependency label, such as the "nsubj" label in Figure 7(a). The above analysis indicates that dependency labels can better capture the relationship between aspects and their corresponding opinions.

To validate the effectiveness of our proposed self-adaptive aggregation module, we compare it with several typical information fusion strategies: "concat", "sum" and "gate". As shown in Figure 8, we can find that "gate" outperforms better than "concat" and "sum" on all datasets. Furthermore, "con-

ChatGPT OpenAI (2023a), powered by GPT-3.5 and GPT-4, can achieve significant zero-shot and few-shot in-context learning (ICL) Brown et al. (2020b) performance on unseen tasks, even without any parameter updates.

In this section, we investigate the performance of ChatGPT on ABSA tasks and its ability for fine-grained understanding of segmental context. We experimented with 3 different prompts and their settings as detailed in A.1. We also compared our results with Han et al. (2023), who investigated the performance of ChatGPT on various information extraction tasks, including ABSA. All results are shown in table 4. From the experiment, we find that when we prompt ChatGPT with some instructions (single aspect and multi-aspect sentences) can better improve the performance, but its best results are still not as good as our model. This situation suggests that ChatGPT possesses the ability to understand fine-grained segmental information to some extent. Perhaps there are ways to better harness this ability, such as incorporating constituent trees and dynamic local attention mechanisms as described in this paper (refer A.1 for details).

To investigate the impact of different layer numbers of the constituent tree, we evaluate the performance of the model with 2 to 5 constituent tree layers on three different datasets. As shown in Figure 9, the best performance of the model is achieved when the number of layers of the constituent tree is 4. When the layer numbers of the constituent tree are lower than 4, the information from the constituent tree cannot fully cover the entire sentence, resulting in the model not being able to fully learn the complete segment information of a sentence. When the layer numbers are greater than 4, the model will repetitively learn redundant segment information, resulting in a decrease in model performance.

We conduct a case study with a few examples, shown in Table 5. The first sentence only has one aspect word, so all models can easily determine the sentiment polarity of the aspect correctly. For the second comparative type of sentence, there is a certain syntactic dependency between the aspect "hamburger with special sauce" and the aspect "big mac", and the SeSG, which lacks syntactic information, cannot handle this type of sentence well. The last cases contain multiple aspects and opinions. DualGCN and SyLG, which lack segment structure awareness, cannot focus on local information around aspects, resulting in incorrect judgments. Our S²GSL correctly predicts all samples, indicating that it effectively considers the complementarity between segment structures and syntax correlations of a sentence.

The rise of large language models (LLMs) such as GPT-3 Brown et al. (2020a), PaLM Chowdhery et al. (2022), Llama Touvron et al. (2023), etc, has greatly facilitated the rapid development of natural language processing (NLP). ChatGPT OpenAI (2023a), powered by GPT-3.5 and GPT-4, can achieve significant zero-shot and few-shot in-context learning (ICL) Brown et al. (2020b) performance on unseen tasks, even without any parameter updates.

In this section, we conducted exhaustive experiments to investigate the performance of ChatGPT on ABSA tasks and its ability for fine-grained understanding of segmental context.

Experiment setting : The version of ChatGPT we utilized in the experiment is gpt-3.5-turbo. To avoid variations in ChatGPT-generated outputs, the temperature parameter was set to 0. For the number of response words, the max tokens parameter was set to 512. We experimented with 3 different prompts and their respective settings as shown in Figure10. We take the laptop dataset as example to explain our prompt settings:

• •

  Prompt-setting 1: Zero-shot. In this prompt setting, we have only given definition: "Recognize the sentiment polarity for the given aspect term in the given review. Answer from the options [ “positive”, “negative”, “neutral” ] without any explanation.", then we sent a test review and all aspect words in this review to ChatGPT, and ChatGPT will give the answer.

• •

  Prompt-setting 2: 5-Shot ICL with single aspect short sentences. The given definition is same as setting 1. What differs from setting 1 is that we randomly selected 5 single-aspect word sentences from the training set as examples to better leverage ChatGPT’s In-Context Learning capability. Sequentially, we provide a test review and all aspect words, asking ChatGPT to output the sentiment polarity for each aspect word.

• •

  Prompt-setting 3: 5-Shot ICL with multiple aspects long sentences. What differs from setting 2 is that the examples chosen from the training set are all multiple aspects long sentences. The purpose of doing this is to explore whether ChatGPT possesses the ability to understand fine-grained segmental context and the strength of this ability.

Result analysis : The results using ChatGPT and our model are shown in table6. Firstly, from the result of prompt-setting 2, we can infer that using a few examples to guide ChatGPT can effectively improve its performance, demonstrating the powerful In-Context-Learning capability of ChatGPT. Secondly, we compared our experimental results with Han et al. (2023), who investigated the performance of ChatGPT on various information extraction tasks, including ABSA. We found that our model has better performance. Finally, from prompt-setting 3, we can observe that providing ChatGPT with long examples containing multiple aspect words can lead to better performance. However, the results still fall short compared to our model. This situation suggests that ChatGPT possesses the ability to understand fine-grained segmental information to some extent. Perhaps there are ways to better harness this ability, such as incorporating constituent trees and dynamic local attention mechanisms as described in this paper.

We conduct a parameter comparison between S²GSL and other GCN-based baseline methods, shown in table 7. Notably, the number of parameters in S²GSL is comparable while the two graph learning branch design of S²GSL does lead to an increase in the parameters, it’s important to note that this does not result in an unnecessary escalation in computational costs when compared with the baselines.