Abstractive Opinion Tagging

A lot of research has been conducted on generating keyphrases to summarize various types of text such as tweets, news reports, research articles, etc. (Meng et al., 2017; Chen et al., 2019; Liu et al., 2020b; Zhang et al., 2018; Wang et al., 2019b, a; Ray Chowdhury et al., 2019; Liu et al., 2020a). Early approaches to keyphrase generation extract important phrases from the document as the results. Sequence tagging models have been applied to identify keyphrases (Zhang et al., 2016; Luan et al., 2017; Gollapalli et al., 2017). Retrieval-based approaches utilize a two-step pipeline to extract and rank candidate keyphrases (Mihalcea and Tarau, 2004; Medelyan et al., 2009; Wang et al., 2016; Le et al., 2016). Sun et al. (2019) adopt an extractive graph-based approach, which applies a point network to generate a set of diverse keyphrases. Recently, abstractive approaches have also been explored. meng2017deep are the first to employ attention-based seq2seq framework with copy mechanism to conduct abstractive keyphrase generation. Chan et al. (2019) propose a reinforcement learning approach for neural keyphrase generation that encourages a model to generate both sufficient and accurate keyphrases. Wang et al. (2019a) propose a topic-aware neural keyphrase generation method to identify topic words.

Unlike the work listed above, which only considers keyphrase generation for single document, we consider opinion tagging from multiple documents, that is, from all of the reviews for a given item.

Opinion summarization has become an emerging research topic in recent years. Early studies on opinion summarization focus on extracting salient sentences from the original review text (Hu and Liu, 2004; Carenini et al., 2006; Lu et al., 2009; Ganesan et al., 2010; Xiong and Litman, 2014; Angelidis and Lapata, 2018): Hu and Liu (2004) identify item features mentioned in the reviews and then extract opinion sentences for the identified features. Xiong and Litman (2014) utilize unsupervised learning methods to extract review summaries by exploiting review helpfulness ratings. Angelidis and Lapata (2018) present a weakly supervised neural framework for aspect-based opinion summarization by combining the tasks of aspect extracting and sentiment predicting. Reflecting the most representative opinions from reviewers, many recent studies have shown that abstractive approaches are more appropriate for summarizing review text (Carenini et al., 2013; Gerani et al., 2014; Fabbrizio et al., 2014; Li et al., 2019b; Tay, 2019): Gerani et al. (2014) utilize a template filling strategy to indirectly generate a review summary; Wang and Ling (2016) apply an attention-based encoder-decoder framework to generate an abstractive summary for opinionated documents. The main objective of the above summarization approaches is to generate coherent sentences to summarize opinions.

In contrast, we propose the abstractive opinion tagging task so as to generate opinion tags from a large number of user-generated reviews. In our scenario, opinion tags are more concise but without loss of essential information; they should help users comprehend reviews quickly and conveniently (Li et al., 2017).

Before providing the details of AOT-Net, our proposed method for abstractive opinion tagging, we first provide an overview in Figure 4. We divide AOT-Net into three main phases: (A) sentence-level salience estimation; (B) review clustering and ranking; and (C) rank-aware opinion tagging. For a set of reviews $\mathcal{X}$ about a given item, in phase A, we derive a salience score $z_{i}$ for each review $X_{i}\in\mathcal{X}$ to estimate its item-aware salience information. In phase B, reviews are first encoded into vector representations and weighted by corresponding salience scores. The weighted vector representations of reviews are clustered into $K$ opinion clusters $\{C_{1},\ldots,C_{K}\}$ and ranked by cluster size. Reviews within each cluster are ranked by their distance to the cluster center. Then we flatten ranked reviews into word-level vector representations. In phase C, we use review representations to generate ranked opinion tags via two alignment constraints, i.e., alignment features and alignment loss. We jointly learn all components in a multi-task learning framework.

The aim of the sentence-level salience estimation component is to compute a salience score for each review $X_{i}\in\mathcal{X}$. We design a sentence-level self-attention mechanism to highlight item-related reviews and reduce noise. First, the component reads each review sequence $X_{i}=[x_{1},\ldots,x_{L_{x_{i}}}]$ and uses a lookup table to convert each review word $x_{p}$ to a word embedding vector $\mathbf{x}_{p}\in\mathbb{R}^{d_{e}}$. To incorporate the contextual information of the review text into the representation of each word, we feed each embedding vector $\mathbf{x}_{p}$ to a bi-directional Gated-Recurrent Unit (GRU) (Cho et al., 2014) to learn a hidden representation $\mathbf{h}_{p}\in\mathbb{R}^{d}$. More specifically, a bi-directional GRU consists of a forward GRU that reads the embedding sequence from $\mathbf{x}_{1}$ to $\mathbf{x}_{L_{x_{i}}}$ and a backward GRU that reads from $\mathbf{x}_{L_{x_{i}}}$ to $\mathbf{x}_{1}$:

where $\overrightarrow{\mathbf{h}}_{p}\in\mathbb{R}^{d/2}$ and $\overleftarrow{\mathbf{h}}_{p}\in\mathbb{R}^{d/2}$ denote the hidden states of the forward $\text{GRU}_{f}$ and backward $\text{GRU}_{b}$, respectively. We concatenate the last forward hidden state $\overrightarrow{\mathbf{h}}_{L_{x_{i}}}$ and last backward hidden state $\overleftarrow{\mathbf{h}}_{1}$ to form the hidden representation for review $X_{i}$, i.e., $\mathbf{h}_{X_{i}}=[\overrightarrow{\mathbf{h}}_{L_{x_{i}}};\overleftarrow{\mathbf{h}}_{1}]$.

Next, we pass hidden representations of all reviews $\{\mathbf{h}_{X_{i}}\}_{i=1:M}$ to a self-attention layer to model more complex interactions among the reviews. We propose a salience context vector $\textbf{c}_{i}$ for each review $X_{i}$ to denote the shared information from other reviews:

where $\textbf{q}_{i},\textbf{k}_{i},\textbf{v}_{i}\in\mathbb{R}^{d\times d}$ refer to query, key, and value vectors, respectively. These vectors are linearly transformed from review hidden representation $\mathbf{h}_{X_{i}}$. Then we apply a residual connection from the review hidden representation $\mathbf{h}_{X_{i}}$ to the salience context vector $\mathbf{c}_{i}$ and feed it to a two-layer feed-forward network with a ReLU as the activation function:

Given the context-enhanced review hidden representation $\mathbf{h}^{\prime}_{X_{i}}$, we can derive the real-valued salience score $z_{i}$ for $X_{i}$:

where $\mathbf{W}_{s}\in\mathbb{R}^{d}$ and $b_{s}\in\mathbb{R}$. $\sigma(\cdot)$ is the sigmoid activation function. The salience scores $\{s^{\prime}_{1},\ldots,s^{\prime}_{M}\}$ serve as the salience weights of review representations for the later review clustering and ranking component.

In order to optimize the sentence-level salience estimating component, we manually label a binary salience label $z^{*}_{i}\in\{0,1\}$ for each review $X_{i}$, where 1 denotes “item-related” whereas 0 denotes “noisy”. Then, sentence-level salience estimation component is trained by minimizing the cross-entropy loss function:

We propose a review clustering and ranking component to learn the ranks of reviews by grouping reviews into ranked opinion clusters, which is the main prerequisite to accurately generate ranked opinion tags.

We use a standard transformer encoder (Vaswani et al., 2017) to convert each review $X_{i}$ into vector representations. Following Vaswani et al. (2017), we first map each word $x\in X_{i}$ into its vectorized representation $\textbf{x}\in\mathbb{R}^{d_{e}}$ using a word embedding layer and a positional embedding layer, as shown in the following equation:

Then we use a transformer layer to encode global contextual information for words within $X_{i}$.

where LayerNorm is the layer normalization proposed by Ba et al. (2016); MHAtt is the multi-head attention mechanism introduced by Vaswani et al. (2017); FFN is a two-layer feed-forward network with ReLU as hidden activation function; and $n$ is the number of transformer block layers. The word-level vector representations of review $X_{i}$ are $\mathbf{X}_{i}=[\mathbf{x}_{1},\ldots,\mathbf{x}_{L_{x_{i}}}]=[\mathbf{x}^{n}_{1},\ldots,\mathbf{x}^{n}_{L_{x_{i}}}]$. To obtain the sentence representation $\tilde{\mathbf{X}}_{i}$ of review $X_{i}$, we perform a hierarchical pooling operation (Shen et al., 2018) across its different words. The hierarchical pooling mechanism can preserve word order information and has demonstrated superior performance over mean-pooling or max-pooling on many semantic analysis tasks (Shen et al., 2018).

To highlight the item-aware reviews and ignore noisy reviews, the sentence representations of reviews are first weighted by the corresponding salience scores, i.e., $\mathbf{\tilde{X}}^{\prime}_{i}=z_{i}\mathbf{\tilde{X}}_{i}$. Then we apply the $k$-means (MacQueen, 1967) algorithm on $\{\mathbf{\tilde{X}}^{\prime}_{1},\ldots,\mathbf{\tilde{X}}^{\prime}_{M}\}$ to group corresponding $\{\mathbf{X}_{1},\ldots,\mathbf{X}_{M}\}$ into $K$ opinion clusters.³³3$K$ is manually assigned according to the number of reviews. If $M\leq 200$, $K=\lceil\frac{M}{20}\rceil$, otherwise, $K=20$. We rank opinion clusters from the largest (representing the highest number of reviews) to the smallest, denoted as $[C_{1},\ldots,C_{K}]$. For each cluster, we rank reviews from the nearest (representing the distance between review and cluster center) to the farthest. Finally, we obtain a ranked list of reviews, represented as $[\mathbf{X}_{1,1},\ldots,\mathbf{X}_{1,{L_{1}}},\ldots,\mathbf{X}_{K,{1}},\ldots,\mathbf{X}_{K,{L_{K}}}]$, where $\mathbf{X}_{k,i}$ is the $i$-th review in the $k$-th opinion cluster and $L_{k}$ is the number of reviews in the $k$-th opinion cluster.

We sequentially concatenate the vector representations of reviews in the ranked list and derive the final word-level representations, i.e., $\mathbf{X}=[\mathbf{x}_{1,{1}},\ldots,\mathbf{x}_{1,{c_{1}}},\ldots,\mathbf{x}_{K,{1}},\ldots,\mathbf{x}_{K,{c_{K}}}]$.⁴⁴4In this paper, we only focus on the ranks of opinion clusters. We regard reviews or words in the same opinion cluster as equally important. Similarly, $x_{k,p}$ is the $p$-th word in the $k$-th opinion cluster and $c_{k}$ is the number of words in the $k$-th opinion cluster. Next, $\mathbf{X}$ will serve as the memory bank for the later rank-aware opinion tagging component.

Accurately generating opinion tags with ranks is challenging. Therefore, we propose a rank-aware opinion tagging component to generate ranked opinion tags.

In the training stage, we add a start token BOS at the beginning of each opinion tag, i.e., $Y^{\prime}_{j}=[\texttt{BOS};Y_{j}]$ where $[;]$ is the concatenation function. Then we concatenate all opinion tags into a sequence of words: $Y=[Y^{\prime}_{1};\ldots;Y^{\prime}_{N}]=[y_{1,0},\ldots,y_{1,L_{y_{1}}},\ldots,y_{N,0},\ldots,y_{N,L_{y_{N}}}]$, where $y_{j,q}$ is the $q$-th word of opinion tag $Y^{\prime}_{j}$, $y_{j,0}$ is the BOS token of the $j$-th opinion tag. Our decoder, i.e., the rank-aware opinion tagging component, follows the transformer architecture (Vaswani et al., 2017).

From the data analysis in Figure 3, we know that the ranks of opinion tags have a strong correlation with the ranks of opinion clusters. The $j$-th opinion tag may pay attention to the $j$-th opinion cluster and its surrounding neighbors simultaneously. Therefore, for each opinion tag $Y^{\prime}_{j}$, we hypothesize that the model needs to focus on the $F$ most related opinion clusters.⁵⁵5$F$ is a hyperparameter. $F$ opinion clusters mean the $j$-th opinion clusters and its surrounding neighbors. If an opinion cluster belongs to the $F$ focused opinion clusters, we call it a Focused Opinion Cluster (FOC). Otherwise, we call it an Outer Opinion Cluster (OOC). We design two alignment strategies between between FOCs and opinion tags, i.e., incorporating alignment features and enforcing alignment loss, which help to improve the generation of ranked opinion tags.

We adopt a multi-task learning framework to jointly minimize the salience classification loss, alignment loss, and generation loss. The objective function is:

where $\lambda_{1},\lambda_{2},\lambda_{3}$ are hyper-parameters that control the weights of these three losses. We set $\lambda_{1}=\lambda_{2}=\lambda_{3}=1$. Thus, each component of our joint model can be trained end-to-end.

We report on five experiments. First, we compare AOT-Net against a number of baselines to assess its overall performance. Then we conduct ablation studies to analyze the influence of different components in AOT-Net as follows: (i) w/o SSEis AOT-Net without the sentence-level salience estimating component (SSE). (ii) w/o RCRis AOT-Net without the review clustering and ranking component (RCR). (iii) w/o AFis AOT-Net without alignment feature (AF). (iv) w/o ALis AOT-Net without alignment loss (AL). To further explore the effectiveness of the sentence-level self-attention mechanism in SSE, we consider AOT-RNN, the method only considers BiGRU in salience score prediction; whereas we write AOT-Embed for the method that employs MLP to replace BiGRU in SSE. Fourth, we analyze the performance of AOT-Net for different sizes of FOCs. Lastly, we provide a case study about abstractive opinion tagging.

We compare AOT-Net with the following methods: (i) TF-IDFis an extractive approach that selects the important words as summary based on term frequency and inverse document frequency; (ii) TextRank(Mihalcea and Tarau, 2004)is an unsupervised algorithm based on weighted-graphs; (iii) RNNis a sequence to sequence model with attention implemented by bi-directional GRU layer (Cho et al., 2014); (iv) PG-Net(See et al., 2017)is a classical opinion summarization model based on the encoder-decoder framework with attention and copy mechanisms; and (v) the Transformer (Vaswani et al., 2017) is a Transformer-based encoder-decoder model with a copy mechanism, which is a strong baseline widely-adopt in opinion summarization.

Since there is no available opinion tagging dataset, we build a new one, named eComTag from several Chinese e-commerce websites. We collect nearly 112k items, sampling from different domains, including Cosmetic ($37.43\%$), Electronics ($29.51\%$), Books ($10.57\%$), Entertainment ($8.23\%$), Food ($7.62\%$), Sports ($3.96\%$), Clothes ($3.16\%$), Medical ($1.62\%$), and Furniture ($0.33\%$). For each domain, there are a set of reviews and a list of opinion tags. Since reviewers may comment on multiple aspects, e.g., “The dim sum tasted extremely fresh, and the price was quite reasonable!”, we split each review into sentences by punctuation. We use a sentence to denote a review in our paper. Then, we remove samples where the number of opinion tags is smaller than 4 or the number of reviews is fewer than 50. Finally, we construct eComTag with 50,068 item samples. Users may write reviews arbitrarily, which results in many meaningless expressions, such as “Love, love, LOVE this space!”, and “come with my boyfriend.” To teach our model to distinguish these noisy sentences, we annotate each review with a binary salience label via human judgment. If a review is item-related, we label the review as 1, otherwise 0. The salience labels are the supervision signals for the sentence-level salience estimating component.

Finally, each item sample consists of a set of reviews, a set of corresponding salience labels, and a sequence of opinion tags. For text preprocessing, we tokenize texts using the Jieba toolkit⁶⁶6https://github.com/fxsjy/jieba and maintain a 50k vocabulary. In eComTag, about 30% of samples have more than $1024$ words in reviews. We randomly split the dataset into training/validation/test sets with 8:1:1 ratio. The statistics are shown in Table 1. Specially, we define the present tag (Pr) as the exact tag that appears in reviews and absent tag (Ab) as the tag unseen in reviews. The proportion of absent tags is close to 75%, which further proves the necessity to apply abstractive methods on opinion tagging.

We employ two information retrieval metrics to evaluate the opinion tag generation: the macro F$@{k}$ score and the normalized discounted cumulative gain (NDCG$@{k}$) score. Both are widely used to measure word overlap (Sun et al., 2019; Chan et al., 2020). To measure diversity of the generated opinion tags, we adopt the Distinct-${2}$ score (Li et al., 2016) and a macro Unique-$N$ score. We compute Unique-$N$ as follows, $\text{Unique-}N=\sum_{i=1}^{T}N_{i}/T$, where $N_{i}$ is the number of distinct opinion tags in the $i$-th sample. Moreover, we design two metrics, Exact Rank Match (ERM) and Fuzzy Rank Match (FRM), to evaluate the rank accuracy of opinion tags. ERM is defined as the one-to-one exact match proportion between true tags and predicted tags. Inspired by the Embedding Score (Liu et al., 2016), FRM first maps predicted tags and the corresponding true tags into the same vector space, and then computes the average cosine similarity between their vector representations.

We adopt the Adam (Kingma and Ba, 2015) optimizer with settings {$\beta_{1}=0.9$,$\beta_{2}=0.999$,$\epsilon=10^{-8}$,$lr=10^{-4}$} and we vary the learning rate following Vaswani et al. (2017). We add dropout (Srivastava et al., 2014) with keeping rate 0.8 and label smoothing (Szegedy et al., 2016) with smoothing factor 0.1. We use the Tencent AI Lab Chinese Embeddings⁷⁷7https://ai.tencent.com/ailab/nlp/embedding.html for initialization of the word embedding layers. The rest of the parameters are randomly initialized. The dimensions of the alignment feature, word embedding layers and positional embedding layers are set to 200. We set the batch size to 16 and use the validation loss for early stopping. When inference, we set the maximum decoding step as 50. We use a bidirectional GRU (Cho et al., 2014) with 2 layers to implement the sentence-level salience estimating component. All RNN-based models have 256 hidden units. All transformer-based models have 300 hidden units; the feed-forward hidden size is set to 50 for all layers. We set the $F$ in Section 4.4 to $3$ ($3$ is the number of Focused Opinion Clusters at each decoding step) to ensure the target tag token have enough relevant reviews to reference and avoid introducing too much interference information simultaneously. AOT-Net was trained on a single Tesla V100 GPU and is implemented using PyTorch. All hyperparameters and models are selected on the validation set and the results are reported on the test set.

As shown in Table 2, AOT-Net achieves a $17.1$% and $2.72$% increase over “w/o SSE” in terms of Micro-Distinct-2 and Macro-Distinct-2, respectively. Similar improvements can be observed for other metrics. This demonstrates that the predicted salience scores help AOT-Net to focus on more valuable reviews. To verify the effectiveness of SSE with more details, in Figure 6 we list the accuracy scores of AOT-Embed, AOT-RNN, and AOT-Net for sentence-level salience estimation. We find that AOT-Net outperforms both AOT-Embed and AOT-RNN, which verifies the effectiveness of BiGRU and self-attention mechanisms. AOT-RNN achieves $7.2$% increase over AOT-Embed in terms of accuracy for all reviews, which verifies the advantage of BiGRU in representing review. In terms of accuracy, we find that AOT-Net gives a $0.4$% and $5.1$% increase over AOT-RNN for all reviews and item-related reviews, respectively. This indicates that AOT-Net benefits from self-attention mechanisms, which capture the shared information to distinguish item-related reviews.

To evaluate the effect of the number of FOCs on the performance of rank-aware opinion tagging, we examine the performance of AOT-Net with different values of $F$ (see Section 4.4) in terms of ERM, FRM, and Distinct-2, respectively. As shown in Table 3, $F=1$ significantly decreases the model rank and diversity performance. This suggests that only focusing on a single opinion cluster ignores many related reviews. We also find that when $F=5$, the performance of AOT-Net slightly decreases; AOT-Net achieves the best performance in terms of all metrics when $F=3$. Hence, we infer that $F$ is a trade-off between focusing on relevant reviews and removing irrelevant noise.

Figure 7 shows an example illustrating the 5 highest attention weights $\alpha_{i,p}$ during the generation of opinion tags. We see that the model transits its focus smoothly from the first review sentence to later review sentences. Sometimes, the model may focus on the same review for two opinion tags, such as “The service ($1$st tag) is good and hairstyle ($2$nd tag) is good.” The first three predicted tags were completely accurate. However, the transformer only predicted the first tag accurately. To validate the effectiveness of the alignment loss, we calculate $\sum_{i,p:x_{i,p}\in\text{FOCs}}$ and $\sum_{i,p:x_{i,p}\in\text{OOCs}}$ for all items in the test set. Results show that $\sum_{i,p:x_{i,p}\in\text{FOCs}}$ and $\sum_{i,p:x_{i,p}\in\text{OOCs}}$ in AOT-Net are $0.7304$ and $0.2696$ on average. However, for AOT-Net w/o AL, $\sum_{i,p:x_{i,p}\in\text{FOCs}}$ and $\sum_{i,p:x_{i,p}\in\text{OOCs}}$ are $0.6509$ and $0.3491$, respectively. This phenomenon indicates that the alignment feature by itself is not enough to force AOT-Net to focus on FOCs. In particular, we compare the ranks of opinion tags generated by the transformer and AOT-Net respectively. For the transformer, items where the first 3 opinion tags have accurate ranks only account for near 6.11%, while for AOT-Net, the proportion can reach 9.68%. Thus, we conclude that the alignment feature and alignment loss in AOT-Net are helpful to capture the ranks of the opinion tags.