Graph-based Extractive Explainer for Recommendations

We first define the notations employed in this paper. Denote the candidate recommendation item set as $\mathcal{C}$, the set of users as $\mathcal{U}$, and the vocabulary of text reviews as $\mathcal{V}$. We use $|\cdot|$ to denote the cardinality of a set. We collect all the reviews from $\mathcal{U}$ about $\mathcal{C}$ and segment them into sentences. Then we denote $\mathcal{S}_{uc}=\{s_{uc}^{i}\}_{i=1}^{N}$ as all the sentences from user $u$ about item $c$, where each sentence $s_{uc}=\{w_{i}\}_{i=1}^{T}$ consists of $T$ words $w\in\mathcal{V}$. We aggregate sentences from user $u$ over all items into a set $\mathcal{S}_{u}=\{\mathcal{S}_{uc}\}_{c\in\mathcal{C}}$; and similarly, we aggregate sentences about item $c$ from all users into $\mathcal{S}_{c}=\{\mathcal{S}_{uc}\}_{u\in\mathcal{U}}$. The attributes $\mathcal{F}$ are items’ popular properties mentioned in the whole review corpus, which are a subset of words in $\mathcal{V}$. For a pair of a user $u\in\mathcal{U}$ and an item $c\in\mathcal{C}$, an extractive explanation model is to select $K$ sentences $\{s^{i}\}_{i=1}^{K}$ from the union of sentences $\mathcal{S}_{u}\bigcup\mathcal{S}_{c}$, such that these $K$ sentences best describe how the item is relevant to the user’s preference.

Measuring the relatedness between a candidate sentence and a target pair of user and item is vital for precise extractive explanation. On a given training corpus, the relatedness can be inferred based on the observed associations between user, item, and sentences in the reviews. For example, how does a user usually describe a type of items; and how is an item typically commented by a group of users. But such observations are expected to be sparse, as a user often provides only a handful of reviews.

GREENer addresses the sparsity issue by introducing item attributes as the intermediary between a user-item pair and associated sentences, and accordingly captures the relatedness from two complementary perspectives. First, the observed sentences written by the user and those describing the item, connected via the attributes appearing in these sentences, suggest the direct dependence between a sentence and the pair of user and item. This co-occurrence relation forms a heterogeneous graph among users, items, attributes, and sentences, suggesting their contextualized dependency. GREENer leverages an attentive graph neural network to model this direct dependence structure. Second, the learnt representations of users, items and sentences also suggest the dependence in the embedding space. GREENer utilizes the feature crossing technique to capture this indirect dependence. In the following, we provide the technical details of these two important components.

Based on the encoded sentence representations, GREENer learns to predict if the sentences are qualified explanations. In addition, GREENer also predicts if an attribute should be mentioned in the extracted explanations, which forms a multi-task objective for parameter estimation.

The relevance of a candidate sentence $s_{i}$ against $\mathcal{S}_{uc}$ can be simply realized via an indicator function, i.e., whether the $s_{i}$ is in $\mathcal{S}_{uc}$. To relax this, we choose to measure the similarity $r_{i}$ between $s_{i}$ and $\mathcal{S}_{uc}$,

where $\textit{sim}(\cdot)$ can be any text similarity metric that measures the semantic similarity between a pair of sentences. In our experiments, we used BLEU score, such that sentences in $\mathcal{S}_{uc}$ always have the highest similarity.

Under this similarity-based notion of sentence relevance, the pairwise ranking loss on a set of candidate sentences can be computed as follows,

where $\sigma(\cdot)$ is the sigmoid function.

In addition to recognizing the qualification of individual sentences, we believe good explanations should also cover important attributes for each user-item pair. This can be achieved by requiring the learnt attribute representations to be predictive about the ground-truth attributes. As a result, we introduce a logistic regression classifier based on the output representation $\hat{x}_{f_{i}}$ for each attribute node $f_{i}$, $p(f_{i})=\sigma\big{(}\langle W^{f}_{o}\,,\hat{x}_{f_{i}}\rangle\big{)}$, to predict if $f_{i}$ should appear in the explanation. We adopt the cross entropy as the loss function of attribute predictions. With the ground-truth label $y_{f_{i}}$, i.e., those appear in the ground-truth review content for a user-item pair, the loss of attribute predictions is,

Combining these two losses, we obtain the objective function as

where $\lambda$ is a hyper-parameter to control the weight of each loss to the objective.

We chose review-based public recommendation datasets Ratebeer (McAuley et al., 2012) and TripAdvisor (Wang et al., 2010) for our evaluation purpose. Both datasets contain user-provided textual reviews about their opinions towards specific items, including user ID, item ID, review text content and opinion ratings. In the Ratebeer dataset, the ratings fall into the range of $[1,20]$; and in the TripAdvisor dataset, the rating’s range is $[1,5]$. Since a recommender system would generally recommend items that are attractive to users, we only focused on user-item interactions with positive ratings. In particular, we used Ratebeer reviews with ratings greater than $10$ and TripAdvisor reviews greater than $3$ to construct the corpus for our experiments. Since GREENer focuses on explanation generation, in the experiments we directly used the observed item in each user-item pair as the recommendation to be explained.

Data Pre-Processing. Review content has been directly used as ground-truth explanations for evaluation in many previous studies (Chen et al., 2018; Wang et al., 2018b). But as suggested in Ni et al. (2019), a large portion of sentences in a review describes personal subjective experience, like “I drank two bottles of this beer”, which does not provide any information about the reason why the user liked this item, and hence they are not qualified as explanations. In contrast, sentences that serve as explanations should describe the important properties of items to help users make informed decisions, like “taste is of bubble gum and some banana.” Therefore, we construct the explanation dataset by keeping informative sentences in the experiments. For both datasets, we used the Sentires toolkit (Zhang et al., 2014b) to extract attribute words from reviews and manually filter out inappropriate ones based on our domain knowledge. Then, for each review, we kept sentences that cover at least one of the selected attribute words as candidate explanations.

We also filtered inactive users and unpopular items: we kept users who at least have fifteen reviews and items are associated with at least fifteen reviews. We keep the 20,000 most frequent words in our vocabulary and use the “unk” token to represent the rest words. The statistics of the processed datasets are reported in Table 2. We split the dataset into training, validation, and testing dataset according to the ratio $70\%\!:\!15\%\!:\!15\%$.

Baselines. We compare our model with five baselines, covering both generation-based and extraction-based methods, which can produce natural language sentences as explanations :

• •

  NRT: Neural Rating and Tips Generation (Li et al., 2017), a generation-based solution. It is originally proposed for tip (a short sentence summary) generation, but can be seamlessly adapted to generating explanations. It utilizes an RNN-based neural language model to generate explanations.

• •

  SAER: Sentiment Aligned Explainable Recommendation (Yang et al., 2021b). This is another generation-based solution. It focuses specifically on the sentiment alignment between the recommendation score and generated explanations. It implements a sentiment regularizer and a constrained decoding method to enforce the sentiment alignment in the explanations in both training and inference phases.

• •

  NARRE: Neural Attentional Regression model with Review-level Explanations (Chen et al., 2018). It is an extraction-based solution. It learns the usefulness of the existing reviews through attention. It selects the review with highest attention score as the explanation.

• •

  SEER: Synthesizing Aspect-Driven Recommendation Explanations from Reviews (Le and Lauw, 2021). This is another extraction-based solution. It takes a user’s sentiment towards item’s aspects as input, which can be obtained from explainable recommendation models such as EFM (Zhang et al., 2014a), and then form an ILP problem to select the $K$ most representative and coherent sentences that fit the user’s demand of $K$ aspects.

• •

  ESCOFILT: Unsupervised Extractive Summarization-Based Representations for Accurate and Explainable Collaborative Filtering (Pugoy and Kao, 2021). This is also an extraction-based solution. It is built on BERT text representation to generate user and item profile by clustering user-side and item-side sentence embeddings using $K$-Means. The $K$ sentences that are the closest to their own cluster centroids are selected to form the explanations.

Implementation Details. For both datasets, we first pre-trained 256-dimension Glove embeddings (Pennington et al., 2014) on the whole vocabulary and fine-tuned BERT model on our dataset using Sentence-BERT (Reimers and Gurevych, 2019) which were later used to initialize the attribute node and sentence node, respectively. User and item node embedding size $d_{u}$ and $d_{c}$ were both set to 256. We stacked 2 GAT layers, with 4 head in the first layer and 1 head in the second layer. The hidden dimension size in GAT was set to 256. For DCN, we combined a 2-layer cross network with a 2-layer MLP whose hidden size and output size were both set to 128. The sentence relevance score $r_{i}$ in Eq (2) used in pairwise loss was the pre-computed maximum sentence BLEU score between $s_{i}$ and $\mathcal{S}_{uc}$.

During training, we used a batch size of 16 and applied Adam optimizer (Kingma and Ba, 2014) with a learning rate of 2e-4. The $\lambda$ in multi-task loss function was set to 0.5. The model was selected according to its performance on the valid set where we took the top-5 predicted sentences and computed the BLEU score with the ground-truth review. When finally generating the explanations, in order to reduce the computation complexity of the ILP problem, we selected the top-100 predicted sentences for each user-item pair on the test set and use the Gurobi¹¹1https://www.gurobi.com/products/gurobi-optimizer/ solver to select the top-5 most relevant yet non-repetitive sentences. We computed the cosine similarity based on the tf-idf representation of a pair of sentences. $\alpha$ is set to 2.0 according to the performance on the validation set.

To comprehensively evaluate the quality of the generated explanations at both word-level and attribute-level, we employed different types of metrics, including BLEU-{1, 2, 4} and F1 score of ROUGE-{1,2,L}, which are used to measure word-level content generation quality, and Precision, Recall and F1 score of mentioned attributes, which are used to measure the attribute-level content generation quality. The results are reported in Table 3 and Table 4 respectively.

Word-level Content Generation Quality. GREENer outperformed baselines under every BLEU and ROUGE metric on both datasets. As BLEU is a precision-based metric, a larger BLEU achieved by a model suggests that a larger portion of content in sentences selected by the model is relevant to the ground-truth sentences. In other words, more content in sentences selected by GREENer can reflect/predict the users’ opinions towards the recommended items. RNN-based generative methods such as NRT and NARRE suffer from generating short and generic content, such as “the staff was very friendly and helpful.” Although such generic content has a larger chance to match with the ground-truth sentences, the brevity penalty in BLEU penalizes methods focusing too much on such less information sentences. The comparison of GREENer against NARRE shows that though both methods extract sentences from an existing corpus, the graph structure employed in GREENer can better recognize sentences matching users’ criteria of items than the attention structure used in NARRE. Both GREENer and SEER select the final $K$ sentences using ILP. The comparison between them shows that GREENer’s learnt sentence relevance score better reflects the utility of a sentence as an explanation than directly modeling it from pre-computed user sentiment scores over item aspects. ESCOFILT is enforced to cover $K$ different aspects of an item for its explanations to all users, while GREENer learns to predict what the most important sentences are for each user-item pair. This hard requirement in ESCOFILT makes it one of the most competitive baselines, but its lack of flexibility also leaves it behind GREENer. It is noteworthy that GREENer achieves much larger BLEU-4 than all baselines. It suggests the explanations generated by GREENer not only have more overlaps with the ground-truth, but also cover longer segments in the ground-truth; and therefore its content is more coherent as a whole.

On the other hand, as ROUGE is a recall-based metric, a higher ROUGE score achieved by a model suggests that more content in ground-truth sentences are included in the sentences selected by the model. This further demonstrates the effectiveness of GREENer in recognizing the relevance of candidate sentences to a user-item pair. In particular, GREENer achieved much higher ROUGE-L than all baselines. This indicates that GREENer is more successful in identifying sentences with long consecutive spans that match users’ comments about items. The graph structure that fuses heterogeneous information from users, items, attributes and sentences contributes the most to GREENer’s better extraction performance, which will be validated in our ablation analysis in Section 4.3.

Attribute-level Content Generation Quality. To better understand whether the explanations generated by GREENer cover more important information about the target items’ attributes, we also evaluated the Precision, Recall and F1 score of the attributes contained in the synthesized explanation with respect to the corresponding ground-truth. As we can observe in Table 4, GREENer outperformed baselines with a large margin in recall and comparable precision on both datasets. This strongly suggests GREENer can better recognize the relevance of candidate sentences with respect to the important item attributes, and maximize the coverage of those attributes. On the other hand, it confirms that GREENer’s encouraging performance on BLEU and ROUGE is not simply because of its extractive nature, but also due to its ability to more accurately identify and cover important target attributes.

We include four variants of our solution to study the contribution of each component to the performance of GREENer:

• •

  $\neg$ GAT. This variant excludes the graph neural network model and only preserves the user, item, attribute and sentence information. For each sentence $\mathcal{S}_{uc}$, we concatenate its embedding with user $u$, item $c$ and mean-pooling of $f_{\mathcal{S}_{uc}}$, where $f_{\mathcal{S}_{uc}}$ represents the attributes in sentence $\mathcal{S}_{uc}$. Then the concatenated embedding is directly fed into DCN to predict the relevance score of sentence $\mathcal{S}_{uc}$. The comparison between this variant and GREENer indicates the necessity of modeling the user, item, attributes and sentences into a heterogeneous graph to learn their intermediate relationships.

• •

  $\neg$ BERT. This variant replaces BERT with the average word embeddings for sentence representations. The comparison between this variant and GREENer demonstrates the importance of using a pre-trained language model to encode sentences as input sentence node representations.

• •

  $\neg$ DCN. This variant replaces DCN in GREENer with a single linear layer. The comparison between this variant and GREENer presents the effectiveness of including direct feature-level interaction among user, item and sentences when learning the final sentence representations.

• •

  $\neg$ ILP. This variant replaces the ILP-based sentence selection strategy with a vanilla strategy which selects sentences solely based on the predicted probabilities of sentences in a descending order. The comparison between this variant and GREENer shows the utility of considering both sentence relevance and redundancy when selecting sentences as explanations.

The results of our ablation analysis about GREENer are reported in Tabel 5. The same as Table 3, we report BLEU-{1, 2, 4} and F1 score of ROUGE-{1,2,L}. All the variants performed worse than GREENer. The most important component turns out to be ILP, which is expected. As the learnt sentence embedding in GREENer reflects the relevance of a sentence to a given user-item pair independently from other sentences, simply counting on this sentence representation for selecting sentences can hardly avoid repetitions. When we looked into the output of $\neg$ ILP, most of its top ranked sentences are very similar to each other and therefore can hardly cover a comprehensive set of aspects of the target item. The next most important component is the heterogeneous graph structure, which helps GREENer capture the complex relationship between a user’s preference and item’s aspects, which cannot be tackled by directly performing feature crossing. In addition, using BERT to obtain sentence representations enables GREENer to learn a better sentence representation. This analysis suggested that DCN introduced least impact on GREENer’s final performance; but combining DCN with GAT can still boost the model’s performance. This indicates feature-level interactions in the embedding space provide a complementary view for the final sentence representation and selection.

We present two group of example explanations produced by GREENer and other baselines in Table 6. The ground-truth explanations are also included for reference. We manually labeled the overlapping attributes in the generated sentences from the 3 methods. From the table, we can clearly observe that the extracted sentences by GREENer are much more relevant to the ground-truth explanations. The features in extracted sentences match those in ground-truth explanations, especially those uncommon attributes such as “roasty chocolate” in the first example and “hot tub” in the second example. With the explicitly mentioning of features, the explanations generated by GREENer can help users make more informed decisions on which item better suits their preferences and needs.