On the Relationship between Explanation and Recommendation: Learning to Rank Explanations for Improved Performance

Personalized recommendation aims at providing a user with a ranked list of items that he/she never interacted with before. For each user $u\in\mathcal{U}$, the list of $M$ items can be generated as follows,

where $\hat{r}_{u,\underline{i}}$ is the predicted score for a user $u$ on item $i$, and $\mathcal{I}/\mathcal{I}_{u}$ denotes the set of items on which user $u$ has no interactions. In Eq. (1), $i$ is underlined, which means that we aim to rank the items.

Explanation ranking is the task of finding a list of appropriate explanations for a user-item pair to justify the recommendation. Formally, given a user $u\in\mathcal{U}$ and an item $i\in\mathcal{I}$, the goal of this task is to rank the entire collection of explanations $\mathcal{E}$, and select the top $N$ to reason why the item $i$ is recommended. Specifically, we define this list of top $N$ explanations as:

where $\hat{r}_{u,i,\underline{e}}$ is the estimated score of explanation $e$ for a given user-item pair $(u,i)$, which could be given by a recommendation model or by the user’s true behavior.

The preceding tasks solely rank either items or explanations. In this task, we further investigate whether it is possible to find an ideal item-explanation pair for a user, to whom the explanation best justifies the item that he/she likes the most. To this end, we treat each pair of item-explanation as a joint unit, and then rank these units. Specifically, for each user $u\in\mathcal{U}$, a ranked list of $M$ item-explanation pairs can be produced as follows,

where $\hat{r}_{u,\underline{i,e}}$ is the predicted score for a given user $u$ on the item-explanation pair ($i$, $e$).

We see that either item ranking task or explanation ranking task is a special case of this item-explanation joint-ranking task. Concretely, Eq. (3) degenerates to Eq. (1) when explanation $e$ is fixed, while it reduces to Eq. (2) if item $i$ is already known.

Suppose we have an ideal model that can perform the aforementioned joint-ranking task. During the prediction stage as in Eq. (3), there would be $\left|\mathcal{I}\right|\times\left|\mathcal{E}\right|$ candidate item-explanation pairs to rank for each user $u\in\mathcal{U}$. The runtime complexity is then $O(\left|\mathcal{U}\right|\cdot\left|\mathcal{I}\right|\cdot\left|\mathcal{E}\right|)$, which makes this task impractical, compared with the traditional recommendation task’s $O(\left|\mathcal{U}\right|\cdot\left|\mathcal{I}\right|)$ complexity.

To reduce the complexity, we reformulate the joint-ranking task by performing ranking for items and explanations simultaneously but separately. In this way, we are also able to investigate the relationship between item ranking and explanation ranking, e.g., improving the performance of both. Specifically, during the testing stage, we first follow Eq. (1) to rank items for each user $u\in\mathcal{U}$, which has the runtime complexity of $O(\left|\mathcal{U}\right|\cdot\left|\mathcal{I}\right|)$. After that, for $M$ recommendations for each user, we can rank and select explanations to justify each of them according to Eq. (2). The second step’s complexity is $O(\left|\mathcal{U}\right|\cdot M\cdot\left|\mathcal{E}\right|)$, but since $M$ is a constant and $\left|\mathcal{E}\right|\ll\left|\mathcal{I}\right|$ (see Table 2), the overall complexity of the two steps is $O(\left|\mathcal{U}\right|\cdot\left|\mathcal{I}\right|)$.

In the following, we first analyze the drawback of a conventional Tensor Factorization (TF) model when being applied to the explanation ranking problem, and then introduce our solution BPER. Second, we show how to further enhance BPER by utilizing the semantic features of textual explanations (denoted as BPER+). Third, we illustrate their relation to two typical TF methods CD and PITF. At last, we integrate the explanation ranking with item ranking into a multi-task learning framework as a joint-ranking task.

To perform explanation ranking, the score $\hat{r}_{u,i,e}$ on each explanation $e\in\mathcal{E}$ for a given user-item pair $(u,i)$ must be estimated. As the user-item-explanation ternary relations $\mathcal{T}=\{(u,i,e)|u\in\mathcal{U},i\in\mathcal{I},e\in\mathcal{E}\}$ form an interaction cube, we are inspired to employ factorization models to predict this type of scores. There are a number of tensor factorization techniques (Bhargava et al., 2015; Ioannidis et al., 2019), such as Tucker Decomposition (TD) (Tucker, 1966), Canonical Decomposition (CD) (Carroll and Chang, 1970) and High Order Singular Value Decomposition (HOSVD) (De Lathauwer et al., 2000). Intuitively, one would adopt CD, because of its linear runtime complexity in terms of both training and prediction (Rendle and Schmidt-Thieme, 2010) and its close relation to Matrix Factorization (MF) (Mnih and Salakhutdinov, 2007) that has been extensively studied in recent years for item recommendation. Formally, according to CD, the score $\hat{r}_{u,i,e}$ of user $u$ on item $i$’s explanation $e$ can be estimated by the sum over the element-wise multiplication of the user’s latent factors $\mathbf{p}_{u}$, the item’s $\mathbf{q}_{i}$ and the explanation’s $\mathbf{o}_{e}$:

where $\odot$ denotes the element-wise multiplication of two vectors.

However, this method may not be effective enough due to the inherent sparsity problem of the ternary data as we discussed before. Since each user-item pair $(u,i)$ in the training set $\mathcal{T}$ is unlikely to have interactions with many explanations in $\mathcal{E}$, the data sparsity problem for explanation ranking is severer than that for item recommendation. Simply multiplying the three vectors would hurt the performance of explanation ranking, which is evidenced by our experimental results in Section 6.

To mitigate such an issue and to improve the effectiveness of explanation ranking, we propose to separately estimate the user $u$’s preference score $\hat{r}_{u,e}$ on explanation $e$ and the item $i$’s appropriateness score $\hat{r}_{i,e}$ for explanation $e$. To this end, we perform two sets of matrix factorization, rather than employing one single TF model. In this way, the sparsity problem would be considerably alleviated, since the data are reduced to two collections of binary relations, both of which are similar to the case of item recommendation discussed above. At last, the two scores $\hat{r}_{u,e}$ and $\hat{r}_{i,e}$ are combined linearly through a hyper-parameter $\mu$. Specifically, the score of user $u$ for item $i$ on explanation $e$ is predicted as follows,

where $\{\mathbf{o}_{e}^{U},b_{e}^{U}\}$ and $\{\mathbf{o}_{e}^{I},b_{e}^{I}\}$ are two different sets of latent factors for explanations, corresponding to users and items respectively.

Since selecting explanations that are likely to be perceived helpful by users is inherently a ranking-oriented task, directly modeling the relative order of explanations is thus more effective than simply predicting their absolute scores. The Bayesian Personalized Ranking (BPR) criterion (Rendle et al., 2009) meets such an optimization requirement. Intuitively, a user would be more likely to appreciate explanations that cater to her own preferences, while those that do not fit one’s interests would be less attractive to the user. Similarly, some explanations might be more suitable to describe certain items, while other explanations might not. To build such type of pair-wise preferences, we use the first two rows in Eq. (5) to compute the difference between two explanations for both user $u$ and item $i$ as follows,

which respectively reflect user $u$’s interest in explanation $e$ over $e^{\prime}$, and item $i$’s appropriateness for explanation $e$ over $e^{\prime\prime}$.

With the scores $\hat{r}_{u,ee^{\prime}}$ and $\hat{r}_{u,ee^{\prime\prime}}$, we can then adopt the BPR criterion (Rendle et al., 2009) to minimize the following objective function:

where $\sigma(\cdot)$ denotes the sigmoid function, $\mathcal{I}_{u}$ represents the set of items that user $u$ has interacted with, $\mathcal{E}_{u,i}$ is the set of explanations in the training set for the user-item pair $(u,i)$, $\mathcal{E}/\mathcal{E}_{u}$ and $\mathcal{E}/\mathcal{E}_{i}$ respectively correspond to explanations that user $u$ and item $i$ have not interacted with, $\Theta$ is the model parameter, and $\lambda$ is the regularization coefficient.

From Eq. (7), we can see that there are two explanation tasks to be learned respectively, corresponding to users and items. During the training stage, we allow them to be equally important, since we have a hyper-parameter $\mu$ in Eq. (5) to balance their importance during the testing stage. The effect of this parameter is studied in Section 6.1. After the model parameters are estimated, we can rank explanations according to Eq. (2) for each user-item pair in the testing set. As we model the explanation ranking task under BPR criterion, we accordingly name our method Bayesian Personalized Explanation Ranking (BPER). To learn the model parameter $\Theta$, we draw on the widely used stochastic gradient descent algorithm to optimize the objective function in Eq. (7). Specifically, we first randomly initialize the parameters, and then repeatedly update them by uniformly taking samples from the training set and computing the gradients w.r.t. the parameters, until the convergence of the algorithm. The complete learning steps are shown in Algorithm 1.

The BPER model only exploits the IDs of users, items and explanations to infer their relation for explanation ranking. However, this makes the rich semantic features of the explanations, which could also capture the relation between explanations, under-explored. For example, “the acting is good” and “the acting is great” for movie recommendation both convey a positive sentiment with a similar meaning, so their ranks are expected to be close. Hence, we further investigate whether such features could help to enhance BPER. As a feature extractor, we opt for BERT (Devlin et al., 2019), a well-known pre-trained language model, whose effectiveness has been demonstrated on a wide range of natural language understanding tasks. Specifically, we first add a special [CLS] token at the beginning of a textual explanation $e$, e.g., “[CLS] the acting is great”. After passing it through BERT, we can obtain the aggregate representation (corresponding to [CLS]) that encodes the explanation’s overall semantics. To match the dimension of latent factors in our model, we apply a linear layer to this vector, resulting in $\mathbf{o}_{e}^{BERT}$. Then, we enhance the two ID-based explanation vectors $\mathbf{o}_{e}^{U}$ and $\mathbf{o}_{e}^{I}$ in Eq. (5) by multiplying $\mathbf{o}_{e}^{BERT}$, resulting in $\mathbf{o}_{e}^{U+}$ and $\mathbf{o}_{e}^{I+}$.

To predict the score for $(u,i,e)$ triplet, we replace $\mathbf{o}_{e}^{U}$ and $\mathbf{o}_{e}^{I}$ in Eq. (5) with $\mathbf{o}_{e}^{U+}$ and $\mathbf{o}_{e}^{I+}$. Then we use Eq. (7) as the objective function, which can be optimized via back-propagation. In Eq. (8), we adopt the multiplication operation simply to verify the feasibility of incorporating semantic features. The model may be further improved by more sophisticated operations, e.g., multi-layer perceptron (MLP), but we leave the exploration for future work.

Notice that, BPER is a general method that only requires the IDs of users, items and explanations, which makes it very flexible when being adapted to other explanation styles (e.g., images (Chen et al., 2019b)). However, it may suffer from the common cold-start issue as with other recommender systems. BPER+ could mitigate this issue to some extent, because besides IDs it also considers the semantic relation between textual explanations via BERT, which can connect new explanations with existing ones. As the first work on ranking explanations for recommendations, we opt to make both methods relatively simple for reproducibility purpose. In this way, it is also easy to observe the experimental results (such as the impact of explanation task on recommendation task), without the interference of other factors.

In fact, our Bayesian Personalized Explanation Ranking (BPER) model is a type of Tensor Factorization (TF), so we analyze its relation to two closely related TF methods: Canonical Decomposition (CD) (Carroll and Chang, 1970) and Pairwise Interaction Tensor Factorization (PITF) (Rendle and Schmidt-Thieme, 2010). On the one hand, in theory BPER can be considered as a special case of the CD model. Suppose the dimensionality of BPER is $2\cdot d+2$, we can reformulate it as CD in the following,

where the parameter $\mu$ is a constant.

On the other hand, PITF can be seen as a special case of our BPER. Formally, its predicted score $\hat{r}_{u,i,e}$ for the user-item-explanation triplet $(u,i,e)$ can be calculated by:

We can see that our BPER degenerates to PITF if in Eq. (5) we remove the bias terms $b_{e}^{U}$ and $b_{e}^{I}$ and set the hyper-parameter $\mu$ to 0.5, which means that the two types of scores for users and items are equally important to the explanation ranking task.

Although CD is more general than our BPER, its performance may be affected by the data sparsity issue as discussed before. Our BPER could mitigate this problem given its explicitly designed structure that may be difficult for CD to learn from scratch. When comparing with PITF, we can find that the parameter $\mu$ in BPER is able to balance the importance of the two types of scores, corresponding to users and items, which makes our BPER more expressive than PITF and hence likely reach better ranking quality.

In a similar way, BPER+ can also be rewritten as CD or PITF. Concretely, by revising the last part of Eq. (9) as the following formula, BPER+ can be seen as CD. When $\mathbf{o}_{e}^{BERT}=[1,...,1]^{\top}$, BPER+ is equal to BPER, so it can be easily converted into PITF. The graphical illustration of the four models is shown in Fig. 3.

Owing to BPER’s flexibility to accommodate various explanation styles as discussed before, we perform the joint-ranking on it. Specifically, we incorporate the two tasks of explanation ranking and item recommendation into a unified multi-task learning framework, so as to find a good solution that benefits both of them.

For recommendation, we adopt Singular Value Decomposition (SVD) model (Koren et al., 2009) to predict the score $\hat{r}_{u,i}$ of user $u$ on item $i$:

where $b_{i}$ is the bias term for item $i$. Notice that, the latent factors $\mathbf{p}_{u}$ and $\mathbf{q}_{i}$ are shared with those for explanation ranking in Eq. (5). In essence, item recommendation is also a ranking task that can be optimized using BPR criteria (Rendle et al., 2009), so we first compute the preference difference $\hat{r}_{u,ii^{\prime}}$ between a pair of items $i$ and $i^{\prime}$ to a user $u$ as follows,

which can then be combined with the task of explanation ranking in Eq. (7) to form the following objective function for joint-ranking:

where the parameter $\alpha$ can be fine-tuned to balance the learning of the two tasks.

We name this method BPER-J where J stands for joint-ranking. Similar to BPER, we can update each parameter of BPER-J via stochastic gradient descent (see Algorithm 2).

To compare the ranking performance of different methods, it is expected that the datasets contain user-item-explanation interaction triplets. The datasets could be manually constructed as in (Xu et al., 2020), but we are not given access to such datasets. Therefore, we adopt three public datasets³³3https://github.com/lileipisces/EXTRA (Li et al., 2021b), where the explanations are automatically extracted from user reviews via near-duplicate detection, which ensures that the explanations are commonly used by users. Specifically, the datasets are from different domains, including Amazon Movies & TV⁴⁴4http://jmcauley.ucsd.edu/data/amazon, TripAdvisor⁵⁵5https://www.tripadvisor.com for hotels and Yelp⁶⁶6https://www.yelp.com/dataset/challenge for restaurants. Each record in the three datasets consists of user ID, item ID, and one or multiple explanation IDs, and thus results in one or multiple user-item-explanation triplets. Moreover, each explanation ID appears no less than 5 times. The statistics of the three datasets are presented in Table 2. As it can be seen, the data sparsity issue on the three datasets is very severe.

Table 3 shows 5 example explanations taken from the three datasets. As we can see, all the explanations are quite concise and informative, which could prevent from overwhelming users, a critical issue for explainable recommendation (Herlocker et al., 2000). Also, short explanations can be mobile-friendly, since it is difficult for a small screen to fit much content. Moreover, the explanations from different datasets well suit the target application domains, such as “a wonderful movie for all ages” for movies and “comfortable hotel with good facilities” for hotels. Explanations with negative sentiment can also be observed, e.g., “the place is awful”, which can be used to justify why some items are dis-recommended (Zhang et al., 2014). Hence, we believe that the datasets are very suitable for our explanation ranking experiment.

To evaluate the performance of explanation ranking task, where the user-item pairs are given, we adopt the following baselines. Notice that, we omit the comparison with Tucker Decomposition (TD) (Tucker, 1966), because it takes cubic time to run and we also find that it does not perform better than CD in our trial experiment.

• •

  RAND: It is a weak baseline that randomly picks up explanations from the explanation collection $\mathcal{E}$. It is devised to examine whether personalization is needed for explanation ranking.

• •

  RUCF: Revised User-based Collaborative Filtering. Because traditional CF methods (Resnick et al., 1994; Sarwar et al., 2001) cannot be directly applied to the ternary data, we make some modifications to their formula, following (Jäschke et al., 2007). The similarity between two users is measured by their associated explanation sets via Jaccard Index. When predicting a score for the $(u,i,e)$ triplet, we first find users associated with the same item $i$ and explanation $e$, i.e., $\mathcal{U}_{i}\cap\mathcal{U}_{e}$, from which we then find the ones appearing in user $u$’s neighbor set $\mathcal{N}_{u}$.

  (15)

  $$\hat{r}_{u,i,e}=\sum_{u^{\prime}\in\mathcal{N}_{u}\cap(\mathcal{U}_{i}\cap\mathcal{U}_{e})}s_{u,u^{\prime}}\mbox{ where }s_{u,u^{\prime}}=\frac{|\mathcal{E}_{u}\cap\mathcal{E}_{u^{\prime}}|}{|\mathcal{E}_{u}\cup\mathcal{E}_{u^{\prime}}|}$$

• •

  RICF: Revised Item-based Collaborative Filtering. This method predicts a score for a triplet from the perspective of items, whose formula is similar to Eq. (15).

• •

  CD: Canonical Decomposition (Carroll and Chang, 1970) as shown in Eq. (4). This method only predicts one score instead of two for the triplet $(u,i,e)$, so its objective function shown below is slightly different from ours in Eq. (7).

  (16)

  $$\min_{\Theta}\sum_{u\in\mathcal{U}}\sum_{i\in\mathcal{I}_{u}}\sum_{e\in\mathcal{E}_{u,i}}\sum_{e^{\prime}\in\mathcal{E}/\mathcal{E}_{u,i}}-\ln\sigma(\hat{r}_{u,i,ee^{\prime}})+\lambda\left|\left|\Theta\right|\right|_{F}^{2}$$

  where $\hat{r}_{u,i,ee^{\prime}}=\hat{r}_{u,i,e}-\hat{r}_{u,i,e^{\prime}}$ is the score difference between a pair of interactions.

• •

  PITF: Pairwise Interaction Tensor Factorization (Rendle and Schmidt-Thieme, 2010). It makes prediction for a triplet based on Eq. (10), and its objective function is identical to CD’s in Eq. (16).

To verify the effectiveness of the joint-ranking framework, in addition to our method BPER-J, we also present the results of two baselines: CD (Carroll and Chang, 1970) and PITF (Rendle and Schmidt-Thieme, 2010). Since CD and PITF are not originally designed to accomplish the two tasks of item recommendation and explanation ranking together, we first allow them to make prediction for a user-item pair $(u,i)$ via the inner product of their latent factors, i.e., $\hat{r}_{u,i}=\mathbf{p}_{u}^{T}\mathbf{q}_{i}$, and then combine this task with explanation ranking in a multi-task learning framework whose objective function is given below:

where $\hat{r}_{u,ii^{\prime}}=\hat{r}_{u,i}-\hat{r}_{u,i^{\prime}}$ is the difference between a pair of records. We name them CD-J and PITF-J respectively, where J denotes joint-ranking.

To evaluate the performance of both recommendation and explanation, we adopt four commonly used ranking-oriented metrics in recommender systems: Normalized Discounted Cumulative Gain (NDCG), Precision (Pre), Recall (Rec) and F1. We evaluate on top-10 ranking for both recommendation and explanation tasks. For the former task, it is easy to find the definition of the metrics in previous works, so we define those for the latter. Specifically, the scores for a user-item pair on the four metrics are computed as follows,

where $\text{rel}_{p}$ indicates whether the $p$-th explanation in the ranked list $\text{Top}(u,i,N)$ can be found in the ground-truth explanation set $\mathcal{E}_{u,i}^{te}$. Then, we can average the scores for all user-item pairs in the testing set.

We randomly divide each dataset into training (70%) and testing (30%) sets, and guarantee that each user/item/explanation has at least one record in the training set. The splitting process is repeated for 5 times. For validation, we randomly draw 10% records from training set. After hyper-parameters tuning, the average performance on the 5 testing sets is reported.

We implemented all the methods in Python⁷⁷7https://www.python.org. For TF-based methods, including CD, PITF, CD-J, PITF-J, and our BPER and BPER-J, we search the dimension of latent factors $d$ from [10, 20, 30, 40, 50], regularization coefficient $\lambda$ from [0.001, 0.01, 0.1], learning rate $\gamma$ from [0.001, 0.01, 0.1], and maximum iteration number $T$ from [100, 500, 1000]. As to joint-ranking of CD-J, PITF-J and our BPER-J, the regularization coefficient $\alpha$ on explanation task is searched from [0, 0.1, …, 0.9, 1]. For the evaluation of joint-ranking, we first evaluate the performance of item recommendation for users, followed by the evaluation of explanation ranking on those correctly predicted user-item pairs. For our methods BPER and BPER-J, the parameter $\mu$ that balances user and item scores for explanation ranking is searched from [0, 0.1, …, 0.9, 1]. After parameter tuning, we use $d=20$, $\lambda=0.01$, $\gamma=0.01$ and $T=500$ for our methods, while the other parameters $\alpha$ and $\mu$ are dependent on the datasets.

The configuration of BPER+ is slightly different, because of the textual content of the explanations. We adopted the pre-trained BERT from huggingface⁸⁸8https://huggingface.co/bert-base-uncased, and implemented the model in Python with PyTorch⁹⁹9https://pytorch.org. We set batch size to 128, $d=20$ and $T=5$. After parameter tuning, we set learning rate $\gamma$ to 0.0001 on Amazon, and 0.00001 on both TripAdvisor and Yelp.

Experimental results for explanation ranking on the three datasets are shown in Table 4. We see that each method’s performance on the four metrics (i.e., NDCG, Precision, Recall, F1) are fairly consistent across the three datasets. The method RAND is among the weakest baselines, because it randomly selects explanations without considering user and item information, which implies that the explanation ranking task is non-trivial. CD performs even worse than RAND, because of the sparsity issue in the ternary data (see Table 2), for which CD may not be able to mitigate as discussed in Section 4.2. CF-based methods, i.e., RUCF and RICF, largely advance the performance of RAND, as they take into account the information of either users or items, which confirms the important role of personalization for explanation ranking. However, their performance is still limited due to data sparsity. PITF and our BPER/BPER+ outperform the CF-based methods by a large margin, as they not only address the data sparsity issue via their MF-like model structure, but also take each user’s and item’s information into account using latent factors. Most importantly, our method BPER significantly outperforms the strongest baseline PITF, owing to its ability of producing two sets of scores, corresponding to users and items respectively, and its parameter $\mu$ that can balance their relative importance to explanation ranking. Lastly, BPER+ further improves BPER on most of the metrics across the three datasets, especially on NDCG that cares about the ranking order, which can be attributed to the consideration of the semantic features of the explanations as well as BERT’s strong language modeling capability to extract them.

Besides the explanation ranking performance, we also present the training time comparison of the three TF-based methods in Table 4. For fair comparison, the runtime testing is conducted on the same research machine without GPU, because these methods are all implemented in pure Python without involving deep learning framework. From the table, we can see that the training time of the three methods is generally consistent on different datasets. CD takes the least time to train, PITF needs a bit more training time, while the duration of training our BPER is the longest. This is quite expected since the model complexity grows larger from CD to PITF and BPER. However, the slightly sacrificed training time of BPER is quite acceptable because the gap of training duration between the three methods is not very large, e.g., 5h32min for CD, 7h9min for PITF and 9h43min for BPER on TripAdvisor dataset.

At last, we further analyze the parameter $\mu$ of BPER that controls the contributions of user scores and item scores in Eq. (5). As it can be seen in Fig. 4, the curves of NDCG, Precision, Recall and F1 are all bell-shaped, where the performance improves significantly with the increase of $\mu$ until it reaches an optimal point, and then it drops sharply. Due to the characteristics of different application domains, the optimal points vary among the three datasets, i.e., 0.7 for both Amazon and Yelp and 0.5 for TripAdvisor. We omit the figures of BPER+, because the pattern is similar.

As discussed earlier, the sparsity issue of user-item-explanation triple-wise data is severer than that of traditional user-item pair-wise data. To investigate how different methods deal with varying spareness, we further remove certain ratio of the Amazon training set, so that the training triplets to the whole dataset ranges from 30% to 70%, while the testing set remains untouched. For comparison with our BPER and BPER+, we include the most competitive baseline PITF. Fig. 5 shows the ranking performance of the three methods w.r.t. varying spareness. The ranking results are quite consistent on the four metrics (i.e., NDCG, Precision, Recall and F1). Moreover, with the increase of the amount of training triplets, the performance of all three methods goes up linearly. Particularly, the performance gap between our BPER/BPER+ and PITF is quite large, especially when the ratio of training data is small (e.g., 30%). These observations demonstrate our methods’ better capability in mitigating data sparsity issue, and hence prove the rationale of our solution that converts triplets to two groups of binary relation.

To better understand how explanation ranking works, we first present a case study comparing our method BPER and the most effective baseline PITF on Amazon Movies & TV dataset in Table 5. The two cases in the table respectively correspond to recommendation and disrecommendation. In the first case (i.e., recommendation), there are three ground-truth explanations, praising the movie’s “special effects”, “story” and overall quality. Generally speaking, the top-5 explanations resulting from both BPER and PITF are positive, and relevant to the ground-truth, because the two methods are both effective in terms of explanation ranking. However, since PITF’s ranking ability is relatively weaker than our BPER, its explanations miss the key feature “story” that the user also cares about.

In the second case (i.e., disrecommendation), the ground-truth explanation is a negative comment about the target movie’s “acting”. Although the top explanations made by both BPER and PITF contain negative opinions regarding this aspect, their ranking positions are quite different (i.e., top-3 for our BPER vs. bottom-2 for PITF). Moreover, we notice that for this disrecommendation, PITF places a positive explanation in the 1st position, i.e., “good action movie”, which not only contradicts the other two explanations, i.e., “the acting is poor/bad”, but also mismatches the disrecommendation goal. Again, this showcases our model’s effectiveness for explanation ranking.

We further conduct a small scale user study to investigate real people’s perception towards the top ranked explanations. Specifically, we still compare our BPER with PITF on Amazon Movies & TV dataset. We prepared ten different cases and hired college students to do the evaluation. In each case, we provide the movie’s title and the ground-truth explanations, and ask the participants to select one explanation list that is semantically closer to the ground-truth. There are two randomly shuffled options returned by BPER and PITF, respectively. A case is valid only when at least two participants select the same option. The evaluation results are shown in Fig. 6. We can see that on 60% of cases our BPER’s explanations are closer to the ground-truth than PITF’s, which is quite consistent with their explanation ranking performance.

We perform joint-ranking for three TF-based models, i.e., BPER-J, CD-J and PITF-J. Because of the consistency in the experimental results on different datasets, we only show results on Amazon and TripAdvisor. In Fig. 7, we study the effect of the parameter $\alpha$ to both explanation ranking and item ranking in terms of F1 (results on the other three metrics are consistent). In each sub-figure, the green dotted line represents the performance of explanation ranking task without joint-ranking, whose value is taken from Table 4. As we can see, all the points on the explanation curve (in red) are above this line when $\alpha$ is greater than 0, suggesting that the explanation task benefits from the recommendation task under the joint-ranking framework. In particular, the explanation performance of CD-J improves dramatically under the joint-ranking framework, since its recommendation task suffers less from the data sparsity issue than the explanation task as discussed in Section 4.2. It in turn helps to better rank the explanations. Meanwhile, for the recommendation task, all the three models degenerate to BPR when $\alpha$ is set to 0. Therefore, on the recommendation curves (in blue), any points, whose values are greater than that of the starting point, gain profits from the explanation task as well. All these observations show the effectiveness of our joint-ranking framework in terms of enabling the two tasks to benefit from each other.

In Table 6, we make a self-comparison of the three methods in terms of NDCG and F1 (the other two metrics are similar). In this table, “Non-joint-ranking” corresponds to each model’s performance with regard to explanation or recommendation when the two tasks are individually learned. In other words, the explanation performance is taken from Table 4, and the recommendation performance is evaluated when $\alpha=0$. “Best Exp” and “Best Rec” denote the best performance of each method on respectively explanation task and recommendation task under the joint-ranking framework. As we can see, when the recommendation performance is the best for all the models with joint-ranking, the explanation performance is always improved. Although minor recommendation accuracy is sacrificed when the explanation task reaches the best performance, we can always find points where both of the two tasks are improved, e.g., on the top left of Fig. 7 when $\alpha$ is in the range of 0.1 to 0.6 for BPER-J on Amazon. This again demonstrates our joint-ranking framework’s capability in finding good solutions for both tasks.