A Knowledge-Enhanced Recommendation Model
with Attribute-Level Co-Attention

In general, most items and their attributes (e.g., the actor and director of movies) in an open domain can be fetched from large-scale KGs, which constitute knowledge triplets formed as $<h,r,t>$. The head entity ($h$) is just an item, the relation ($r$) corresponds to an attribute and the tail entity ($t$) is an attribute value. For example, a triplet $<Rocky,starred,Stallone>$ describes that Stallone is an actor of movie Rocky. The shared attribute values well indicate the latent relationships between different items (Zhou and et al., 2018; Shen and et al., 2019). Therefore, $M$ significant attributes (values) $\{a_{1},a_{2},...,a_{M}\}$ are first selected to represent an item in ACAM. The $a_{i}$’s embedding is denoted as $\boldsymbol{a}_{i}\in\mathbb{R}^{d}$ ($1\leq i\leq M$). We further use an item embedding (head entity embedding), denoted as $\boldsymbol{a}_{0}$, to supplement the representation of an item. For the convenience of the following introduction, $\boldsymbol{a}_{0}$ is also regarded as an attribute embedding. Thus, an item’s representation is enriched into a matrix of $(M+1)\times d$, as shown in Fig. 1. Please note that an attribute may have multiple values corresponding to different tail entities in KGs. For example, a movie generally involves many actors and each actor corresponds to an entity in KGs and has a unique embedding. Thus, for an attribute with multiple values, we average all of its value embeddings (tail entity embeddings) as its attribute embedding.

A user in ACAM is represented by the recent $L$ items that he/she has interacted with, denoted as $\mathcal{L}=\{v_{1},v_{2},…,v_{L}\}$. As the candidate item’s representation, each interacted item $v_{j}(1\leq j\leq L)$ in $\mathcal{L}$ is also represented by the union of its $M+1$ attribute embeddings. Therefore, the target user $u$’s representation is enriched to be a cube (tensor) of $(M+1)\times L\times d$, denoted as $\boldsymbol{\tilde{E}}_{u}$. For those users with less than $L$ historical interactions, we fill paddings in their representations.

Next, we need to extract the features of $\boldsymbol{\tilde{E}}_{u}$ and compress it into a matrix. It is not only to reduce trainable parameters but also to feed the co-attention layer (module) conveniently. To this end, we aggregate the $i$-th attribute embeddings of $u$’s historical items as $u$’s $i$-th attribute embedding through an attention network. Specifically, we adopt a weighted sum pooling as

where $\boldsymbol{a}^{u}_{i}$ is regarded as $u$’s $i$-th ($0\leq i\leq M$) attribute embedding, and $FFN(\boldsymbol{a}^{j}_{i}\oplus\boldsymbol{a}^{v}_{i})$ is the computation of a feed-forward network fed with the concatenation of item’s attribute embedding $\boldsymbol{a}^{j}_{i}$ and $\boldsymbol{a}^{v}_{i}$. Thus, $u$’s representation consists of $M+1$ attribute embeddings computed by Eq. 1. Such user representations vary given different candidate recommended items, which have been proven more helpful for recommendation performance gains than fixed user representations (Shen and et al., 2019; He and et al., 2018).

We conducted our experiments against two realistic datasets, i.e, Douban movies and NetEase songs²²2Douban: https://movie.douban.com, NetEase: https://music.163.com. The statistics of these two datasets are listed in Table 1. We fetched Douban movies’ attribute values from a large-scale Chinese KG CN-DBpedia (Xu et al., 2017). In our experiments, we selected four significant attributes ($M=4$) for the two datasets, i.e., actor, director, writer and genre for Douban movies, and singer, album, composer and lyricist for NetEase songs. To reproduce our experiment results conveniently, we have published our datasets and ACAM’s source code on https://github.com/DeqingYang/ACAM-model.

For each user, we truncated his/her recent 10 interactions as the positive samples in test set, and the rest interactions as the positive samples in training set. We also used negative sampling (He et al., 2017) to collect negative samples for each user. Note that we inclined to those popular items (with high rating scores or more reviews) when selecting negative samples in random, to avoid such cases that a user did not rate/review an item just due to the unawareness of the item. In both model training and prediction, each positive sample was paired with 4 negative samples, which is the general setting in previous models (He et al., 2017; He and et al., 2018).

NCF (He et al., 2017): This is a DNN-based recommendation model consisting of a GMF (generalized matrix factorization) layer and an MLP, where each user and item is represented only by a single embedding.

NAIS (He and et al., 2018): This is an attention-based recommendation model in which only user representations are refined by attention mechanism, and each item is represented by a single embedding.

AFM (Xiao and et al., 2017): It is a neural version of FM which adds an attention-based pooling layer after the pairwise feature interaction layer.

FDSA (Zhang and et al., 2019): This sequential recommendation model also incorporates feature-level representations but uses self-attention mechanism to only refine user representations rather than item representations.

RippleNet (et al., 2018): It is a representative KG-based recommendation model, which was compared with ACAM to highlight ACAM’s strengths in knowledge (attribute) exploitation.

DIN (Zhou and et al., 2018): It also imports various features to enrich user/item representations. Furthermore, it uses the attention mechanism similar to NAIS to adjust user representations only.

In the following display of experiment results, we adopt three popular metrics evaluating top-n recommendation or ranking, i.e., HR@n (Hit Ratio), nDCG@n (Normailzed Discounted Cumulative Gain) and RR (Reciprocal Rank). To avoid statistics bias, all model’s performance are reported as the average scores of 3 runnings. All baselines’ hyper-parameters were set to the optimal values in their origin papers.

At first, we list the settings of some important hyper-parameters in our experiments in Table 3. The results of hyper-parameter tuning are not displayed due to space limitation. ACAM and other models incorporating attributes enhance their performance a little when more significant attributes are fed, but ACAM still keeps its superiority. In Table 2, we only display the results of $L=3$ and $L=10$ towards the two recommendation tasks. We did not focus on the scenario of $L=1$ because a user’s preference can not be inferred precisely by only one historical interacted item. Given that many users’ preferences may vary as time elapses, using too many historical items to represent a user would induce noises, so we neglected the cases of $L>10$. We also find that almost all models only improve their performance a bit when $L$ increases from 3 to 10. It implies that using 3 recent historical items to represent a user is adequate for many models to generate precise results. In addition, ACAM achieves the best performance when $\lambda_{1}$ is small, implying that too large weight of KGE objective will bias the synthetic objective of Eq. 5.

Table 2 displays all compared models’ top3/5/10 recommendation performance on the two datasets. We find that ACAM outperforms NCF from the table, which justifies the effectiveness of incorporating attribute embeddings. ACAM’s superiority over NAIS shows that the fine-grained user/item representations based on attribute embeddings can improve the effectiveness of attention-based models. It also justifies the rationale of augmenting item representations and user representations simultaneously by co-attention mechanism. Although AFM also captures the interactions (correlations) between different features (attributes) through attention mechanism, it does not perform well as ACAM. It shows that ACAM’s co-attention mechanism captures the correlations between different attributes better than AFM’s attention in terms of recommendation performance. Although FDSA and DIN also incorporate feature embeddings to enrich user/item representations, they do not perform well as ACAM, implying that co-refining user representations and item representations by co-attention mechanism is more effective than only refining user representations by attention mechanism (DIN) or self-attention mechanism (FDSA). Although RippleNet is a state-of-the-art KG-based recommendation model, it is also inferior than ACAM, showing that ACAM’s co-attention mechanism exploits item knowledge (attributes) more effectively.