MVIN: Learning Multiview Items for Recommendation

When MVIN collects information from the neighborhood of the given item in the KG, it scores each relation around the item in a user-specific way. The proposed user-oriented relation attention mechanism utilizes the information of the given user, item, and relations to determine which neighbor connected to the item is more informative. For instance, some users may think the film Iron Man is famous for its main actor Robert Downey Jr.; others may think the film Life of Pi is famous for its director Ang Lee. Thus each entity of neighborhood is weighted by dependent scores ${\pi}^{u}_{r_{v,e}}$, where $u$ denotes a different user; ${r_{v,e}}$ denotes the relation $r$ from entity $v$ to neighboring entity $e$ (the formulation of the scoring method is given below). We aggregate the weighted neighboring entity embeddings and generate the final user-oriented neighborhood information n as

To calculate the neighbor’s dependent score ${\pi}^{u}_{r_{v,e}}$, we first concatenate relation $\textbf{r}\in\mathbb{R}^{s}$, item representation $\textbf{v}\in\mathbb{R}^{s}$, and user embedding $\textbf{u}\in\mathbb{R}^{s}$, and then transform these to generate the final user-oriented score ${\pi}^{u}_{r_{v,e}}$ as

where $W_{r}\in\mathbb{R}^{3s}$ and $\textbf{b}_{r}\in\mathbb{R}$ are trainable parameters.

To further increase user-entity interaction, we propose a user-oriented entity projection module. For different users, KG entities should have different informativeness to characterize their properties. For instance, in a movie recommendation, the user’s impression of actor Will Smith varies from person to person. Someone may think of him as a comedian due to the film Aladdin, while others may think of him as an action actor due to the film Bad Boys. Therefore, the entity projection mechanism refines the entity embeddings by projecting each entity e onto user perspective u, where the projecting function can be either linear or non-linear:

where $W_{e}$ and $\textbf{b}_{e}$ are trainable parameters and $\sigma$ is the non-linear activation.

Thus the user-oriented entity projection module can be seen as an early layer which increases user-entity interactions. Then, the user-oriented relation attention module aggregates the neighboring information in a user-specific way.

To enhance the quality of user-oriented information received from previous sections, we enrich user representations constructed according to KG entities which incorporates user click information (Wang et al., 2018b). For example, if a user watched I, Robot, we find I, Robot is acted by Will Smith, who also acts in Men in Black and The Pursuit of Happyness. Capturing user preference information from the KG relies on consulting all relevant entities in KG and the connections between entities help us to find the potential user interests. The extraction of user preference also fits the proposed user-oriented modules; in the user’s mind, the icon of a famous actor is defined not only by the movies they have watched but also by the movies in the KG that the user is potentially interested in. In our example, if the user has potential interests in Will Smith, the modules would quickly focus on other films he acted. In sum, the relevant KG entities model the user representation and by KG-enhanced user representation, the user-oriented information is enhanced as well.³³3In Section 5.4.2, this design helps MVIN to focus on entities which the user may show interest in given the items that the user has interacted with. The overall process is shown in Figure 2 and Algorithm 1.

Preference Set We first initialize the preference set. For user $u$, the set of items that the user has interacted with, $\mathcal{V}_{u}=\{$$v|y_{uv}=1$$\}$, is treated as the starting point in $\mathcal{G}$, which is then explored along the relations to construct the preference set $\mathcal{S}_{u}$ as

where $\mathcal{E}^{0}_{u}$ = $\mathcal{V}_{u}$; $\mathcal{E}^{p}_{u}$ records the $p$ hop entities linked from entities at previous ${p-1}$ hop.

Preference Propagation The KG-enhanced user representation is constructed by user preference responses $\textbf{o}_{u}$ generated by propagating preference set $\mathcal{S}_{u}$.

First, we define at hop 0 the user preference responses $\textbf{o}^{0}_{i}$ which is calculated from the user-clicked items $h_{i}\in\mathcal{S}^{1}_{u}$; taking into account different items representations v assigns different degrees of impact to the user preference response:

where $W_{a}$ is a trainable parameter.

Second, at hop $p$, where $p>0$, user preference responses $\textbf{o}^{p}_{u}$ are computed as the sum of the tails weighted by the corresponding relevance probabilities $k_{i}$ as

where $\textbf{h}_{i}\in\mathbb{R}^{s}$, $\textbf{R}_{i}\in\mathbb{R}^{s\times s}$, $\textbf{t}_{i}\in\mathbb{R}^{s}$, and $\textbf{v}\in\mathbb{R}^{s}$ are the embeddings of heads $h_{i}$, relations $r_{i}$, tails $t_{i}$, and item $v$. The relation space embedding R helps to calculate the relevance of item representation v and entity representation h.

After integrating all user preference responses $\textbf{o}^{p}_{i}$, we generate the final preference embedding of user u $\in\mathbb{R}^{s}$ as

where $W_{o}$ and $\textbf{b}_{o}$ are trainable parameters.

In a real-world knowledge graph, the size of $N(e)$ varies significantly. In addition, $S^{p}_{u}$ may grow too quickly with the number of hops. To maintain computational efficiency, we adopt a fixed-size strategy (Wang et al., 2019d; Ying et al., 2018) and sample the set of entities for sections 4.1 and 4.1.3.

For Section 4.1, we uniformly sample a fixed-size set of neighbors $\mathcal{N}^{\prime}(v)$ for each entity $v$, where $\mathcal{N}^{\prime}(v)\triangleq\{e|e\thicksim\mathcal{N}(v)\}$ and $\mathcal{N}(v)$ denotes those entities directly connected to $v$, where $|\mathcal{N}^{\prime}(v)|=K_{n}$ and $K_{n}$ is the sampling size of the item neighborhoods and can be modified.⁵⁵5We discuss the performance changes when $K_{n}$ and $K_{m}$ vary. Also, we do not compute the next-order entity representations for all entities $e\in\mathcal{G}$, as shown in line 6 of Algorithm 2, and we sample only a minimal number of entities to calculate the final entity embedding $\textbf{v}^{\prime}$. Per Section 4.1.3, at hop $p$ we sample user preferences set $\mathcal{S}^{p}_{u}$ to maintain a fixed number of relevant entities, where $|\mathcal{S}^{p}_{u}|=K_{m}$ and $K_{m}$ is the fixed neighbor sample size, which can be modified.⁵

To solve the potential issue that the fixed-size sampling strategy may put limitation on the use of all entities, recently stage-wise training has been proposed to collect more entity-relation from KG to approach the panoramic view of the whole neighborhood (Tai et al., 2019). Specifically, in each stage, stage-wise training would resample another set of entities to allow MVIN to collect more entity information from KG. The whole algorithm of stage-wise training is shown in the Algorithm 3 (Line3).

Per batch, the time cost for MVIN mainly comes from generating KG-enhanced user representation and the mixing layer. The user representation generation has a computational complexity of $O(l_{\mathrm{p}}K_{m}s^{2})$ to calculate the relevance probability $k_{i}$ for total of $l_{\mathrm{p}}$ layers. The mixing layer has a computational complexity of $O({K_{n}}^{l_{\mathrm{w}}l_{\mathrm{d}}}s^{2})$ to aggregate through the deep layer $l_{\mathrm{d}}$ and wide layer $l_{\mathrm{w}}$. The overall training complexity of MVIN is thus $O(l_{\mathrm{p}}K_{m}s^{2}+{K_{n}}^{l_{\mathrm{w}}l_{\mathrm{d}}}s^{2})$.

Compared with other GNN-based recommendation models such as RippleNet, KGCN, and KGAT, MVIN achieves a comparable computation complexity level. Below, we set their layers to $l$ and the sampling number to $K$ for simplicity. The computational complexities of RippleNet and KGCN are $O(lKs^{2})$ and $O(K^{l}s^{2})$ respectively. This is at the same level as ours because ${l_{w}l_{d}}$ is a special case of $l$. However, for KGAT, without the sampling strategy, its attention embedding propagation part should globally update the all entities in graph, and its computational complexity is $O(l|\mathcal{G}|s^{2})$.

We conducted experiments to compare the training speed of the proposed MVIN and others on an RTX-2080 GPU. Empirically, MVIN, RippleNet, KGCN, and KGAT take around 6.5s, 5.8s, 3.7s, and 550s respectively to iterate all training user-item pairs in the Amazon-Book dataset. We see that MVIN has a time consumption comparable with RippleNet and KGCN, but KGAT is inefficient because of the whole-graph updates.

For MVIN, $l_{\mathrm{p}}$ = 2, $l_{\mathrm{w}}$ = 1, $l_{\mathrm{d}}$ = 2, $K_{m}$ = 64, $K_{n}$ = 8, $\lambda$ = $1\times{10}^{-7}$ for ML-1M; $l_{\mathrm{p}}$ = 1, $l_{\mathrm{w}}$ = 1, $l_{\mathrm{d}}$ = 2, $K_{m}$ = 64, $K_{n}$ = 4, $\lambda$ = $5\times{10}^{-8}$ for LFM-1b; $l_{\mathrm{p}}$ = 2, $l_{\mathrm{w}}$ = 2, $l_{\mathrm{d}}$ = 2, $K_{m}$ = 16, $K_{n}$ = 8, $\lambda$ = $1\times{10}^{-7}$ for AZ-book; We set function $\sigma$ as ReLU. The embedding size was fixed to 16 for all models except 32 for KGAT because it stacks propagation layers for final output. For stage-wise training, average early stopping stage number is 7, 7, 5 for ML-1M, LFM-1b and AZ-book, respectively. For all models, the hyperparameters were determined by optimizing $\mathit{AUC}$ on a validation set. For all models, the learning rate $\eta$ and regularization weight were selected from [$2\times{10}^{-2}$, $1\times{10}^{-2}$, $5\times{10}^{-3}$, $5\times{10}^{-4}$, $2\times{10}^{-4}$] and from [$1\times{10}^{-4}$, $1\times{10}^{-5}$, $2\times{10}^{-5}$, $2\times{10}^{-7}$, $1\times{10}^{-7}$, $5\times{10}^{-8}$], respectively. For MCRec, to define several types of meta paths, we manually selected user-item-attribute item meta paths for each dataset and set the hidden layers as in (Hu et al., 2018b). For KGAT, we set the depth to 2 and layer size to [16,16]. For RippleNet, we set the number of hops to 2 and the sampling size to 64 for each dataset. For KGCN, we set the number of hops to 2, 2, 1 and sampling size to 4, 8, 8 for ML-1M, AZ-book, and LFM-1b, respectively. Other hyperparameters were optimized according to validation result.

Table 2 and Figure 3 are the results of MVIN and the baselines, respectively (FM, NFM, CKF, GC-MC, MCRec, RippleNet, KGCN, KGAT), in click-through rate (CTR) prediction, i.e., taking a user-item pair as input and predicting the probability of the user engaging with the item. We adopt $\mathit{AUC}$ and $\mathit{ACC}$, which are widely used in binary classification problems, to evaluate the performance of CTR prediction. For those of top-$N$ recommendation, selecting $N$ items with highest predicted click probability for each user and choose $\mathit{Precision@N}$ to evaluate the recommended sets. We have the following observations:

• •

  MVIN yields the best performance of all the datasets and achieves $\mathit{AUC}$ performance gains of 1.2% , 2.5%, and 2.2% on ML-1M, LFM-1b, and AZ-book, respectively. Also, MVIN achieves outstanding performance in top-$N$ recommendation, as shown in Figure 3.

• •

  The two path-based baselines RippleNet and KGAT outperform the two CF-based methods FM and NFM, indicating that KG is helpful for recommendation. Furthermore, although RippleNet and KGAT achieve excellent performance, they still do not outperform MVIN. This is because RippleNet neither incorporates user click history items $h^{1}_{i}$ into user representation nor does it introduces high-order connectivities, and KGAT does not mix GCN layer information and not v consider user preferences when collecting KG information.

• •

  For the other baselines KGCN and MCRec, their relatively bad performance is attributed to their not fully utilizing information from user click items. In contrast, MVIN would first enrich a user representation by user click items and all relevant entities in KG and then weighted the nearby entities and emphasize the most important ones. Also, KGCN only uses GCN in each layer, which does not allow contrast on neighborhood layers. Furthermore, MCRec requires finer defined meta paths, which requires manual labor and domain knowledge.

• •

  To our surprise, the CF-based NFM achieves good performance on LFM-1b and AZ-book, even outperforming the KG-aware baseline KGCN, and achieves results comparable to RippleNet. Upon investigation, we found that this is because we enriched its item representation by feeding the embeddings of its connected entities. In addition, NFM’s design involves modeling higher-order and non-linear feature interactions and thus better captures interactions between user and item embeddings. These observation conform to (Wang et al., 2019a).

• •

  The regularization-based CKE is outperformed by NFM. CKE does not make full use of the KG because it is only regularized by correct triplets from KG. Also, CKE neglects high-order connectivities.

• •

  Although GC-MC has introduced high-order connectivity into user and item representations, it achieves comparably weak performance as it only utilizes a user-item bipartite graph and ignores semantic information between KG entities.

The ablation study results are shown in Table 3. After removing the proposed user-oriented relation attention UO(r) and user-oriented entity projection UO(e) modules, MVIN${}_{\mathrm{w/o\,UO(r)}}$ and MVIN${}_{\mathrm{w/o\,UO(e)}}$ perform worse than MVIN in all datasets. Thus considering user preferences when aggregating entities and relations in KG improves recommendation results.

To enhance user-oriented information, we enrich the user representation using KG information as a pre-processing step. Here, we denote MVIN without KG-enhanced user-oriented information as MVIN${}_{\mathrm{w/o\,UO(k)}}$. We compare the performance of MVIN and MVIN${}_{\mathrm{w/o\,UO(k)}}$. Table 3 shows that the former outperforms the latter by a large margin, which confirms that KG-enhanced user representation improves user-oriented information.

Moreover, we conducted a case study to understand the effect of KG-enhanced user-oriented information incorporated with user-entity interaction. Given the attention weights learned by MVIN ${}_{\mathrm{w/o\,UO(k)}}$ in Figure 4(a), user $\mathrm{u}_{324}$ puts only slightly more value on the author of Life After Life. However, Figure 4(b) shows that MVIN puts much more attention on the author when information on user-interacted items is provided. Furthermore, in Figure 4(b), we find user $\mathrm{u}_{324}$’s interacted items—$v_{734}$ (One Good Turn), $v_{3941}$ (Behind the Scenes at the Museum), and $v_{610}$ (Case Histories)—are all written by Kate Atkinson. This demonstrates that MVIN outperforms MVIN${}_{\mathrm{w/o\,UO(k)}}$ because it captures the most important view that user $\mathrm{u}_{324}$ sees: item Life After Life, a book by Kate Atkinson.

In the mixing layer, the wide part ML(w) allows MVIN to represent general layer-wise neighborhood mixing. To study the effect of ML(w), we remove the wide part from MVIN, denoted as MVIN${}_{\mathrm{w/o\,ML(w)}}$. Table 3 shows a drop in performance, suggesting that the mixing of features from different distances improves recommendation performance.

It has shown that in homogeneous graphs the benefit of the mixing layer depends on the homophily level.⁶⁶6The homophily level indicates the likelihood of a node forming a connection to a neighbor with the same label. In MVIN, the mixing layer works in KGs, i.e., heterogeneous graphs; we also investigate its effect (ML(w)) at different sampling sizes of neighborhood nodes $K_{n}$. With a large $K_{n}$, entities in KG connect to more different entities, which is similar to a low homophily level. Figure 5 shows that ML(w) is effective in heterogeneous graphs. In addition, $K_{n}$ increases the performance gap between MVIN and MVIN${}_{\mathrm{w/o\,ML(w)}}$. We conclude that the mixing layer not only improves MVIN performance but is indispensable for large $K_{n}$.

In addition to the wide part in the mixing layer, the proposed deep part allows MVIN to aggregate high-order connectivity information. Figure 3 shows that after removing the mixing layer (ML), MVIN${}_{\mathrm{w/o\,ML}}$ performs poorly compared to MVIN, demonstrating the significance of high-order connectivity. This observation is consistent with (Wang et al., 2019a; Wang et al., 2019c; Ying et al., 2018).

Removing the stage-wise training (SW) shown by MVIN${}_{\mathrm{w/o\,SW}}$ deteriorates performance, showing that stage-wise training helps MVIN achieve better performance by collecting more entity relations from the KG to approximate a panoramic view of the whole neighborhood. Note that compared to KGAT, the state-of-the-art baseline model which samples the whole neighbor entities in KG, MVIN${}_{\mathrm{w/o\,SW}}$ refers to a limited number of entities in KG but still significantly outperforms all baselines (at $p$-value = 0.01), which confirms again the effectiveness of the proposed MVIN.