A²-GCN: An Attribute-aware Attentive GCN Model for Recommendation

Attributes are widely available and valuable for item description. Thus, many methods have been developed to leverage attributes in recommendation [3, 4, 16, 17]. Factorization Machines (FMs) based models can model the second-order interactions among attributes via the inner product of their factorized vectors [3, 18, 5, 19]. Moreover, they can work with any real-valued feature vectors by modelling all interactions between each pair of features via factorized interaction parameters for prediction. As the inner product violates the triangle inequality and cannot capture finer-grained attribute interactions, TransFM [18] employs the squared Euclidean distance function to replace the inner product for sequential recommendation. The performance of Factorization Machine (FM) [3] is limited by its linearity and the modelling of pairwise feature interactions. To deal with the linearity of FM, NFM [5] is proposed to handle the complex and underlying nonlinear structure in real-world dataset. In this method, the attribute information, such as user and item features, is used as side information and embedded into different vectors. AFM [19] extends NFM by adding an attention mechanism to discriminate the importance of different feature interactions.

Another line of research adopts the deep neural network (DNN) techniques to learn the attribute embeddings and concatenate them for recommendation [4, 20, 6, 21]. Wide&Deep [4] combines the deep neural network and the linear model for recommendation, where the deep part is a multi-layer perceptron (MLP) on the concatenation of feature embedding vectors to learn feature interactions. DeepCross [20] shares a similar framework with Wide&Deep by replacing the MLP with the state-of-the-art residual network. ACCM[6] adaptively adjusts the source information of each user-item pair with an attention mechanism.

All the above methods suffer from the two limitations discussed in Section 1. In this paper, we developed a GCN-based attribute-aware recommendation model, which can effectively tackle those limitations.

In recent years, Graph Convolutional Networks (GCNs) have achieved great success due to the powerful capability on representation learning from non-Euclidean structure [22, 23, 24, 25, 26, 27, 28]. The main idea of GCNs is how to iteratively aggregate feature information from local graph neighborhoods via neural networks.

Recently, the GCNs have attracted increasing attention in recommendation. For example, PinSage [23] combines random walks with multiple graph convolutional layers on the item-item graph for Pinterest image recommendation. Moreover, the CF effect is captured on the level of item relations, rather than the collective user behaviors. SpectralCF [24] reveals the proximity information of a graph and the high-order connectivity information via convolution operation in the spectral domain. A drawback of this method is that the eigen-decomposition of graph adjacency matrix causes a high computational complexity. NGCF[22] exploits high-order proximity by propagating embeddings on the user-item interaction graph. It can simultaneously update the representations for all users and items in an efficient way by implementing the matrix-form rule. GC-MC [26] applies the GCN techniques on user-item graph with edges labeled with the observed ratings. It employs one convolutional layer to exploit the direct connections between users and items.

There are also GCN-based models specially designed for specific recommendation scenarios, such as social recommendation [25], micro-video recommendation [27], and knowledge-aware recommendation [28]. In particular, GraphRec [25] introduces a method to consider heterogeneous strengths of social relations for social recommendation. GCN-PHR [27] leverages GCN techniques to model the complicated interactions among $<$users, hashtags, micro-videos$>$ and learn their representations. KGAT [28] models the high-order connectivities in Knowledge Graph based on an end-to-end learning framework.

In this paper, we presented an attribute-aware attentive GCN recommendation model, which leverages attribute information of items to learn better user and item representation for improving recommendation performance. Besides, our model can be used for different scenarios with attribute information, such as E-commerce product, social images, and videos.

Before describing our model, we would like to introduce the problem setting first. Given a dataset with an interaction matrix $\bm{R}^{N_{u}\times N_{v}}$ of a set of users $\mathcal{U}$ and a set of items $\mathcal{V}$, where $N_{u}$ and $N_{v}$ are the numbers of users and items, respectively. In the dataset, each item is associated with a set of attribute labels $a\in\mathcal{A}$ which describe different attributes of the item. In addition, the number of attribute labels is defined as $N_{a}$. The matrix $\bm{R}$ records the interaction history between users and items. A nonzero entry $r_{uv}\in\bm{R}$ indicates that user $u\in\mathcal{U}$ has interacted with item ${v}\in\mathcal{V}$ before; otherwise, the entry is zero. Notice that the interactions can be implicit (e.g., click) or explicit (e.g., rating). Based on this dataset, the goal is to recommend a user $u\in\mathcal{U}$ with suitable items, which the user did not consume before and will be appealing to.

Let $\mathcal{G}=(\mathcal{W},\mathcal{E})$ be an undirected graph, where $\mathcal{W}$ denotes the set of notes and $\mathcal{E}$ is the set of edges. Specifically, $\mathcal{W}$ consists of three types of nodes: users $u_{i}\in\mathcal{U}$ with $i\in\{1,\cdots,N_{u}\}$, items $v_{j}\in\mathcal{V}$ with $j\in\{1,\cdots,N_{v}\}$, attributes $a_{k}\in\mathcal{A}$ with $k\in\{1,\cdots,N_{a}\}$, and $\mathcal{U}\cup\mathcal{V}\cup\mathcal{A}=\mathcal{W}$. For the ease of presentation, $i$, $j$ and $k$ will be assigned to index user, item and attribute, respectively. In the graph, user nodes and item nodes are linked based on their interaction history; item nodes and attribute nodes are linked based on the association of attributes to the corresponding items. Note that there is no edge between user nodes and attribute nodes. Fig. 3 illustrates a toy example of an interaction graph which is based on $<$users, items, attributes$>$ and connectivity information for user $u_{1}$ and item $i_{4}$. We take user $u_{1}$ and item $i_{4}$ as examples for illustration. User $u_{1}$ has neighbor nodes $i_{1}$ and $i_{2}$; $i_{1}$ has attributes $a_{1}$, $a_{3}$ and $i_{2}$ links to its attribute node $a_{1}$. For item $i_{4}$, it has interactions with user $u_{2}$, $u_{3}$, as well as attributes $a_{1}$ and $a_{3}$.

Typically, model-based CF methods learn the representations of users and items by collectively exploiting the user-item interactions. Here, the attribute provides valuable information about item features, which can be leveraged to capture user preferences. The attribute information of an item can affect a user’s preference for the item, because different users may prefer different aspects of an item [29] and attributes deliver item information from different aspects. For example, some users like a movie because of its director while others favor it because of the leading actor. Therefore, different users focus on different features of the movie because of its attribute information, or in other words, the attributes of director and actor pass different information of the movie to different users. Therefore, to accurately model user and item representations, a desired model should capture the complex interactions among users, items, and attributes.

In this paper, we refer to the graph convolutional network (GCN) techniques, because of its powerful representation learning capability, to learn user and item representation in the graph. The core idea behind the node representation learning in GCN is how to recursively aggregate feature information from local graph neighborhoods via neural networks. In our cases, an item node collects information from the connected user nodes and attribute nodes to learn its representation. In particular, given that different users and attributes provide information from different aspects and contribute to an item with different importance, an attention mechanism is designed to distill useful information passed from user nodes and attribute nodes to the item. Similarly, the representation of a user node is learned by distilling information from the connected item nodes which represent the ones this user has consumed before. As discussed above, the attributes of an item can have important effects on a user’s preference for this item. To model the effects, we propose a novel attribute-aware attention mechanism which leverages the information of the attribute nodes connected to the items to help learn the attention vector of this item with respect to the target user. To this end, our model can well exploit the interactions among users, items, and attributes for user and item representation learning.

We adopt the messaging-passing strategy [26] to learn user and item representation. Let $\bm{e_{u}}\in\mathds{R}^{d}$, $\bm{e_{v}}\in\mathds{R}^{d}$, and $\bm{e_{a}}\in\mathds{R}^{d}$ be the embeddings of user $u$, item $v$, and attribute $a$, respectively. $d$ is the embedding size. In the next, we detail the algorithm for user and item representation learning⁵⁵5 Given that the primary goal of this work is to study the effectiveness of our model on exploiting attribute information for user and representation learning, our model does not learn the attribute embeddings in this work. However, the embedding of attribute nodes can also be updated in a similar way as user or item nodes do..

As we can see, the attribute information has been integrated in our A²-GCN model for user and item embedding learning based on their connections to items (and second-order user neighbors). It is worth mentioning that our model can naturally avoid the limitations in existing attribute-aware methods, which are either using a default value to represent the missing attributes (e.g., “Other”) or simply assuming that the item does not possess the properties of the missing attributes (e.g., “Comedy” for the movie “Waiting to Exhale”) as discussed in Section 1. In our model, we do not make any assumption on the missing attributes and only take the available attributes into considerations, because only the linked attribute nodes are directly leveraged for the embedding learning. In this way, it can avoid introducing misleading information to the model by inappropriate assumptions, which may hurt the performance. Besides, our model enables the items with missing attributes to benefit from the attribute information propagated from other items via the graph connections. Our experiments validate the superiority of our model over other attribute-aware recommendation models on dealing with the attribute missing problem (see Section 4.3).

Prediction. After learning the user and item representations, given a user $u$ and a target item $v$, the user preference for this item is predicted by:

Objective function. we target at the top-$n$ recommendation, which aiming to recommend a set of $n$ top-ranked items which match the target user’s preferences. Similar to other rank-oriented recommendation work [31, 22], we adopt the pairwise-based learning method for optimization. Given two user-item pairs: a positive pair $(u,v^{+})$ and a negative pair $(u,v^{-})$, the positive pair indicates that user $u$ has consumed item $v^{+}$ before; and the negative pair means that no interaction exists between $u$ and $v^{-}$. The assumption of pairwise learning is that the user $u$ should prefer $v^{+}$ to $v^{-}$, namely, $\hat{r}_{uv^{+}}>\hat{r}_{uv^{-}}$.

The objective function is formulated as:

where $\mathcal{O}=\{(u,v^{+},v^{-})|(u,v^{+})\in\mathcal{R^{+}},(u,v^{-})\in\mathcal{R^{-}}\}$ denotes the training set; $\mathcal{R^{+}}$ indicates the observed interactions between user $u$ and $v^{+}$ in the training dataset, and $\mathcal{R^{-}}$ is the sampled unobserved interaction set. $\lambda$ and $\Theta$ represent the regularization weight and the parameters of the model, respectively. $\phi$ is the sigmoid function. And the $L_{2}$ regularization is used to prevent overfitting.

Model training. To prevent the overfitting, we adopt message dropout and node dropout in A²-GCN. Message dropout randomly blocks the messages passed from one node to another node in the training. It makes the model more independent of edges. Similarly, node dropout randomly removes some nodes, and thus all outgoing messages from those nodes are blocked. This technique enables the embeddings more robust against the presence or absence of particular nodes. They have been successfully applied in previous models [26, 22]. The drop ratios of massage dropout and node dropout are empirically tuned in practice.

The mini-batch Adam [32] is adopted to optimize the prediction model and update the model parameters. Specifically, for a batch of randomly sampled triplets $(u,v^{+},v^{-})\in\mathcal{O}$, we first learn the representations of those users and items based on the propagation rules, and then update the model parameters by using the gradients of the loss function.

Matrix-form propagation rule. In order to update the representation for all users and items in an efficient way, we implement the matrix-form of layer-wise propagation rule [33], which can be described as follows:

where $\mathbf{E}\in\mathcal{R}^{\left(N_{u}+N_{v}+N_{a}\right)\times d}$ is the representations for users, items and attributes; and $\mathbf{I}$ denotes an identity matrix. $\mathbf{\mathcal{L}}$ represents the Laplacian matrix for the $<$user, item, attribute$>$ based graph, which is formulated as:

where $\mathbf{Y_{uv}}\in\mathcal{R}^{N_{u}\times N_{v}}$ is the attention weight matrix which represents the weights from item nodes to each user node, where $\gamma_{v}^{u}\in\mathbf{Y_{uv}}$. Analogously, $\mathbf{Y_{vu}}\in\mathcal{R}^{N_{v}\times N_{u}}$ and $\mathbf{Y_{va}}\in\mathcal{R}^{N_{v}\times N_{a}}$ are the attention weight matrices that represent the weights from user nodes to each item node and the weights from attribute nodes to each item node, respectively.

Table III reports the results of all the considered methods. In the table, both the best and second best performance is highlighted in bold. In addition, we conducted pairwise significant test between our method and the baseline with the best performance. From the results, we have some interesting observations.

Overall, A²-GCN surpasses all the competitors consistently and significantly across all the cases, demonstrating its effectiveness on leveraging attributes for recommendation. Another interesting observation is that the performance on the Office Products and Kindle Store is much better than those on the other two datasets. The main reason might be that the item features on the other two datasets are more diverse than that on the Office Products and Kindle Store. Therefore, it is more difficult to model user preferences on those two datasets, resulting in worse performance. In the next, we analyze the results in detail.

For the methods based on matrix factorization, NeuMF obtains consistently better performance over BPR-MF on all the datasets. The main reason is that the BPR-MF uses the inner products as the interaction function, which cannot capture the complex relations between users and items. In contrast, NeuMF adopts multiple layers of neural networks above the element-wise interaction and concatenation of user and item embeddings to capture the non-linear interactions. This demonstrates the importance of modeling the complicated interactions between users and items. NGCF outperforms BPR-MF and NeuMF across all the datasets. This is attributed to the utilization of GCN techniques to learn user and item embeddings on the user-item interaction graph. In addition, NGCF exploits high-order connectivities between users and items through embedding propagation over the graph structure to improve the representation learning. For relatively denser datasets (e.g., Office Products), more neighbor nodes (e.g., interacted items) are available for each node’s embedding learning, and thus better performance can be achieved.

Next, we introduce the performance of the baselines using attribute information (i.e., NFM, Wide&Deep, ACCM). In general, these methods yield much better performance than those without using attribute information, demonstrating the great value of attributes for recommendation. NFM outperforms NeuMF on all the datasets but underperforms NGCF on the Office Products, which might because that Office Products is relatively denser. With the rich interactions, NGCF can learn good user and item representations by leveraging high-order proximity information, thus achieving better performance. Wide&Deep yields better performance over the above methods on all the datasets. ACCM obtains consistently much better performance on Office Products and Clothing, owing to the adoption of the attention mechanism to control the ratio of information from different sources. This also indicates the importance of differentiating the influence of different information on user/item representation learning. However, it underperforms NFM and Wide$\&$Deep on Toys Games and Kindle Store. This is because the way of ACCM leveraging attributes makes it only deal with the limited number of attributes. Here, we followed the setting of the ACCM in [6] to consider only the top 100 most frequent attribute labels, which causes great information loss. In contrast, NFM and Wide&Deep can tackle more attribute labels, and thus yield better performance on the two datasets.

A²-GCN outperforms all the baselines consistently over all the datasets. In particular, comparing to the strongest baseline in terms of NDCG@20, A²-GCN can achieve a relative improvement by 6.77%, 12.80%, 11.82%, 9.88% on Office Products, Toys Games, Clothing and Kindle Store, respectively. The representation learning process of A²-GCN is the same as NGCF with one embedding propagation layer. The great improvement over NGCF demonstrates the effectiveness of leveraging attribute information in representation learning. Besides, our proposed method yields substantial improvement over the baselines (NFM, Wide&Deep and ACCM), which also exploit attribute information. This should be credited with the following reasons. Firstly, the GCN technique is used in our method to exploit the attribute information. Due to its powerful capability on representation learning, better performance is expected. Besides, our model can naturally avoid the limitations of those baselines on dealing with the attribute missing problem. This also contributes to the performance improvement (see Section 4.3). Last but not least, our model also benefits from the attention mechanism, which differentiates influence of attributes on user preference in the learning process (See Section 4.5).

Based on the above discussions, we can have following findings. The good performance of NeuMF reveals the importance of modeling non-linear features between users and items. The performance improvement achieved by NGCF on all datasets demonstrates the advantages of GCN-based models and the value of high-order information. Besides, NFM, Wide&Deep and ACCM indicate the effectiveness of leveraging attribute information to deal with the sparsity problem. Our A²-GCN achieves the best performance because of its effectiveness on exploiting the attribute information by via an attentive GCN method and advantages on tackling the attribute missing problem.

In this section, we study the influence of the attribute missing problem on attribute-aware recommendation methods. To simulate the attribute missing problem, we randomly removed attribute labels from the datasets by a ratio in {0, 0.2, 0.4, 0.6, 0.8}. A ratio of 0.2 indicates that 20% of the attribute labels of items are randomly removed from the datasets. Due to the space limitation, we only present the performance on the dataset Kindle Store and Office Products. The results on the other two datasets show similar trends.

We compare our model with the three baselines using attribute information. Fig. 4 shows the results in terms of HR@20 and NDCG@20 on two datasets. As we can see, A²-GCN outperforms all the baselines by a large margin, especially when more attribute labels are missing, which indicates the effectiveness of our proposed method on modeling user preferences with attribute information. The performance of three baselines declines rapidly when more attribute labels are missing, especially for the Office Products. In particular, the performance of the three baselines deteriorates substantially when $k>0.4$ on Office Products. According to statistics (see Table II), the average numbers of attribute labels per item in Kindle Store and Office Products are 6.83 and 3.39, respectively. This may explain the dramatical performance drop on Office Products. The reasons for performance degradation are two-fold. On the one hand, the removal of attribute labels reduces information source for item and user representation learning. This equally affects all models. On the other hand, the strategies of those baselines for dealing with missing attributes introduce misleading information into the recommendation model, resulting in inferior performance.

In contrast, the performance of our model is also negatively affected by the reduction of attribute information, but it is much more stable and the declination is much smaller, especially on the Kindle Store because of the relatively rich attribute information. This demonstrates that our model can exploit the attribute information in a much more efficient way via the information propagation through the graph structure. Besides, it also demonstrates the superiority of our model on dealing with the missing attribute problem over the baselines.

As important side information, attributes can help alleviate the data sparsity problem. To demonstrate the capability of A²-GCN for users with limited interactions, we conducted experiments to study the performance of our method and other competitors over user groups with different sparsity levels. In particular, for each dataset, we clustered the users into four groups based on their interaction numbers in the training data. Taking dataset Clothing as example, users are divided into four groups based on the number of interactions in the training dataset: less than 5, 10, 15, and more than 15. Fig. 5 shows the performance in terms of HR@20 on different user groups on dataset Clothing and Kindle Store. These figures also show the number of users in different groups. We can see that most users have less than 10 interactions in Clothing and Kindle Store, and many users even have less than 5 interactions in Clothing, which further validates the common sparsity problem in real datasets. From the results, we can have the following observations.

ACCM, NFM and Wide&Deep consistently obtain better performance than all other baselines. This demonstrates that attribute information is indeed very useful in alleviating sparsity problem. In addition, NGCF surpasses NeuMF and BPR-MF, validating the effectiveness of exploiting collaborative signals from high-order connectivities. A²-GCN yields significantly better performance than all the other baselines over all user groups, which verifies the better capability of our model on exploiting attributes.

By further analyzing the performance improvement on different groups over the two datasets, we can see that the improvement of our model over that of the best baseline becomes larger with the increasing of interactions. Notice that our model benefits from both the attribute information and the GCN technique, which leverages the graph structure to learn user and item representations. With more connections (interactions) in the graph, our model can leverage the graph structure to better exploit the attribute information. It is also interesting to find that our model obtains better performance in the third group over the best baseline than that of other groups in Kindle Store. Specifically, relative improvements of A²-GCN over the second best method Wide&Deep on the four groups are 2.35%, 6.71%, 9.76% and 5.56%, respectively. The reason might be that, when the interactions are relatively rich, the benefits from attribute and neighbors become limited since interactions provide quit sufficient information for representation learning.

In this section, we examine the contribution of different components to the performance of our model. Our analyses are based on the performance comparisons to following variants.

• •

  GCN_b: In this model, attribute information and attention mechanism are removed from our method. It is a baseline model which merely applies the GCN technique to the user-item bipartite graph.

• •

  A-GCN_am: This variant is designed to investigate the effectiveness of the attention mechanism without attribute information.

• •

  A-GCN_att: This model removes the attention mechanism from our model. This variant is designed to investigate the effects of attributes without attention mechanism.

• •

  A²-GCN_v: This model replaces the attribute-aware attention mechanism described in Eq. 12 by the common mechanism described in Eq. 10, where the attribute information has not been integrated into the attention mechanism. This variant is designed to study the effectiveness of our proposed attribute-aware attention mechanism.

The results of those variants and our method are reported in Table IV. A-GCN_am outperforms GCN_b over all the datasets, which indicates the importance of differentiating the information distilled from different neighbor nodes in GCN. From the perspective of recommendation, this validates the varying preferences of a user towards different items. The performance of A-GCN_att indicates the effectiveness of leveraging attribute information for recommendation. In other words, the attributes can provide valuable information in learning better user and item representations. Although A²-GCN_v adopts a simplified attention mechanism, it consistently and substantially outperforms the A-GCN_att. This also demonstrates the effectiveness of the attention mechanism. More importantly, our A²-GCN further improves the performance based on the proposed attribute-aware attention mechanism. This validates the assumption that items’ attribute information affects a user’s preference for this item. It also indicates our designed attention model can effectively capture this effect.