Attention on Global-Local Representation Spaces in Recommender Systems

The core idea of the CF approach is to model the preference of users toward items according to their historical interactions. Several techniques are proposed to address this issue. In addition to the above-mentioned matrix factorization and neighborhood model, here we introduce other popular techniques, namely, ranking formulation and deep learning.

Rendle et al. [14] presented a generic optimization criterion by Bayesian personalized ranking, called BPR-Opt, for personalized ranking in the item recommendation task. BPR-Opt is the maximum posterior estimator derived from a Bayesian analysis of the problem. The main idea is to rank observed items higher than unobserved items. They also provided a generic learning algorithm for optimizing models with respect to BPR-Opt. Following the similar ranking formulation, Hsien et al. [6] proposed collaborative metric learning (CML), which encoded user preferences and user-user/item-item similarity in a joint metric space. CML achieves significant speedup for top-N recommendation tasks using off-the-shelf approximate nearest-neighbor search with negligible accuracy reduction.

More recent approaches tend to use the deep learning technique. He et al. [5] presented a general framework called neural network-based collaborative filtering (NeuMF) for recommender systems. They used nonlinear neural networks as the user–item interaction function. NeuMF expresses and generalizes matrix factorization and leverages a multi-layer perceptron to learn the user–item interaction function. The authors transformed the identity of a user/item to a binarized sparse vector with one-hot encoding, which only explored the user/item feature in a limited manner. Due to the complexity of the scoring function, the NeuMF retrieval process is generally hard to accelerate. Rendle et al. [15] revisited the NeuMF experiments, and concluded that a dot product might be a better default choice for combining embeddings than learned similarities using MLP or NeuMF. Xue et al. [16] proposed a neural network framework to model higher-order item relations for item-based collaborative filtering by using multiple nonlinear layers above pairwise interaction modeling. Under this framework, the authors also used an attention model to differentiate the importance of pairwise interactions. Deng et al. [17] proposed a general framework called deep collaborative filtering (DeepCF) to combine the representation learning and matching function learning. They also proposed a model called a collaborative filtering network (CFNet) based on a plain-vanilla MLP model under the DeepCF framework. CFNet has great flexibility in learning complex matching functions, with good efficiency in learning low-rank relations between users and items.

To handle the high-dimensional and sparse matrix efficiently, Luo et al. presented a series studies for latent factor-based CF methods [18] [19] [20] [21]. For example, they investigated the stochastic gradient descent algorithm and develop several extensions to improve the accuracy [18]. In addition, some non-negative factorization algorithms are proposed to overcome the slow convergence problem under non-negativity constraints [19][20]. Wu et al. designed a deep structure to construct the hierarchical latent factor models that can achieve high computation and storage efficiency [21].

The clustering-based approach is well known for its scalability to large and sparse systems. The basic steps include calculating the similarity based on rating data, and then applying a clustering algorithm to group similar users and items into clusters. Some advanced methods are introduced in the following.

Ji et al. [22] proposed a reconstructive method for improving matrix approximation by introducing clustering and transfer learning techniques. The proposed method compresses low-rank approximation into a cluster-level rating-pattern referred to as a codebook and then constructs an improved approximation by expanding the codebook. Wu et al. [23] proposed a scalable co-clustering method called CCCF. The idea is users may have different preferences over different subsets of items, where these subsets of items may overlap. CCCF can explore subgroups of users who share interests over similar subsets of items through user-item co-clustering. Chen et al. [12] proposed a recommender system called N2VSCDNNR based on node2vec [24] and clustering technology. N2VSCDNNR recommends item clusters with high weights to the target user cluster. The weights are obtained from the interaction frequency between items and users. Then the items in the high-weight clusters are recommended to the target user. Kang and McAuley [13] proposed a clustering-based framework called CIGAR, which learns a preference-preserving binary embedding model and a candidate item re-ranking model. Although using binary codes can significantly reduce the inference cost, due to their constrained capability, the performance is limited compared to models that use real-valued representations.

The attention mechanisms are components of prediction systems that enable a system to focus sequentially on different subsets of the input [25]. They have been widely adopted in recommender systems recently.

Chen et al. [26] presented an attentive collaborative filtering model to address implicit feedback in multimedia recommendation. They introduced item-level and component-level attention models to assign attentive weights for inferring the underlying user preferences encoded in implicit user feedback. Zhu et al. [27] presented a tree-based deep recommendation model that predicts user preferences using a max-heap-like tree probability formulation with an attention-based deep network. The underlying deep model and tree structure are learned in turn. Tay et al. [28] proposed a metric learning approach called latent relational metric learning for recommendations. They adopt the idea of translating users to items by translation vectors while relying on a memory attention network to learn the relation vectors. He et al. [29] proposed a user-generated list recommendation model that leverages the hierarchical structure of items, lists, and users to capture the containment relationship between lists and items. They applied a self-attention network to refine the item and list representations by considering the consistency of neighboring items and lists. Hu et al. [30] introduced a model named BCFNet, which is an extended version of DeepCF [17]. The BCFNet consists of three sub-models, including representation learning, matching function learning, and balance module. In particular, an attention mechanism is integrated to improve the ability of the representation learning and matching function learning.

For group recommendation, the attention mechanisms can be employed to aggregate the properties of a set of user/item group. Cao et al. [31] proposed using NeuMF with attention models to characterize user and user group interest at the same time. They further integrated the modeling of user-item interactions into their method, making it possible to reinforce the two tasks of recommending items for users and user groups. Chen et al. [1] considered the task of recommending a set of items (bundle) to a user. They designed a factorized attention network to fuse the set of item representations into a bundle representation.

The attention mechanisms are also applied in sequential recommendation. For example, Zheng et al. [2] proposed a memory-augmented hierarchical attention network (MAHAN) for next point-of-interest (POI) recommendation. MAHAN incorporates two networks to tackle short-term check-in sequences and long-term memories for preference modeling. A co-attention network is used to characterize the interaction between these long-term and short-term preferences. Wu et al. [32] presented a similar idea, where the long-term preference is formulated by attention weight evaluation. Hao et al. [33] explored user’s short-term preference based on the local and global features of the annular-graph of the user behavior sequence by the self-attention mechanism. Fan et al. [34] introduced the low-rank decomposed self-attention to generate content-aware representations, where $n$ items are aggregated into $k$ latent interests $(n>k)$ to perform a lightweight self-attention computation.

To sum up, the CF techniques that employ neural networks to model user-item interactions become a popular choice. However, these techniques mainly focus on a single representation space. We argue that it is insufficient to model the complicated user-item interactions. The clustering-based methods are proposed in several studies to address the sparsity and scalability problems. Few of them take advantage of global and local representations from the cluster structure. We point out that multiple representations are beneficial to provide multi-views for observing user-item interactions. The attention mechanisms can provide an effective way to fuse these representations into a blended one. We develop the representation-level attention, which is different from the item-level attention used by existing methods.

The goal of the CF approach with implicit feedback is to explore the unobserved relations between user $u$ and item $i$ and to answer the question: will the user interact with the item in the future? A user-item interaction consists of, for example, the user assigning a score to an item or checking information about an item. The interaction history is usually described by an interaction matrix, denoted as $A=\left[a_{ui}\right]$, which contains the user ID $u$, the item ID $i$, and the interaction $a_{ui}$ that represents the preference of user $u$ for item $i$. This study focuses on evidence that the user interacted with an item, regardless of the extent of the interaction. Therefore, the matrix element $a_{ui}$ is defined by:

A triplet training set is sampled from $A$ and expressed as:

where $i^{+}$ and $i^{-}$ are the positive and negative items, respectively. In practice, it is impossible to enumerate all triplet combinations and many of them are trivial for training. We adopt a simple strategy in this study [6][35]. Given the row vector of user $u$ in interaction matrix $A$, for each positive item $i^{+}$ (where $a_{ui^{+}}=1$), we randomly choose an negative item $i^{-}$ (where $a_{ui^{-}}=0$) to produce a triplet $(u,i^{+},i^{-})$. The iteration process will repeat until the number of triplets is sufficient.

To derive the global representation, local information such as cluster assignment of the training data is ignored during the training phase. Let $G\in\mathbb{R}^{d}$ be a global representation space of $d$-dimensions. To project user $u$ and item $i$ into $G$, the representation network is designed based on the Siamese network [36], as shown in Fig. 2. The embedding layer transforms a user or item ID into a dense vector of a fixed size. The embedding vector is then forwarded to hidden layers for nonlinear transformation. Note that we modify the typical Siamese network so that the variant can accept a triplet $\left(u,i^{+},i^{-}\right)\in T$ as the input in the training process. In addition, we separate the user from the item without sharing the hidden layers. In other words, the network parameters can be very different between the user part and the item part. This is because users and items are different in nature, they should be processed through different paths in order to be projected to the common representation space. Through the respective embedding and hidden layers, inputs $u$, $i^{+}$, and $i^{-}$ are transformed with L2 normalization into $G$ as representation outputs $\hat{u}$, $\hat{i}^{+}$, and $\hat{i}^{-}$, respectively.

The training of the representation network is guided by a triplet loss function. Using different loss functions to generate variant representations can provide multiple views to characterize each user and item. We introduce two loss functions for this purpose, the dot product loss and the Euclidean distance loss; more loss functions can be conducted if necessary. The dot product loss originates from BPR-Opt [14]. Its variant adopts the conventional matrix factorization, referred to as BPR-MF, which is generally used as an underlying preference estimator. BPR-MF can be optimized by a contrastive pairwise ranking objective function of a triplet, referred to as product loss:

where $\sigma$ is the sigmoid function and $\left\langle\cdot\right\rangle$ denotes the dot product computation. Because enumerating all triplets in $T$ is typically intractable, BPR-MF uses stochastic gradient descent (SGD) to optimize the model. Specifically, in each step of SGD, a batch of triplets is dynamically sampled from $T$. In addition, L2 regularization was adopted on the user and item representations, which is crucial to alleviate overfitting.

The second global representation was generated based on the Euclidean distance metric, where the optimization function is distance loss expressed by [37]:

where $\alpha$ is the margin.

To make the optimization effective, hard examples of triplets that result in large loss can be selected frequently during training. The network parameters are updated through SGD to minimize the loss function iteratively. In the following, the global representations of user $u$ and item $i$ are denoted as $\hat{u}$ and $\hat{i}$, respectively.

Roughly speaking, the global representations provide a macro view of users and items. However, they are insufficient to model complex user-item interactions. The local representations, on the other hand, are derived by learning local data, which are extracted from the clusters that are close to the user. They can model the user-item interactions from a micro view.

For simplicity, we assume the global representation space $G$ is a Euclidean space. K-means clustering is used to construct a cluster set of size $M$. Given a query (user) $u$ and its global representation $\hat{u}$, $J$ clusters are chosen from the $M$ clusters that are the closest to $\hat{u}$ and are denoted as candidate clusters $\{C_{j}\}_{j=1}^{J}$. The corresponding codebook of centroids is $\{\mu_{j}|\mu_{j}\in G\}_{j=1}^{J}$. To construct the local representation network, we employ the same network architecture defined in Fig. 2 but train with the local properties of the candidate clusters.

Suppose that a positive item $\hat{i}^{+}$ and a negative item $\hat{i}^{-}$ belong to two candidate clusters, i.e., $\hat{i}^{+}\in C_{j}$ and $\hat{i}^{-}\in C_{k}$, where $j,k\in\{1,\cdots,J\}$. There are two considerations for generating the triplet training set. The first case is $j=k$, i.e., $\hat{i}^{+}$ and $\hat{i}^{-}$ are in the same cluster. They can be included in the local intra-triplet set $T^{\dagger}\in T$ when satisfying the following criterion:

The intuition is that, if the distance between $u$ and $i^{+}$ is larger than that between $u$ and $i^{-}$, it will incur positive triplet loss. We can use this hard example to tune the representation network so that the user is close to its positive item and far away from its negative item. The other case is $j\neq k$, i.e., $\hat{i}^{+}$ and $\hat{i}^{-}$ are in different clusters. A similar definition for the local inter-triplet set $T^{\ddagger}\in T$ is given:

Unlike Eq. (5) that considers the relation between the user and items, Eq. (6) addresses the relation between the user and clusters. That is, if the distance between $u$ and $C_{j}$ (where $i^{+}$ belongs to) is larger than that between $u$ and $C_{k}$ (where $i^{-}$ belongs to), we take this hard example to tune the user representation network so that the user is close to its positive cluster $C_{j}$ and far away from its negative cluster $C_{k}$. Consequently, both $T^{\dagger}$ and $T^{\ddagger}$ serve as the local training dataset for learning local representations.

The outputs of the local representation network are expressed as the user $\tilde{u}$, the positive item $\tilde{i}^{+}$, and the negative item $\tilde{i}^{-}$. In a similar way to Eq. 3 and Eq. 4, the loss functions for training local representations are expressed as $l_{prod}(\tilde{u},\tilde{i}^{+},\tilde{i}^{-})$ for product loss and $l_{dist}(\tilde{u},\tilde{i}^{+},\tilde{i}^{-})$ for distance loss.

By leveraging the global and local properties with different loss functions, multiple representation networks can be trained to generate the respective representations for each user/item. Moreover, an attention network is proposed to find the best combination based on these representations. That is, the proposed attention network can map a set of user-item representations to weighted sum representations of the user and item. The weight is assigned to each representation space by computing a compatibility function of the user and item. Details are described in the following.

Suppose that we construct $R$ representation networks to derive corresponding representation spaces; each user/item has $R$ representations. Fig. 3 gives an overview of the proposed network consisting of $R$ representation networks followed by an attention network, where four representation spaces $(R=4)$ are taken as an example. In the training process, the input layer receives a triplet $(u,i^{+},i^{-})$. Let $g^{(r)}$ be the $r$th representation network, $r\in{1,2,...,R}$, which transforms the user and items to their $r$th representations. We express the $r$th representations of the triplet as $\big{(}g^{(r)}(u),g^{(r)}(i^{+}),g^{(r)}(i^{-})\big{)}$, and forward them to the $r$th attention channel.

Each attention channel can be divided into three parts. The first part is a transformation function, denoted as $f^{(r)}$, which maps the user and positive item to the respective vectors $u^{(r)}$ and $i^{(r)}$:

Note that the negative item is excluded in the attention network. Consequently, the second part is to calculate the compatibility between the user and positive item. The idea is paying more attention to the representation space where the user and the positive item are more compatible, as measured by the dot product. The compatibility is then convert to a normalized weight, denoted as $\alpha^{(r)}$, by applying the softmax function over all representation spaces.

In the last part, a weighted sum is calculated as the linear combinations of the $R$ representation spaces for each user/item:

The outputs $\bar{u}$, $\bar{i}^{+}$, and $\bar{i}^{-}$ are denoted as attentive representations. The proposed model is trained by the triplet loss function based on either product loss $l_{prod}(\bar{u},\bar{i}^{+},\bar{i}^{-})$ or distance loss $l_{dist}(\bar{u},\bar{i}^{+},\bar{i}^{-})$, and the training data are randomly sampled from the global and local triplet sets $T$, $T^{\dagger}$, and $T^{\ddagger}$.

We treat the recommendation task as a retrieval process that searches for similar items from multi-view representation spaces for a given user. In clustering-based recommendation, a coarse-to-fine search strategy can be applied without scanning the dataset exhaustively, making recommendation more efficient and scalable. For each item $i$ in the dataset, $R$ representations are extracted by the set of representation networks $\{{g}^{(r)}(i)|r=1,2,...,R\}$. We choose one of the global representations and perform k-means clustering to create $M$ clusters and to build the index structure with $M$ clusters, each of which has attached an inverted list of associated items.

When a user query $u$ is submitted, the user representations $\{{g}^{(r)}(u)\}$ are extracted and used to search for the $K$ nearest item clusters. We denote the items stored in the nearest clusters as the candidate items. For each pair of user $u$ and candidate item $i^{\prime}$, the attention network is used to obtain the attentive representations $\bar{u}$ and $\bar{i}^{\prime}$ defined in Eq. 9 and to compute their Euclidean distance or dot product to estimate their similarity. The candidates are sorted to output the top-$N$ items as the recommendation result for user $u$. The retrieval process is summarized in Algorithm 1.

By taking advantage of the coarse-to-fine search strategy, only some dataset items are selected as candidates for the attention and re-ranking computation. Although applying the attention network for each user-item pair requires additional computations, these can be accelerated by GPUs, and thus the runtime is usually minor. Therefore, the time complexity of the retrieval process can be considered sublinear with respect to the size of the item dataset.

Four popular benchmark datasets, including MovieLens, Yelp, and Amazon, were used in the experimental evaluation. We briefly introduce these datasets, as well as the corresponding training/test data partitions:

• •

  MovieLens-100k. This dataset consists of 100,000 records of user-movie ratings from 943 users and 1,682 movies. Each user has rated at least 20 movies. We randomly sampled half of the user-movie views as test data, and used the other half for training.

• •

  Yelp Madison. The Yelp dataset is a set of user reviews of businesses in local cities. Here, we selected the city of Madison. Users who had more than five business interactions were kept; the others were removed from the dataset. For each of the remaining users, we randomly sampled three user-business interactions as the test data, and used the rest for training.

• •

  Yelp Pittsburgh. Yelp Pittsburgh is another dataset containing user reviews of businesses in Pittsburgh. The user filtering and training/test data generation were the same as for Yelp Madison.

• •

  Amazon. This dataset contains product reviews from customers of Amazon.com. For this study, the category Movie and TV was chosen. Users with fewer than six reviews were removed from the dataset. The training/test data were generated in the same way as for Yelp Madison and Yelp Pittsburgh.

Table I summarizes some statistic properties of these datasets. Their scales and densities are very different from each other, providing a diverse way for observation.

Several widely used accuracy metrics in top-N recommendation, including precision, recall, hit rate (HR), average-reciprocal hit rank (ARHR), and normalized discounted cumulative gain (NDCG), were used for experiment evaluation. For a user query $u$, the top-N recommended items are denoted as $Q_{u}=\{i_{1},i_{2},...,i_{N}\}$ and the ground truth (positive) items as $G_{u}=\{i^{\prime}_{1},i^{\prime}_{2},...,i^{\prime}_{|G_{u}|}\}$, where $|\cdot|$ is the set cardinality. These metrics are defined as follows:

where $rank_{i}$ is the ranking position of item $i$. Although a higher value reflects a better accuracy in general, it is recommended to assess the model performance from multiple metrics rather than any single metric.

To study the impact of model parameters, we take the following four global and local representations, as well as two attentive representations, for consideration:

• •

  Global representation / distance loss (GD) was generated based on the global training set (Eq. (2)) and the distance loss function (Eq. (4)).

• •

  Global representation / product loss (GP) was generated based on the global training set and product loss function (Eq. (3)).

• •

  Local representation / distance loss (LD) was generated based on the local training set (Eqs. (5) and (6)) and the distance loss function.

• •

  Local representation / product loss (LP) was generated based on the local training set and the product loss function.

• •

  Attentive representation / distance loss (AD) was blended from the above four representations by the proposed attention network and the distance loss function.

• •

  Attentive representation / product loss (AP) was blended by the proposed attention network and the product loss function.

Except for using different training datasets and loss functions, these representations adopted the same representation network shown in Fig. 2, which contains one embedding layer and two hidden layers, each of which has 64 neurons. The dimensionality of the output layer was also set to 64. Each model was trained for a maximum of 200 epochs with an early stopping strategy and optimized with the Adam optimizer, where the batch size was fixed at 512 and the learning rate was 0.00017. The margin of the loss function in Eq. (4) was set $\alpha=0.5$.

We first evaluate the impact of the number of clusters $M$. We evaluate three different $M$ for each dataset: $M\in\{10,15,20\}$ for MovieLens, $M\in\{10,20,30\}$ for Yelp Madison, $M\in\{10,20,30\}$ for Pittsburgh, and $M\in\{30,100,256\}$ for Amazon. The range of $M$ has been carefully chosen: too large or small $M$ will incur a bad clustering result. The performance is plotted in Fig. 4, where each column shows a particular dataset with different $M$. Each plot shows six recall curves of the proposed representations under the top-$K$ candidate clusters, $K\in\{1,2,3,4,5\}$. We ranked the items in the candidate clusters to return the top-20 items for recommendation.

As the number of candidate clusters $K$ is increased, more positive items are found from the candidate clusters, and therefore the recall curves is generally improved. In addition, most of them show a sharp rise at $K=2$. However, some recall curves have a degraded performance as $K$ increases. This means that the corresponding model does not generate good representation spaces, so that the relevance between the user and positive item is less than that between the user and negative item. The proposed attentive representations (AD and AP) are relatively stable under these configurations, showing that they became more robust by adaptively fusing multiple representations.

In general, local representations (LD and LP) are better than global representations (GD and GP). However, no single representation always performs better than others because of the variety of user-item interaction behaviors. Again, the proposed attention network combining different views of representation spaces yield the best recall on all the benchmark datasets. Based on the above result, we choose the following $M$ in subsequent experiments: $M=20$ for MovieLens, $M=20$ for Yelp Madison, $M=10$ for Yelp Pittsburgh, and $M=100$ for Amazon.

Next, we observe the recommendation result from more evaluation metrics, as shown in Figs. 5, 6, 7 and 8 for MovieLens, Yelp Madison, Yelp Pittsburgh, and Amazon, respectively. Each figure contains four subplots, including recall, precision, HR, and ARHR (from left to right). The X-axis represents the number of top-$N$ recommendation items, $N\in\{5,10,15,20,25,30\}$, and the Y-axis represents the accuracy. The number of candidate clusters is fixed at $K=2$. Again we see the proposed attention networks outperform the other representation networks under these accuracy metrics. Particularly, the proposed method AP is superior to AD under the ARHR metric, even they are comparable under the other three metrics. Note that ARHR measures the ranking quality that assigns a high score to hits at top position ranks. That is, AP could provide a better ranking quality. Another interesting phenomenon is, for the HR metric, most of the curves show a sharp rise at $N=10$. Since HR is an indication whether at least one positive item is in the top-$N$ set, we suggest $N\geq 10$ would be a good number of recommendation items.

An ablation study is introduced for the attention network of the global and local representations. Three representation types are considered: global, local, and all. The global type consists of GD, GP, GAD (global attentive representations with distance loss), and GAP (global attentive representations with product loss). The local type follows the similar definition. We fixed the number of candidate clusters at $K=2$ and the number of recommended items at $N=15$. Tables II and III list the performance in the four benchmark datasets. The number in boldface indicates the best accuracy in each metric column. It is clear to see that with the attention network, the global attentive representations (i.e., GAD and GAP) gain further improvement by combining the two global representations. The local attentive representations (i.e., LAD and LAP) also gain the similar improvement. By combining all global and local representations together, the attention network can yield the best accuracy, as demonstrated by AD and AP.

We compared several CF recommendation systems, including network-based, factorization-based, and clustering-based approaches:

• •

  MetaPath2Vec++ [38]: This system uses a network embedding model that characterizes user-item interactions by heterogeneous network representation learning.

• •

  BiNE [39]: This system is also a network embedding model that characterizes user-item interactions by bipartite representation learning.

• •

  CoFactor [4]: This factorization model generates the representation by jointly decomposing the user-item interaction matrix and the item-item co-occurrence matrix with shared item latent factors.

• •

  N2VSCDNNR [12]: This clustering-based method uses the node2vec technique to generate a common representation space and builds user and item clusters for matching.

• •

  CIGAR [13]: This is also a clustering-based method that uses a hashing technique to retrieve candidate items and learns a ranking model to re-rank the candidate items.

Tables IV and V present their performances in terms of recall, precision, HR, and ARHR. The clustering-based approach (i.e., N2VSCDNNR, CIGAR, AP and AD) yields a better performance than the other approaches in the diverse benchmark datasets, demonstrating its superiority to address the sparsity and scalability problems to a certain extent. Furthermore, CIGAR performs better than N2VSCDNNR in the sparser and larger datasets, i.e., Yelp and Amazon. However, these compared methods employ a single representation, which is insufficient to model complex user-item interactions. The proposed attention model combines multiple representations to regenerate joint representations for each user-item pair dynamically, thus gain the best performance in various benchmark datasets.

Finally, we performed leave-one-out evaluation to compare our methods with two other CF methods. The compared methods are end-to-end learning-based recommender systems:

• •

  NeuMF [5]: This system generalizes matrix factorization by nonlinear neural networks to model user-item interactions.

• •

  BCFNet [30]: This system, which is an extension of DeepCF [17], uses an attention mechanism to improve the representation learning and matching function learning.

In leave-one-out evaluation, we held-out the latest interaction as a test item for each user and utilized the remaining data for training. For testing we followed the same way of the above two methods: we randomly selected 99 unobserved interactions as the negative examples for each user, together with the latest interaction as the positive example. In total 100 test are ranked, and the top-10 items were evaluated by HR and NDCG.

Table VI lists the leave-one-out evaluation for the four benchmark datasets. Note the BCFNet’s result for Amazon is not available because the memory consumption is too large to be executed in our computer. For each dataset, we generated three representation spaces of 32, 64, and 128 dimensions. Our methods yield better HR and NDCG scores in the high-dimensional representation space. In particular, our NDCG scores outperform NeuMF and BCFNet with a large gap, indicating our methods can rank the positive example at a superior position. The proposed method is able to find better representations for users and items through separate representation networks. Separating items into positive and negative ones also has an important impact on the recommendation task. In addition, similar to BCFNet, the attention mechanism can be used to highlight important features. However, because the local representations are derived by learning local data, which are extracted from the clusters close to the user, when the density vector is low-dimensional, it may be easier to mix some dissimilar data points in a cluster than that is high-dimensional. The local representations that come out may be poor, and in turn affect the final recommendation results.