SocialTrans: A Deep Sequential Model with Social Information for Web-Scale Recommendation Systems

The personal preference modeling module tries to capture how users’ dynamic interests evolve over time. To be specific, this module generates a user’s personal preference embedding at the current timestamp given his behavior sequence.

We use a multi-layer Transformer (Kang and McAuley, 2018) to capture users’ personal preferences. Transformer is widely used in sequence modeling. It is able to capture the correlation between any pairs in sequences. As shown in Figure 4, the Transformer layer contains three sub-layers, a Multi-Head Attention sub-layer, a Feed-Forward Network sub-layer, and an Add & Norm sub-layer. We now describe the input, the output, and sub-layers in Transformer in detail.

Input Embedding. The input matrix $\mathbf{H}^{(0)}\in\mathbb{R}^{m\times d}$ to the multi-layer Transformer is mainly constructed from items in user’s behavior sequence $\hat{S}_{t-1}^{u}=(v_{0},v_{1},...,v_{m-1})$. Here $d$ is the hidden dimension and each item $v\in V$ is represented as a row $\mathbf{w}_{v}$ in the item embedding matrix $\mathbf{W}\in\mathbb{R}^{|V|\times d}$. Since the Transformer can’t be aware of items’ position, each position $\tau$ is associated with a learnable position embedding vector $\mathbf{p}_{\tau}\in\mathbb{R}^{d}$ to carry location information for the corresponding item. Each row $\mathbf{h}^{(0)}_{\tau}$ in $\mathbf{H}^{(0)}$ is defined as:

Multi-head Self-Attention. Attention mechanisms are widely used in sequence modeling tasks. They allow a model to capture the relationship between any pairs in the sequences. Recent work (Li et al., 2018; Devlin et al., 2019) has shown that attention to different representation subspaces simultaneously is beneficial. In this work, we adopt the multi-head self-attention as in work (Vaswani et al., 2017). This allows the model to jointly attend to information from different representation subspaces.

First, the scaled dot-product attention is applied to each head. This attention function can be described as mapping a set of query-key-value tuples to an output. It is defined as:

Here $\mathbf{Q},\mathbf{K},\mathbf{V}\in\mathbb{R}^{m\times d_{s}}$ represent the query, key, and value matrices. Moreover, $d_{s}$ is the dimensionality for each head, and we have $d_{s}=d/r$ for an $r$-head model. The scaled dot-product attention computes a weighted sum of all values, where the weight is calculated by the query matrix $\mathbf{Q}$ and the key matrix $\mathbf{K}$. The scaling factor $\sqrt{d_{s}}$ is used to avoid large weights, especially when the dimensionality is high.

For an attention head $i$ in layer $l$, all inputs to the scaled dot-product attention come from layer $l-1$’s output. This implies a self-attention mechanism. The query, key, and value matrices are linear projection of $\mathbf{H}^{(l-1)}$. The head is defined as:

In the above equation, $\mathbf{W}^{(l,i)}_{Q},\mathbf{W}^{(l,i)}_{K},\mathbf{W}^{(l,i)}_{V}\in\mathbb{R}^{d\times d_{s}}$ are the corresponding matrices that project input $\mathbf{H}^{(l-1)}$ into the latent space of query, key, and value. Row $\tau$ of $\text{head}_{i}^{(l)}$ corresponds to an intermediate representation of a user’s behavior sequence at timestamp $\tau$.

Items in a user behavior sequence are produced one by one. The model should only consider previous items when predicting the next item. However, the aggregated value in Equation (3) contains information of subsequent items, which makes the model ill-defined. Therefore we remove all links between row $\tau$ in $\mathbf{Q}$ and row $\tau^{\prime}$ in $\mathbf{V}$ if $\tau>\tau^{\prime}$.

After all attention heads are computed in layer $l$, their outputs are concatenate and projected by $\mathbf{W}^{(l)}_{O}\in\mathbb{R}^{d\times d}$, resulting in the final output $\mathbf{A}^{(l)}\in\mathbb{R}^{m\times d}$ of a multi-head attention sub-layer:

Feed Forward Network. Although previous items’ information can be aggregated in the multi-head self-attention, it’s still a linear model. To improve the representation power of our model, we apply a two-layer feed forward network to each position:

where $\boldsymbol{W}_{\text{FFN}}^{(k,1)},\boldsymbol{W}_{\text{FFN}}^{(k,2)}$ are both $d\times d$ matrices and $\boldsymbol{b}_{\text{FFN}}^{(k,1)},\boldsymbol{b}_{\text{FFN}}^{(k,2)}$ are $d$ dimensional vectors. Moreover $f$ is an activation function and we choose Gaussian Error Linear Unit as in work (Hendrycks and Gimpel, 2016). Nonlinear transformation is applied to all positions independently, meaning that no information will be exchanged across positions in this sub-layer.

Add & Norm Sub-layer. Training a multi-layers network is difficult because the vanishing gradient problem may occur in back-propagation. Residual neural network (He et al., 2016) has shown its effectiveness in solving this problem. The core idea behind the residual neural network is to propagate outputs of lower layers to higher layers simply by adding them. If lower layer outputs are useful, the model can skip through higher layers to get necessary information. Let $\mathbf{X}$ be the output from lower layer and $\mathbf{Y}$ be the output from higher layer. The residual layer (or add) layer is defined as:

In addition, to stabilize and accelerate neural network training, we apply Layer Normalization (Ba et al., 2016) to the residual layer output. Assuming the input is a vector $\boldsymbol{z}$, the operation is defined as:

where $\mu$ and $\sigma$ are the mean and variance of $\boldsymbol{z}$, $\boldsymbol{\alpha}$ and $\boldsymbol{\beta}$ are learned scaling and bias terms, and $\odot$ represents an element-wise product.

Personal Preference Embedding Output. SocialTrans encapsulates a user’s previous behavior sequence into a $d$ dimensional embedding. Let $l_{T}$ be the number of layers in Transformer. The output of the $l_{T}$-th layer is $\mathbf{H}^{(l_{T})}$. We take $\mathbf{h}^{(l_{T})}_{m-1}$ as the user’s personal preference embedding, where $\mathbf{h}^{(l_{T})}_{m-1}$ is the last row of $\mathbf{H}^{(l_{T})}$. The personal preference embedding is expressive because $\mathbf{h}^{(l_{T})}_{m-1}$ aggregated all previous items information in multi-head self-attention layer. Moreover, stacking multiple layers provides personal preference embedding with the highly non-linearly expressive power.

Birds of a feather flock together. A user’s behavior is influenced by his friends (Marsden and Friedkin, 1993; McPherson et al., 2001). We should incorporate the social information to further model user latent factors. Meanwhile, different social connections or friends have different influence on a user. In other words, the learning of social-space user latent factors should consider different strengths in social relations. Therefore, we introduce a multi-head graph attention network (GAT) (Veličković et al., 2018) to select friends that are representative in characterizing users’ socially influenced preferences. We also consider edge attributes to learn the context-dependent social influence. Next, we describe the input, output and our modified GAT in detail.

Input Embedding. In §3.1, we described how to obtain a user’s personal preference embedding given his behavior sequence. Suppose we want to generate the socially influenced preference embedding of a user $u$. This module’s inputs are personal preference embeddings of $u$ and his friends. Specifically, for a user $u$, the input of this module is $\{\hat{\mathbf{h}}^{(0)}_{u^{\prime}}|u^{\prime}\in\{u\}\cup N(u)\}$, where $\hat{\mathbf{h}}^{(0)}_{u}=\mathbf{h}^{(l_{T})}_{m-1}$ and $N(u)$ is the set of user $u$’s friends.

Graph Attention Network. Veličković et al. (Veličković et al., 2018) introduces graph attention networks (GAT) which specifies different weights to different nodes in the neighborhood. Social influence is often context-dependent. It depends on both friends’ preferences and the degree of closeness with friends. We use the GAT to aggregate contextual information from the user’s friends. In this work, we propose an extended GAT that uses edge attributes and the multi-heads mechanisms. Figure 5 shows our modified version of GAT.

We first calculate the similarity score $\delta_{u,u^{\prime}}^{(l)}$ between the target user’s embedding $\hat{\mathbf{h}}^{(l-1)}_{u}$ and all of his neighbors’ embedding $\hat{\mathbf{h}}^{(l-1)}_{u^{\prime}}$:

Then we normalized the similarity score to a probability distribution:

where $\hat{\mathbf{W}}_{Q}^{(l)},\hat{\mathbf{W}}_{K}^{(l)}\in\mathbf{R}^{d\times d}$ in Equation (8) are the query and key projection matrices similar to Equation (3) in Transformer. $\mathbf{e}_{u,u^{\prime}}$ is a vector of attributes corresponding to the edge between $u$ and $u^{\prime}$, and $\hat{\mathbf{w}}_{E}^{(l)}$ is a weighted vector applied to these attributes. Equation (8) computes similarity score based on user’s and friend’s representation and their corresponding attributes. This enables us to combine both the preferences of friends and the degree of closeness with friends.

Intuitively, $\kappa_{u,u^{\prime}}^{(l)}$ is the social influence strength of a friend $u^{\prime}$ on the user $u$. We aggregate social influence of all friends as:

where $\hat{\mathbf{W}}_{V}^{(l)}\in\mathbf{R}^{d\times d}$ is value projection matrix and $f$ is a Gaussian Error Linear Unit as in Equation (5).

In practice, we find that the multi-head attention mechanism is useful to jointly capture context semantic at different subspaces. We extend our previous Equations (8), (9), and (10) to:

where $\hat{\mathbf{W}}_{Q}^{(l,i)},\hat{\mathbf{W}}_{K}^{(l,i)},\hat{\mathbf{W}}_{V}^{(l,i)}\in\mathbb{R}^{d_{s}\times d}$ and $\hat{\mathbf{w}}_{E}^{(l,i)}$ are corresponding parameters for head $i$. Finally, all $r$ heads are stacked and projected by $\hat{\mathbf{W}}^{(l)}_{O}\in\mathbf{R}^{d\times d}$ to get the final output embedding in layer $l$:

Social Embedding Output. We stack $l_{G}$ layers of GAT and take the final representation $\hat{\mathbf{h}}^{(l_{G})}_{u}$ as the user $u$ socially influenced preference embedding. It encapsulates social information from user $u$ and his friends. Note that stacking too many layers of GAT may harm the model performance, because our analysis result shown in Figure 2 suggests that most social information come from one-hop friends. Thus, we use at most two layers of GAT.

A user’s decisions depend on the dynamic personal preference and socially influenced preference from his friends. The final embedding representation is obtained by merging personal preference embedding and socially influenced preference embedding, which is defined as:

where $\mathbf{W}_{F}\in\mathbf{R}^{d\times 2d}$ and $\tilde{\mathbf{h}}_{u}$ is user $u$’s final representation.

The probability that the next item will be $v$ is computed using a softmax function:

We train the model parameters by maximizing the log-probability of all observed sequences:

To predict which item user $u$ will click, we compute all items’ scores according to Equation (16) and return a list of items with top $K$ scores for the recommendation. To avoid large computational cost in a real-world application, the approximate nearest neighbor search method (Andoni and Indyk, 2006) is used.

We evaluate our model in three data sets. For offline evaluation, we tested on a benchmark data set Yelp and a data set WeChat Official Accounts. For online evaluation, we conducted experiments on WeChat Top Stories, a major article recommendation application in China³³3All users in data sets of WeChat are anonymous.. The statistics of those data sets are summarized in Table 1. We describe the detailed information as follows:

Yelp⁴⁴4https://www.yelp.com/dataset. Yelp is a popular review website in the United States. Users can review local businesses including restaurants and shops. We treat each review from 01/01/2012 to 11/14/2018 as an interaction. This data set includes 3.3 million user reviews, more than 170,000 businesses, more than 250,000 users and 4.7 million friendship links. The last 150 days of reviews are used as a test set and the training set is constructed from the remaining days. Items that do not exist in the training set are removed from the test set. For each recommendation, we use the user’s own interaction and friends’ interactions before the recommendation time.

WeChat Official Accounts. WeChat is a Chinese messaging mobile app with more than one billion active users. WeChat users can register an official account, which can push articles to subscribed users. We sample users and their social networks restricted to an anonymous city in China. We treat each official account reading activity as an item click event. The training set is constructed from users’ reading logs in June 2019. The first four days of July 2019 are kept for testing. We remove items appeared less than five times in training data to ensure the quality of recommendation. After processing, this data set contains more than 736,000 users, 48,000 items and each user has an average of 81 friends.

WeChat Top Stories. WeChat users can receive articles recommendation service in Top Stories, whose contents are provided by WeChat official accounts. Here we treat each official account reading activity as an item click event. Training logs are constructed from users’ reading logs in June 2019. Testing is conducted on an online environment in five consecutive days of July 2019. This data set contains billions of users and millions of items. In this data set, we keep the specific values for business secret.

In this subsection, we evaluate the performance of different models on Yelp and WeChat Official Accounts data sets.

We are interested in the performance of SocialTrans under different circumstances. First, we analyze the performance of different models for users with different number of friends. Then, we analyze the performance of SocialTrans with different number of layers.

Sequential recommendation algorithms use previous interactions to predict a user’s next interaction in the near future. Different from general recommendation, sequential recommendation always views the interactions as a sequence of items in time order. Previous sequential recommendation methods have been based on Markov chains, these methods construct a transition matrix to solve sequential recommendation. Factorizing Personalized Markov Chains (FPMC) (Rendle et al., 2010) mixes Markov chains (MC) and matrix factorization (MF) together and estimates a transition cube to learn an transition matrix for each user. FPMC cannot catch relationships between items in a sequence, because it assumes each item independently affects the user’s next interaction. HRM (Wang et al., 2015) could well capture both sequential behavior and users’ global interest by including transaction and user representations in prediction. Recently, there have been many methods that apply deep learning to recommendation systems emerged. Restricted Boltzmann Machine (RBM) (Salakhutdinov et al., 2007) and Autoencoders (Sedhain et al., 2015) have been the earliest methods, and RBM has been proved to be one of the best performing Collaborative Filtering models. Caser (Tang and Wang, 2018) converts embedding of items in a sequence into a square matrix and uses CNN to learn user local features as a sequential pattern. However, CNN could hardly capture long-term dependencies between items and users’ global features are always important for the recommendation. On the other hand, RNN has been widely used to model global sequential interactions (Yu et al., 2016). GRU4Rec (Hidasi et al., 2016) has been a representative method based on RNN for the sequential recommendation, which uses the final hidden state of GRU to delineate the user current preference. NARM and EDRec (Li et al., 2017; Loyola et al., 2017) use the attention mechanism to improve the effect of GRU4Rec. SASRec (Kang and McAuley, 2018) use a multi-layer Transformer to capture user’s dynamic interest evolved over time. SHAN (Ying et al., 2018b) proposed a two-layer hierarchical attention network to take both user-item and item-item interactions into account. These models assume that a user’s behavior sequence is dynamically determined by his personal preference. However, they do not utilize social information, which is important in social recommendation tasks.

In online social networks, one common assumption is that a user’s preference is influenced by his friends. So introducing social relationships could improve the effectiveness of the model. Besides, for recommendation systems, using friends’ information of users could effectively alleviate cold start and data sparsity problems. There have been many studies that models the influence of friends on user interests from different aspects. Most proposed models are based on Gaussian or Poisson matrix factorization. Ma et al. (Ma et al., 2011) proposed a matrix factorization framework with social regularization such that the distances of connected users’ embedding vectors are small. TrustMF (Yang et al., 2016) adopts a matrix factorization technique to map users into low-dimensional potential feature spaces according to their trust relationship. SBPR (Zhao et al., 2014) is an approach that utilizes social information for training instance selection. Recently, many studies leveraged deep neural networks and network embedding approaches to solve social recommendation problems because of their powerful performance. The NCF model (He et al., 2017) leverages a multi-layer perceptron to learn the user-item interaction function. NSCR (Wang et al., 2017a) enhances the NCF model by plugging a pairwise pooling operation and extends NCF to cross-domain social recommendations. Previous methods neglected the weight of the relationship edge, and different types of edges should play different roles. mTrust (Tang et al., 2012a) and eTrust (Tang et al., 2012b) could model trust evolution, integrating multi-faceted trust relationships into traditional rating prediction algorithms in order to reliably evaluate their strengths. TBPR (Wang et al., 2016) and PTPMF (Wang et al., 2017b) distinguish between strong and weak ties of users for the recommendation in social networks. These approaches assume that a user’s behavior sequence is determined by his personal preference and his socially influenced preference. However, these methods model users’ personal preferences as static and social influences as context-independent.

Graph Convolutional Networks (GCN) has been a powerful technique to encode graph nodes into low-dimensional space and has been proven to have the ability to extract features from graph-structured data (Defferrard et al., 2016). Kipf and Welling (Kipf and Welling, 2017) used GCNs for semi-supervised graph classification and achieved the state-the-of-art performance. GCNs combine the features of the current node and its neighbors, and handle edge features easily. However, in the original GCNs, all neighbors’ weights are fixed when using convolution filters to update the node embeddings. GATs (Veličković et al., 2018) use an attention mechanism to assign different weights to different nodes in the neighborhood. In the area of recommendation, PinSage (Ying et al., 2018a) uses efficient random walks to structure the convolutions and is scalable to recommendations on large-scale networks. GCMC (Berg et al., 2017) proposes a graph auto-encoder framework based on differentiable message passing on the bipartite user-item graph and provides users’ and items’ embedding vectors for the recommendation. SocialGCN (Wu et al., 2019) uses the ability of GCNs to catch how users’ interests are influenced by the social diffusion process in social networks. DGRec (Song et al., 2019) proposes a method based on a dynamic-graph-attention neural network. DGRec uses RNN to model user short-term interest and uses GAT to model social influence between users and their friends. SocialTrans uses a multi-layers Transformer to model users’ personal preference. We show by experiments that SocialTrans outperforms DGRec both on the Yelp data set and on the WeChat Official Accounts data set.