Multi-Behavior Enhanced Recommendation with Cross-Interaction Collaborative Relation Modeling ^††thanks: *Corresponding author: Yong Xu.

We suppose the output embedding of the $l$-th GNMR graph layer as $\textbf{H}^{l}$ which is then fed into the $(l+1)$-th layer as input representation. Given edges between users and items under the behavior type of $k$, we construct the passed message as below:

where $\textbf{H}_{i\leftarrow}^{k,(l+1)}\in\mathbb{R}^{d}$ and $\textbf{H}_{j\leftarrow}^{k,(l+1)}\in\mathbb{R}^{d}$ denotes the embeddings passed to $u_{i}$ and $v_{j}$, respectively. $\eta(\cdot)$ represents the embedding layer which preserves the unique characteristics of each type (i.e., $k$) of user behaviors. During the message propagation process, we initialize the embeddings $\textbf{H}_{i}^{0}$ and $\textbf{H}_{j}^{0}$ for user $u_{i}$ and item $v_{j}$ by leveraging Autoencoder-based pre-training scheme [9] for generating low-dimensional representations based on multi-behavior interaction tensor X.

In our message passing architecture, $\eta(\cdot)$ is designed to obtain representations for each behavior type of $k$, by considering type-specific behavior contextual signals (e.g., behavior frequency). We formally represent $\eta(\cdot)$ as follows:

where $\alpha_{c,k}$ represents the learned weight of the $k$-th type of user behavior from the projected $c$-th latent dimension. $N(i,k)$ denotes the neighboring item nodes connected with user $u_{i}$ under behavior type of $k$ in the interaction graph $G$. $\textbf{W}_{1}\in\mathbb{R}^{C\times d}$ and $\textbf{b}_{1}\in\mathbb{R}^{C}$ are learn hyperparameters. We define $\delta(\cdot)$ as ReLU activation function. In our embedding process, we aggregate embeddings from different latent dimensions with weight $\alpha_{c,k}$ and transformation parameter $\textbf{W}_{2,c}\in\mathbb{R}^{d\times d}$. The message passing procedure for the target item node $v_{j}$ from its adjacent user nodes $N(j,k)$ under the behavior type of $k$ is conducted in an analogous way in Eq 2.

After performing the propagation of type-specific behavior embeddings between user and item, we propose to aggregate representations from different behavior types, by exploiting the underlying dependencies. In GNMR framework, our message aggregation layer is built upon the attentional neural mechanism. In particular, we first define two transformation weight matrices Q and K for embedding projection between different behavior types $k$ and $k^{\prime}$. The explicit relevance score between type-specific behavior embeddings is represented as $\beta_{k,k^{\prime}}$, which is formally calculated as follows:

We perform the embedding projection process with multiple latent spaces ($s\in S$), to enable the behavior dependency modeling from different hidden dimensions. $\textbf{Q}^{s}\in\mathbb{R}^{\frac{d}{S}\times d}$ and $\textbf{K}^{s}\in\mathbb{R}^{\frac{d}{S}\times d}$ correspond to the transformation matrices of $s$-th projection space. We further apply the softmax function on $\beta_{k,k^{\prime}}^{s}$. Then, we recalibrate the type-specific behavior embedding by concatenating representations from different learning subspaces with the following operation:

where $\mathop{\Bigm{|}\Bigm{|}}$ represents the concatenation operation and $\textbf{V}_{s}\in\mathbb{R}^{\frac{d}{S}\times d}$ denotes the transformation matrix. We define the propagated recalibrated embedding as $\tilde{\textbf{H}}_{i}^{k,(l)}$. To preserve the original type-specific behavioral patterns, the element-wise addition is utilized between the original embedding $\textbf{H}_{i\leftarrow}^{k,(l)}$ and recalibrated representation $\tilde{\textbf{H}}_{i\leftarrow}^{k,(l)}$, i.e., $\hat{\textbf{H}}_{i\leftarrow}^{k,(l)}=\tilde{\textbf{H}}_{i\leftarrow}^{k,(l)}\oplus\textbf{H}_{i\leftarrow}^{k,(l)}$, where $\hat{\textbf{H}}_{i\leftarrow}^{k,(l)}$ is the updated embedding propagated through the connection edge of $k$-th behavior type.

To fuse the behavior type-specific representations during the embedding propagation process, we develop our message aggregation layer with the following functions as:

where $\psi(\cdot)$ represents the message aggregation function. To fuse user/item embeddings from different behavior types, we aim to identify the importance score of individual behavior type-specific representation in assisting the recommendation phase between users and items. To achieve this goal, we feed $\hat{\textbf{H}}_{i\leftarrow}^{k,(l)}$ into a feed-forward neural network to calculate the importance weights as follows (take the user side as example):

where $\gamma_{k}$ represents the intermediate values which are fed into a softmax function to generate the importance weight $\hat{\gamma}_{k}$. In addition, $\delta(\cdot)$ denotes the ReLU activation function. $\textbf{W}_{1}\in\mathbb{R}^{d^{\prime}\times d}$ and $\textbf{w}_{2}\in\mathbb{R}^{d^{\prime}}$ represents trainable transformation matrices. $\textbf{b}_{1}\in\mathbb{R}^{d^{\prime}}$ and $b_{2}\in\mathbb{R}$ are bias terms. $d^{\prime}$ denotes the embedding dimension of hidden layer. After obtaining the weight $\hat{\gamma}_{k}$ corresponding to behavior type of $k$, the embedding aggregation process is performed as: $\textbf{H}^{(l+1)}_{i}=\sum_{k=1}^{K}\hat{\gamma}_{k}\hat{\textbf{H}}_{i\leftarrow}^{k,(l+1)}$ and $\textbf{H}^{(l+1)}_{j}=\sum_{k=1}^{K}\hat{\gamma}_{k}\hat{\textbf{H}}_{j\leftarrow}^{k,(l+1)}$, where $\textbf{H}^{(l+1)}_{i}$ and $\textbf{H}^{(l+1)}_{j}$ serve as the input user/item embedding for $(l+1)$-th graph layer.

Given the generated graph structure of user-item interactions, we learn the high-order multi-behavioral relations over $G=\{U,V,E\}$ by stacking multiple information propagation layers. The embedding propagation between the $(l)$-th and $(l+1)$-th graph layers can be formally represented as below:

where $\textbf{H}^{(l+1)}_{(i)}\in\mathbb{R}^{I\times d}$ denotes the embeddings of all users for the $(l+1)$-th graph layer. Given the behavior type of $k$, the adjacent relations are represented as $\textbf{X}^{k}\in\mathbb{R}^{I\times J}$. Similarly, embeddings of items ($v_{j}\in V$) can be generated based on the above propagation and aggregation operations.

To optimize our GNMR model and infer the hyperparameters, we perform the learning process with the pairwise loss which has been widely used in item recommendation task. Specifically, for each user $u_{i}$ in the mini-batch training, we define the positive interacted items (i.e., $(v_{p_{1}},v_{p_{2}},...,v_{p_{S}})$) of user $u_{i}$ as $S$. For generating negative instances, we randomly sample $S$ non-interacted items $(v_{n_{1}},v_{n_{2}},...v_{n_{S}})$ of user $u_{i}$. Given the sampled positive and negative instances, we define our loss function as follows:

we incorporate the regularization term $|\mathbf{\Theta}\|_{\text{F}}^{2}$ with the parameter $\lambda$. The learnable parameters are denoted as $\mathbf{\Theta}$. The training process of our model is elaborated in Algorithm 1.

Type-specific Behavior Embedding Component. We first design a contrast method GNMR-be without the type-specific behavior embedding layer, to generate the propagated feature embedding of individual type of user behavior during the message passing process. The performance gap between GNMR-be and GNMR indicates the effectiveness of our attention-based projection layer to capture the type-specific behavioral features.

Message Aggregation Component. Another dimension of model ablation study aims to evaluate the efficacy of our message aggregation layer. GNMR-ma is the variant of our framework, in which the behavior dependency modeling component is removed in our graph-structured embedding propagation. From Figure 2, we observe that GNMR outperforms GNMR-ma, which shows that capturing the type-wise behavior dependencies is helpful for learning relation-aware collaborative signals and improving the recommendation performance.