When Multi-Level Meets Multi-Interest: A Multi-Grained Neural Model for Sequential Recommendation

Compared with the general recommendation, the scenario of sequential recommendation is different, and its main task is simplified to predict what the user prefers for a commodity pool in the future by using considering the sequential nature of the user historical behaviors. During the early phase, traditional reasoning methods are utilized, such as Markov Chain, which assumes that the next action depends on the previous action sequence. For example, Rendle et al. (Rendle, 2010) propose to combine matrix factorization with Markov Chains (MC) to achieve better performance in sequence recommendation. And some works assume that the next action only relies on the last behavior, using first-order Markov chain (Cheng et al., 2013). Note that these methods are not capable to capture the long-term interests of users effectively due to the limitation of the capability to simulate the dynamic changes of user preferences over time. Then, the emergence of neural networks further enhances recommender systems’ ability to extracte the preference of users, so another paradigm of sequence recommendation method based on neural networks in addition to MC-based methods has gradually become the mainstream. The most basic multi-layer perceptions (MLPs) structure extracts the non-linear correlations from user-item interactions (He and Chua, 2017). Then a series of models (Zhang et al., 2016; Qu et al., 2016; Shan et al., 2016; Cheng et al., 2016) represented by DeepFM (Guo et al., 2017) are put forward. For the DeepFM model, the FM module is used for a low-order combination of features, and the deep network module is used for the high-order combination of features. By combining the two methods in parallel, the final architecture can learn low-order and high-order combination features at the same time. Referring to the feature extraction mechanism in texts, audios, and pictures, CNN is used to improve the model capability in sequence recommendation. The CNN architectures are also verified to be effective in this regard to a certain extent, by mapping item sequences to embedding matrices. A representative work is Caser (Tang and Wang, 2018), which treats the use’s behavior sequence as an ”image” and adopts a convolutional neural network to extract user representation. Nevertheless, this mechanism ignores the sequential relations in sequence.

Compared with approaches based on DNN and CNN, RNN is able to capture dynamic time series information (Wang et al., 2020; Zhou et al., 2018). Hidasi et al. (Hidasi et al., 2015) first introduce RNN to the sequential recommendation and achieve impressive performance gain over previous methods. Due to the appearance and excellent performance of the RNN network, more and more methods based on the RNN structure are proposed. GRU4Rec (Hidasi et al., 2015) first applies Gated Recurrent Units to model the whole session for a more accurate recommendation. To quantify the different importance of past interactions on the next prediction, attention mechanism (Vaswani et al., 2017) is adopted. Specifically, attention mechanism makes it easy to memorize various remote dependencies or focus on important parts of the input. In addition, the attention-based methods are often more interpretable (Sha and Wang, 2017) than traditional deep learning models. There are some other works that introduce specific neural modules for particular recommendation scenarios, which are mainly based on the combination of RNNs, CNNs, and attention structure, leading to the applications of some emerging network models coming into vogue. For example, memory networks (Chen et al., 2018; Huang et al., 2018), graph neural networks (GNN) (Wu et al., 2019; Ying et al., 2018) that cooperate with the attention mechanism are used to extract short-term features with more consistency or adjacency consideration. SRGNN (Wu et al., 2019) regards the session history as a directed graph. In addition to considering the relationship between an item and its adjacent previous items, it also considers the relationship with other interactive items. What’s more, Fan and Liu et al. (Fan et al., 2021) integrate the sequence information and collaboration information, use a transformer to capture the temporal relationship in the sequence, and construct a continuous-time bipartite graph. SLi_Rec (Yu et al., 2019) utilized the fine-grained temporal characteristics of interactive data in the sequence recommendation to stress the ability to modeling sequential behaviors. The recent work represented by TGSRec (Fan et al., 2021) combines graph and temporal information to further greatly improve the performance of the model.

In a word, most of the existing general sequential approaches are learning to get a single representation of users from an RNN and attention-based model according to the historical behaviors. And graph models, which are capable to aggregate neighbor information, have also been proved to be very effective. Nevertheless, the user history interaction sequence contains more than one discrete interest of the user, and a single vector can not fully express the user preferences. In addition, the noise in the process of graph construction and information aggregation is also an important reason to limit the performance of graph-based sequential models.

For a stronger ability to learn the complex behaviors precisely, recently researchers consider that representing user preferences as a single vector is insufficient, more and more sequential recommendation models based on multi-interest, therefore, appear in our field of vision. Li et al. (Li et al., 2020) consider that users’ interests are dynamic and evolve over time. A pre-trained model based on transformer structure is designed, using the item of the next time step as the label of the interest at the current time step, and then obtains the interest of each time step. The final interest representation is generated by the attentional fusion structure. Pi et al. (Pi et al., 2019) propose MIMN system which contains modules Neural Turing Machine (NTM), Memory Induction Unit (MIU), etc. In the MIU module, an additional storage unit s is also included, which contains M memory slots. It is considered that each memory slot is a user interest channel. Besides, both MIND (Li et al., 2019) and ComiRec (Cen et al., 2020) devise multi-interest recommendation models on the basis of CapsNet, which uses the idea of neural routing to realize interest decomposition. Note that ComiRec introduces two multi-interest extraction mechanisms including CapsNet and self-attention. At the same time, they also have good applications in the industry. The above methods are multi-interest methods based on sequence models. With the popularity of graph neural networks, the undeniable role of neighbor information has also been proved to be effective obviously. Therefore, the approach of combining graph and multi-interest has also attracted extensive attention in recent years. For example, in SURGE (Chang et al., 2021), it forms dense clusters in the interest graph to distinguish users’ core interests and performs cluster-aware and query-graph graph convolutional propagation to fuse users’ current core interests from behavior sequences. These mentioned approaches have also been successfully applied in many recommendation applications and are rather useful and efficient in real-world application tasks.

Let $\mathcal{V}=\{x_{1},x_{2},...,x_{M}\}$ denotes the set of all items, $\mathcal{U}=\{u_{1},u_{2},...,u_{N}\}$ be the set of all users, and $\{b_{u}\}_{M}^{N}$ be the behavior sequence set between these $N$ users and $M$ items. Here, for each user $u$, $b_{u}=[x_{1},x_{2},...,x_{m}]$ is the sequence of her clicked items following the chronological order, and $m$ is the predefined maximum capacity. The sequential recommendation is to precisely identify the next item $x_{m+1}$ that user $u$ will click in terms of $\{b_{u}\}_{M}^{N}$.

In order to extract complex and high-order interests from user click sequences, we consider the graph structure and the aggregation of neighbor information of the target node at different distances in the graph. So at the first step, we convert discrete history behavior into a fully connected item-item graph. Compared with existing methods, we do not artificial use co-occurrence, click of the same user, and other relationships to enhance the graph, because this approach often introduces noise, which affects the performance of information aggregation in the later convolution process to some extent. In the MGNM, the nodes and users embedding would be updated by using the neural aggregation of CapsNet through gradient feedback and then generate an adaptive graph structure.

After extracting multi-level item representations $\{\mathbf{H}^{(0)},\cdots,\mathbf{H}^{(L)}\}$, where $\mathbf{H}^{l}=[\mathbf{h}_{1}^{(l)},\cdots,\mathbf{h}_{m}^{(l)}]$ and $\mathbf{h}_{i}^{(l)}\in R^{d}$, we choose to utilize CapsNet to generate the user’s multiple interests for each level. Actually, the existing works for multi-interest-based recommendation utilize CapsNet to composite each interest representation through the built-in dynamic routing mechanism. The output of each capsule is equivalent to specific user interests. However, the standard dynamic routing mechanism mainly achieves the function of iterative soft-clustering. It is well validated that temporal information is critical for the sequential recommendation. This is why the application of CapsNet in fine-tuning CTR tasks is limited(Cen et al., 2020; Li et al., 2019).

Here, we repatch this defect by introducing a sequential encoding layer for CapsNet. Specifically, given the item representations at level $l$, the $i$th capsule firstly performs a linear projection over $\mathbf{H}^{(l)}$ as follows:

where $\mathbf{Z}_{i}=[\mathbf{z}_{1}^{(l)},\cdots,\mathbf{z}_{m}^{(l)}]$ and $\mathbf{W}_{i}\in R^{d\times d}$ is the trainable parameter for the projection.

We then initialize $\mathbf{g}=[g_{1},\cdots,g_{m}]$ by truncated normal distribution, where $g_{i}$ is the agreement score indicating the relevance of item $x_{i}$ towards the capsule. The coupling coefficient $\mathbf{c}\in R^{d}$ for the corresponding dynamic routing mechanism is then derived via a softmax function:

Then, the capsule derive its output $\mathbf{o}_{i}$ via a nonlinear squashing function as follows:

where $c_{j}$ is the $j$th element of $\mathbf{c}$. We then update the agreement score $g_{i}$ as follows:

After this first iteration, we utilize a BiLSTM¹¹1Any other sequential modeling techniques like GRU and Transformer can be straightforwardly applied here. module to encode sequential patterns and update $\mathbf{Z}_{i}$ via a residual structure:

We then repeat the above routing process following Equation 6-10 for $\tau-1$ without further applying BiLSTM encoding over $\mathbf{Z}_{i}$. That is, total $\tau$ iterations are performed for each capsule. The output $\mathbf{o}_{i}$ in the last iteration is fed into a full-connected layer to derive the $i$th interest representation $\mathbf{q}_{i}^{(l)}$ in the $l$ level as follows:

where $\mathbf{W}_{i}^{\prime}\in R^{d\times d}$ is the trainable parameter. Assuming the number of interests is $K$, we obtain $K$ interest representations $[\mathbf{q}_{1}^{(l)},\cdots,\mathbf{q}_{K}^{(l)}]$ for the $l$th level. That is, we extract in total $(L+1)\cdot K$ interest representations.

The other two datasets are from Amazon product datasets³³3http://snap.stanford.edu/data/amazon/: Musical Instruments and Toys and Games. Here, each user interaction in the Amazon dataset is associated with a user rating score. Following the previous works (Rendle et al., 2010; He and McAuley, 2016; He et al., 2017), we take each user interaction with a rating score larger than $2$ as the positive. We then organize these interactions and split the interaction sequence with the ratio of $7$:$1$:$2$ to form the training, validation, and test set respectively following the chronological order. We further remove users whose length of history sequence is $1$.

Table 1 summarizes detailed statistics of the three datasets after preprocessing. The Micro-video dataset includes a large number of items, while Toys and Games is much smaller. Also, the Musical Instruments is the smallest according to the interaction number. We can see that these three real-world datasets hold different characteristics, covering a broad range of real-world scenarios.

The overall performance of all methods is reported in Table 2. Here, we make the following observations.

As for traditional sequential models that utilize RNN and attention mechanism, they are difficult to achieve better performance. Compared with temporal-aware models and multi-interest models, they are not suitable for complex and various user interest modeling. The temporal-aware models perform very well in Amazon datasets. Specifically, on the Music Instruments dataset, TGSRec and Sli_Rec achieve the best performance in terms of GAUC and NDCG@$5$ respectively. They also achieve strong performance in terms of the other six metrics for both Toys and Games and Music Instruments. It is worthwhile to mention that TGSRec needs to build a global graph and takes the interactions at different time points as edges. This design choice requires much more computation cost for graph retrieval and convolution. Note that the graph constructed on the Micro-video dataset contains more than $200$ million edges. We utilize the implementation released by the original authors for evaluation. The time of an epoch training exceeds $1,200$ hours. Hence, we do not obtain results on the Micro-video dataset.

Also, given the superiority of these temporal-aware models on both Amazon datasets, the multi-interest models perform better on the Micro-video dataset. This suggests that neither temporal-aware models nor multi-interest models are robust enough to achieve precise preference understanding across different scenarios. Considering the semantic space in the Micro-video recommender scenario could be much broader than commodities in E-Commerce scenarios, the interest of each user also becomes more complex. It is reasonable that the multi-interest models could achieve better recommendation performance instead.

Our proposed MGNM has obvious improvement in most settings for the three datasets including Micro-video, Toys and Games, and Music Instruments. In detail, MGNM performs significantly better than all the baselines on $10$ out of $12$ dataset and metric combinations. Although our MGNM achieves only comparable NDCG@$5$ performance against SLi_Rec and performs a bit worse to TGSRec in terms of GAUC both on the Music Instruments dataset, we need to emphasize that both SLi_Rec and TGSRec exploit additional timestamp features to seize more discriminative capacity. In other words, our model MGNM lacks one-dimensional timestamp characteristics than these two models (i.e. SLi_Rec and TGSRec). Note that modeling multi-grained multi-interest in MGNM and exploiting timestamp information are not mutually excluded. As shown in Equation 10, it is straightforward to include fine-grained timestamp information in the sequential capsule network component⁴⁴4We leave it as a part of our future work.. Moreover, we can see that our proposed MGNM performs increasingly better on larger datasets. The relative performance gain by MGNM against the best baseline is in the range of $1.63\%-2.44\%$ and $2.09\%-4.35\%$ on Toys and Games and Micro-video datasets respectively. This further confirms that our MGNM is effective to capture multi-grained user interests for large-scale real-world scenarios that are rich in semantics.

Here, we investigate the impact of each design choice and important parameter settings to the performance of MGNM.

• •

  SCN$\to$ BiLSTM: We replace the sequential capsule network with BiLSTM to encode the user behavior sequence in different levels. The last hidden state generated by the BiLSTM is taken the user interest in the corresponding level.

• •

  SCN$\to$ SumPool: We replace the sequential capsule network with a sum pooling mechanism. Similar to SCN$\to$ BiLSTM, the resultant representation is taken the user interest for the corresponding level.

• •

  SCN$\to$ SelfAtt: We replace the sequential capsule network with a self-attention mechanism. The candidate item is utilized to derive the user interest for each level by using an attention mechanism.

• •

  SCN (Transformer): We replace the built-in BiLSTM in the sequential capsule network with a powerful transformer module.

Table 3 reports the performance of these variants and the full MGNM model on Toys and Games dataset⁵⁵5Similar observations are also made in the other two datasets. Here, we can make the following observations.

Firstly, The $L_{1}$ regularization indeed improves the discriminative capacity of user-aware graph convolution. The experimental results show that the performance degradation without it is obvious. When we remove the multi-level item representation learning supported by user-aware graph convolution (i.e., w/o UGCN), substantial performance degradation is also experienced by MGNM, which illustrates the effectiveness of user-aware graph convolution and multi-level preference learning significantly. Also, we can find that MGNM experiences a large performance degradation by removing the sequential encoding layer (i.e., w/o BiLSTM). This is reasonable since the sequential patterns have been well validated to be effective for the sequential recommendation. Now, we further validate that sequential patterns are also very useful for multi-interest learning. At last, the max-pooling-based prediction plays a great role in improving all four performance metrics. As we described earlier, the max-pooling mechanism is flexible in capturing the complex user preference from multi-grained interests.

Secondly, we further dive deep into the effectiveness of the sequential capsule network component. The first three variants (i.e., SCN$\to$ BiLSTM, SCN$\to$ SumPool, SCN$\to$ SelfAtt) aim to remove the multi-interest modeling by considering only the multi-level user preferences. We can see that these three variants all experience significant performance degradation across the four metrics. Recall that MGNM w/o UGCN also produces a substantial performance degradation above. These two observations suggest that both user-aware graph convolution and sequential capsule network works as a whole and either of them complements the other, leading to better user preference understanding. At last, we find that encoding sequential patterns with a heavy module like transformer achieves better performance than SCN$\to$ BiLSTM, SCN$\to$ SumPool, and SCN$\to$ SelfAtt in terms of NDCG@$5$, HIT@$5$ and MRR@$5$. This also validates the benefit of modeling sequential patterns for multi-interest learning. However, the huge number of parameters involved in a transformer module could complicate the model optimization process. Note that we also derive multi-level item representations by the user-aware graph convolution, a lightweight sequential model like BiLSTM is sufficient for the next step. At the same time, to the best of our knowledge, there are no previous methods to integrate sequential modeling with CapsNet.