Ensemble Modeling with Contrastive Knowledge Distillation for Sequential Recommendation

Traditional methods for sequential recommendation adopt the Markov Chains (MCs) to model item transitions, such as MDP (Shani et al., 2005), FPMC (Rendle et al., 2010) and Fossil (He and McAuley, 2016). With the advancements in deep learning, RNN-based model (Hidasi et al., 2016) and CNN-based model (Tang and Wang, 2018) have all been shown as effective sequence encoders. The recent success of Transformer (Vaswani et al., 2017) also motivates the designing of Transformer-based sequence encoders. For example, SASRec (Kang and McAuley, 2018) adopts unidirectional Transformer to automatically assign attention weights to different items, BERT4Rec (Sun et al., 2019) adopts bidirectional Transformer with the cloze task (Devlin et al., 2019) to adapt for interactions that do not follow a rigid sequential order. Some works also leverage the attribute information about items in order to provide contextual data for sequential recommendation. For example, FDSA (Zhang et al., 2019) adopts feature-level self-attention blocks to fuse attribute data into sequences, $\rm{S}^{3}$-Rec (Zhou et al., 2020) proposes self-supervised objectives to model the correlations between items, sequences and attributes, MMInfoRec (Qiu et al., 2021) combines the attribute encoder with a memory module and a sampling strategy to capture long-term preferences. Different from these works that only focus on a single encoder, our framework trains an ensemble of networks with contrastive knowledge distillation.

Ensemble modeling (Srivastava et al., 2014; Huang et al., 2017; Lan et al., 2018; Song and Chai, 2018; Zhang et al., 2018; Tramèr et al., 2018; Garipov et al., 2018) has long been established as an effective approach for improving the accuracy and robustness of neural networks. One commonly adopted technique is Dropout (Srivastava et al., 2014), which creates a self-ensemble of networks by randomly dropping neurons at the training stage. As for the ensemble of multiple parallel networks, some works have recognized the necessity of facilitating knowledge when training multiple parallel networks, which is often implemented via knowledge distillation (Hinton et al., 2015). For example, DML (Zhang et al., 2018) proposes a collaborative learning strategy by distilling knowledge between parallel networks, ONE (Lan et al., 2018) proposes an on-the-fly native ensemble method for one-stage online distillation. However, these methods are all proposed for the compression of large-scale models on computer vision tasks, and how to design efficient and effective methods to facilitate knowledge transfer for ensemble sequential models remains to be explored.

Contrastive learning has achieved great success in computer vision (van den Oord et al., 2018; He et al., 2020; Chen et al., 2020a, 2021b, b; Chen and He, 2021; Tian et al., 2020; Fang et al., 2021; Chen et al., 2021a), natural language processing (Gao et al., 2021; Yan et al., 2021), and recommender systems (Yu et al., 2022; Wei et al., 2022; Xie et al., 2022; Qiu et al., 2022). In sequential recommendation, contrastive learning has also been shown as an effective approach for alleviating data sparsity and learning better representations. For example, CL4SRec (Xie et al., 2022) first introduces contrastive learning into sequential recommendation by proposing three data augmentation operators, DuoRec (Qiu et al., 2022) performs contrastive learning from the model level to mitigate the representation degeneration problem, CML (Wei et al., 2022) combines contrastive learning with meta learning for personalized multi-behavior recommendation. Different from these works, our framework formulates contrastive learning as a way of contrastive representation distillation (Tian et al., 2020) to facilitate knowledge transfer between the ensemble networks.

In sequential recommendation, we denote $\mathcal{U}$ as a set of users, $\mathcal{V}$ as a set of items, and $\mathcal{A}$ as a set of attributes. Each user $u\in\mathcal{U}$ is associated with an interaction sequence $s_{u}=[v_{1},v_{2},\cdots,v_{t},\cdots,v_{T}\big{]}$ sorted in the chronological order, where $v_{t}\in\mathcal{V}$ denotes the item that user $u$ has interacted with at the $t$-th timestamp and $T$ indicates the sequence length. Each item $v_{t}$ has its own attribute set ${\alpha}_{t}=\{a_{1},a_{2},\cdots,a_{\lvert{\alpha}_{t}\rvert}\}\subset\mathcal{A}$ as a subset of all the attributes. The task of sequential recommendation is to predict the next item that user $u$ will probably interact with, and it can be formulated as generating the probability distribution over all candidate items for user $u$ at the time step $T+1$:

A sequential recommendation model encodes essential sequence information via a sequence encoder, which takes user sequences as input and outputs hidden representations of sequences. In our framework, we adopt bidirectional Transformer as the base sequence encoder, which is able to model item correlations from both directions. The architecture of bidirectional Transformer consists of an embedding layer and a bidirectional Transformer module. For the input sequence $s_{u}=[v_{1},v_{2},\cdots,v_{T}]$, an item embedding matrix $\boldsymbol{V}\in{\mathbb{R}}^{{\lvert}\mathcal{V}{\rvert}\times d}$ and a positional embedding matrix ${\emph{{P}}}\in{\mathbb{R}}^{T\times d}$ and are combined together to represent items in the hidden space, where $T$ denotes the maximum sequence length of our model, $d$ is the hidden dimensionality. The computation of the embedding layer is formulated as follows:

We then feed $\boldsymbol{E}(s_{u})$ into the bidirectional Transformer module $Trm$ and fetch the output $\boldsymbol{H}\in{\mathbb{R}^{T\times d}}$ as the hidden representations of the sequence. In general, we denote a sequence encoder $f(\cdot)$ that takes user sequence $s_{u}$ as the input and outputs the hidden representations $\boldsymbol{H}(s_{u})$ of this sequence:

As we have mentioned above, an ensemble of parallel networks can generate different predictions, which will be helpful for improving the overall accuracy rate. We adopt $N$ parallel networks as an ensemble of sequence encoders. Each parallel network $f^{n}(\cdot)$ is a bidirectional Transformer sequence encoder, where $1{\leq}n{\leq}N$. We initialize these parallel networks with different random seeds so that they will generate different prediction results.

We adopt a multi-task learning scheme to enhance sequential recommendation with contextual information. Specifically, we adopt masked item prediction as the main task for recommendation training. An attribute prediction task is also introduced as the auxiliary task to fuse attribute information.

To transfer knowledge between parallel networks, we design a contrastive knowledge distillation approach that transfers both the representational knowledge and the logits-level knowledge.

One possible reason why ensemble methods are superior for sequential recommendation is that it can handle behavior uncertainty. Users’ sequential behaviors are uncertain rather than deterministic, and users’ preferences may shift under different circumstances (Fan et al., 2022). For example, a user favours movies with Action and Comedy genres. However, there are many candidate movies that fall under these genres, and the user may choose different movies under different situations. In such cases, the diversity of prediction results brought by ensemble methods can consider all these situations and generate a more accurate recommendation result.

Another possible reason is that ensemble methods can mitigate the adverse effects of noisy data. The datasets for sequential recommendation are usually sparse and noisy. In such cases, the knowledge transfer mechanisms between the ensemble networks can filter out the incorrect predictions introduced by noisy labels and transfer the useful knowledge to the whole network (Han et al., 2018), thus improving the overall accuracy rate and model robustness.

Table 3 presents the overall performance of EMKD and baseline methods. Based on the experiment results, we can see:

• •

  EMKD outperforms all baseline methods on both sparse and dense datasets with the relative performance improvements ranging from 13.05% to 33.48%. We attribute the performance improvement to these factors: (1) ensemble modeling uses multiple parallel networks to yield diverse predictions, and combining a diversity of prediction results is more accurate than the prediction of a single network; (2) contrastive knowledge distillation facilitates knowledge transfer between parallel networks, and each individual network benefits from this collaborative learning paradigm; (3) the attribute information provides rich contextual data and can benefit the main task of sequential recommendation.

• •

  Among general sequential methods, Transformer-based sequence encoders (e.g., SASRec, BERT4Rec) outperform RNN-based (e.g., GRU4Rec) or CNN-based (e.g., Caser) sequence encoders. This suggests that the self-attention mechanism can effectively model sequential patterns. Moreover, attribute-aware sequential methods outperform general sequential methods owing to the incorporation of attribute information. Besides, contrastive sequential methods show performance improvements compared with general sequential methods. This is probably because contrastive learning serves as a regularization objective that can alleviate the data sparsity issue and improve model robustness.

In this section, we study the influences of important hyper-parameters, including the number of masked sequences $M$, temperature $\tau$, weight hyper-parameters $\lambda$ and $\mu$, hidden dimensionality $d$. To control variables we only change one hyper-parameter at one time while keeping the other hyper-parameters optimal.

In this section, we conduct an ablation study to investigate the effectiveness of the key components in our framework. Table 4 presents the performance of EMKD under the default setting and its variants. Based on the experiment results, we can see that:

• •

  Ensemble networks outperform a single network. We can see that (6) shows the worst performance compared with other ensemble methods. This observation verifies our claim that ensemble modeling is more powerful than a single network.

• •

  Ensemble networks learn better with effective knowledge transfer. Our method for contrastive knowledge distillation contains three objectives—ICL, CCL and KD. From (2)-(4) we can see that each objective plays a crucial role in facilitating knowledge transfer between parallel networks, and removing any objective leads to performance decrease. Compared with (5), we can see that ensemble networks with knowledge transfer methods outperform the simple ensemble of independently-trained networks.

• •

  Training EMKD with 3 parallel networks is the optimal setting. Increasing the number of parallel networks can improve performance because training more parallel networks can increase the diversity of prediction results. However, we cannot train an infinite number of parallel networks due to the limitation of computational cost. From (1)(6)(7)(8) we can see that training EMKD with 3 parallel networks almost reaches the best performance. Although (8) shows a slight improvement compared with (1), training EMKD with 4 parallel networks is too costly and also increases the risk of overfitting.

• •

  Sequential recommendation benefits from the auxiliary attribute information. From (9) we can see that removing the auxiliary task of attribute prediction will hurt performance. This observation is also consistent with previous works (Zhang et al., 2019; Zhou et al., 2020; Qiu et al., 2021) that utilizing attribute information can enhance the performance of sequential recommendation models.

EMKD trains an ensemble of bidirectional Transformers as sequence encoders, which can be viewed as an ensemble of BERT4Rec models. Since ensemble modeling and knowledge transfer is a model-agnostic approach, it can also be adapted to other sequence encoders. In this section, we apply EMKD to the ensemble of various base sequence encoders with a slight modification. Different from the Masked Language Modeling (MLM) style in bidirectional Transformers, other sequence encoders (e.g., GRU4Rec, Caser and SASRec) are autoregressive models that predict the next item based on previous items. Therefore, we need to switch the masked item prediction task to the next item prediction task. The attribute prediction task is also modified so that the model will predict the associate attributes at the next position.

Figure 4 presents the performances of different models enhanced by EMKD on Beauty and ML-1M datasets. We can see that all the base sequence encoders enhanced by our approach achieve better performance. This shows that ensemble modeling with contrastive knowledge distillation is a generalized approach that can be utilized to improve the performance of sequential recommenders.

A major concern for ensemble modeling is the training efficiency. To evaluate the training efficiency of our framework, we compare EMKD with two best-performed baselines: MMInfoRec and DuoRec. Training efficiency is evaluated from two aspects: training speed (average training time required for one epoch), and convergence speed (time required to achieve satisfactory performance). To visualize the performance, we save the checkpoint from every epoch and test them afterwards. All experiments use the default hyper-parameters and are carried out on a single Tesla V100 GPU.

Figure 5 presents the training efficiency of each model on Beauty and Toys datasets. We find out that EMKD with 2 parallel networks shows similar training speed (69.4s/epoch and 62.1s/epoch) with MMInfoRec (65.3s/epoch and 60.2s/epoch), while EMKD with 3 parallel networks (109.2s/epoch and 98.6s/epoch) is about 30% slower than MMInfoRec. Besides, DuoRec shows the best training speed (40.8s/epoch and 37.9s/epoch) on both datasets.

We also find out that EMKD almost converges as fast as DuoRec. The performance curve of EMKD coincides with the performance curve of DuoRec around the first 1000 seconds, indicating that EMKD and DuoRec show similar convergence speed during this interval. However, the performance of EMKD continues to improve afterwards, while DuoRec reaches its best performance. By comparison, MMInfoRec takes about 4000 seconds to converge to a satisfactory performance, which is slower than DuoRec and EMKD.

Compared with DuoRec and MMInfoRec, EMKD only brings a minor reduction in training efficiency. However, the performance improvement can be very significant compared with training a single network. Therefore, we think that it is worthwhile to adopt ensemble modeling methods in sequential recommendation.

EMKD trains multiple bidirectional Transformer encoders as an ensemble network, which will increase the total number of parameters. To determine whether the good performance of EMKD results from the proposed knowledge transfer mechanisms or simply comes from the increase in the parameter size, we compare the performance of EMKD with other Transformer-based sequence encoders (SASRec and BERT4Rec) of similar parameter sizes. SASRec and BERT4Rec adopt two layers of Transformer blocks by default, and to scale the number of parameters, we stack more layers of Transformer blocks for SASRec and BERT4Rec so that they will have parameter sizes comparable to EMKD. Note that we count the parameters of both the embedding table and the sequence encoder(s) for each model, and we set the hidden dimensionality of each model as 256 for fair comparisons. Table 5 shows the parameter sizes and the performances of different models on three datasets. We can see that EMKD with 2 parallel networks has similar parameter size with a 6-layer SASRec or a 4-layer BERT4Rec, while EMKD with 3 parallel networks has similar parameter size with a 10-layer BERT4Rec. However, EMKD consistently outperforms SASRec and BERT4Rec of any parameter sizes, which verifies the effectiveness of the knowledge transfer mechanisms between the ensemble networks. We also find out that stacking more transformer blocks will not improve performance or even hurt performance in some cases, indicating that simply increasing the parameter size will not necessarily result in better performance for sequential recommendation.