A Multi-Strategy based Pre-Training Method for Cold-Start Recommendation

The basic pre-training GNN model reconstructs the cold-start embeddings under the meta-learning setting (Vinyals et al., 2016) to simulate the cold-start scenario. Specifically, for each target user $u$, we mask some neighbors and only maintains at most $K^{l}$ neighbors ($K$ is empirically selected as a small number, e.g., $K$=3) at the $l$-th layer. Based on which we perform graph convolution multiple steps to predict the target user embedding. Take GraphSAGE (Hamilton et al., 2017) as the backbone GNN model as an example, the graph convolution process at the $l$-th layer is:

We perform graph convolution $L$-1 steps, aggregate the refined embeddings of the first-order neighbors $\{\textbf{h}_{1}^{L-1},\cdots,\textbf{h}_{K}^{L-1}\}$ to obtain the smoothed embedding $\textbf{h}^{L}_{\mathcal{N}(u)}$, and transform it into the target embedding $\textbf{h}_{u}^{L}$, i.e., $\textbf{h}^{L}_{\mathcal{N}(u)}=\text{AGGREGATE}(\{\textbf{h}^{L-1}_{i},\forall i\in\mathcal{N}(u)\})$, $\textbf{h}_{u}^{L}=\sigma(\mathbf{W}^{L}\cdot\textbf{h}_{\mathcal{N}(u)}^{L})$. Then we calculate the cosine similarity between the predicted embedding $\textbf{h}_{u}^{L}$ and the ground-truth embedding $\textbf{h}_{u}$ to optimize the parameters of the GNN model:

The basic pre-training strategy has two problems. On one hand, it can not explicitly deal with the high-order cold-start neighbors during the graph convolution process. On the other hand, the GNN sampling strategies such as random sampling (Hamilton et al., 2017) or importance sampling (Chen et al., 2018) strategies may fail to sample high-order relevant cold-start neighbors due to their sparse interactions.

To solve the first challenge, we incorporate a meta aggregator to enhance the aggregation ability of each graph convolution step. Specifically, the meta aggregator uses self-attention mechanism (Vaswani et al., 2017) to encode the initial embeddings $\{\textbf{h}_{1}^{0},\cdots,\textbf{h}_{K}^{0}\}$ of the $K$ first-order neighbors for $u$ as input, and outputs the meta embedding $\tilde{\textbf{h}}_{u}$ for $u$. The process is:

We use the same cosine similarity described in Eq. (2) to measure the similarity between the predicted meta embedding $\tilde{\textbf{h}}_{u}$ and the ground truth embedding $\textbf{h}_{u}$.

To solve the second challenge, we propose an adaptive neighbor sampler to select proper high-order neighbors according to the feedbacks from the GNN model. The adaptive neighbor sampler is formalized as a hierarchical Markov Sequential Decision Process, which sequentially samples from the low-order neighbors to the high-order neighbors and results in at most $K^{l}$ neighbors in the $l$-th layer for each target user $u$. After sampling the neighbors of the former $L$ layers, the GNN model accepts the sampled neighbors to produce the reward, which denotes whether the sampled neighbors are reasonable or not. Through maximizing the expected reward from the GNN model, the adaptive neighbor sampler can sample proper high-order neighbors. Once the meta aggregator and the adaptive neighbor sampler are trained, the meta embedding $\tilde{\textbf{h}}_{u}$ and the averaged sampled neighbor embedding $\tilde{\textbf{h}}^{l}_{\mathcal{N}(u)}$ at the $l$-th layer are added into each graph convolution step in Eq. (1):

For the target user $u$, Eq. (4) is repeated $L-1$ steps to obtain the embeddings $\{\textbf{h}_{1}^{L-1},\cdots,\textbf{h}_{K}^{L-1}\}$ for its $K$ first-order neighbors, then the predicted embedding $\textbf{h}_{u}^{L}$ is obtained by averaging the first-order embeddings. Finally, Eq. (2) is used to optimize the parameters of the pre-training GNN model. The pre-training parameter set $\Theta=\{\Theta_{gnn},\Theta_{f},\Theta_{s}\}$, where $\Theta_{gnn}$ is the parameters of the vanilla GNN, $\Theta_{f}$ is the parameters of the meta aggregator and $\Theta_{s}$ is the parameters of the neighbor sampler. The item embedding $\textbf{h}^{L}_{i}$ can be obtained in a similar way.

We fine-tune ${\rm PT}$-${\rm GNN}$ in the downstream recommendation task. Specifically, for each target user $u$ and his neighbors $\{\mathcal{N}^{1}(u),\cdots,\mathcal{N}^{L}(u)\}$ of different order, we first use the pre-trained adaptive neighbor sampler to sample proper high-order neighbors $\{\hat{\mathcal{N}}^{1}(u),\hat{\mathcal{N}}^{2}(u)\cdots,\hat{\mathcal{N}}^{L}(u)\}$, and then use the pre-trained meta aggregator to produce the user embedding $\textbf{h}_{u}^{L}$. The item embedding $\textbf{h}_{i}^{L}$ can be obtained in the same way. Then we transform the embeddings to calculate the relevance score between a user and an item, i.e., $y(u,i)={\sigma(\mathbf{W}\cdot\textbf{h}_{u}^{L})}^{\mathrm{T}}\sigma(\mathbf{W}\cdot\textbf{h}_{i}^{L})$ with parameters $\Theta_{r}=\{\mathbf{W}\}$. Finally, we adopt the BPR loss (He et al., 2020) to optimize $\Theta_{r}$ and fine-tune $\Theta$:

However, as shown in Fig. 1, due to the limitation of the GNN model, ${\rm PT}$-${\rm GNN}$ can only capture the short-range dependencies of users and items; besides, the embedding reconstruction task only focuses on exploring the intra-correlations within the subgraph of the target user or item. Therefore, it is necessary to fully capture both short- and long-range dependencies of users and items, and both intra- and inter-correlations of users or items.

We select on three public datasets MovieLens-1M (Ml-1M)³³3https://grouplens.org/datasets/movielens/ (Harper and Konstan, 2016), MOOCs⁴⁴4http://moocdata.cn/data/course-recommendation (Zhang et al., 2019) and Gowalla⁵⁵5https://snap.stanford.edu/data/loc-gowalla.html (Liang et al., 2016). Table 1 illustrates the statistics of these datasets.

We select three types of baselines, including the neural collaborative filtering model, the GNN models and the self-supervised graph learning models.

• •

  NCF (He et al., 2017): is a neural collaborative filtering model which uses multi-layer perceptron and matrix factorization to learn the representations of users/items.

• •

  PinSage (Ying et al., 2018): employs the GraphSAGE (Hamilton et al., 2017) model, which samples neighbors randomly and aggregates them by the AVERAGE function.

• •

  GCMC (van den Berg et al., 2018): employs the standard GCN (Kipf and Welling, 2017) model to learn the embeddings of users and items.

• •

  NGCF (Wang et al., 2019b): primarily follows the neural passing based GNN model (Gilmer et al., 2017), but additionally adds second-order interactions during the message passing process.

• •

  LightGCN (He et al., 2020): discards the feature transformation and nonlinear activation functions in NGCF.

• •

  SGL (Wu et al., 2021): contrasts the node representation within the graphs from multiple views, where node dropout, edge dropout and random walk are adopted to generate these views. We find edge dropout has the best performance.

• •

  PT-GNN (Hao et al., 2021): takes the embedding reconstruction as the pretext task to explicitly improve the embedding quality.

For each vanilla GNN model (e.g., GCMC, NGCF), notation ${\rm GNN}^{+}$ means we apply GNN in ${\rm PT}$-${\rm GNN}$, and notation ${\rm GNN}^{*}$ means we apply GNN in ${\rm MPT}$. Since SGL adopts multi-task learning paradigm for recommendation, for fair comparison, we compare it in the extrinsic recommendation task and use notation ${\rm GNN}$-${\rm SGL}$ to denote it.

Following ${\rm PT}$-${\rm GNN}$ (Hao et al., 2021), we divide each dataset into the meta-training set $D_{T}$ and the meta-test set $D_{N}$. We train and evaluate the proposed ${\rm MPT}$ model in the intrinsic user/item embedding inference task on $D_{T}$. Once the the proposed model ${\rm MPT}$ is trained, we fine-tune it in the extrinsic task and evaluate its performance on $D_{N}$. We select the users/items from each dataset with sufficient interactions as the target users/items in $D_{T}$, as the intrinsic evaluation needs the ground-truth embeddings of users/items inferred from the sufficient interactions. In the cold-start user scenario, we divide the users with the number of the direct interacted items (first-order neighbors) more than $n_{i}$ into $D_{T}$ and leave the rest users into $D_{N}$. We set $n_{i}$ as 25, 10 and 25 for the dataset Ml-1M, MOOCs and Gowalla, respectively. Similarly, in the cold-start item scenario, we divide the items with the number of the direct interacted users (first-order neighbors) more than $n_{u}$ into $D_{T}$ and leave the rest items into $D_{N}$, where we set $n_{u}$ as 15, 10 and 15 for Ml-1M, MOOCs and Gowalla, respectively. We set $K$ as 3 and 8 in the intrinsic task, and 8 in the extrinsic task. By default, we set $d$ as 32, the learning rate as 0.003, $L$ as 4, $T$ as 6, $a$ as 0.2, $b$ as 0.2 and $\tau$ as 0.2.

We conduct the intrinsic evaluation task, which aims to infer the embeddings of cold-start users and items by the proposed ${\rm MPT}$ model. Both the evaluations on user embedding inference and item embedding inference are performed.

We train NCF transductively on the merged dataset $Train_{T}$ and $Test_{T}^{\prime}$. We train the vanilla GNN models by BPR loss (He et al., 2020) on $Train_{T}$. We train ${\rm PT}$-${\rm GNN}$ on $Train_{T}$, where we first perform Eq. (4) $L$-1 steps to obtain the refined first-order neighbors, and then average them to reconstruct the target embedding; finally we use Eq. (2) to measure the quality of the predicted embedding. We train ${\rm MPT}$ on $Train_{T}$, where we first perform four pretext tasks by Eq. (7), Eq. (9), Eq. (10) and Eq. (11), and then use Eq. (12) to fuse the generated embeddings; finally we use Eq. (2) to measure the quality of the fused embedding. Note that the embeddings in all the models are randomly initialized. We use cosine similarity to measure the agreement between the ground-truth and predicted embeddings, due to its popularity as an indicator for the semantic similarity between embeddings.

We apply the pre-training GNN model into the downstream recommendation task and evaluate its performance.

The vanilla GNN and the NCF models are trained by the BPR loss on $D_{T}$ and $Train_{N}$. For each user in $Test_{N}$, we calculate his relevance score to each of the rest 1-$c$% items. We adopt Recall@$\mathcal{K}$ and NDCG@$\mathcal{K}$ as the metrics to evaluate the items ranked by the relevance scores. By default, we set $\mathcal{K}$ as 20 for Ml-1M, MOOCs and Gowalla.

We report the overall performance of intrinsic embedding inference and extrinsic recommendation tasks in Table 2 and Table 3. We find that:

• •

  ${\rm MPT}$ (denoted as ${\rm GNN}^{*}$) is better than NCF and the vanilla GNN model, which indicates the effectiveness of the pre-training strategy. Compared with the pre-training model ${\rm PT}$-${\rm GNN}$ (denoted as ${\rm GNN}^{+}$) and SGL (denoted as ${\rm GNN}$-${\rm SGL}$), ${\rm MPT}$ also performs better than them. This indicates the superiority of simultaneously considering the intra- and inter- correlations of nodes as well as the short- and long-range dependencies of nodes.

• •

  In Table 2, when $K$ decreases from 8 to 3, the performance of all the baseline methods drops by a large scale, while ${\rm MPT}$ drops a little. This indicates ${\rm MPT}$ is able to learn high-quality embeddings for users or items that have extremely sparse interactions.

• •

  In the intrinsic task (Cf. Table 2), we notice that most models perform generally better on the item embedding inference task than the user embedding inference task. The reason is that the users in $D_{T}$ have more interacted neighbors than the items in $D_{T}$, thus the learned ground-truth user embedding shows much more diversity than the learned ground-truth item embedding, and only use $K^{l}$ (K is small) neighbors to learn the user embedding is more difficult than learning the item embedding. To validate our point of view, we calculate the average number of interacted neighbors of users and items in $D_{T}$ among these three datasets. In ML-1M, MOOCs and Gowalla, when the layer size $L=1$, the users interact with an average of 178.26, 134.90 and 178.27 items, while the items interact with an average of 32.14, 15.75 and 32.14 users. When the layer size $L$ gets larger, the interacted neighbors’ number of users and items show the same trend. The results indicate the abundant users have much more interacted neighbors than the abundant items, which verifies the above analysis.

• •

  Aligning Table 2 and Table 3, we notice that using GCMC, LigtGCN as the backbone GNN model can always obtain better performance than using PinSage and NGCF. The possible reason is that compared with PinSage and NGCF, GCMC and LightGCN have discarded the complex feature transformation operation, which will contribute to learn high-quality embeddings and better recommendation performance. This phenomena is consistent with LightGCN’s (He et al., 2020) findings.

It is still unclear how does ${\rm MPT}$ handle the cold-start users with different interacted items $n_{i}$ and sparse rate $c$%. To this end, we change the interaction number $n_{i}$ in the range of $\{5,10,15\}$ and the sparse rate $c$% in the range of $\{20\%,40\%,60\%\}$, select LightGCN as the backbone GNN model and report the cold-start recommendation performance on MOOCs dataset in Fig. 3. The smaller $n_{i}$ and $c$% are, the cold-start users in $D_{N}$ have fewer interactions. We find that:

• •

  ${\rm MPT}$ (denoted as ${\rm LightGCN}^{*}$) is consistently superior to all the other baselines, which justifies the superiority of ${\rm MPT}$ in handling cold-start recommendation with different $n_{i}$ and $c$%.

• •

  When $n_{i}$ decreases from 15 to 5, ${\rm MPT}$ has a larger improvement compared with other baselines, which verifies its capability to solve the cold-start users with extremely sparse interactions.

• •

  When $c$% decreases from 60% to 20%, ${\rm MPT}$ has a larger improvement compared with other baselines, which again verifies the superiority of ${\rm MPT}$ in handling cold-start users with different sparse rate.

It is still not clear which part of the pretext tasks is responsible for the good performance in ${\rm MPT}$. To answer this question, we apply LightGCN in ${\rm MPT}$, perform individual or compositional pretext tasks, report the performance of intrinsic and extrinsic task in Fig. 4, where notation $R_{g}$, $C_{g}$, $R_{p}$ and $C_{p}$ denote the pretext task of reconstruction with GNN (only use the objective function Eq. (7)), contrastive learning with GNN (only use the objective function Eq. (9)), reconstruction with Transformer (only use the objective function Eq. (10)) and contrastive learning with Transformer (only use the objective function Eq. (11)), respectively; notation ${\rm LightGCN}^{*}$-$R_{g}$ means we only discard reconstruction task with GNN encoder and use the other three pretext tasks (use the objective functions Eq. (9), Eq. (10) and Eq. (11)) as the variant model. Other variant models are named in a similar way. Aligning Table 2 with Fig. 4, we find that:

• •

  Combining all the pretext tasks can benefit both embedding quality and recommendation performance. This indicates simultaneously consider intra- and inter-correlations of users and items can benefit cold-start representation learning and recommendation.

• •

  Compared with other variant models, performing the contrastive learning task with the GNN or Transformer encoder independently, or performing the contrastive learning task with both the GNN and Transformer encoder leads poor performance in the intrinsic task, but has satisfactory performance in the recommendation task. The reason is that contrastive learning does not focus on predicting the target embedding, but can capture the inter-correlations of users or items, and thus can benefit the recommendation task.

We study the effect of the proposed dynamic sampling strategy. In this paper, we compare four variants of the LightGCN model: ${\rm LightGCN}^{+}$ that adaptively samples neighbors (Hao et al., 2021), ${\rm LightGCN}^{i}$ that samples neighbors according to the importance sampling strategy (Chen et al., 2018), ${\rm LightGCN}^{r}$ that randomly samples neighbors (Hamilton et al., 2017) and ${\rm LightGCN}^{*}$ that dynamically samples neighbors. Notably, for fair comparison, for each target node, at the $l$-th layer, the number of the sampled neighbors are at most $K^{l}$. We report the average sampling time and the average training time per epoch (the average training time includes the training time of all the four pretext tasks, but does not include the neighbor sampling time) for these sampling strategies on MOOCs and Ml-1M in Table 4, and report the recommendation performance on MOOCs and Ml-1M in Fig. 5. We find that:

• •

  The sampling time of the proposed dynamic sampling strategy is in the same magnitude with the importance sampling strategy and the random sampling strategy, which is totally acceptable. While the adaptive sampling strategy takes too much sampling time, since it adopts a Monte-Carlo based policy gradient strategy, and has the complex action-state trajectories for the entire neighbor sampling process. Besides, we report the average training time in each epoch and find that the training time of all the variant models is almost the same, since each variant model performs the same pretext tasks., i.e., the graph convolution operation in the GNN encoder and self-attention operation in the Transformer encoder. By aligning Table 4 and Fig. 5, we find that ${\rm MPT}$ has a quick convergence speed.

• •

  Although the adaptive sampling strategy has competitive performance than the random and importance sampling strategies, it has slow convergence speed due to the complex RL-based sampling process.

• •

  Compared with the other sampling strategies, the proposed dynamic sampling strategy not only has the best performance, but also has quick convergence speed.

As mentioned in Section 2, when performing embedding reconstruction with GNN and Transformer encoders, we choose NCF to learn the ground-truth embeddings. However, one may consider whether ${\rm MPT}$’s performance is sensitive to the ground-truth embedding. To this end, we use the baseline GNN models to learn the ground-truth embeddings, and only perform embedding reconstruction task with LightGCN or Transformer. For NCF, we concatenate embeddings produced by both the MLP and GMF modules as ground-truth embeddings. While for PinSage, GCMC, NGCF and LightGCN, we combine the embeddings obtained at each layer to form the ground-truth matrix, i.e., $\textbf{E}=\textbf{E}^{(0)}$ + $\cdots$+ $\textbf{E}^{(L)}$, where $\textbf{E}^{l}\in\mathcal{R}^{(|\mathcal{U}|+|\mathcal{I}|)\times d}$ is the concatenated user-item embedding matrix at $l$-th convolution step. Fig. 6(a) and Fig. 6(b) shows the recommendation performance using LightGCN and Transformer encoder, respectively. Suffix +${\rm NCF}$, +${\rm L}$, +${\rm G}$, +${\rm P}$ and +${\rm N}$ denote that the ground-truth embeddings are obtained by NCF, LightGCN, GCMC, PinSage and NGCF, respectively. We find that:

• •

  All the models that equipped with different ground-truth embeddings achieve almost the same performance. This indicates our model is not sensitive to the ground-truth embeddings, as the NCF and vanilla GNN models are good enough to learn high-quality user or item embeddings from the abundant interactions.

• •

  Besides, when LightGCN is used to obtain the ground-truth embeddings (denoted as ${\rm LightGCN}^{*}$+${\rm L}$ and ${\rm Trans}^{*}$+${\rm L}$), we can obtain marginal performance gain compared with other variant models.

To understand the effects of individual or compositional data augmentations in the contrastive learning (CL) task using GNNs or Transformer encoder, we investigate the performance of CL pretext tasks in ${\rm MPT}$ when applying augmentations individually or in pairs. We report the performance in Fig. 7. Notation ${\rm LightGCN}^{*}{\rm D}$, ${\rm LightGCN}^{*}{\rm S}$ and ${\rm LightGCN}^{*}{\rm SD}$ mean we apply LightGCN into the CL task and use deletion, substitution and their combinations, respectively; Notation ${\rm Trans}^{*}{\rm D}$, ${\rm Trans}^{*}{\rm S}$ and ${\rm Trans}^{*}{\rm SD}$ denote we use Transformer encoder into the CL task and use deletion, substitution and their combinations, respectively. We find that:

• •

  Composing augmentations can lead to better recommendation performance than performing individual data augmentation.

• •

  Aligning Fig. 7 and Table 3, we find combining substitution and deletion using GNNs has competitive performance than SGL. The reason is that, same as SGL, our proposed contrastive data augmentation can also change the graph structure, and thus inter-correlations of nodes can be captured.

We move on to study different designs of the layer depth $L$, the deletion ratio $a$%, the substitution ratio $b$% and the user-item path length $T$ in ${\rm MPT}$. Due to the space limitation, we omit the results on MOOCs and Gowalla which have a similar trend to Ml-1M.

• •

  We only perform embedding reconstruction with GNNs in ${\rm MPT}$, and report the results in Fig. 8. We find that the performance first increases and then drops when increasing $L$ from 1 to 4. The peak point is 3 at most cases. This indicates GNN can only capture short-range dependencies, which is consistent with LightGCN’s (He et al., 2020) finding.

• •

  We only perform embedding reconstruction with Transformer encoder in ${\rm MPT}$, and report the results in Fig. 9. We find that the performance first improves when the path length $T$ increases from 3 to 8, and then drops from 8 to 10. This indicates Transformer encoder is good at capturing long-range rather than short-range dependencies of nodes.

• •

  We perform contrastive learning with LightGCN and contrastive learning with Transformer encoder under different deletion ratio $a$% and substitution $b$% independently, and report the recommendation results in Fig 10. Based on the results, we find that the recommendation performance is not sensitive to the substitution ratio $b$% but is a little bit sensitive to the deletion ratio $a$%. In terms of the deletion ratio $a$%, the recommendation performance first increases and then drops. The peak point is around 0.2-0.3 for most cases. The reason is that when we perform the substitution operation, we randomly replace $u$ or $i$ with one of its parent’s interacted first-order neighbors, thus the replaced nodes can still reflect its parent’s characteristic to some extent. However, once the deletion ratio gets larger (e.g., 0.9), it will result in a highly skewed graph structure, which can not bring much useful collaborative signals to the target node.