Fusion Self-supervised Learning for Recommendations

Given $\mathcal{U}(|\mathcal{U}|=M)$ and $\mathcal{I}(|\mathcal{I}|=N)$ denote the set of users and items, respectively. In recommender system (Wu et al., 2023a), Graph Convolutional Networks (GCN) (Wang et al., 2019; Wu et al., 2019; He et al., 2020) are increasingly favored, which through graph convolution operations, effectively gather high-order neighborhood information to accurately capture user preference behaviors. LightGCN (He et al., 2020) as currently the most popular GCN encoder, which effectively captures high-order information through neighborhood aggregation. For efficient training, graph Laplace matrix is proposed: $\tilde{A}=D^{-\frac{1}{2}}A{D^{-\frac{1}{2}}}$, where $\tilde{A}\in\mathbb{R}^{(M+N)\times(M+N)}$ denotes graph Laplacian matrix, and $A$ denotes adjacency matrix, and $D$ denotes diagonal matrix of $A$. To ensure that diverse neighborhood information is integrated, final embeddings are aggregated from all layers: $\mathbf{E}=\frac{1}{L+1}(\mathbf{E}^{(0)}+...+\tilde{A}^{L}\mathbf{E}^{(0)}),$ where $L$ denotes the number of GCN layers. The point-wise Bayesian Personalisation Ranking (BPR) loss (Rendle et al., 2009) is adopted to optimize model parameters:

where $\langle u,i,j\rangle$ represents a triplet input to the model.

Recent studies (Wu et al., 2021) have demonstrated that contrastive learning (CL), the primary paradigm in self-supervised learning, effectively addresses the challenge of data sparsity in recommender systems through the generation of self-supervised signals (Xia et al., 2023). CL-based methods create contrastive views through data augmentation and seek to optimize the mutual information between positive pairs, thereby obtaining self-supervised signals. Most of the existing methods mainly use InfoNCE (Chen et al., 2020) loss for CL:

where $\langle a,b\rangle$ denote positive pair, assumed here as $\langle a_{2},b_{3}\rangle$ in same batch $\mathcal{B}=\{\langle a_{1},b_{2}\rangle,\langle a_{2},b_{3}\rangle,...,\langle a_{m},b_{n}\rangle\}$, $c$ denotes all samples of $\mathcal{B}_{b}=\{b_{2},b_{3},...,b_{n}\}$ in batch $\mathcal{B}$, $\tau$ denotes the temperature coefficient. CL loss aims to maximizes the mutual information between positive pairs $\mathbf{e}_{a}$ and $\mathbf{e}_{b}$, and to increase the distance between $\mathbf{e}_{a}$ and other negative sample $\mathbf{e}_{c}$.

Existing researches on CL underscore the critical role of data augmentations (Wu et al., 2021; Yu et al., 2022; Yang et al., 2023b; Zhang et al., 2024b), which is attributed to CL that necessitates the creation of diverse contrastive views for effective implementation. In recommendation, there are two main types in Fig. 1: (1) graph augmentation (Wu et al., 2021) (2) feature augmentation (Yu et al., 2022).

Graph augmentation typically involves generating two distinct subgraphs by randomly discarding edges/nodes from original graph:

where $\mathcal{G}$ denotes the original graph, $\mathcal{N}$ and $\mathcal{E}$ denote the nodes and edges, respectively, $\mathcal{G}^{\prime}$ and $\mathcal{G}^{\prime\prime}$ denotes the augmentation subgraphs, $p$ denotes keeping rate. Then, we perform graph convolution operations by $\mathcal{G}^{\prime}$ and $\mathcal{G}^{\prime\prime}$ to obtain different views $\mathbf{e}^{\prime}$ and $\mathbf{e}^{\prime\prime}$ for the same node $\mathbf{e}$. Unlike graph augmentation, feature augmentation has garnered significant attention in numerous studies by introducing noises into graph convolution process:

where $\triangle$ denotes the added noise ($e.g.$, gaussian noise or uniform noise). After obtaining the contrastive views, contrastive learning is performed using the CL loss in Eq. 2, $\mathcal{L}_{cl}=\mathcal{C}(\mathbf{e}^{\prime},\mathbf{e}^{\prime\prime})$, effectively extracting self-supervised signals from different views to alleviate the data sparsity problem.

Despite these contrastive views are effective in improving performance (as shown in Table 2), data augmentation has a significant drawback: generating contrastive views each epoch adds extra time, significantly increasing training costs. Table 1 provides the per-epoch time consumption for the GCN-based method ($e.g.,$ LightGCN (He et al., 2020)) and GCL-based methods with ($e.g.,$ SGL-ED (Wu et al., 2021), SimGCL (Yu et al., 2022), NCL (Lin et al., 2022), CGCL (He et al., 2023) and VGCL (Yang et al., 2023b)). For fair comparison, all methods are tested under the same experimental setup, with details provided in Section 5.1. SGL-ED, which utilizes the graph augmentation technique to generate subgraphs, is extremely time-consuming. Additionally, most GCL-based ($e.g.,$ SimGCL) methods require repetitive graph convolution or modeling operations to obtain the contrastive views, further contributing to the high time cost. This raises an important question: Is it possible to propose a novel contrastive learning paradigm that does not rely on data augmentation techniques, but still generates high-quality contrastive views and improves training efficiency? While data augmentations have proven effective in generating contrastive views and enhancing recommendation performance, efficiently generating high-quality contrastive views remains a key challenge in contrastive learning.

To further investigate the optimization mechanism of GCL-based recommendation methods, we present the number of contrastive views generated by these methods in Table 2. Specifically, the listed methods obtain different perspectives of the same node ($e.g.,$ e) and treat each pair of generated views ($e.g.,$ view1=$\mathbf{e}^{\prime}$ and view2=$\mathbf{e}^{\prime\prime}$) as a positive pair ($e.g.,$ $\langle\mathrm{viwe1},\mathrm{viwe2}\rangle$) for contrastive learning. This approach allows them to leverage additional self-supervised signals to optimize model performance. As shown in the statistics, SimGCL, compared to SGL-ED, does not generate additional contrastive views but instead focuses on optimizing data augmentation strategies. In contrast, subsequent methods such as NCL, CGCL, and VGCL, shift their focus toward generating more contrastive views ($i.e.,$ significantly increasing the number of views). While these methods demonstrate improvements over baselines in their original studies, our experimental results (Table 2) reveal that the gains in recommendation performance are modest. Notably, these newer methods often fail to surpass the earlier SimGCL.

Our experimental results suggest that while generating more contrastive views can provide richer self-supervised signals, the subtle differences among these signals pose significant challenges for effective utilization. Consequently, methods that prioritize generating a larger number of contrastive views often exhibit relatively inferior performance compared to SimGCL, which with high-quality contrastive views. As discussed in the previous section, identifying and generating high-quality contrastive views remains a critical yet often overlooked challenge in contrastive learning. Furthermore, as suggested by previous studies (Yang et al., 2023b), although self-supervised signals from different views demonstrate potential in alleviating data sparsity, the nuanced discrepancies between these signals significantly hinder their seamless fusion, ultimately limiting the effectiveness of these new methods.

Following the analysis in Section 3, we urgently need to find a novel contrastive learning paradigm. Inspired by the natural language processing (NLP) (Gao et al., 2022b), which generates contrastive views by applying the dropout function to input sentence. We believe that similar contrastive views also exist in graph-based recommendation tasks.

Considering a different perspective, we directly treat users and items as contrastive views. Given that users and items are first-order neighbors of each other, their high-order information are essentially identical. As graph convolution (GCN) progresses, users and items gradually capture neighborhood information, resulting in their embeddings becoming progressively closer. Therefore, GCN process ultimately generates a set of distinctive contrastive views.

The previous contrastive views originate from the same sample ($e.g.,$ $\mathbf{e}_{u}^{\prime}$ and $\mathbf{e}_{u}^{\prime\prime}$ from $\mathbf{e}_{u}$, as shown in Eq. 4), necessitating that a set of views can be selected as negative samples ($e.g.,$ $\sum_{i\in B}\mathbf{e}_{u_{i}}^{\prime\prime}$):

However, considering that the user-item contrastive views differs from the previous contrastive views, we utilize the item views and user views as negative samples for the user and item, respectively.

Formally, we propose the CL loss of user-item contrastive views is different from Eq. 5 as follows:

where $\mathbf{e}_{u}$ and $\mathbf{e}_{i}$ are from $\mathbf{E}=\frac{1}{L+1}(\mathbf{E}^{(0)}+...+\tilde{A}^{L}\mathbf{E}^{(0)})$.

Many studies (Wang et al., 2019) use aggregation functions to integrate layer-wise embeddings into the final encoder. Though the encoder effectively aggregates information from both the sample itself and its neighborhoods, we find that the low-order information is extremely unique. This uniqueness can seriously affect the similarity between positive pairs, thereby interfering with the mutual information of them.

To make the positive pairs more similar, we eliminate the low-order information ($l\leq h-1$) and aggregate high-order information ($l\geq h$) as the final embeddings.

where $h$ denotes high-order information beginning at layer $h$. Unlike various GCL-based methods, by mining the properties in the existing embedding, we find the high-order contrastive views in the graph convolution process. This idea reduces the time cost while preserving the original embedding information, thus improving the efficiency and performance of the model.

In recommender systems, data augmentation typically involve augmenting the original user or item embeddings with multiple GCN and model modeling operations to generate different contrastive views. When calculating CL loss, different views of the same sample are regarded as positive pairs, while a set of augmented views are selected as negative pairs. Among them, it is not difficult to find that the negative pairs used in the previous CL loss come from a set of augmented views. In contrast, for the high-order contrastive views propose in Section 4.2, we give the corresponding CL loss in Eq. 6. It is differently from the previous negative pairs sampling approach, the negative pairs come from the opposing views. This difference leads to the self-supervised signals of users and items themselves are missing.

To eliminate such biases, we propose a fused embedding representation $\mathbf{e}^{*}_{ui}$. Specifically, we dynamically reconstruct and fuse the high-order neighborhood information of users and items:

where $\mathcal{Z}_{u}$ and $\mathcal{Z}_{i}$ denote the neighbors of users and items, respectively, $\alpha$ represents the parameter to adjust the contribution of user and item embeddings in the fusion process, $\mathbf{e}_{u}^{(h)}$ and $\mathbf{e}_{i}^{(h)}$ denote the embeddings obtained by stacking multiple GCN layers on the initial embeddings $\mathbf{e}_{u}^{(0)}$ and $\mathbf{e}_{u}^{(0)}$, respectively. Subsequently, to better integrate the contrastive objectives generated by user, item, and user-item contrastive views, we present an advanced fusion CL loss that is specifically adapted to high-order contrastive views:

By calculating the similarity between fused embedding $\mathbf{e}^{*}_{ui}$ and richer negative pairs $\mathcal{B}_{ui}$, we can effectively integrate the self-supervised signals from diverse views and maximize the mutual information between the positive pair $\langle u,i\rangle$. Meanwhile, to effectively illustrate how HFGCL optimizes the model, we provide the training process in Fig. 3 and demonstrate the signal fusion process. Since our high-order contrastive views select user-item interactions as positive pairs, unlike traditional contrastive methods that can only choose one set as negative pairs, we can simultaneously treat two sets of samples as negative pairs $\mathcal{B}_{ui}$ for contrastive learning. Furthermore, our fusion CL loss can effectively aggregate this additional negative pairs information and naturally fuse all the self-supervised signals.

Formally, we propose the evolution of HFGCL’s CL loss:

Through studying the relationships between multi-scale contrastive objectives, we simplify the various contrastive objectives traditionally used to capture different self-supervised signals. By utilizing a fusion CL loss, we efficiently obtain the richer self-supervised signals, enabling personalized recommendations.

For the purpose of enhancing our contrastive method’s suitability for the primary recommendation task, we implement a multi-task training strategy to optimize the method’s parameters. The overall loss function for HFGCL is defined as follows:

where $\lambda_{1}$ and $\lambda_{2}$ denote the weights for $\mathcal{L}_{cl}$ and $\|\Theta\|^{2}_{2}$, $\Theta$ denotes the regularization parameter applied to the naive embedding $\mathbf{E}^{(0)}$.

To illustrate the superior performance of HFGCL, we compare with all baselines in Table 6. The following observations are made:

• •

  Our HFGCL achieves the most superior performance compared to all baselines on three sparse datasets. Specifically, compared to the strongest baseline, HFGCL improves $w.r.t.$ NDCG@20 by 10.63%, 3.83%, and 3.41% on the Amazon-book, Yelp2018, and Tmall datasets, respectively. The experimental results are sufficient to show that HFGCL has a strong recommendation performance, enabling the provision of personalized recommendations.

• •

  Traditional MF-based method generally underperform compared to GNN-based methods, underscoring the substantial improvements that graph structures bring to recommender systems. Among these, LightGCN serves as the encoder for most GNN-based methods due to its straightforward and efficient architectural design. HFGCL uses this encoder, and by extracting the high-order information, we obtain the high-quality contrastive views thus further improving the method efficiency.

• •

  All CL-based methods such as SimGCL, RecDCL, and BIGCF demonstrate clear superiority over traditional methods, largely due to the self-supervised signals derived from CL. Within this group, HFGCL achieves the best performance. This is mainly attributed to the high quality contrastive views and the ability to fuse different self-supervised signals with CL loss. Additionally, while RecDCL achieves sub-optimal performance $w.r.t.$ Recall@20 on the Amazon-book dataset, the use of the 2048 embedding size may compromise the fairness of comparisons with other methods. In contrast, HFGCL achieves superior recommendation performance by leveraging high-order contrastive for CL, without the need for any data augmentation.

To demonstrate the effectiveness of HFGCL’s components, we introduce various variants and conduct an ablation study:

• •

  $\mathrm{HFGCL}_{b}$: replace the CL loss with Eq. 6. ($0\leq l\leq L$)

• •

  $\mathrm{HFGCL}_{h}$: replace the CL loss with Eq. 6. ($l\geq h$)

• •

  $\mathrm{HFGCL}_{s}$: replace the CL loss with Eq. 11. ($l\geq h$)

• •

  $\mathrm{HFGCL_{w/o\,high}}$: use user-item contrstive views. ($0\leq l\leq L$)

• •

  $\mathrm{HFGCL_{w/o\,GCN}}$: remove the GCN encoder. ($l=0$)

• •

  $\mathrm{HFGCL_{w/o\,view}}$: replace the CL loss with Eq. 10. ($l\geq h$)

• •

  $\mathrm{HFGCL_{w/o\,CL}}$: remove the CL loss. ($l\geq h$)

From the results shown in Table 7, we draw the following conclusions. Firstly, HFGCL outperforms LightGCN across all performance metrics. For variant $\mathrm{HFGCL}_{b}$, which utilizes user-item contrastive views, the performance not only matches that of SGL-ED but also significantly surpasses LightGCN. For variant $\mathrm{HFGCL}_{h}$, which eliminates low-order information, the CL loss effectively gets self-supervised signals from high-order information. For variant $\mathrm{HFGCL}_{s}$, with the addition self-sueprvies signals, there is a discernible enhancement in the method’s recommendation quality. These variants adequately address the question raised in Section 3.1 regarding the necessity of data augmentations, demonstrating that it is not essential for creating contrastive views.

Furthermore, for the fusion CL loss, we conduct a series of ablation studies to validate the effectiveness. For $\mathrm{HFGCL_{w/o\,high}}$, the results show that low-order information hinders similarity between positive pairs. For $\mathrm{HFGCL_{w/o\,GCN}}$, $\mathrm{HFGCL_{w/o\,view}}$ and $\mathrm{HFGCL_{w/o\,CL}}$, we remove the widely used GCN encoder, use the original CL loss and remove the CL loss, respectively, and the experimental results demonstrate the power of the fusion CL loss.

To verify the efficiency of HFGCL, we compare the total training cost of the state-of-the-art methods. As shown in Fig. 4, our HFGCL has the shortest training cost on three datasets. This efficiency is primarily due to HFGCL quickly fuse self-supervised signals over the contrastive views, which enables it to converge exceedingly quickly. For SGL-ED and SimGCL, both data augmentation necessitate graph convolution, a process that is notably time-consuming. For NCL and CGCL, though they also do not require data augmentations, it is again extremely time-consuming to compute all users/items from the user-item interaction as negative samples. Moreover, for most of the GCL-based methods (using data augmentations), the overall convergence speed is significantly impacted because the quality of the generated embedding information is perturbed to varying degrees. It is noteworthy that despite the absence of training cost data for RecDCL, it ranks as the most time-consuming method in the comparison, primarily because of its embedding size of 2048.

To show the ability of HFGCL in addressing the data sparsity issue, we categorize the users of the dataset into three groups—sparse, common, and popular—following methodologies from previous research. The experimental results are shown in Fig. 5. In the sparsity study, we focus on the performance of the method in the sparse user group. Compared to existing SOTA methods, SimGCL and BIGCF, HFGCL shows significant performance improvements across all three datasets. Additionally, HFGCL maintains excellent performance across different levels of sparsity. On the Amazon-book dataset, we observe a general decline in performance for the regular user category, possibly due to higher noise levels in this group. Nevertheless, HFGCL still demonstrates strong recommendation performance, proving its robustness and resistance to sparsity.

In this section, we provide the sensitivity of HFGCL to various hyperparameters in Fig. 6 and Table 8. As depicted in Fig. 6, the method’s performance is significantly affected by the value of $\tau$, which is slightly higher compared to the previous method. This increase is primarily due to the elevated number of negative samples, necessitating a larger $\tau$ to ensure optimal acquisition of mutual information. However, HFGCL’s performance is barely affected by $\lambda_{1}$. We attribute this to our CL loss’s robust adaptation to the high-order contrastive views, ensuring the method performs optimally regardless of the $\lambda_{1}$ value. This efficacy underscores the strength of HFGCL’s CL loss. Table 8 shows the performance results when the high-order initialization layer $h$ of HFGCL is set to 1, 2 and 3. The results confirm the effectiveness of high-order information, indicating that when $h\leq l\leq L$, HFGCL significantly improves performance. It also shows that stacking low-order information causes nodes to contain excessive self-information, reducing the similarity between positive pairs and resulting in suboptimal recommendations.