High-order Fusion Graph Contrastive Learning for Recommendation

Given $\mathcal{U}(|\mathcal{U}|=M)$ and $\mathcal{I}(|\mathcal{I}|=N)$ denote the set of users and items, respectively. In recommender system (Wu et al., 2023), Graph Convolutional Networks (GCN) (Wang et al., 2019; Wu et al., 2019; He et al., 2020) are increasingly favored, which through graph convolution operations, effectively gather high-order neighborhood information to accurately capture user preference behaviors. LightGCN (He et al., 2020) as currently the most popular GCN encoder, which effectively captures high-order information through neighborhood aggregation. For efficient training, graph Laplace matrix is proposed: $\tilde{A}=D^{-\frac{1}{2}}A{D^{-\frac{1}{2}}}$, where $\tilde{A}\in\mathbb{R}^{(M+N)\times(M+N)}$ denotes graph Laplacian matrix, and $A$ denotes adjacency matrix, and $D$ denotes diagonal matrix of A. To ensure that diverse neighborhood information is integrated, final embeddings are aggregated from all layers: $\mathbf{E}=\frac{1}{L+1}(\mathbf{E}^{(0)}+...+\tilde{A}^{L}\mathbf{E}^{(0)}),$ where $L$ denotes the number of GCN layers. The point-wise Bayesian Personalisation Ranking (BPR) loss (Rendle et al., 2009) is adopted to optimize model parameters:

where $<u,i,j>$ represents a triplet input to the model.

Recent studies have demonstrated that contrastive learning (CL), through the generation of self-supervised signals, effectively addresses the challenge of data sparsity in recommender systems (Xia et al., 2023). CL-based methods create contrastive views through data augmentation and seek to optimize the mutual information between positive pairs, thereby obtaining self-supervised signals. Most of the existing methods mainly use InfoNCE (Chen et al., 2020) loss for CL:

where $a$ and $b$ denote samples ($e.g.,$ user or item) in same batch $\mathcal{B}$, $\tau$ denotes the temperature coefficient. CL loss aims to maximizes the mutual information between positive pairs $\mathbf{e}_{a}$ and $\mathbf{e}_{b}$, and to increase the distance between $\mathbf{e}_{a}$ and other negative sample $\mathbf{e}_{c}$.

Existing researches on CL underscore the critical role of data augmentations (Wu et al., 2021; Yu et al., 2022b; Yang et al., 2023b; Zhang et al., 2024b), which is attributed to CL that necessitates the creation of diverse contrastive views for effective implementation. In recommendation, there are two main types in Fig. 1: (1) graph augmentation (Wu et al., 2021) (2) representation augmentation (Yu et al., 2022b).

Graph augmentation typically involves generating two distinct subgraphs by randomly discarding edges or nodes from the original graph: $\mathcal{G}^{\prime}=\mathrm{Dropout}(\mathcal{G(\underline{N},\underline{E})},p),$ where $\mathcal{G}$ denotes the original graph, $\mathcal{N}$ and $\mathcal{E}$ denote the nodes and edges, respectively, $\mathcal{G}^{\prime}$ denotes the augmentation subgraphs, $p$ denotes keeping rate. Representation augmentation by introducing noises into graph convolution process: $\mathbf{e}^{\prime}=\mathbf{e}+\triangle,$ where $\mathbf{e}$ denotes embedding, $\triangle$ denotes the added noise ($e.g.$, gaussian noise or uniform noise).

While these contrastive views are effective, data augmentations corrupt the embedding’s quality. In Fig. 2, graph augmentation methods discard edges when generating contrastive views, leading to relatively poorer embedding feature distributions and density distributions. Representation augmentation methods by adding noise to the graph convolution. Although the effect of some noise on the distribution is less pronounced than that of graph augmentation, it still results in less smooth density distribution. Additionally, data augmentations are exceptionally time-consuming. Table 1 provides the per-epoch time consumption for the GCN-based method ($e.g.,$ LightGCN) and GCL-based methods with ($e.g.,$ SGL and SimGCL) and without ($e.g.,$ HFGCL) data augmentations. SGL (Wu et al., 2021), which utilizes the graph augmentation technique to generate subgraphs, is extremely time-consuming. Additionally, most GCL-based ($e.g.,$ SimGCL and VGCL) methods require repetitive graph convolution or modeling operations to obtain the contrastive views, further contributing to the time consumption. This raises the question: Are data augmentation techniques necessary for contrastive learning in recommendation? In fact, despite the demonstrated effectiveness of data augmentations, how to efficiently generate high quality contrastive views remains a key challenge in CL.

Following the analysis in Section 2.3, we shift our attention to the traditional recommendation task. Inspired by the natural language processing (NLP) (Gao et al., 2022b), which generates contrastive views by applying the dropout function to input sentence. We believe that the contrastive views same exist in traditional recommendation task.

Considering a different perspective, we directly treat users and items as contrasting views. Given that users and items are first-order neighbors of each other, their high-order information are essentially identical. As graph convolution progresses, users and items gradually capture neighborhood information, resulting in their embeddings becoming progressively closer. Therefore, GCN process ultimately generates a set of distinctive contrastive views.

The previous contrastive views originate from the same sample ($e.g.,$ $\mathbf{e}_{u}^{\prime}$ and $\mathbf{e}_{u}^{\prime\prime}$ from $\mathbf{e}_{u}$), necessitating that a set of views can be selected as negative samples ($e.g.,$ $\sum_{i\in B}\mathbf{e}_{u_{i}}^{\prime\prime}$) to calculate CL losses.

However, considering that the user-item contrastive views differs from the previous contrastive views, we utilize the item views and user views as negative samples for the user and item, respectively.

Formally, we propose the CL loss of user-item contrastive views is different from Eq. 3 as follows:

where $\mathbf{e}_{u}$ and $\mathbf{e}_{i}$ are from $\mathbf{E}=\frac{1}{L+1}(\mathbf{E}^{(0)}+...+\tilde{A}^{L}\mathbf{E}^{(0)})$.

Many studies use aggregation methods to combine layer-wise embeddings into the final embeddings. Though this method effectively fuses information from both the sample itself and its neighborhoods, we find that the low-order information is extremely unique. This uniqueness can seriously affect the similarity between positive pairs, thereby interfering with the mutual information of them.

To make the positive pairs more similar, we eliminate the low-order information ($l\leq h-1$) and aggregate high-order information ($l\geq h$) as the final embeddings.

where $h$ denotes high-order information beginning at layer $h$. Unlike various GCL-based methods, by mining the properties in the existing embedding, we find the high-order contrastive views in the graph convolution process. This idea reduces the time cost while preserving the original embedding information, thus improving the efficiency and performance of the model.

In recommender systems, data augmentation typically involve augmenting the original user or item embeddings with multiple GCN operations to generate different contrastive views. When calculating CL loss, different views of the same sample are regarded as positive pairs, while a set of augmented views are selected as negative samples. Among them, it is not difficult to find that the negative samples used in the previous CL loss come from a set of augmented views. In contrast, for the high-order contrastive views propose in Section 3.2, we give the corresponding CL loss in Eq. 4. It is differently from the previous negative samples sampling approach, the negative samples come from the opposing views. This difference leads to the self-supervised signals of users and items themselves are missing.

To enable our method to better aggregate with more negative samples $\mathcal{B}_{ui}$, we present an advanced fusion CL loss that is specifically adapted to high-order contrastive views:

By separately calculating the similarity between view $a$ and view $b$ with all negative samples, we effectively fuse the self-supervised signals of different views and maximize the mutual information between positive pairs. Meanwhile, to effectively illustrate how HFGCL combines more negative samples, we show the positive pairs and selected negative samples for various contrastive methods in Fig. 4. Since our high-order contrastive views are derived from two different samples, in the selection of negative samples, unlike traditional contrastive views where only one set can be chosen as negative samples, we can simultaneously use both as negative samples for contrastive loss. Furthermore, our fusion CL loss can effectively aggregate this additional negative sample information and naturally fuse all the self-supervised signals.

Formally, we propose the evolution of HFGCL’s CL loss:

Through the study of relationships between positive pairs and negative samples, we have simplified the diverse contrastive objectives traditionally used to capture different self-supervised signals. By utilizing a fusion CL loss, we efficiently obtain the richer self-supervised signals, enabling personalized recommendations.

For the purpose of enhancing our contrastive model’s suitability for the primary recommendation task, we implement a multi-task training strategy to optimize the model’s parameters. The overall loss function for HFGCL is defined as follows:

where $\lambda_{1}$ and $\lambda_{2}$ denote the weights for $\mathcal{L}_{cl}$ and $\|\Theta\|^{2}_{2}$, $\Theta$ denotes the regularization parameter applied to the naive embedding $\mathbf{E}^{(0)}$.

The core motivation of this paper is to construct high-quality contrastive views and fuse self-supervised signals from different CL objectives. For the former, after proposing the user-item contrastive views, to find high-quality contrastive views, we utilize cosine similarity to measure the similarity between positive pairs ($i.e.,$ $\mathbf{e}_{u}$ and $\mathbf{e}_{i}$) in Table 2. For the latter, to better integrate different self-supervised signals, we propose a fusion paradigm applicable to contrastive views. Specifically, the fusion CL loss simultaneously considers the distance between both positive pairs and more negative samples, further improving the alignment of positive pairs and the uniformity of the sample space (Wang et al., 2022a). To further explain, we present the gradient of user $u$ in fusion CL loss:

where $f=\mathrm{exp}((\mathbf{e}_{u}^{\top}\mathbf{e}_{i})/\tau)$, $c(\cdot)$ is the contribution provided. For the first term $c(u)$, the contribution of item $i$ to the gradient encourages user $u$ to move closer to item $i$, directly strengthening the connection between the positive pair. For the second term $c(i,j)$, the contribution of item $i$ and negative sample $j$ to the gradient indicates that user $u$ must consider the relationship between positive item $i$ and negative sample $j$, thereby improving the distancing from hard negative samples filtered by item $i$. The above demonstrates the effectiveness of proposed fusion paradigm:

• •

  For user-item pairs, maximizing the mutual information directly reflects users’ deeper preferences.

• •

  For more negative sample pairs, effective fusion of information between positive and negative pairs makes the feature space more uniform distribution.

The same conclusion holds for the gradient of item $i$. Furthermore, for the temperature coefficient $\tau$, unlike previous work, we must assign a slightly larger $\tau$ to the fusion CL loss due to the inclusion of information from both positive pairs and more negative samples.

To illustrate the superior performance of HFGCL, we compare with all baselines in Table 4. The following observations are made:

• •

  Our HFGCL achieves the most superior performance compared to all baselines on three sparse datasets. Specifically, compared to the strongest baseline, HFGCL improves $w.r.t.$ NDCG@20 by 10.63%, 3.83%, and 3.41% on the Amazon-book, Yelp2018, and Tmall datasets, respectively. The experimental results are sufficient to show that HFGCL has a strong recommendation performance, enabling the provision of personalized recommendations.

• •

  Traditional MF-based method generally underperform compared to GNN-based methods, underscoring the substantial improvements that graph structures bring to recommender systems. Among these, LightGCN serves as the encoder for most GNN-based methods due to its straightforward and efficient architectural design. HFGCL uses this encoder, and by extracting the high-order information, we obtain the high-quality contrastive views thus further improving the method efficiency.

• •

  All GCL-based methods such as SimGCL, RecDCL, and BIGCF demonstrate clear superiority over traditional methods, largely due to the self-supervised signals derived from CL. Within this group, HFGCL achieves the best performance. This is mainly attributed to the high quality contrastive views and the ability to fuse different self-supervised signals with CL loss. Additionally, while RecDCL achieves sub-optimal performance $w.r.t.$ Recall@20 on the Amazon-book dataset, the use of the 2048 embedding size may compromise the fairness of comparisons with other methods. In contrast, HFGCL achieves superior recommendation performance by leveraging high-order contrastive for CL, without the need for any data augmentation.

To demonstrate the effectiveness of HFGCL’s components, we introduce various variants and conduct an ablation study:

• •

  $\mathrm{HFGCL}_{b}$: replace the CL loss with Eq. 4. ($0\leq l\leq L$)

• •

  $\mathrm{HFGCL}_{h}$: replace the CL loss with Eq. 4. ($l\geq h$)

• •

  $\mathrm{HFGCL}_{s}$: replace the CL loss with Eq. 9. ($l\geq h$)

• •

  $\mathrm{HFGCL_{w/o\,high}}$: use user-item contrstive views. ($0\leq l\leq L$)

• •

  $\mathrm{HFGCL_{w/o\,GCN}}$: remove the GCN encoder. ($l=0$)

• •

  $\mathrm{HFGCL_{w/o\,view}}$: replace the CL loss with Eq. 8. ($l\geq h$)

• •

  $\mathrm{HFGCL_{w/o\,CL}}$: remove the CL loss. ($l\geq h$)

From the results shown in Table 5, we draw the following conclusions. Firstly, HFGCL outperforms LightGCN across all performance metrics. For variant $\mathrm{HFGCL}_{b}$, which utilizes user-item contrastive views, the performance not only matches that of SGL but also significantly surpasses LightGCN. For variant $\mathrm{HFGCL}_{h}$, which eliminates low-order information, the CL loss effectively gets self-supervised signals from high-order information. For variant $\mathrm{HFGCL}_{s}$, with the addition self-sueprvies signals, there is a discernible enhancement in the method’s recommendation quality. These variants adequately address the question raised in Section 2.3 regarding the necessity of data augmentations, demonstrating that it is not essential for creating contrastive views.

Furthermore, for the fusion CL loss, we conduct a series of ablation studies to validate the effectiveness. For $\mathrm{HFGCL_{w/o\,high}}$, the results show that low-order information hinders similarity between positive pairs. For $\mathrm{HFGCL_{w/o\,GCN}}$, $\mathrm{HFGCL_{w/o\,view}}$ and $\mathrm{HFGCL_{w/o\,CL}}$, we remove the widely used GCN encoder, use the original CL loss and remove the CL loss, respectively, and the experimental results demonstrate the power of the fusion CL loss. Finally, compared to HFGCL, all ablations result in significant performance declines, which clearly demonstrate the superiority of the fusion CL loss.

To verify the efficiency of HFGCL, we compare the total training cost of the state-of-the-art methods. As shown in Fig. 5, our HFGCL has the shortest training cost on three datasets. This efficiency is primarily due to HFGCL quickly fuse self-supervised signals over the contrastive views, which enables it to converge exceedingly quickly. For SGL and SimGCL, both data augmentation necessitate graph convolution, a process that is notably time-consuming. For NCL and CGCL, though they also do not require data augmentations, it is again extremely time-consuming to compute all users/items from the user-item interaction as negative samples. Moreover, for most of the GCL-based methods (using data augmentations), the overall convergence speed is significantly impacted because the quality of the generated embedding information is perturbed to varying degrees. It is noteworthy that despite the absence of training cost data for RecDCL, it ranks as the most time-consuming method in the comparison, primarily because of its embedding size of 2048.

To show the ability of HFGCL in addressing the data sparsity issue, we categorize the users of the dataset into three groups—sparse, common, and popular—following methodologies from previous research. The experimental results are shown in Fig. 6. HFGCL maintains superior performance across different sparsity degrees, and performs exceptionally well within the sparsest user category. Additionally, on Amazon-book dataset, we observe a general decrease in performance in the normal category, possibly attributed to high levels of noise in this user segment. Even so, HFGCL maintains superior recommendation performance, demonstrating its robustness and resistance to sparsity.

In this section, we provide the sensitivity of HFGCL to various hyperparameters in Fig. 7 and Table 6. As depicted in Fig. 7, the method’s performance is significantly affected by the value of $\tau$, which is slightly higher compared to the previous method. This increase is primarily due to the elevated number of negative samples, necessitating a larger $\tau$ to ensure optimal acquisition of mutual information. However, HFGCL’s performance is barely affected by $\lambda_{1}$. We attribute this to our CL loss’s robust adaptation to the high-order contrastive views, ensuring the method performs optimally regardless of the $\lambda_{1}$ value. This efficacy underscores the strength of HFGCL’s CL loss. The results presented in Table 6 confirm the efficacy of high-order information, demonstrating that HFGCL significantly enhances method performance when $l\geq h$. This also confirms that the low-order information decreases the similarity between positive pairs, leading to suboptimal recommendation.