Overcoming Pitfalls in Graph Contrastive Learning Evaluation: Toward Comprehensive Benchmarks

Graph Neural Networks (GNNs) (Li et al., 2015; Gilmer et al., 2017; Veličković et al., 2017) have emerged as a powerful tool for learning representations from graph-structured data, demonstrating remarkable success across various fields including social network analysis (Kipf and Welling, 2016; Fan et al., 2019; Goldenberg, 2021), molecular biology (Duvenaud et al., 2015; You et al., 2018; Kearnes et al., 2016), recommendation systems (Monti et al., 2017; Ying et al., 2018; Ma et al., 2019), and traffic prediction (Wu et al., 2019; Ma et al., 2023; Li et al., 2017). By leveraging the rich relational information inherent in graphs, GNNs can capture complex patterns that traditional neural network architectures struggle to process. However, the efficacy of GNNs is largely contingent upon the availability of task-dependent labels to learn meaningful representations (Hamilton, 2020; Xu et al., 2018; Zhang et al., 2020; Liu et al., 2022). Unlike labeling in more common modalities such as images, videos, texts, and audio, annotating graphs presents significant challenges due to the complex and often domain-specific nature of graph data (Liu et al., 2022; Hassani and Khasahmadi, 2020; Dai et al., 2022). This has led to a growing interest in self-supervised learning methods as a means to circumvent the limitations imposed by the need for extensive labeled datasets. Among the various categories of self-supervised learning, graph contrastive learning (GCL) has emerged as a particularly promising approach (Veličković et al., 2018; Zhu et al., 2020, 2021c; Zhang et al., 2021; Thakoor et al., 2021; Mo et al., 2022; Zhang et al., 2023).

The primary goal of GCL is to pre-train an encoder capable of generating high-quality graph representations without relying on label information, with the hope that these representations can be effectively utilized across a wide array of downstream tasks (He et al., 2020; Chen et al., 2020; Gao et al., 2021; Liu et al., 2022). However, the current evaluation protocols for GCL methods exhibit significant shortcomings that fail to align with these fundamental goals as detailed below:

• •

  GCL methods typically consist of numerous hyper-parameters due to their unique designs in augmentation methods or contrastive objectives. We provide a detailed review of several representative GCL methods in Section 2. In existing evaluation protocols, these hyper-parameters are often tuned for each graph dataset, which plausibly involves the use of a validation set from a downstream task. However, in practice, this process contradicts the premise of pre-training without task-specific labels. This would be particularly problematic if the GCL methods are highly sensitive to hyper-parameter configurations.

• •

  In existing evaluation procedures, the evaluation of GCL methods is predominantly focused on a single downstream task, usually node classification, conducted on the same dataset used for encoder pre-training. Such a limited evaluation framework may not accurately reflect the encoder’s versatility across diverse tasks, potentially leading to misleading comparisons among different GCL methods.

In our study, we conduct a thorough empirical analysis focusing on two crucial elements: (a) how GCL methods’ performance is affected by hyper-parameter adjustments in the pre-training phase, and (b) the extent to which a single downstream task can accurately reflect the overall efficacy of GCL methods. Our investigation, detailed in Section 4, confirms that the current evaluation systems are indeed compromised by these concerns. Particularly, we observe that some GCL methods are highly sensitive to the settings of hyper-parameters. Although these methods can perform well when hyper-parameters are optimally tuned, their effectiveness can substantially decrease with less-than-ideal settings, making them less practical for pre-training situations where adapting hyper-parameters for each specific downstream task is impractical. Despite this, such methods may still show strong results within the current evaluation models, which do not fully account for the challenges posed by their sensitivity to hyper-parameters (Zhang et al., 2023; Zhu et al., 2020). Additionally, evaluating these methods based on a single downstream task often does not provide a complete picture of their capabilities, as the performance of GCL methods can vary significantly across different tasks. This inconsistency highlights the limitations of current evaluation approaches that focus on singular tasks, thus failing to offer a comprehensive assessment of GCL methods’ overall performance.

Hence, to address the identified issues, we propose a new evaluation protocol aimed at a more comprehensive and accurate assessment of GCL methods. This improved protocol incorporates a comprehensive analysis across diverse hyper-parameter configurations and extends the evaluation to include multi-label dataset evaluation. Through these enhancements, our protocol seeks to mitigate the impact of hyper-parameter sensitivity and broaden the scope of evaluation beyond a single task, thereby offering a more comprehensive understanding of GCL method performance.

Graph Contrastive Learning (GCL) focuses on learning superior node representations by distinguishing between node pairs that share similar semantics and those that do not. An anchor node, which can be any selected node in augmented graphs, is paired with semantically similar nodes or graphs (positive examples) to create positive pairs and with dissimilar nodes or graphs (negative examples) to establish negative pairs. GCL models aim to map graph elements such as nodes or graphs into an embedding space where positive pairs are pulled closely together while negative pairs are pushed far apart. To accomplish this, GCL methods are designed with a variety of components. As well-summarized in a prior work (Zhu et al., 2021a), GCL methods are normally designed differently in the following key components:

• •

  Data Augmentation: Data augmentation in GCL aims to create variations of a graph that preserve its major semantic information, thereby helping models to create positive and negative examples. This involves two main approaches: topology transformations and feature transformations. Topology augmentations adjust the graph structure through methods like edge dropping etc. Feature augmentations alter node features by masking to generate diverse representations. The common hypeter-parameters derived from these components are drop edge rate and drop feature rate.

• •

  Contrastive Mode: In GCL, contrasting modes define how an anchor node’s similar and dissimilar samples are selected for comparison. There are three primary modes: local-local contrasts nodes at the same level, global-global contrasts entire graph embeddings, and global-local contrasts nodes with their graph-level representation. The choice of mode depends on the task, with node-focused tasks typically using local-local and global-local modes.

• •

  Contrastive Objective: Contrastive objectives quantify the similarity between positive pairs and the difference from negative pairs. Common objectives include Information Noise Contrastive Estimation (InfoNCE) and Jensen-Shannon Divergence (JSD), which require explicit negative sampling. Other objectives like Bootstrapping Latent Loss (BL) and Barlow Twins (BT) do not require negatives, focusing instead on enhancing positive pair similarity and feature diversity. The temperature hyper-parameter $\tau$ in InfoNCE is typically adjustable in GCL methods utilizing it as the contrastive objective.

Building on the general paradigm of GCL outlined above, we next briefly introduce the benchmarked methods, highlighting their unique design components and the corresponding hyper-parameters used to enhance graph representation learning. (1) DGI: As a pioneer work of GCL, DGI (Veličković et al., 2018) maximizes mutual information between global graph embeddings and local node embeddings, leveraging JSD as its contrastive loss. Under its design, it has two hyper-parameters to vary, hidden dimension and layer number of the backbone GCN used to extract representations. (2) MVGRL (Hassani and Khasahmadi, 2020) leverages multiple graph views to capture comprehensive structural patterns, applying different GCN configurations for each view to enrich node representations. MVGRL is with the same set of adjustable hyper-parameters as the DGI, but two backbone GCNs are applied on different augmented views. (3) GRACE (Zhu et al., 2020) focuses on local node-level embedding contrast between two randomly augmented graph views. Each view is copied from the input graph first, next undergoing modifications where a certain percentage of edges and node features are removed and masked, determined by the specified drop edge rate and drop feature rate, respectively. Different from JSD loss adopted DGI and MVGRL, the adopted contrastive loss for GRACE is InfoNCE and therefore it also has one additional hyper-parameter $\tau$ to be determined. Further, (4) BGRL (Thakoor et al., 2021) utilizes Bootstrapping Latent loss for graph learning without negative samples, with similar two augmented graph views adjustable with drop edge rate and drop feature rate. (5) Graph Barlow Twins (Bielak et al., 2022) aims to make the cross-correlation matrix of embeddings from two views as close to the identity matrix as possible, using different hyper-parameters drop edge rate and drop node rate for graph augmentations on two views. (6) CCA-SSG (Zhang et al., 2021) first generates shared node representations from two augmented graph views with similar hyper-parameters mentioned before, then using Canonical Correlation Analysis to maximize correlation between views and decorrelate feature dimensions within each view. CCA-SSG has an additional hyper-parameter loss reweighting factor for the trade-off between its losses. (7) SUGRL (Mo et al., 2022) simplifies the contrastive learning process by omitting graph augmentation and similarity determination steps, focusing instead on increasing inter-class differences and decreasing intra-class differences with a triplet loss and an upper bound loss. Given this specific design, there are four unique hyper-parameters for SUGRL, loss reweighting factor, iteration number for its unique negative samples generation module, margin parameter, a hyper-parameter of the triplet loss that penalizes a small gap in distances between a positive pair and a negative pair, and a bound parameter, a non-negative tuning parameter used in the upper bound loss. Recently, (8) COSTA (Zhang et al., 2022) is proposed to address biases in graph augmentation by employing feature augmentation, allowing for better control over data distribution in the latent space. COSTA has $\tau$ for its InfoNCE contrastive loss and drop edge rate and drop node rate for two different augmented views. In addition, (9) SFA (Zhang et al., 2023) proposed spectral feature augmentation for GCL, which is designed to re-balance the feature spectrum by iteratively removing low-rank information from the feature matrix. Therefore, apart from drop edge rate and drop node rate, there are two more hyper-parameters for SFA: $k$ as the number of iterations used in its spectral feature augmentation, and again $\tau$ in its adopted InfoNCE loss.

The primary objective of graph contrastive learning is to pre-train an encoder capable of producing high-quality graph representations without label information, with the anticipation that these representations can be efficiently applied to a diverse range of downstream tasks. However, the existing evaluation protocol for graph contrastive learning methods deviates from the aforementioned goals, thus inadequately assessing their efficacy. Specifically, existing evaluation protocols have the following deficiencies:

• •

  As elucidated in Section 2, GCL methods typically involve numerous hyper-parameters during the pre-training stage. In the current evaluation protocol, these hyper-parameters are typically optimized for each graph dataset (Zhu et al., 2020; Zhang et al., 2023). The hyper-parameter selection is plausibly carried out with a validation set of a downstream task—an approach that is ideally not assumed during the pre-training phase of the encoder. This is essentially problematic especially if the encoders are sensitive to the hyper-parameters.

• •

  The existing evaluation protocol predominantly concentrates on a single downstream task, i.e., node classification on the same dataset the encoder is trained. Such a constrained assessment framework may inadequately capture the encoder’s performance across the entire task space, rendering comparisons between different GCL methods potentially misleading. In particular, if the downstream task does not effectively represent the diverse range of possible applications, the basis for comparing GCL methods becomes flawed, as it fails to reflect the true generality and adaptability of the encoder’s learned representations.

In this section, we undertake empirical experiments to substantiate the aforementioned concerns. Our experiments seek to investigate the following two research questions:

• •

  What is the extent of sensitivity of GCL methods to the hyper-parameters during the pre-training stage?

• •

  Is a single downstream task representative enough to comprehensively assess the performance of GCL methods?

To address the aforementioned issues, we introduce an improved protocol for a more comprehensive evaluation. This protocol is designed to address the limitations of hyper-parameter sensitivity and the narrow focus on single downstream tasks. The improved evaluation framework consists of the following components.

• •

  Comprehensive Analysis Across Diverse Hyper-parameter Configurations. We advocate for evaluating GCL methods based on their performance across a variety of hyper-parameter configurations. This approach includes analyzing both the average performance and the variability (or variance) of these performances. By considering the variance, we gain insights into the stability of the GCL methods under different hyper-parameter settings, providing a more holistic view of their effectiveness.

• •

  Extension to Multi-label Dataset Evaluation. Our protocol expands the scope of evaluation to include multi-label datasets, which allows us to simulate an environment with multiple downstream tasks. Evaluating GCL methods in this context offers a richer perspective on their capacity to generalize and adapt to diverse downstream tasks.

In this paper, we illustrate the intrinsic limitations of the existing evaluation protocol of GCL methods, highlighting its divergence from the overarching goals of self-supervised learning. We approach this assessment from two angles: the numerous hyper-parameters during the pre-training phase and the sole evaluation based on a single downstream task. We verify the prominent issues by investigating the sensitivities of GCL methods to hyper-parameters during the pre-training stage and the representativeness of the downstream task. Moreover, to rectify the aforementioned issues, we propose an improved evaluation protocol for better benchmarking to comprehensively evaluate the GCL methods.