GC-Bench: An Open and Unified Benchmark for Graph Condensation

Regarding evaluation datasets, we adapt the 12 widely-used datasets in the current literature. The node level dataset include 5 homogeneous dataset (Cora [16], Citeseer [16], ogbn-arxiv [13], Flickr [43], Reddit [12]) and 2 heterogeneous datasets (ACM [46] and DBLP [7]). The graph-level dataset include NCI1 [32], DD [4], ogbg-molbace [13], ogbg-molbbbp [13], ogbg-molhiv [13]. We leverage the public train, valid, and test split of these datasets. We report the dataset statistics in Appendix A.1.

We selected 12 representative and competitive GC methods across 6 categories for evaluation. The main ideas of these methods are shown in Figure 1. The evaluated methods include: (1) traditional core-set methods including Random, Herding [36], K-Center [30], (2) gradient matching methods including DosCond [14], GCond [15] and SGDD [42], (3) trajectory matching methods including SFGC [47] and GEOM [45], (4) distribution matching methods including GCDM [20] and DM [22; 24], (5) Kernel Ridge Regression (KRR) based method KiDD [41], and (6) Computation Tree Compression (CTC) method Mirage [11]. The details of evaluated methods are in Appendix A.2.

We systematically design the GC-Bench to comprehensively evaluate the existing GC algorithms and inspire future research. In particular, we aim to investigate the following research questiones.

RQ1: How much progress has been made by existing GC methods?

Motivation and Experiment Design. Previous GC methods always adopt different experimental settings, making it difficult to compare them fairly. Given the unified settings of GC-Bench, the first question is to revisit the progress made by existing GC methods and provide potential enhancement directions. A good GC method is expected to perform well consistently under different datasets and different condensation ratios. To answer this question, we evaluate GC methods’ performance on 7 node-level datasets and 5 graph-level datasets with a broader range of condensation ratio $r$ than previous works. The results are shown in Sec. 3.1 and Appendix B.1.

RQ2: How do the potential flaws of the structure affect the graph condensation performance?

Motivation and Experiment Design. Most of the current GC methods borrow the idea of image condensation and overlook the specific non-IID properties of irregular graph data. The impact of structural properties in GC is still not thoroughly explored. On one hand, the structure itself possesses various characteristics such as homogeneity and heterogeneity, as well as homophily and heterophily. It remains unclear whether these properties should be preserved in GC and how to preserve them. On the other hand, some structure-free GC methods [47] suggest that the condensed dataset may not need to explicitly preserve graph structure, and preserving structural information in the generated samples is sufficient. To answer this question, we evaluated GC methods on both homogeneous and heterogeneous as well as homophilic and heterophilic datasets to further explore the impact of structural properties. The results are shown in Sec. 3.2 and Appendix B.2.

RQ3: Can the condensed graphs be transferred to different types of tasks?

Motivation and Experiment Design. Most existing GC methods are primarily designed for node classification and graph classification tasks. However, there are numerous downstream tasks on graph data, such as link prediction and anomaly detection, which focus on different aspects of graph data. The transferability of GC across various graph tasks has yet to be thoroughly explored. To answer this question, we perform condensation guided by node classification and use the condensed dataset to train models for 3 classic downstream tasks: link prediction, node clustering, and anomaly detection. The evaluation results are shown in Sec. 3.3 and Appendix B.3.

RQ4: How does the backbone model architecture used for condensation affect the performance?

Motivation and Experiment Design. The backbone model plays an important role in extracting the critical features of the original dataset and guiding the optimization process of generating the condensed dataset. Most GC methods choose a specific graph neural network (GNN) as the backbone. The impact of the backbone model architecture and its transferability is under-explored. A high-quality condensed dataset is expected to be used for training models with not only the specific one used for condensation but also various architectures. To answer this question, we evaluate the transferability performance for 5 representative GNN models (SGC [37], GCN [16], GraphSAGE [12], APPNP [18], and ChebyNet [2]) and 2 non-GNN models (the popular Graph Transformer [31]) and simple MLP). We also investigated the performance variation of different backbones with the number of training steps. The evaluation results are shown in Sec. 3.4 and Appendix B.4.

RQ5: How does the initialization mechanism affect the performance of graph condensation?

Motivation and Experiment Design. The initialization mechanism of the condensed dataset is crucial for convergence and performance in image dataset condensation but remains unexplored for irregular graph data. To answer this question, we adopt 5 distinct initialization strategies (Random Noise, Random Sample, Center, K-Center, and K-Means) to evaluate their impact on condensation performance and converge speed. The results are shown in Sec. 3.5 and Appendix B.5.

RQ6: How efficient are these GC methods in terms of time and space?

Motivation and Experiment Design. As the GC methods aim to achieve comparable performance on the condensed dataset and the original dataset, they always rely on the training process on the original datasets. The efficiency and scalability of GC methods are overlooked by existing methods, which is crucial in practice since the original intent of GC is to reduce computation and storage costs for large graphs. To answer this question, we evaluate the time and memory consumption of these GC methods. Specifically, we record the overall time when achieving the best result, the peak CPU memory, and the peak GPU memory. The results are shown in Sec. 3.6 and Appendix B.6.

To the best of our knowledge, GC-Bench is the first comprehensive benchmark for both node-level and graph-level graph condensation. There are a few image dataset condensation benchmark works for image classification task [1] and condensation adversarial robustness [38]. A recent work GCondenser [23] evaluates some node-level GC methods for node classification on homogeneous graphs with limited evaluation dimensions in terms of performance and time efficiency. Our GC-Bench analyzes more GC methods on a wider variety of datasets (both homogeneous and heterogeneous) and tasks (node classification, graph classification), encompassing both node-level and graph-level methods. In addition to performance and efficiency analysis, we further explore the transferability across different tasks (link prediction, node clustering, anomaly detection) and backbones. With GC-Bench covering more in-depth investigation over a wider scope, we believe it will provide valuable insights into existing works and future directions. A comprehensive comparison with GCondenser can be found in Appendix A.5.

We evaluate the GC methods across a spectrum of condensation ratios¹¹1The condensation ratio is defined as $r=N^{\prime}/N$ for node-level experiments and $r=n^{\prime}/n$ for graph-level experiments. We use Graph/Cls to denote the number of condensed graphs per class in graph-level experiments. to identify their progress and effective ranges. Prior studies primarily utilize three ratios of labeling rates, a selection that is far from comprehensive. We broaden the evaluation of existing condensation methods and the node-level and graph-level GC results are shown in Table 2 and Table 3, respectively. The experiment setting and additional results can be found in Appendix B.1.

For node-level experiments (Table 2), we observe that: (1) GC methods can achieve lossless results [45] compared to the whole dataset in 5 out of 7 cases (highlighted in red), generally outperforming traditional core-set methods, especially at smaller condensation ratios (e.g., Citeseer with $r$=0.18%, ACM with $r$=0.003%, DBLP with $r$=0.002%). (2) Distribution matching methods underperform compared to gradient matching and trajectory matching methods in 6 out of 7 datasets. The gradient matching methods and the trajectory matching methods perform well in our benchmark.

From graph-level experiments on GIN (Table 3) and GCN (Section B.1), our observations are: (1) KiDD with GIN shows significant advantages in 18 out of 25 cases, while DosCond and Mirage do not consistently outperform traditional core-set methods, indicating room for improvement in future work. (2) KiDD performs well when GIN is used as the model for downstream tasks but performs poorly with GCN. This is because KiDD does not rely on the backbone and depends solely on the structure. Consequently, the stronger the downstream model’s expressive ability, the better the results.

From both node-level and graph-level results, we observe that as the condensation ratio increases, traditional core-set methods improve, narrowing the performance gap with deep methods. However, deep GC methods show a saturation point or even a decline in performance beyond a certain threshold, suggesting that larger condensed data may introduce noise and biases that degrade performance.

Key Takeaways 1: Current node-level GC methods can achieve nearly lossless condensation performance. However, there is still a significant gap between graph-level GC and whole dataset training, indicating there is substantial room for improvement.

Key Takeaways 2: A large condensation ratio does not necessarily lead to better performance with current methods.

We analyze the impact of structure in terms of heterogeneity and heterophily. Experimental settings and additional results can be found in Appendix B.2.

(1) Heterogeneity v.s. Homogeneity. For the heterogeneous datasets ACM and DBLP, we convert the heterogeneous graphs into homogeneous ones for evaluation. From the results in Table 2, we observe that GC methods designed for homogeneous graphs preserve most of the semantic information and perform comparably to models training on the whole dataset.

(2) Heterophily v.s. Homophily. From the results of the heterophilous dataset Flickr (with homophily ratio $\mathcal{H}=0.24$) in Table 2, we can observe that current GC methods can achieve almost the same accuracy as models training on the whole dataset. However, there is still a significant gap compared to the state-of-the-art results of the model designed for heterophilic graphs.

Key Takeaways 3: Existing GC methods primarily address simple graph data. However, the conversion process to specific data types is non-trivial, leaving significant room for improvement in preserving complex structural properties.

To evaluate the transferability of GC methods, we condense the dataset by node classification (NC) and use the condensed dataset to train models for link prediction (LP), node clustering (NClu), and anomaly detection (AD) tasks. The results on Citeseer are shown in Figure 2. Settings and additional results can be found in Appendix B.3.

As shown in Figure 2, performance with condensed datasets was significantly lower compared to original datasets in all transferred tasks. This decline may be due to the task-specific nature of the condensation process, which retains only task-relevant information while ignoring other semantically rich details. For instance, AD task prioritizes high-frequency graph signals more than NC and LP tasks, leading to poor performance when transferring condensed datasets from NC to AD tasks. Among the methods, gradient matching methods (GCond, DosCond, and SGDD) demonstrated better transferability in downstream tasks. In contrast, while structure-free methods (SFGC and GEOM) perform well in node classification (Section 3.1), they show a significant performance gap in AD tasks compared to gradient matching methods.

Key Takeaways 4: All condensed datasets struggle to perform well outside the context of the specific tasks for which they were condensed, leading to limited applicability.

We adopt one model (SGC or Graph Transformer) as the backbone for condensation and use the various models in downstream tasks evaluation. Details and additional results are in Appendix B.4.

As shown in Figure LABEL:fig:arch-SGC and  LABEL:fig:arch-GTrans, each column shows the generalization performance of a condensed graph generated by different methods for various downstream models. We can observe that datasets condensed with SGC generally maintain performance when transferred across models. However, datasets condensed with Graph Transformer (GTrans) consistently underperform across various methods, and other models also exhibit reduced performance when adapted to Graph Transformer. Intuitively, SGC’s basic neighbor message-passing strategy may overlook global dependencies critical to more complex models, and similarly, complex models may not perform well when adapted to simpler models. As we can observe, DosCond exhibits generally better transferability compared to other gradient-matching methods. Since it can be regarded as the one-step gradient matching variant of GCond, we further test the impact of gradient matching steps on transferability (Figure LABEL:fig:Gcond-steps). Increasing the number of matching steps was found to correlate with reduced performance across architectures, indicating that extensive gradient matching may encode model-specific biases.

Key Takeaways 5: Current GC methods exhibit significant performance variability when transferred to different backbone architectures. Involving the entire training process potentially may lead to encoding backbone-specific details in the condensed datasets.

Key Takeaways 6: Despite their strong performance in general graph learning tasks, transformers surprisingly yield suboptimal results in graph condensation.

We evaluate 5 distinct initialization strategies, namely: Random Noise, Random Sample, Center, K-Center, and K-Means. The results of GCond on Cora and ogbn-arxiv are shown in Figure LABEL:fig:init. Detailed settings and additional results can be found in Appendix B.5.

As shown in Figure LABEL:fig:initial_0.5 and Figure LABEL:fig:initial_0.05, the choice of the initialization method can significantly influence the efficiency of the condensation process but with little impact on the final accuracy. For instance, using Center on Cora reduces the average time to reach the same accuracy by approximately 25% compared to Random Sample and 71% compared to Random Noise. However, this speed advantage diminishes as the scale of the condensed graph increases. Additionally, different datasets have their preferred initialization methods for optimal performance. For example, Center is generally faster for Cora condensed by GCond while K-Means performs better on ogbn-arxiv.

Key Takeaways 7: Different datasets have their preferred initialization methods for optimal performance even for the same GC method.

Key Takeaways 8: The initialization mechanism primarily affects the convergence speed with little impact on the final performance. The smaller the condensed graph, the greater the influence of different initialization strategies on the convergence speed.

In this subsection, we evaluate the condensation time and memory consumption of GC methods. The results on ogbn-arxiv are shown in Figure 5, where the x-axis denotes the overall condensation time (min) when achieving the best validation performance, the y-axis denotes the test accuracy (%), the inner size of the marker represents the peak CPU memory usage (MB), while the outer size represents the peak GPU memory usage (MB). As we can observe, the gradient matching methods have higher time and space consumption compared to other types of methods. However, Table 2 shows that current gradient and distribution matching GC methods may trigger OOM (Out of Memory) errors on large datasets with high condensation ratios, making them unsuitable for large-scale scenarios, which contradicts the goal of applying graph condensation to extremely large graphs. More detailed results in Appendix B.6.

Key Takeaways 9: GC methods that rely on backbones and full-scale data training have large time and space consumption.