Self-supervised Graph-level Representation Learning
with Local and Global Structure

An ideal representation should preserve the local structure among various data instances. More specifically, we define it as follows:

Definition 1 (Local-instance Structure). The local structure aims to preserve the pairwise similarity between different instances after mapping from the high-dimensional input space to the low-dimensional latent space (Roweis & Saul, 2000; Belkin & Niyogi, 2002). For a pair of similar graphs/subgraphs, $\mathcal{G}$ and $\mathcal{G^{\prime}}$, their embeddings are expected to be nearby in the latent space, as illustrated in Fig. 1(a), while the dissimilar pairs should be mapped to far apart.

The pursuit of local-instance structure alone is usually insufficient to capture the semantics underlying the entire data set. It is therefore important to discover the global-semantic structure of the data, which is concretely defined as follows:

Definition 2 (Global-semantic Structure). A real-world data set can usually be organized as various semantic clusters (Furnas et al., 2017; Ji et al., 2019), especially in a hierarchical way for graph-structured data (Ashburner et al., 2000; Chen et al., 2012b). After mapping to the latent space, the embeddings of a set of graphs are expected to form some global structures reflecting the clustering patterns of the original data. A graphical illustration can be seen in Fig. 1(b).

Problem Definition. The problem of self-supervised graph representation learning considers a set of unlabeled graphs $\mathbf{G}=\{\mathcal{G}_{1},\mathcal{G}_{2},\cdots,\mathcal{G}_{M}\}$, and we aim at learning a low-dimensional vector $h_{\mathcal{G}_{m}}\in\mathbb{R}^{\delta}$ for each graph $\mathcal{G}_{m}\in\mathbf{G}$ under the guidance of the data itself. In specific, we expect the graph embeddings $\mathbf{H}=\{h_{\mathcal{G}_{1}},h_{\mathcal{G}_{2}},\cdots,h_{\mathcal{G}_{M}}\}$ follow both the local-instance and global-semantic structure.

Graph Neural Networks (GNNs). A typical graph can be represented as $\mathcal{G}=(\mathcal{V},\mathcal{E},X_{\mathcal{V}},X_{\mathcal{E}})$, where $\mathcal{V}$ is a set of nodes, $\mathcal{E}$ denotes the edge set, $X_{\mathcal{V}}=\{X_{v}|v\in\mathcal{V}\}$ stands for the attributes of all nodes, and $X_{\mathcal{E}}=\{X_{uv}|(u,v)\in\mathcal{E}\}$ represents the edge attributes. A GNN aims to learn an embedding vector $h_{v}\in\mathbb{R}^{\delta}$ for each node $v\in\mathcal{V}$ and also a vector $h_{\mathcal{G}}\in\mathbb{R}^{\delta}$ for the entire graph $\mathcal{G}$. For an $L$-layer GNN, a neighborhood aggregation scheme is performed to capture the $L$-hop information surrounding each node. As suggested in Gilmer et al. (2017), the $l$-th layer of a GNN can be formalized as follows:

where $\mathcal{N}(v)$ is the neighborhood set of $v$, $h_{v}^{(l)}$ denotes the representation of node $v$ at the $l$-th layer, $h_{v}^{(0)}$ is initialized as the node attribute $X_{v}$, and $f^{(l)}_{M}$ and $f^{(l)}_{U}$ stand for the message passing and update function at the $l$-th layer, respectively. Since $h_{v}$ summarizes the information of a subgraph centered around node $v$, we will refer to $h_{v}$ as subgraph embedding to underscore this point. The entire graph’s embedding can be derived as below:

where $f_{R}$ is a permutation-invariant readout function, e.g. mean pooling or more complex graph-level pooling function (Ying et al., 2018; Zhang et al., 2018).

General EM Algorithm. The basic objective of EM algorithm (Dempster et al., 1977; Krishnan et al., 1997) is to find the maximum likelihood solution for a model containing latent variables. In such a problem, we denote the set of all observed data as $\mathbf{X}$, the set of all latent variables as $\mathbf{Z}$ and the set of all model parameters as $\bm{\theta}$.

In the E-step, using the data $\mathbf{X}$ and the model parameters $\bm{\theta}_{t-1}$ estimated by the last EM cycle, the posterior distribution of latent variables is derived as $p(\mathbf{Z}|\mathbf{X},\bm{\theta}_{t-1})$ which can also be regarded as the responsibility that a specific set of latent variables are taken for explaining the observations.

In the M-step, employing the posterior distribution given by the E-step, the expectation of complete-data log-likelihood, denoted as $Q(\bm{\theta})$, is evaluated for the general model parameters $\bm{\theta}$ as follows:

The model parameters are updated to maximize this expectation in the M-step, which outputs:

Each cycle of EM can increase the complete-data likelihood expected by the current model, and, considering the whole progress, the EM algorithm has been demonstrated to be capable of maximizing the marginal likelihood function $p(\mathbf{X}|\bm{\theta})$ (Hathaway, 1986; Neal & Hinton, 1998).

Following the existing methods for dimensionality reduction (Tenenbaum et al., 2000; Roweis & Saul, 2000; Belkin & Niyogi, 2002), the goal of local-structure learning is to preserve the local similarity of the data before and after mapping to a low-dimensional latent space. In specific, it is expected that similar graphs or subgraphs are embedded close to each other, and dissimilar ones are mapped to far apart. Using a similarity measurement defined in the latent space, we formulate this problem as maximizing the similarity of correlated graph/subgraph pairs while minimizing that of negative pairs.

In specific, given a graph $\mathcal{G}=(\mathcal{V},\mathcal{E},X_{\mathcal{V}},X_{\mathcal{E}})$ sampled from the data distribution $P_{\mathcal{G}}$, we obtain its correlated counterpart $\mathcal{G}^{\prime}=(\mathcal{V}^{\prime},\mathcal{E}^{\prime},X_{\mathcal{V}^{\prime}},X_{\mathcal{E}^{\prime}})$ through randomly masking a part of node/edge attributes in the graph (Hu et al., 2019) (see appendix for the detailed scheme). In addition, for a subgraph $\mathcal{G}_{v}$ constituted by node $v$ and its $L$-hop neighborhoods in graph $\mathcal{G}$, we regard the corresponding subgraph $\mathcal{G}^{\prime}_{v}$ in graph $\mathcal{G}^{\prime}$ as its correlated counterpart. Through applying an $L$-layer GNN model $\mathrm{GNN}_{\theta}$ ($\theta$ stands for GNN’s parameters) upon graph $\mathcal{G}$ and $\mathcal{G}^{\prime}$, the graph and subgraph embeddings are derived as follows:

where $h_{\mathcal{V}}=\{h_{\mathcal{G}_{v}}|v\in\mathcal{V}\}$ and $h_{\mathcal{V}^{\prime}}=\{h_{\mathcal{G}^{\prime}_{v}}|v\in\mathcal{V}^{\prime}\}$ represent the set of subgraph embeddings within graph $\mathcal{G}$ and $\mathcal{G}^{\prime}$, respectively.

In this phase, the objective of learning is to enhance the similarity of correlated graph/subgraph pairs while diminish that of negative pairs. Using a specific similarity measure (e.g. cosine similarity $s(x,y)=x^{\top}y/||x||\ \!||y||$ in our practice), we seeks to optimize the following two objective functions for graph and subgraph, respectively:

where $p_{n}(\mathbf{\mathcal{G}},\mathbf{\mathcal{G}^{\prime}})$ and $p_{n}(\mathbf{\mathcal{G}_{v}},\mathbf{\mathcal{G}^{\prime}_{v}})$ denote the noise distribution from which negative pairs are sampled. In practice, for a correlated graph pair $(\mathcal{G},\mathcal{G}^{\prime})$ or correlated subgraph pair $(\mathcal{G}_{v},\mathcal{G}^{\prime}_{v})$, we substitute $\mathcal{G}$ ($\mathcal{G}_{v}$) randomly with another graph from the data set (a subgraph centered around another node in the same graph) to construct negative pairs.

For learning the local-instance structure of graph representations, we aim to minimize both objective functions (Eqs. 6 and 7) with respect to the parameters of GNN:

It is worth noticing that the graphs in a data set may possess hierarchical semantic information. For example, drugs (i.e. molecular graphs) are represented by a five-level hierarchy in the Anatomical Therapeutic Chemical (ATC) classification system (Chen et al., 2012b). After mapping to the latent space, the embeddings of all graphs in the data set are also expected to form some global structures corresponding to the hierarchical semantic structures of the original data.

However, for the lack of explicit semantic labels in the self-supervised graph representation learning, such global structure cannot be attained via label-induced supervision. To tackle this limitation, we introduce an additional set of model parameters, hierarchical prototypes, to represent the feature clusters in the latent space in a hierarchical way. They are formally defined as $\mathbf{C}=\{c^{l}_{i}\}_{i=1}^{M_{l}}$ ($l=1,2,\cdots,L_{p}$), where $c^{l}_{i}\in\mathbb{R}^{\delta}$ stands for the $i$-th prototype at the $l$-th layer, $L_{p}$ is the depth of hierarchical prototypes, and $M_{l}$ denotes the number of prototypes at the $l$-th layer. These prototypes are structured as a set of trees (Fig. 1(b)), in which each node corresponds to a prototype, and, except for the leaf nodes, each prototype possesses a set of child nodes, denoted as $\mathbb{C}(c^{l}_{i})$ ($1\leqslant i\leqslant M_{l}$, $l=1,2,\cdots,L_{p}-1$).

The goal of global-semantic learning is to encourage the graphs to be compactly embedded around corresponding prototypes and, at the same time, refine hierarchical prototypes to better represent the data. We formalize this problem as optimizing a latent variable model. Specifically, for the observed data set $\mathbf{G}=\{\mathcal{G}_{1},\mathcal{G}_{2},\cdots,\mathcal{G}_{M}\}$, we consider a latent variable set, i.e. the prototype assignments of all graphs $\mathbf{Z}=\{z_{\mathcal{G}_{1}},z_{\mathcal{G}_{2}},\cdots,z_{\mathcal{G}_{M}}\}$ ($z_{\mathcal{G}_{m}}$ is a set of prototypes that best represent $\mathcal{G}_{m}$ in the latent space). The model parameters in this problem are the GNN parameters $\theta$ and hierarchical prototypes $\mathbf{C}$. Since the corresponding latent variable of each graph is not given, it is hard to directly maximize the complete-data likelihood function $p(\mathbf{G},\mathbf{Z}|\theta,\mathbf{C})$. Therefore, we seek to maximize the expectation of complete-data likelihood through the EM algorithm.

The vanilla EM algorithm (Dempster et al., 1977; Krishnan et al., 1997) requires a full pass through the data set before each parameter update, which is computationally inefficient when the size of data set is large like in our case. Therefore, we consider an online EM variant (Sato & Ishii, 2000; Cappé & Moulines, 2009; Liang & Klein, 2009) which operates on mini-batches of data. This approach is based on the i.i.d. assumption of the data set, where both the complete-data likelihood and the posterior probability of latent variables can be factorized over each observed-latent variable pair:

First, we introduce the initialization scheme of model parameters.

Initialization of model parameters. Before triggering the global structure exploration, we first pre-train the GNN by minimizing $\mathcal{L}_{\mathrm{local}}$ and employ the derived GNN model as initialization, which establishes a locally smooth latent space. After that, we utilize this pre-trained GNN model to extract the embeddings of all graphs in the data set, and the K-means clustering is applied upon these graph embeddings to initialize the bottom layer prototypes (i.e. $\{c^{L_{p}}_{i}\}_{i=1}^{M_{L_{p}}}$) with the output cluster centers. The prototypes of upper layers are initialized by iteratively applying K-means clustering to the prototypes of the layer below. For each time of clustering, we discard the output cluster centers assigned with less than two samples to avoid trivial solutions (Bach & Harchaoui, 2007; Caron et al., 2018).

Next, we state the details of the E-step and M-step applied in our method.

E-step. In this step, we first randomly sample a mini-batch of graphs $\widetilde{\mathbf{G}}=\{\mathcal{G}_{1},\mathcal{G}_{2},\cdots,\mathcal{G}_{N}\}$ ($N$ denotes the batch size) from the data set $\mathbf{G}$, and $\widetilde{\mathbf{Z}}=\{z_{\mathcal{G}_{n}}\}_{n=1}^{N}$ stands for the latent variables corresponding to these sampled graphs. Each latent variable $z_{\mathcal{G}_{n}}=\{z^{1}_{\mathcal{G}_{n}},z^{2}_{\mathcal{G}_{n}},\cdots,z^{L_{p}}_{\mathcal{G}_{n}}\}$ is a chain of prototypes from top layer to bottom layer that best represent graph $\mathcal{G}_{n}$ in the latent space, and it holds that $z^{l+1}_{\mathcal{G}_{n}}$ is the child node of $z^{l}_{\mathcal{G}_{n}}$ in the corresponding tree structure, i.e. $z^{l+1}_{\mathcal{G}_{n}}\in\mathbb{C}(z^{l}_{\mathcal{G}_{n}})$ ($l=1,2,\cdots,L_{p}-1$). The posterior distribution of $\widetilde{\mathbf{Z}}$ can be evaluated in a factorized way because of the i.i.d. assumption:

where $\theta_{t-1}$ and $\mathbf{C}_{t-1}$ are the model parameters from the last EM cycle. Directly evaluating each posterior distribution $p(z_{\mathcal{G}_{n}}|\mathcal{G}_{n},\theta_{t-1},\mathbf{C}_{t-1})$ is nontrivial, which requires to traverse all the possible chains in hierarchical prototype. Instead, we adopt the idea of stochastic EM algorithm (Celeux & Govaert, 1992; Nielsen et al., 2000) and draw a sample $\hat{z}_{\mathcal{G}_{n}}\sim p(z_{\mathcal{G}_{n}}|\mathcal{G}_{n},\theta_{t-1},\mathbf{C}_{t-1})$ for Monte Carlo estimation. In specific, we sequentially sample a prototype from each layer in a top-down manner, and all the sampled prototypes form a connected chain from top layer to bottom layer in hierarchical prototypes. Formally, we first sample a prototype from top layer according to a categorical distribution over all the top layer prototypes, i.e. $\hat{z}^{1}_{\mathcal{G}_{n}}\sim\mathrm{Cat}(z^{1}_{\mathcal{G}_{n}}|\{\alpha_{i}\}_{i=1}^{M_{1}})$ ($\alpha_{i}=\mathrm{softmax}(s(c^{1}_{i},h_{\mathcal{G}_{n}}))$), where $s$ denotes the cosine similarity measurement; for the sampling at layer $l$ ($l\geqslant 2$), we draw a prototype from that layer based on a categorical distribution over the child nodes of prototype $\hat{z}^{l-1}_{\mathcal{G}_{n}}$ sampled from the layer above, i.e. $\hat{z}^{l}_{\mathcal{G}_{n}}\sim\mathrm{Cat}(z^{l}_{\mathcal{G}_{n}}|\{\alpha_{c}\})$ ($\alpha_{c}=\mathrm{softmax}(s(c,h_{\mathcal{G}_{n}}))$, $\forall c\in\mathbb{C}(\hat{z}^{l-1}_{\mathcal{G}_{n}})$), such that we sample a latent variable $\hat{z}_{\mathcal{G}_{n}}=\{\hat{z}^{1}_{\mathcal{G}_{n}},\hat{z}^{2}_{\mathcal{G}_{n}},\cdots,\hat{z}^{L_{p}}_{\mathcal{G}_{n}}\}$ which is a connected chain in hierarchical prototypes. Using the latent variables inferred as above, we seek to maximize the expectation of complete-data log-likelihood in the M-step.

M-step. In this step, we aim at maximizing the expected complete-data log-likelihood with respect to the posterior distribution of latent variables, which is defined as follows:

This expectation needs the computation over all data points, which cannot be attained in the online setting. As a substitute, we propose to maximize the expected log-likelihood on mini-batch $\widetilde{\mathbf{G}}$, which can be estimated using the latent variables $\widetilde{\mathbf{Z}}_{est}=\{\hat{z}_{\mathcal{G}_{n}}\}_{n=1}^{N}$ sampled in the E-step:

We would like to point out that $\widetilde{Q}(\theta,\mathbf{C})$ is a decent proxy for $Q(\theta,\mathbf{C})$, where a proportional relation approximately holds between them (see appendix for the proof):

We further scale $\widetilde{Q}(\theta,\mathbf{C})$ with the batch size to derive the log-likelihood function $\mathcal{L}(\theta,\mathbf{C})$ that is more stable in terms of computation:

To estimate $\mathcal{L}(\theta,\mathbf{C})$, we need to define the joint likelihood of a graph $\mathcal{G}$ and a latent variable $z_{\mathcal{G}}$, which is represented with an energy-based formulation in our method:

where $Z(\theta,\mathbf{C})$ denotes the partition function. We formalize the negative energy function $f$ by measuring the similarities between graph embedding $h_{\mathcal{G}}$ and the prototypes in $z_{\mathcal{G}}$ and also measuring the similarities between the prototypes in $z_{\mathcal{G}}$ that are from consecutive layers:

Intuitively, $f$ evaluates how well a latent variable $z_{\mathcal{G}}$ represents graph $\mathcal{G}$ in the latent space, and it also measures the affinity between the consecutive prototypes along a chain from top layer to bottom layer in hierarchical prototypes.

It is nontrivial to optimize with $p(\mathcal{G},z_{\mathcal{G}}|\theta,\mathbf{C})$ due to the intractable partition function. Inspired by Noise-Contrastive Estimation (NCE) (Gutmann & Hyvärinen, 2010, 2012), we seek to optimize with the unnormalized likelihoods, i.e. $\tilde{p}(\mathcal{G},z_{\mathcal{G}}|\theta,\mathbf{C})=\exp(f(h_{\mathcal{G}},z_{\mathcal{G}}))$, by contrasting the positive observed-latent variable pair with the negative pairs sampled from some noise distribution, which defines an objective function that well approximates $\mathcal{L}(\theta,\mathbf{C})$:

where $p_{n}(\mathcal{G},z_{\mathcal{G}})$ is the noise distribution. In practice, we compute the outer expectation with all the positive pairs in the mini-batch, i.e. $(\mathcal{G}_{n},\hat{z}_{\mathcal{G}_{n}})$ ($1\leqslant n\leqslant N$), and, for computing the inner expectation, we construct $L_{p}$ negative pairs for the positive pair $(\mathcal{G}_{n},\hat{z}_{\mathcal{G}_{n}})$ by fixing the graph $\mathcal{G}_{n}$ and randomly substituting one of $L_{p}$ prototypes in $\hat{z}_{\mathcal{G}_{n}}$ with another prototype at the same layer each time. For global-semantic learning, we aim to minimize the global objective function $\mathcal{L}_{\textrm{global}}$ with respect to both the GNN parameter $\theta$ and hierarchical prototypes $\mathbf{C}$:

In general, the proposed online EM algorithm seeks to maximize the joint likelihood $p(\mathbf{G},\mathbf{Z}|\theta,\mathbf{C})$ governed by model parameters $\theta$ and $\mathbf{C}$. For a step further, we propose the following proposition that this algorithm can indeed maximize the marginal likelihood function $p(\mathbf{G}|\theta,\mathbf{C})$.

The proof of Proposition 1 is provided in the appendix.

The GraphLoG framework seeks to learn the graph representations preserving both the local-instance and global-semantic structure on an unlabeled graph data set $\mathbf{G}$. For model optimization under this framework, we first pre-train the GNN by minimizing the local objective function $\mathcal{L}_{\textrm{local}}$ and initialize the model parameters with the pre-trained GNN. After that, for each learning step, we sample a mini-batch $\widetilde{\mathbf{G}}$ from the data set and conduct an EM cycle. In the E-step, the latent variables corresponding to the mini-batch, i.e. $\widetilde{\mathbf{Z}}_{est}$, are sampled from the posterior distribution defined by the current model. In the M-step, model parameters are updated to maximize the expected log-likelihood on mini-batch $\widetilde{\mathbf{G}}$. Also, we add the local objective function to the optimization of M-step, which guarantees the local smoothness when pursuing the global-semantic structure and performs well in practice. We summarize the optimization algorithm in Alg. 1.

After the self-supervised pre-training on massive unlabeled graphs, the derived GNN model can be applied to various downstream tasks for producing effective embedding vectors of different graphs. For example, we can first pre-train a GNN model with GraphLoG on a large number of unlabeled molecules (i.e. molecular graphs). After that, for a downstream task of molecular property prediction where a small set of labeled molecules are available, we can learn a linear classifier upon the pre-trained GNN model to perform the specific graph classification task.

Pre-training details. Following Hu et al. (2019), we adopt a five-layer Graph Isomorphism Network (GIN) (Xu et al., 2019) with 300-dimensional hidden units and a mean pooling readout function for performance comparisons (Secs. 5.2 and 5.3). We use an Adam optimizer (Kingma & Ba, 2015) (learning rate: $1\times 10^{-3}$) to pre-train the GNN with $\mathcal{L}_{\textrm{local}}$ for one epoch and then train the whole model with both $\mathcal{L}_{\textrm{local}}$ and $\mathcal{L}_{\textrm{global}}$ for 10 epochs. For each time of K-means clustering in the initialization of hierarchical prototypes, we adopt 50 initial cluster centers. Unless otherwise specified, the batch size $N$ is set as 512, and the hierarchical prototypes’ depth $L_{p}$ is set as 3. These hyperparameters are selected by the grid search on the validation sets of four downstream molecule data sets (i.e. BBBP, SIDER, ClinTox and BACE), and their sensitivity is analyzed in Sec. 5.4.

Fine-tuning details. For fine-tuning on a downstream task, a linear classifier is appended upon the pre-trained GNN, and an Adam optimizer (learning rate: $1\times 10^{-3}$, fine-tuning batch size: 32) is employed to train the model for 100 epochs. We utilize a learning rate scheduler with fix step size which multiplies the learning rate by 0.3 every 30 epochs. All the reported results are averaged over five independent runs. The source code is available at https://github.com/DeepGraphLearning/GraphLoG.

Performance comparison. For the experiments on both chemistry and biology domains, we compare the proposed method with existing self-supervised graph representation learning algorithms (i.e. EdgePred (Kipf & Welling, 2016), InfoGraph (Sun et al., 2019), AttrMasking (Hu et al., 2019), ContextPred (Hu et al., 2019), GraphPartition (You et al., 2020b) and GraphCL (You et al., 2020a)) to verify its effectiveness. We report the results of EdgePred, AttrMasking and ContextPred from Hu et al. (2019) and examine the performance of InfoGraph, GraphPartition and GraphCL based on the released source code.

Data sets. For fair comparison, we use the same data sets as in Hu et al. (2019). In specific, a subset of ZINC15 database (Sterling & Irwin, 2015) with 2 million unlabeled molecules is employed for self-supervised pre-training. Eight binary classification data sets in MoleculeNet (Wu et al., 2018) serve as downstream tasks, where the scaffold split scheme (Chen et al., 2012a) is used for data set split.

Results. In Tab. 1, we report the performance of the proposed GraphLoG method compared with other works, where ‘Random’ denotes the GIN model with random initialization. Among all self-supervised learning strategies, our approach achieves the best performance on six of eight tasks, and a $2.1\%$ performance gain is obtained in terms of average ROC-AUC. We deem that this improvement over previous works is mainly from the global structure modeling in GraphLoG, which is not included in existing methods.

Data sets. For biology domain, following Hu et al. (2019), 395K unlabeled protein ego-networks are utilized for self-supervised pre-training. The downstream task is to predict 40 fine-grained biological functions of 8 species.

Results. Tab. 2 reports the test ROC-AUC of various self-supervised learning techniques. It can be observed that the proposed GraphLoG method outperforms existing approaches with a clear margin, i.e. a $1.7\%$ performance gain. This result illustrates that the proposed approach is able to learn effective graph representations that benefit the downstream task involving fine-grained classification.

In Tab. 3, we further compare GraphLoG with four existing methods under four GNN architectures (i.e. GCN (Kipf & Welling, 2017), GraphSAGE (Hamilton et al., 2017), GAT (Velickovic et al., 2018) and GIN (Xu et al., 2019)). We can observe that GraphLoG outperforms the existing approaches on all configurations, and, compared to EdgePred, AttrMasking and ContextPred, it avoids the performance decrease relative to random initialization baseline on GAT.

Effect of different objective functions. In Tab. 4, we analyze the effect of three objective functions on the biology domain, and we continue using the GIN depicted in Sec. 5.1 in this experiment. When each objective function is individually applied (1st, 2nd and 3rd row), the one for global-semantic learning performs best, which probably benefits from its exploration of the semantic structure of the data. Through simultaneously applying different objective functions, the full model (last row) achieves the best performance, which illustrates that the learning of local and global structure are complementary to each other.

Sensitivity of hierarchical prototypes’ depth $L_{p}$. In this part, we discuss the selection of parameter $L_{p}$ which controls the number of discovered semantic hierarchies. In Fig. 3(a), we plot model’s performance under different $L_{p}$ values. It can be observed that deeper hierarchical prototypes (i.e. $L_{p}\geqslant 3$) achieve stable performance gain compared to the shallow ones (i.e. $L_{p}\leqslant 2$).

Sensitivity of batch size $N$. In this experiment, we evaluate the effect of the batch size $N$ on our method. Fig. 3(b) shows the test ROC-AUC on downstream task using different batch sizes. From the line chart, we can observe that large batch size (i.e. $N\geqslant 256$) can promote the performance of GraphLoG. Under such condition, the sampled mini-batches can better represent the whole data set and thus derive more precise likelihood expectation in Eq. 14.

Visualization. In Fig. 2, we utilize the t-SNE (Maaten & Hinton, 2008) to visualize the graph embeddings and hierarchical prototypes on ZINC15 data set. Compared with only using the local constraints $\mathcal{L}_{\textrm{sub}}$ and $\mathcal{L}_{\textrm{graph}}$ (configurations (a) and (b)), more obvious feature separation is achieved after applying the global constraint $\mathcal{L}_{\textrm{global}}$ (configuration (c)), which illustrates its effectiveness on discovering the underlying global-semantic structure of the data.