Instruction-based Hypergraph Pretraining

Interactions in most real-work networks are more complex than dyadic edges used in traditional graphs, in which cases a hypergraph provides a more expressive structure to represent such relations (Lung et al., 2018; Jia et al., 2021; Feige et al., 2015). Beyond the basic hypergraph convolution and attention neural networks(Feng et al., 2019; Bai et al., 2021), hypergraph learning demonstrates adaptability for customization across a variety of graph-based tasks. DHCF (Ji et al., 2020) employs residual connections to effectively capture hybrid multi-order correlations in the user-item graph, simultaneously considering aggregated representations from the original graph and the hypergraph. MHCN (Yu et al., 2021) introduces a multi-channel hypergraph convolutional network that leverages different types of high-order relations among users to predict social links. Seq-HyGAN (Saifuddin et al., 2023) proposes sequential hypergraph attention to capture dependencies between extracted subsequences for sequence classification. To the best of our knowledge, no previous method has applied hypergraph learning for instruction-based pretraining.

Graph learning has attracted interest owing to the prevalence of graph-structured data in various fields. To improve the learning efficacy of such graphs, researchers have been investigating graph pretraining utilizing unlabeled graph data (Jiang et al., 2021; Lu et al., 2021; Hu et al., 2020b). GCC (Qiu et al., 2020) applies contrastive learning to capture the universal topological properties across multiple networks. GPT-GNN (Hu et al., 2020a) introduces a graph generation task during pretraining to capture the characteristics of the graph through the utilization of node features.

With the prevalence of prompt learning, prompting frameworks are applied to mitigate the training objective gap and catastrophic forgetting. A pioneering work GPPT (Sun et al., 2022) designs a pairwise prompting function generating learnable token pairs from standalone nodes for downstream tasks. GraphPrompt (Liu et al., 2023a) multiplies each node embedding by a prompt vector into graph aggregation functions for different tasks. Sun et al. (Sun et al., 2023) introduce a prompting framework that reformulates nodes and edges to induced subgraphs to narrow the gap between different tasks during pretraining and finetuning. However, these works are limited to homogeneous graphs, and learnable prompts provide no explicit semantic clues on graph structures and tasks. On the contrary, our method leverages instruction-based prompts based on plain text instead of learnable prompts. With the guidance of instructions delivered by hyperedges, IHP is able to capture the high-order relations among different nodes.

Instructions can be regarded as additional text information to the graph structure. Incorporating text into graph learning offers extra contextual understanding and semantic insights, especially when the downstream graph structure is sparse. With the breakthroughs in PLMs, it is straightforward to leverage PLMs and GNNs for textual graph representation following a cascaded architecture (Hamilton et al., 2017). In the cascaded architecture, text information is first encoded by text encoders and then aggregated by graph models to obtain the final representation (Zhu et al., 2021; Li et al., 2021; Jin et al., 2021). In contrast with cascaded architectures, GraphFormers (Yang et al., 2021) and EdgeFormers (Jin et al., 2023a) propose GNN-nested Transformers to jointly encode text and node features in textual graphs. Different from previous works straightforwardly incorporating text encoding in heterogeneous graph learning (Zhang et al., 2019; Jin et al., 2023b), IHP focuses on leveraging instructions to transfer learning capabilities from pretext tasks to downstream tasks. By pretraining with instruction-based prompts, IHP enables the model to develop a generalized understanding of graph properties, which is beneficial when dealing with unseen graph data.

Let the graph for the pretraining task be $G=(V,E)$ where $V$ and $E$ are sets of nodes and edges, respectively. Similarly, let the graph for the downstream task be $G^{\prime}=(V^{\prime},E^{\prime})$, and the two instruction sets be $I_{p}$ and $I_{d}$ for the pretraining task and the downstream task. Our target is to pretrain the representations of the overlapping nodes, which are defined as target nodes $\mathcal{V}_{\text{t}}=V\cap V^{\prime}$. For example, the target nodes can be the overlapping molecules of different proteins in protein classification, or the overlapping users purchasing items of different domains in cross-domain recommendation. Given the target node set $\mathcal{V}_{\text{t}}$, we define the nodes other than target nodes in the pretraining task as pretraining context nodes $\mathcal{V}_{\text{c}}=V\setminus V_{t}$ in the pretraining task, and the unseen nodes in the downstream task as downstream context nodes $\mathcal{V}_{\text{c}}^{\prime}=V^{\prime}\setminus V_{t}$. Besides, we define the descriptions of pretraining nodes as $\mathcal{A}$ and the descriptions of downstream nodes as $\mathcal{A}^{\prime}$. The descriptions of different tasks are denoted as $\mathcal{T}$, with each task related to arbitrary nodes in graphs. These descriptions are used to construct instructions if available.

Compared with an edge connecting only two nodes in a traditional graph, a hyperedge connects multiple nodes simultaneously in a hypergraph. Hence, hypergraph naturally possesses the ability to model higher-order connections.

We maintain an embedding layer $\mathbf{E}\in\mathbb{R}^{(|{\mathcal{V}_{\text{t}}}|+|\mathcal{V}_{\text{c}}|)\times d}$ during pretraining, where $d$ is the feature dimension and columns represent all trainable node embeddings. The initial target node embeddings are denoted as $\mathbf{E}_{\text{t}}$ and the initial context node embeddings are denoted as $\mathbf{E}_{\text{c}}$. Since target nodes exist in pretraining and downstream tasks, prior knowledge learned from the pretraining tasks can be preserved in $\mathbf{E}_{\text{t}}$ and then leveraged in the downstream tasks.

We construct the target hypergraph and the context hypergraph based on the original graph. Compared to a graph where edges only connect two nodes, the advantage of a hypergraph is that hyperedges can simultaneously connect multiple nodes as the objectives of an instruction. According to the type of nodes in the graph, we construct two hypergraphs based on four basic types of hyperedges as below:

• •

  Target Hypergraph. The target hypergraph $\mathcal{H}_{\mathbf{t}}$ consists of target nodes $\mathcal{V}_{\mathbf{t}}$ and hyperedges $\mathcal{E}_{\mathbf{t}}$. Each target-target hyperedge is used to connect a target node with its one-hop target node neighbors. For each group of target nodes connected to one context node, a target-context hyperedge is used to connect them in the hypergraph. Its incidence matrix is denoted as $\mathbf{H}_{\text{t}}$.

• •

  Context Hypergraph. Similar to the target hypergraph, the context hypergraph $\mathcal{H}_{\mathbf{c}}$ consists of context nodes $\mathcal{V}_{\mathbf{c}}$ and hyperedges $\mathcal{E}_{\mathbf{c}}$. Each context-context hyperedge is used to connect a context node with its one-hop context node neighbors. For each group of context nodes connected to one target node, a context-target hyperedge is used to connect them simultaneously in the hypergraph. Its incidence matrix is denoted as $\mathbf{H}_{\text{c}}$.

In other words, we use the two hypergraphs to ensure the distinction between the target nodes and the context nodes. In this way, the framework can learn from the homogeneity of target nodes and context nodes in a discriminative manner during the following hypergraph pretraining. On the one hand, the information from pretraining is preserved in the target node embeddings and directly transferred to the downstream task. On the other hand, although context nodes are not in the downstream task, they provide the necessary information for the model to capture broader contextual patterns that are not limited to the nodes seen during the downstream task. The relation between target nodes can depend on the similarity among context nodes in pretraining, which is reflected by the context hypergraph. For example, two target nodes related to two context nodes are similar if the two context nodes are similar.

Similarly, a target hypergraph $\mathcal{H}_{\text{t}}^{\prime}=(\mathcal{V}_{\text{t}},\mathcal{E}_{\text{t}}^{\prime})$ and a context hypergraph $\mathcal{H}_{\text{c}}^{\prime}=(\mathcal{V}_{\text{c}}^{\prime},\mathcal{E}_{\text{c}}^{\prime})$ are defined for the downstream task with their corresponding incidence matrices $\mathbf{H}_{\text{t}}^{\prime}$ and $\mathbf{H}_{\text{c}}^{\prime}$. For both pretraining and downstream tasks, the isolation between the two hypergraphs separates the information propagation paths. In this way, the information of context nodes will not be directly aggregated to the target nodes through hyperedges in hypergraph learning, and vice versa. This design prevents the over-smoothing issue (Bodnar et al., 2022; Keriven, 2022) among target and context nodes, preventing the representations of target nodes from converging to the same values, especially within dense graph data.

In hypergraph learning, hypergraph convolution is prevalently used for information propagation across nodes in hypergraphs. Nonetheless, current hypergraph convolution methods focus on the direct aggregation of structural information from neighbors connected by hyperedge. To integrate the information from instructions into the hypergraph convolution process, we devise a novel Prompt Hypergraph Convolution layer. First, we construct the hyperedge information based on connected nodes. This step is denoted as hyperedge initialization of the PHC layer. Then, a information prompting step is proposed to endow the fusion of hyperedge information from resources other than the hypergraph structure, such as instructions. Finally, fused information is aggregated back to update the node embeddings in the node update step. To be more concrete, we present the calculation details of a PHC layer in the target hypergraph. The illustration is in Figure 2(c).

To provide explicit guidance for hypergraph learning, instruction-based prompt embeddings are constructed based on available task-related information and fed into the PHC layer for further information propagation. Each instruction is expected to assist in capturing the relationship between the nodes connected by one hyperedge. Hence, the instructions and the formulated hyperedges are designed in one-to-one correspondence. Instructions can be formulated from task descriptions, node descriptions, or a concatenation of both.

We show an example of leveraging the concatenation of task and node descriptions as instructions in Figure 2. Given that target-context hyperedges are context nodes in the original graph, we concatenate the node descriptions $\mathcal{A}_{c}$ and the descriptions $\mathcal{T}_{c}$ of tasks related to these nodes as instructions $\mathcal{I}_{c}$, where $|\mathcal{I}_{c}|=|\mathcal{E}_{t,c}|$. In this paper, descriptions of context nodes are obtained from the dataset, and the task descriptions are manually created for different tasks. For instance, if target nodes are readers and context nodes are history books in pretraining, the task description can be formulated as ’promote history books to readers.’ In other words, the task description consists of node information (e.g., readers) and/or other task information (e.g., promote or classify).

After the construction of instructions, a pretrained sentence transformer (Wang et al., 2020) is deployed to encode the instructions as text embeddings $\mathbf{C}\in\mathbb{R}^{|\mathcal{I}_{c}|\times d^{\prime}}$, which are frozen in both pretraining and downstream tasks. A linear layer is applied to adaptively learn the transformation from text embeddings to prompt embeddings:

where $\mathbf{W}_{\text{p}}\in\mathbb{R}^{d^{\prime}\times d}$ and $\mathbf{b}_{\text{p}}\in\mathbb{R}^{d}$ are the learnable weight matrix and bias for prompt transformation. And $\mathbf{P}$ is the output prompt embedding, which prompts the target-context hyperedges with instruction information in every PHC layer for the target hypergraph.

We show the overall comparison of link prediction performance in Table 2. The best results are in boldface. We observe that the proposed IHP framework achieves the best results and outperforms all the baselines in the three datasets. We hypothesize these large and stable gains result from prior knowledge learned from pretraining under the explicit guidance of instructions. In addition, baseline pretraining frameworks achieve better performance than models without pretraining in Goodreads-P and Amazon. However, they are surpassed by other baselines in Goodreads-H. Since the pretrained edges in Goodreads-H are densest, and pretrained node types in it are fewer than in Amazon, the risk of overfitting for baseline pretraining frameworks is high in such a dataset. In contrast, instructions provide explicit information about certain tasks, which explains IHP’s better generalization and adaptation for the downstream task.

We also evaluate IHP on node classification as the downstream task, optimized by cross-entropy loss (Good, 1952). Besides the previous baselines designed for general graph learning, HGNN (Feng et al., 2019) leveraging hypergraph convolution for classification is also used as a baseline. For Goodreads-H, books in the categories of children and young adults are pretraining context nodes, and history, biography, mystery, thriller and crime books are used in downstream classification. For Amazon, items in the categories of arts, crafts, sewing, grocery, gourmet food and office products are used for pretraining context nodes, and items in the categories of electronics, sports, outdoors, toys, games and instruments are used in downstream classification. We split $70\%$ of nodes for training and the remaining $30\%$ for testing. Accuracy and F1 scores are adopted as evaluation metrics. To prevent data leakage, we removed node category information from task descriptions in the instructions. The parameters remain the same as the settings for the link prediction.

The comparison is shown in Table 4. The performance verifies the efficacy of IHP in node classification tasks. The pretraining framework baselines have better performance than other baselines in most cases, which indicates that the node features can be accurately captured with the prior knowledge learned during pretraining. HGNN surpasses all other non-pretrained baselines, showing the benefit of modeling high-order relations by hypergraph learning. In addition, with inconsistency in pretraining and finetuning structures, pretrained baselines slightly degrade the downstream performance of hypergraph learning in Amazon, compared with HGNN. However, IHP with frozen prompt transformation maintains the consistency between pretraining and finetuning stages, and leverages instructions to offer information on downstream node features, which is a more comprehensive pretraining framework for node classification.

To validate the effectiveness of IHP in dynamically evolving graph structures, we add inductive target nodes which account for $25\%$, $40\%$, and $50\%$ of the original target nodes into Amazon, Goodreads-P, and Goodreads-H datasets. The embeddings of inductive nodes are never trained in either pretraining or finetuning stages. Instead, the inductive nodes are connected to other downstream training nodes by hyperedges defined in Section 4.2 and fed into IHP only for inference. Then, we evaluate the link prediction performance on those inductive nodes. In addition to the pretraining baselines (Hu et al., 2020b; Qiu et al., 2020; Hou et al., 2022), we also compare IHP with GraphSage (Hamilton et al., 2017) which is a general inductive graph learning method.

The results are demonstrated in Table 6. One can observe that IHP exhibits the best performance on inductive nodes in all the datasets, especially in larger Goodreads-H and Amazon datasets. With prior knowledge from abundant pretrained data in these two datasets, pretraining baselines always perform better than scratch-training GraphSage. Nonetheless, they are worse than IHP, which better captures dependencies between inductive and trained nodes by directly prompting text-based instruction to hyperedges.

To maintain the balance between retaining prior knowledge and adapting for various downstream tasks, we deploy the adaptation intensity coefficient $\lambda_{\text{t}}$ to differentiate the learning rates of target and context node embeddings during finetuning. We demonstrate the impact of this coefficient on three datasets in Figure 7. It becomes obvious that the performance is always suboptimal when the learning rates of target and context node embeddings remain the same, i.e., $\lambda_{\text{t}}=1$. For the two Goodreads datasets, a small $\lambda_{\text{t}}$ prevents the framework from forgetting the prior knowledge during finetuning. For Amazon, a large $\lambda_{\text{t}}$ is required to endow the model with the higher adaptivity to the downstream data.

We believe the difference of $\lambda_{\text{t}}$ in Goodreads and Amazon results from the discrepancy between the pretraining and downstream data in each dataset. For verification, we evaluate the model performance with the same learning rates of target and context nodes after merging the pretraining and downstream data. To be specific, we randomly select a subset $E_{\text{s}}$ of edges from the set $E$ of all the pretraining edges, add them to the downstream data, and then finetune the model with $\lambda_{\text{t}}=1$ on the downstream tasks. In other words, the edges become $E_{\text{s}}\cup E^{\prime}$ for the downstream tasks. The proportion of the selected edges in all the pretrained edges is defined as pretraining edges retention rate $k=|E_{\text{s}}|/|E|$. The ratio of these selected edges to all the downstream edges is denoted as pretraining edges ratio $r=|E_{\text{s}}|/(|E_{\text{s}}|+|E^{\prime}|)$. Pretraining context nodes connected by $E_{\text{s}}$ are also added to the downstream task. In this way, the model is finetuned directly with the original pretraining data.

The results are demonstrated in Figure 5. In the two Goodreads datasets, We observe that the downstream performance is generally improved with the pretraining data added to the downstream task. In Amazon, on the contrary, the performance is degraded after adding the pretraining data when $\lambda_{\text{t}}=1$. The reason is that all the pretraining and downstream context nodes are books in Goodreads, so the discrepancy between the pretraining and downstream data in Goodreads is minor compared to the discrepancy in Amazon, where pretraining and downstream context nodes are items in more diverse categories. That explains why a small $\lambda_{\text{t}}$ is necessary to prevent the framework from forgetting the prior knowledge in Goodreads, while a large $\lambda_{\text{t}}$ can help the pretrain model faster adapt to the significantly different downstream data in Amazon.