Heterogeneous Graph Transformer

Heterogeneous graphs (Sun and Han, 2012) (a.k.a., heterogeneous information networks) are an important abstraction for modeling relational data for many real-world complex systems. Formally, it is defined as:

Notice that, to better model real-world heterogeneous networks, we assume that there may exist multiple types of relations between different types of nodes. For example, in OAG there are different types of relations between the author and paper nodes by considering the authorship order, i.e., “the first author of”, “the second author of”, and so on.

In other words, we assume that the timestamp of an edge is unchanged, denoting the time it is created. For example, when a paper published on a conference at time $T$, $T$ will be assigned to the edge between the paper and conference nodes. On the contrary, different timestamps can be assigned to a node accordingly. For example, the conference node “WWW” can be assigned any year. $WWW@1994$ means that we are considering the first edition of WWW, which focuses more on internet protocol and Web infrastructure, while $WWW@2020$ means the upcoming WWW, which expands its research topics to social analysis, ubiquitous computing, search & IR, privacy and society, etc.

There have been significant lines of research on mining heterogenous graphs, such as node classification, clustering, ranking and representation learning (Sun and Han, 2012; Sun et al., 2011, 2012; Dong et al., 2017a), while the dynamic perspective of HGs has not been extensively explored and studied.

Recent years have witnessed the success of graph neural networks for relational data (Kipf and Welling, 2017; Velickovic et al., 2018; Hamilton et al., 2017). Generally, a GNN can be regarded as using the input graph structure as the computation graph for message passing (Gilmer et al., 2017), during which the local neighborhood information is aggregated to get a more contextual representation. Formally, it has the following form:

The most important GNN operators are Extract($\cdot$) and Aggregate($\cdot$). Extract($\cdot$) represents the neighbor information extractor. It extract useful information from source node’s representation $H^{l-1}[s]$, with the target node’s representation $H^{l-1}[t]$ and the edge $e$ between the two nodes as query. Aggregate($\cdot$) gather the neighborhood information of souce nodes via some aggregation operators, such as mean, sum, and max, while more sophisticated pooling and normalization functions can be also designed.

Various (homogeneous) GNN architectures have been proposed following this framework. Kipf et al. (Kipf and Welling, 2017) propose graph convolutional network (GCN), which averages the one-hop neighbor of each node in the graph, followed by a linear projection and non-linear activation operations. Hamilton et al. propose GraphSAGE that generalizes GCN’s aggregation operation from average to sum, max and a RNN unit. Velickovi et al. propose graph attention network (GAT) (Velickovic et al., 2018) by introducing the attention mechanism into GNNs, which allows GAT to assign different importance to nodes within the same neighborhood.

Recently, studies have attempted to extend GNNs for modeling heterogeneous graphs. Schlichtkrull et al. (Schlichtkrull et al., 2018) propose the relational graph convolutional networks (RGCN) to model knowledge graphs. RGCN keeps a distinct linear projection weight for each edge type. Zhang et al. (Zhang et al., 2019b) present the heterogeneous graph neural networks (HetGNN) that adopts different RNNs for different node types to integrate multi-modal features. Wang et al. (Wang et al., 2019) extend graph attention networks by maintaining different weights for different meta-path-defined edges. They also use high-level semantic attention to differentiate and aggregate information from different meta paths.

Though these methods have shown to be empirically better than the vanilla GCN and GAT models, they have not fully utilized the heterogeneous graphs’ properties. All of them use either node type or edge type alone to determine GNN weight matrices. However, the node or edge counts of different types can vary greatly. For relations that don’t have sufficient occurrences, it’s hard to learn accurate relation-specific weights. To address this, we propose to consider parameter sharing for a better generalization. Given an edge $e=(s,t)$ with its meta relation as $\langle\tau(s),\phi(e),\tau(t)\rangle$, if we use three interaction matrices to model the three corresponding elements $\tau(s),\phi(e)$, and $\tau(t)$ in the meta relation, then the majority of weights could be shared. For example, in “the first author of” and “the second author of” relationships, their source and target node types are both author to paper, respectively. In other words, the knowledge about author and paper learned from one relation could be quickly transferred and adapted to the other one. Therefore, we integrate this idea with the powerful Transformer-like attention architecture, and propose Heterogeneous Graph Transformer.

To summarize, the key differences between HGT and existing attempts include:

1. (1)

   Instead of attending on node or edge type alone, we use the meta relation $\langle\tau(s),\phi(e),\tau(t)\rangle$ to decompose the interaction and transform matrices, enabling HGT to capture both the common and specific patterns of different relationships using equal or even fewer parameters.

2. (2)

   Different from most of the existing works that are based on customized meta paths, we rely on the nature of the neural architecture to incorporate high-order heterogeneous neighbor information, which automatically learns the importance of implicit meta paths.

3. (3)

   Most previous works don’t take the dynamic nature of (heterogeneous) graphs into consideration, while we propose the relative temporal encoding technique to incorporate temporal information by using limited computational resources.

4. (4)

   None of the existing heterogeneous GNNs are designed for and experimented with Web-scale graphs, we therefore propose the heterogeneous Mini-Batch graph sampling algorithm designed for Web-scale graph training, enabling experiments on the billion-scale Open Academic Graph.

Figure 2 shows the overall architecture of Heterogeneous Graph Transformer. Given a sampled heterogeneous sub-graph (Cf. Section 4), HGT extracts all linked node pairs, where target node $t$ is linked by source node $s$ via edge $e$. The goal of HGT is to aggregate information from source nodes to get a contextualized representation for target node $t$. Such process can be decomposed into three components: Heterogeneous Mutual Attention, Heterogeneous Message Passing and Target-Specific Aggregation.

We denote the output of the $(l)$-th HGT layer as $H^{(l)}$, which is also the input of the $(l$+$1)$-th layer. By stacking $L$ layers, we can get the node representations of the whole graph $H^{(L)}$, which can be used for end-to-end training or fed into downstream tasks.

The first step is to calculate the mutual attention between source node $s$ and target node $t$. We first give a brief introduction to the general attention-based GNNs as follows:

where there are three basic operators: Attention, which estimates the importance of each source node; Message, which extracts the message by using only the source node $s$; and Aggregate, which aggregates the neighborhood message by the attention weight.

For example, the Graph Attention Network (GAT) (Velickovic et al., 2018) adopts an additive mechanism as Attention, uses the same weight for calculating Message, and leverages the simple average followed by a nonlinear activation for the Aggregate step. Formally, GAT has

Though GAT is effective to give high attention values to important nodes, it assumes that $s$ and $t$ have the same feature distributions by using one weight matrix $W$. Such an assumption, as we’ve discussed in Section 1, is usually incorrect for heterogeneous graphs, where each type of nodes can have its own feature distribution.

In view of this limitation, we design the Heterogeneous Mutual Attention mechanism. Given a target node $t$, and all its neighbors $s\in N(t)$, which might belong to different distributions, we want to calculate their mutual attention grounded by their meta relations, i.e., the $\langle\tau(s),\phi(e),\tau(t)\rangle$ triplets.

Inspired by the architecture design of Transformer (Vaswani et al., 2017), we map target node $t$ into a Query vector, and source node $s$ into a Key vector, and calculate their dot product as attention. The key difference is that the vanilla Transformer uses a single set of projections for all words, while in our case each meta relation should have a distinct set of projection weights. To maximize parameter sharing while still maintaining the specific characteristics of different relations, we propose to parameterize the weight matrices of the interaction operators into a source node projection, an edge projection, and a target node projection. Specifically, we calculate the $h$-head attention for each edge $e=(s,t)$ (See Figure 2 (1)) by:

First, for the $i$-th attention head $ATT\text{-}head^{i}(s,e,t)$, we project the $\tau(s)$-type source node $s$ into the $i$-th Key vector $K^{i}(s)$ with a linear projection K-Linear${}^{i}_{\tau(s)}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{\frac{d}{h}}$, where $h$ is the number of attention heads and $\frac{d}{h}$ is the vector dimension per head. Note that K-Linear${}^{i}_{\tau(s)}$ is indexed by the source node $s$’s type $\tau(s)$, meaning that each type of nodes has a unique linear projection to maximally model the distribution differences. Similarly, we also project the target node $t$ with a linear projection Q-Linear${}^{i}_{\tau(t)}$ into the $i-$th Query vector.

Next, we need to calculate the similarity between the Query vector $Q^{i}(t)$ and Key vector $K^{i}(s)$. One unique characteristic of heterogeneous graphs is that there may exist different edge types (relations) between a node type pair, e.g., $\tau(s)$ and $\tau(t)$. Therefore, unlike the vanilla Transformer that directly calculates the dot product between the Query and Key vectors, we keep a distinct edge-based matrix $W^{ATT}_{\phi(e)}\in\mathbb{R}^{\frac{d}{h}\times\frac{d}{h}}$ for each edge type $\phi(e)$. In doing so, the model can capture different semantic relations even between the same node type pairs. Moreover, since not all the relationships contribute equally to the target nodes, we add a prior tensor $\mu\in\mathbb{R}^{|\mathcal{A}|\times|\mathcal{R}|\times|\mathcal{A}|}$ to denote the general significance of each meta relation triplet, serving as an adaptive scaling to the attention.

Finally, we concatenate $h$ attention heads together to get the attention vector for each node pair. Then, for each target node $t$, we gather all attention vectors from its neighbors $N(t)$ and conduct softmax, making it fulfill $\sum_{\forall s\in N(t)}\textbf{Attention}_{HGT}(s,e,t)=\mathbf{1}_{h\times 1}$.

Parallel to the calculation of mutual attention, we pass information from source nodes to target nodes (See Figure 2 (2)). Similar to the attention process, we would like to incorporate the meta relations of edges into the message passing process to alleviate the distribution differences of nodes and edges of different types. For a pair of nodes $e=(s,t)$, we calculate its multi-head Message by:

To get the $i$-th message head $MSG\text{-}head^{i}(s,e,t)$, we first project the $\tau(s)$-type source node $s$ into the $i$-th message vector with a linear projection M-Linear${}^{i}_{\tau(s)}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{\frac{d}{h}}$. It is then followed by a matrix $W^{MSG}_{\phi(e)}\in\mathbb{R}^{\frac{d}{h}\times\frac{d}{h}}$ for incorporating the edge dependency. The final step is to concat all $h$ message heads to get the $\textbf{Message}_{HGT}(s,e,t)$ for each node pair.

With the heterogeneous multi-head attention and message calculated, we need to aggregate them from the source nodes to the target node (See Figure 2 (3)). Note that the softmax procedure in Eq. 3 has made the sum of each target node $t$’s attention vectors to one, we can thus simply use the attention vector as the weight to average the corresponding messages from the source nodes and get the updated vector $\widetilde{H}^{(l)}[t]$ as:

This aggregates information to the target node $t$ from all its neighbors (source nodes) of different feature distributions.

The final step is to map target node $t$’s vector back to its type-specific distribution, indexed by its node type $\tau(t)$. To do so, we apply a linear projection A-Linear_τ(t) to the updated vector $\widetilde{H}^{(l)}[t]$, followed by residual connection (He et al., 2016) as:

In this way, we get the $l$-th HGT layer’s output $H^{(l)}[t]$ for the target node $t$. Due to the “small-world” property of real-world graphs, stacking the HGT blocks for $L$ layers ($L$ being a small value) can enable each node reaching a large proportion of nodes—with different types and relations—in the full graph. That is, HGT generates a highly contextualized representation $H^{(L)}$ for each node, which can be fed into any models to conduct downstream heterogeneous network tasks, such as node classification and link prediction.

Through the whole model architecture, we highly rely on using the meta relation—$\langle\tau(s),\phi(e),\tau(t)\rangle$—to parameterize the weight matrices separately. This can be interpreted as a trade-off between the model capacity and efficiency. Compared with the vanilla Transformer, our model distinguishes the operators for different relations and thus is more capable to handle the distribution differences in heterogeneous graphs. Compared with existing models that keep a distinct matrix for each meta relation as a whole, HGT’s triplet parameterization can better leverage the heterogeneous graph schema to achieve parameter sharing. On one hand, relations with few occurrences can benefit from such parameter sharing for fast adaptation and generalization. On the other hand, different relationships’ operators can still maintain their specific characteristics by using a much smaller parameter set.

By far, we present HGT—a graph neural network for modeling heterogeneous graphs. Next, we introduce the Relative Temporal Encoding (RTE) technique for HGT to handle graph dynamic.

The traditional way to incorporate temporal information is to construct a separate graph for each time slot. However, such a procedure may lose a large portion of structural dependencies across different time slots. Meanwhile, the representation of a node at time $t$ might rely on edges that happen at other time slots. Therefore, a proper way to model dynamic graphs is to maintain all the edges happening at different times and allow nodes and edges with different timestamps to interact with each other.

In light of this, we propose the Relative Temporal Encoding (RTE) mechanism to model the dynamic dependencies in heterogeneous graphs. RTE is inspired by Transformer’s positional encoding method (Vaswani et al., 2017; Shaw et al., 2018), which has been shown successful to capture the sequential dependencies of words in long texts.

Specifically, given a source node $s$ and a target node $t$, along with their corresponding timestamps $T(s)$ and $T(t)$, we denote the relative time gap $\Delta T(t,s)=T(t)-T(s)$ as an index to get a relative temporal encoding $RTE(\Delta T(t,s))$. Noted that the training dataset will not cover all possible time gaps, and thus $RTE$ should be capable of generalizing to unseen times and time gaps. Therefore, we adopt a fixed set of sinusoid functions as basis, with a tunable linear projection T-Linear^***For simplicity, we denote a linear projection L $:\mathbb{R}^{a}\rightarrow\mathbb{R}^{b}$ as a function to conduct linear transformation to vector $x\in\mathbb{R}^{a}$ as: L$(x)=Wx+b$, where matrix $W\in\mathbb{R}^{a+b}$ and bias $b\in\mathbb{R}^{b}$. $W$ and $b$ are learnable parameters for L.$:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$ as $RTE$:

Finally, the temporal encoding relative to the target node $t$ is added to the source node $s$’ representation as follows:

In this way, the temporal augmented representation $\widehat{H}^{(l-1)}$ will capture the relative temporal information of source node $s$ and target node $t$. The RTE procedure is illustrated in the Figure 3.

The full-batch GNN (Kipf and Welling, 2017) training requires the calculation of all node representations per layer, making it not scalable for Web-scale graphs. To address this issue, different sampling-based methods (Hamilton et al., 2017; Chen et al., 2018a, b; Zou et al., 2019) have been proposed to train GNNs on a subset of nodes. However, directly using them for heterogeneous graphs is prone to get sub-graphs that are extremely imbalanced regarding different node types, due to that the degree distribution and the total number of nodes for each type can vary dramatically.

To address this issue, we propose an efficient Heterogeneous Mini-Batch Graph Sampling algorithm—HGSampling—to enable both HGT and traditional GNNs to handle Web-scale heterogeneous graphs. HGSampling is able to 1) keep a similar number of nodes and edges for each type and 2) keep the sampled sub-graph dense to minimize the information loss and reduce the sample variance.

Algorithm 1 outlines the HGSampling algorithm. Its basic idea is to keep a separate node budget $B[\tau]$ for each node type $\tau$ and to sample an equal number of nodes per type with an importance sampling strategy to reduce variance. Given node $t$ already sampled, we add all its direct neighbors into the corresponding budget with Algorithm 2, and add $t$’s normalized degree to these neighbors in line 8, which will then be used to calculate the sampling probability. Such normalization is equivalent to accumulate the random walk probability of each sampled node to its neighborhood, avoiding the sampling being dominated by high-degree nodes. Intuitively, the higher such value is, the more a candidate node is correlated with the currently sampled nodes, and thus should be given a higher probability to be sampled.

After the budget is updated, we then calculate the sampling probability in Algorithm 1 line 9, where we calculate the square of the cumulative normalized degree of each node $s$ in each budget. As proved in (Zou et al., 2019), using such sampling probability can reduce the sampling variance. Then, we sample $n$ nodes in type $\tau$ by using the calculated probability, add them into the output node set, update its neighborhood to the budget, and remove it out of the budget in lines 12–15. Repeating such procedure for $L$ times, we get a sampled sub-graph with $L$ depth from the initial nodes. Finally, we reconstruct the adjacency matrix among the sampled nodes. By using the above algorithm, the sampled sub-graph contains a similar number of nodes per type (based on the separate node budget), and is sufficiently dense to reduce the sampling variance (based on the normalized degree and importance sampling), making it suitable for training GNNs on Web-scale heterogeneous graphs.

Till now we have assumed that each node $t$ is assigned with a timestamp $T(t)$. However, in real-world heterogeneous graphs, many nodes are not associated with a fixed time. Therefore, we need to assign different timestamps to it. We denote these nodes as plain nodes. For example, the WWW conference is held in both 1974 and 2019, and the WWW node in these two years has dramatically different research topics. Consequently, we need to decide which timestamp(s) to attach to the WWW node.

There also exist event nodes in heterogeneous graphs that have an explicit timestamp associated with them. For example, the paper node should be associated with its publication behavior and therefore attached to its publication date.

We propose an inductive timestamp assignment algorithm to assign plain nodes timestamps based on event nodes that they are linked with. The algorithm is shown in Algorithm 2 line 6. The idea is that plan nodes inherit the timestamps from event nodes. We examine whether the candidate source node is an event node. If yes, like a paper published at a specific year, we keep its timestamp for capturing temporal dependency. If no, like a conference that can be associated with any timestamp, we inductively assign the associated node’s timestamp, such as the published year of its paper, to this plain node. In this way, we can adaptively assign timestamps during the sub-graph sampling procedure.

To examine the performance of the proposed model and its real-world applications, we use the Open Academic Graph (OAG) (Sinha et al., 2015; Tang et al., 2008; Zhang et al., 2019a) as our experimental basis. OAG consists of more than 178 million nodes and 2.236 billion edges—the largest publicly available heterogeneous academic dataset. In addition, all papers in OAG are associated with their publication dates, spanning from 1900 to 2019.

To test the generalization of the proposed model, we also construct two domain-specific subgraphs from OAG: the Computer Science (CS) and Medicine (Med) academic graphs. The graph statistics are listed in Table 1, in which P–A, P–F, P–V, A–I, and P–P denote the edges between paper and author, paper and field, paper and venue, author and institute, and the citation links between two papers.

Both the CS and Med graphs contain tens of millions of nodes and hundreds of millions of edges, making them at least one magnitude larger than the other CS (e.g., DBLP) and Med (e.g., Pubmed) academic datasets that are commonly used in existing heterogeneous GNN and heterogeneous graph mining studies. Moreover, the three datasets used are far more distinguishable than previously wide-adopted small citation graphs used in GNN studies, such as Cora, Citeseer, and Pubmed (Kipf and Welling, 2017; Velickovic et al., 2018), which only contain thousands of nodes.

There are totally five node types: ‘Paper’, ‘Author’, ‘Field’, ‘Venue’, and ‘Institute’. The ‘Field’ nodes in OAG are categorized into six levels from $L_{0}$ to $L_{5}$, which are organized with a hierarchical tree. Therefore, we differentiate the ‘Paper–Field’ edges corresponding to the field level.

In addition, we differentiate the different author orders (i.e., the first author, the last one, and others) and venue types (i.e., journal, conference, and preprint) as well. Finally, the ‘Self’ type corresponds to the self-loop connection, which is widely added in GNN architectures. Except the ‘Self’ relationship, which are symmetric, all other relation types $\phi$ have a reverse relation type $\phi^{-1}$.

For all tasks, we use papers published before the year 2015 as the training set, papers between 2015 and 2016 for validation, and papers between 2016 and 2019 as testing. We choose NDCG and MRR, which are two widely adopted ranking metrics (Liu, 2011; Li, 2014), as the evaluation metrics. All models are trained for 5 times and, the mean and standard variance of test performance are reported.

The first class of GNN baselines is designed for homogeneous graphs, including:

• •

  Graph Convolutional Networks (GCN) (Kipf and Welling, 2017), which simply averages the neighbor’s embedding followed by linear projection. We use the implementation provided in PyG.

• •

  Graph Attention Networks (GAT) (Velickovic et al., 2018), which adopts multi-head additive attention on neighbors. We use the implementation provided in PyG.

The second class considered is several dedicated heterogeneous GNNs as baselines, including:

• •

  Relational Graph Convolutional Networks (RGCN) (Schlichtkrull et al., 2018), which keeps a different weight for each relationship, i.e., a relation triplet. We use the implementation provided in PyG.

• •

  Heterogeneous Graph Neural Networks (HetGNN) (Zhang et al., 2019b), which adopts different Bi-LSTMs for different node type for aggregating neighbor information. We re-implement this model in PyG following the authors’ official code.

• •

  Heterogeneous Graph Attention Networks (HAN) (Wang et al., 2019) design hierarchical attentions to aggregate neighbor information via different meta paths. We re-implement this model in PyG following the authors’ official code.

In addition, to systematically analyze the effectiveness of the two major components of HGT, i.e., Heterogeneous weight parameterization (Heter) and Relative Temporal Encoding (RTE), we conduct an ablation study, but comparing with models that remove these components. Specifically, we use $-Heter$ to denote models that uses the same set of weights for all meta relations, and use $-RTE$ to denote models that doesn’t include relative temporal encoding. By considering all the permutations, we have: HGT${}_{-Heter}^{-RTE}$, HGT${}_{-Heter}^{+RTE}$, HGT${}_{+Heter}^{-RTE}$ and HGT${}_{+Heter}^{+RTE}$^‡‡‡Unless other stated, HGT refers to HGT${}_{+Heter}^{+RTE}$..

We use our HGSampling algorithm proposed in Section 4 for all baseline GNNs to handle the large-scale OAG graph. To avoid data leakage, we remove out the links we aim to predict (e.g., the Paper-Field link as the label) from the sub-graph.

The homogeneous GNN baselines assume the node features belong to the same distribution, while our feature extraction doesn’t fulfill this assumption. To make a fair comparison, we add an adaptation layer between the input features and all used GNNs. This module simply conducts different linear projections for nodes of different types. Such a procedure can be regarded to map heterogeneous data into the same distribution, which is also adopted in literature (Zhang et al., 2019b; Wang et al., 2019).

We summarize the experimental results of the proposed model and baselines on three datasets in Table  2. All experiments for the four tasks are evaluated in terms of NDCG and MRR.

The results show that in terms of both metrics, the proposed HGT significantly and consistently outperforms all baselines for all tasks on all datasets. Take, for example, the Paper–Field ($L_{1}$) classification task on OAG, HGT achieves relative performance gains over baselines by 15–19% in terms of NDCG and 18–21% in terms of MRR (i.e., the performance gap divided by the baseline performance). When compared to HAN—the best baseline for most of the cases, the average relative NDCG improvements of HGT on the CS, Med and OAG datasets are 11$\%$, 10$\%$ and 8$\%$, respectively.

Overall, we observe that on average, HGT outperforms GCN, GAT, RGCN, HetGNN, and HAN by 20% for the four tasks on all three large-scale datasets. Moreover, HGT has fewer parameters and comparable batch time than all the heterogeneous graph neural network baselines, including RGCN, HetGNN, and HAN. This suggests that by modeling heterogeneous edges according to their meta relation schema, we are able to have better generalization with fewer resource consumption.

In addition, we also try to implement a baseline that keeps a unique weight matrix for each relation. However, such a baseline contains too many parameters so that our experimental setting doesn’t have enough GPU memory to optimize it. This also indicates that using the meta relation to parameterize weight matrices can achieve competitive performance with fewer resources.

To further evaluate how Relative Temporal Encoding (RTE) can help HGT to capture graph dynamics, we conduct a case study showing the evolution of conference topic. We select 100 conferences in computer science with the highest citations, assign them three different timestamps, i.e., 2000, 2010 and 2020, and construct sub-graphs initialized by them. Using a trained HGT, we can get the representations for these conferences, with which we can calculate the euclidean distances between them. We select WWW, KDD, and NeurIPS as illustration. For each of them, we pick the top-5 most similar conferences (i.e., the one with the smallest euclidean distance) to show how the conference’s topics evolve over time.

As shown in Table 3, these venues’ relationships have changed from 2000 to 2020. For example, WWW in 2000 was more related to some database conferences, i.e., SIGMOD and VLDB, and some networking conferences, i.e., NSDI and GLOBECOM. However, WWW in 2020 would become more related to some data mining and information retrieval conferences (KDD, SIGIR, and WSDM), in addition to SIGMOD and GLOBECOM. Also, KDD in 2000 was more related to traditional database and data mining venues, while in 2020 it will tend to correlate with a variety of topics, i.e. machine learning (NeurIPS), database (SIGMOD), Web (WWW), AI (AAAI), and NLP (EMNLP). Additionally, our HGT model can capture the difference brought by new conferences. For example, NeurIPS in 2020 would relate with ICLR, which is a newly organized deep learning conference. This case study shows that the relative temporal encoding can help capture the temporal evolution of the heterogeneous academic graphs.

To illustrate how the incorporated meta relation schema can benefit the heterogeneous message passing process, we pick the schema that has the largest attention value in each of the first two HGT layers and plot the meta relation attention hierarchy tree in Figure 5. For example, to calculate a paper’s representation, $\langle$Paper, $is\_published\_at$, Venue, $is\_published\_at^{-1}$, Paper$\rangle$, $\langle$Paper, $has\_L_{2}\_field\_of$, Field, $has\_L_{5}\_field\_of^{-1}$, Paper$\rangle$, and $\langle$Institute, $is\_affiliated\_with^{-1}$, Author, $is\_first\_author\_of$, Paper$\rangle$ are the three most important meta relation sequences, which can be regarded as meta paths PVP, PFP, and IAP, respectively. Note that these meta paths and their importance are automatically learned from the data without manual design. Another example of calculating an author node’s representation is shown on the right. Such visualization demonstrates that Heterogeneous Graph Transformer is capable of implicitly learning to construct important meta paths for specific downstream tasks, without manual customization.