Shifu2: A Network Representation Learning Based Model for Advisor-advisee Relationship Mining

Nodes attributes are highly correlated with the network structure in social networks [29]. In order to preserve the proximity of each scholar from the perspective of nodes connection, motivated by the method proposed by Tu et al. [26], we adopt a deep autoencoder to get the embedding representation $A^{\prime}$. The hidden layers of the autoencoder are considered as two parts: the encoder part and the decoder part. The layers consistently encode and decode the input data. The output of the $(i-1)th$ layer is considered as the input of the $i$th layer, which can ensure that the output is equal to the input. The hidden layers can automatically capture the characteristics of input data and keep them unchanged.

For each scholar and his/her collaborators in the collaboration network, we use the adjacency matrix represented by $\bm{A}$ as the input of the node autoencoder, where $\bm{A}_{ij}=1$ if $(v_{i},v_{j})\in E$ and $\bm{A}_{ij}=0$, otherwise. Meanwhile, we consider the attributes listed in TABLE II for each node in the network.

$aa$ represents a scholar’s academic age when he/she began to collaborate with his/her collaborators. It can be calculated as (2):

where $y_{f}$ is the year when $i$ published the first paper and $y_{c}$ is the year when $i$ first co-authored with his/her collaborators. For example, if scholar $i$ published his/her first paper in 1987 and the first co-authored paper with $j$ is published in 2000, then his $aa$ when he first collaborated with $j$ is 13 (2000-1987 = 13).

In each hidden layer, we adopt the following no-linear transformation function:

where $f_{a}$ is the activation function and $\bm{W}_{(i)}^{a}$, $\bm{b}_{(i)}^{a}$ represent the transformation matrix and the bias vector, respectively. $k$ is the total number of hidden layers. Besides, we utilize the Sigmoid function to map vectors of arbitrary real values to the range $[0,1]$.

The goal of the node representation part is to minimize the reconstruction error between the representation vectors and the original input. However, the number of non-zero elements in the input is far less than that of zero elements. Therefore, it is easy to reconstruct zero elements. To address this problem, we impose more penalties on reconstruction error of non-zero elements. According to Wang et al. [30], we add a non-zero penalty matrix $\bm{A^{\prime\prime}}$ and define the objective function as:

where $\bm{A_{n}\parallel A}$ is the concatenation operation of $\bm{A_{n}}$ and $\bm{A}$. Specifically, the concatenation operation $\parallel$ in this equation is to connect two matrices in the horizontal axis. Here we use a toy example to explain this operation clearly. Suppose that there are three nodes $n_{1},n_{2},n_{3}$ in the network $G$ nd the adjacency matrix $\bm{A}=[[0,1,0],[1,0,1],[0,1,0]]$. Each node in G is associated with two attributes and the node attributes matrix is $\bm{A_{n}}=[[2,1],[1,0],[1,1]]$. Then $\bm{A_{n}\parallel A}=[[0,1,0,2,1],[1,0,1,1,0],[0,1,0,1,1]]$. The concatenation operation is implemented by the function $numpy.concatenate((),axis=1)$ in Python. If $\bm{A}_{ij}>0$, then $\bm{A^{\prime\prime}}_{ij}=\rho,\rho>1$, else $\bm{A^{\prime\prime}}_{ij}=1$. By using the loss function as (4), the nodes with similar characteristics will be close in the embedding space.

We adopt Adaptive moment estimation (Adam) [31] method for first-order gradient-based optimization. For each step in the iteration process, we need to compute the gradients $g_{t}$ at time $t$. Then we compute the first moment estimate $m_{t}$ and the second raw moment estimate $v_{t}$, where $m_{t}$ is the mean of the gradient $g$ and $v_{t}$ is the non-central variance of $g_{t}$. According to the bias-corrected $\widehat{m}_{t}$ and $\widehat{v}_{t}$, we can update the parameter $\theta_{t}$. In summary, the optimization process can be described as:

Based on experience and experimental results, in our optimization process, we set 0.9 for $\beta_{1}$, 0.999 for $\beta_{2}$, and $10^{-8}$ for $\epsilon$.

In the edge representation construction process, we also employ the deep autoencoder to convert the attribute matrix to the low-dimensional vector representations. The reconstruction process and the implementation details are presented below.

In order to preserve the edge proximity, we take the edge attribute matrix as the input of edge autoencoder. We consider attribute information listed in TABLE III for each edge in the network.

$kulc_{y}^{ij}$ is the collaboration similarity between scholar $i$ and his/her collaborator $j$ after $t$ years since they first co-authored a paper [32]. It aims to depict the dynamics of collaboration. We compute $kulc$ as:

where $np_{ij}$ is the number of co-authored papers in $t$ years, and $np_{i}$, $np_{j}$ represent the number of publications of $i$, $j$, respectively.

The autoencoder adopts the edge attribute matrix as input. It encodes and decodes the vector in each no-linear transformation layers as:

$f_{e}$ is the activation function and $m$ is the number of hidden layers of the encoder. In the $j$th layer, we use $\bm{W}_{(j)}^{e}$ and $\bm{b}_{(j)}^{e}$ to represent the transformation matrix and the bias vector, respectively. Specifically, the input of the edge autoencoder is dense, so we utilize the dropout [33] to overcome the overfitting. Similar to the process of nodes representation, Sigmoid function is employed as the activation function to get the edge representation matrix $\bm{E}^{\prime}$. The reconstructed loss is computed as:

Hence, we can minimize the distances between the reconstructed representations and the original input.

Some fields contain thousands of scholars. It is difficult to ensure the computational time and memory for the running control with numerous vectors. As a result, we add a pooling layer to compress input features. We first reduce each adjacency vector to 1000 dimensions and calculate the mean of reduced vectors accordingly as the input of encoders.

After the processes of nodes and edges reconstruction, we can obtain the low-dimensional representations of the collaboration network. Then we need to use a supervised classifier to predict the label for each edge.

More specifically, for the unlabeled edge $e=(i,j)$, Shifu2 aims to identify the label for $e$. To achieve this goal, we add a supervised classifier in our Shifu2 model, which takes the output of the last hidden layer as input and returns the classification results. We adopt logistic regression [34] as the classification method in our model. Mathematically, the loss function $L_{lr}$ is computed as:

where $\bm{D}$ is the concatenation of two encoders. $\bm{W}^{l}$ is the weight of logistic regression and $\bm{B}^{l}$ is the bias. In this part, we also use Adam method for stochastic optimization.

After the representation process from the perspectives of both nodes and edges, in order to preserve nodes and edges properties at the same time, we calculate the joint loss function to optimize the objective as:

where $\alpha$ and $\beta$ are two hyperparameters to adjust the weights of the edge encoder and the node encoder.

Since the edge encoder is a process of dimension rise and the node encoder is a process of dimension reduction, the sum of weight and bias of the node encoder is much larger than the edge side. We use the regularization parameter $\gamma$ to balance in the middle, to prevent the value of the node side overweight the edge side. In the experiment, we set $\gamma$ as 0.01. If we combine (10) with (1), (4), (8) and (9), then the overall loss function can be rewritten as:

The goal of the aforementioned model is to minimize the loss function $L_{sum}$. In this paper, we achieve this goal by using back propagation algorithm (BP) with stochastic gradient descent, e.g., $W_{ji}=W_{ji}-\delta\frac{\partial}{\partial W_{ji}}L(X,Y)$. If $z=Wx+b$, the gradient can be defined as:

where $\rho_{j}=\partial L(X,Y)/\partial z_{j}$ is the reconstruction error between the activation and the target. $n$ represents the number of samples.

If $f^{\prime}(.)$ represents the partial derivative of $f(.)$, then the updated error can be summarized as:

The overall architecture of the algorithm is summarized in Algorithm 1.

As mentioned previously, we use the MAG which is a widely used dataset containing scientific publication records to construct the collaboration network. In order to investigate the accuracy and the effectiveness of Shifu2, we need some ground truth advisor-advisee pairs. We extract the ground truth advisor-advisee pairs from The Academic Family Tree (AFT)⁶⁶6https://academictree.org/, which is a user content-driven web database storing academic genealogy. We crawl the realistic advisor-advisee pairs in the fields of Chemistry, Computer Science, Economics, Engineering, Mathematics, and Physics. TABLE IV lists the statistics for advisor-advisee pairs in major research fields.

Based on the ground truth advisor-advisee dataset, which only partially covers the authors in the MAG, we further randomly separate the dataset into two sub-datasets: the training set and the test set. The training set contains advisor-advisee pairs who co-authored the first paper during 2000-2006, and the others are used as the test set.

Before applying Shifu2 onto the entire MAG dataset to generate the academic genealogy, there are a set of challenges that must be addressed:

• •

  Author name disambiguation. The MAG only provides the publication related information such as title, authors, published year, and so on. Authors’ names have not been pre-processed which means that if two scholars have the same name, their publications cannot be distinguished. Duplicated scholars’ personal information challenges the application of the model.

• •

  Disciplinary differences. It is now well established from a variety of studies, that diverse collaborative ties exist in different research fields [35, 36]. The disciplinary differences may affect the performance of the model. It is a challenge that how to eliminate the disciplinary differences and make the model universal.

• •

  Temporal effect. Fig. 4 illustrates that the number of publications in the MAG steadily grows over time, which may introduce influence on the law of advisor-advisee’s collaboration. It is important to make sure that the results are not affected by the temporal effect.

In order to address these challenges, we pre-process the dataset as follows:

Author name disambiguation. To gain the complete and accurate publication records for each scholar, we need to infer their actual identities. Since the MAG dataset does not contain unique author identifiers, we conduct name disambiguation to overcome the problem of duplicate names. Accordingly to Sinatra et al [37], there are two main steps in the author name disambiguation: separation and mergence. The first step is separating all authors apart. It means that we should regard each author in the records as a unique one. The ultimate goal is to reduce duplicate identifies by merging authors iteratively. We consider the authors with identical names as the same individual if they meet one of the following criteria:

1. 1.

   The two authors have been cited each other at least once;

2. 2.

   The two authors have at least one co-author;

3. 3.

   The two authors have at least one identical affiliation.

The name disambiguation process ends up with the condition that there are no author pairs to merge.

The matching process consists of two steps: the first step is to find the scholar in the MAG according to scholars’ names obtained from AFT; the second step is to obtain features we need, such as publications and collaborators of these scholars. In the first step, we adopt a regular expression to present the name of each advisee and match them in the MAG dataset in their own fields. For example, if the scholars crawled from FTA major in computer science, then we match them in the field of computer science in the MAG dataset. We add all matching results in a dictionary as “the scholar’s name in AFT: the scholar’s name in the MAG”, where “the scholar’s name in the AFT” is the primary key of the dictionary. In addition, we establish a set to store all the information of the matched advisees. We match the advisor’s name from matched advisee’s collaborators in the same way. If we can match the advisor in the advisee’s collaborators, we define these two collaborators as the ground truth advisor-advisee pair. We use this method to ensure that the matching of advisor-advisee pairs in the MAG dataset is what we really need. Finally, we can obtain the relevant information and digitalize it as the features for training.

Disciplinary differences elimination. As shown in TABLE IV, the training dataset contains six independent research fields. To observe the model performance, experiments are conducted independently in each field. When applying the model to the entire dataset, we add a disciplinary parameter $\delta^{f}_{y}$ for the field $f$ in $y$ ($y$ represents the year), which is calculated as:

where $np^{f}_{y}$ is the total number of publications in $y$ for $f$, and $\langle np\rangle_{y}$ is the average $np$ calculated over all fields published in $y$. $|F|$ is the number of research categories. Since the papers in the MAG dataset are divided into 19 major disciplines, thus $|F|=19$ in this paper.

Re-scaled number of publications. From Fig. 4, we observe that the growth of publications is in line with exponential distribution. The equation achieved by curve fitting is:

where $p$ is the number of publication in year $y$. In order to compare the collaboration attributes between advisors and advisees starting their collaboration at the different time period, we use a re-scaled parameter, $\tilde{p}$, to gauge the publication records for each scholar:

where $\tau$ is the temporal correction factor, which is calculated as:

When calculating the collaboration features, we use the re-scaled dataset to prevent the model performance from temporal bias.

We compare Shifu2 with the following baseline models:

• •

  DeepWalk [19]. DeepWalk uses random walks to obtain the local information of the network and treats the walks as equivalent sentences. By employing skip-gram, it learns the latent representations for vertices.

• •

  LINE [20]. LINE eliminates the limitation of classical stochastic gradient descent by using the edge-sampling algorithm. It can preserve both local and global structures for following networks: direct/undirect networks and weighted/unweighted networks.

• •

  Node2vec [38]. Node2vec is a semi-supervised method focusing on preserving neighbors’ information for each node in the network. It can capture the diversity of connectivity patterns and learn low-dimensional feature representations for nodes effectively.

• •

  TransNet [26]. TransNet is a knowledge graph-based framework, which translates the interactions between vertices to a translation operation. It considers the semantic information for each edge as a binary.

We also compare Shifu2 with existing solutions for identifying advisor-advisee relationships:

• •

  Shifu [15]. Shifu is a deep learning model based on the stacked autoencoder. It takes scholars’ personal properties and network characteristics as the input.

• •

  TPFG [16]. TPFG is a time-constrained probabilistic model. It considers the task of advisor-advisee relationship as a jointly likelihood objective optimization problem.

• •

  tARMM [17]. tARMM is a deep model based on the improved Refresh Gate Recurrent Units. It is inspired by the idea of variance Recurrent Neural Network models.

• •

  tARMM-Shifu2. In tARMM-Shifu2, we modify the input of tARMM. We use node and edge attributes proposed in Shifu2 as the input of tARMM.

Since the task of advisor-advisee relationship mining can be regarded as a binary prediction problem, i.e., for each collaboration pair $(i,j)$, this is the binary classification of whether $j$ is $i$’s advisor. To evaluate the performance of Shifu2, four widely used metrics for classification tasks are used in our experiments: Accuracy, Precision, Recall, and F1-score.

TABLE V presents the results of comparing Shifu2 with all NRL based baselines. Note that, in the process of parameters setting in node2vec, $p$ and $q$ are selected in the set $\{0.25,0.5,1,2,4\}$. Finally, they are decided by the maximum of accuracy for each discipline. From the results, we can see that the task of advisor-advisee relationship identification achieves significant improvements by our proposed model. It demonstrates the effectiveness of Shifu2.

Besides, we vary the nodes for training from 10% to 100% of the training set. From Fig. 5 we can observe that Shifu2 consistently achieves stable performance in contrast to other baselines under different sizes of training data. Taking the accuracy rate as an example, Shifu2 is 30%, 38%, 28%, and 8% higher than deepwalk, line, node2vec, and Transnet in the field of Chemistry, respectively.

Finally, we compare our model with existing methods for advisor-advisee relationships identification mentioned in Section 4.1.3. As shown in Fig. 6, Shifu2 achieves better performance consistently across all fields. Furthermore, we can observe that by using the node attributes and edges attributes proposed in Shifu2, the performance of the existing method such as tARMM has been improved.

In this subsection, we study the effect of the following parameters: (1) number of hidden layers of the node auto-encoder, (2) number of hidden layers of the edge autoencoder, (3) learning rate, (4) number of nodes used for training, (5) embedding dimension, and (6) input features.

Number of hidden layers. While the training set is determined, we should consider how to grid-search the number of hidden layers. Studies demonstrate that models will achieve better performance with more hidden layers. The highest accuracy can even reach 100%. However, models will be prone to over-fitting, thus the prediction performance will be sharply reduced on the test data. So we need to consider the effective depth of the training to make Shifu2 achieve better prediction performance without over-fitting. The number of units in each hidden layer needs to be determined accordingly.

In this paper, we need to determine the number of hidden layers for the node autoncoder and edge autoencoder, respectively. For the node autoencoder and the edge autoencoder, we choose the hidden layers from 1 to 5. The number of hidden units in each layer is presented in TABLE VI. Fig. 7 and Fig. 8 present the experimental results. Thus we can obtain the best architecture of our proposed model with three hidden layers for the node autoencoder, and two layers for the edge autoencoder.

Learning rate. Learning rate is a crucial parameter in Shifu2 because it controls the update speed of the model. If the learning rate is overlarge, the value of the loss function will move backwards and forwards around the minimum and will not converge. Otherwise, it will lead to a slow learning process. In TABLE VII, we discover that if the learning rate is set to 0.01, the model can achieve the best performance across almost all fields.

Number of nodes used for training. We set the number of nodes for training to be 20%, 40%, 60%, and 80% of the entire training data, respectively, to evaluate the sensitivity with respect to the size of training set. TABLE VIII presents the identification performance of the model. From the results, we observe that our model is well performed even though the training data is sparse. Shifu2 can reach the accuracy higher than 90% in all research fields with only 1000 nodes, which indicates the robustness of Shifu2.

Embedding dimension. After learning, we will get node representations and edge representations in terms of $n$-dimensional vectors, where $n$ is the artificially set embedding dimension. To verify whether it influences the performance of the model, we vary it from 20 to 160 in Shifu2. The results are shown in Fig. 9. From the results we can see that, our model performs well with both low-dimensional representations and high-dimensional representations, which can be proved by the accuracy higher than 90% with different embedding dimensions. We also observe that the disciplinary differences do exist. For example, the performance of the model in terms of the accuracy keeps increasing with larger embedding dimensions for all research fields except chemistry. For chemistry, the accuracy first presents an increasing trend. After reaching a peak, it begins to decline.

Input features. A key ingredient of Shifu2 is the exploitation of node attributes and edge attributes. To validate the effectiveness of this mechanism, here we focus on examining the performance of Shifu2 without the node autoencoder (represented as Shifu2-E in TABLE IX) first. From TABLE IX, we can conclude that if we only use the edge autoencoder, it will achieve a relatively poor performance. The use of node attributes clearly improves the performances.

An advisee is usually supervised by an advisor for a specific period of time (i.e., not forever), during which they collaborate with each other closely. In many (if not most) cases, for instance, it takes about 3 to 5 years for a PhD student to graduate from the university. After graduation, many students carry out their own research work, with much less or even no collaboration with their (PhD) supervisors. As a result, the attributes of collaboration networks will change. For example, collaboration frequency might decrease. On the other hand, it is difficult to determine when advisees will collaborate with their advisors since the collaboration pattern may vary from case to case.

Considering the above observations, we conduct experiments to examine the influence of edge attributes with different time lengths. Fig. 10 describes the effect of feature selection period with different durations in the edge autoencoder. Note that here we use only the edge autoencoder to avoid the influence of node attributes (i.e., academic age).

We observe that the overall trend of all metrics increases with more investigated years in the fields of Physics, Computer Science, and Economics. Particularly, the edge autoencoder can achieve better performance if we adopt the collaboration information of first 7 years. It is noticeable that the performance differs from one field to another. The reason accounting for this phenomenon is the differences in collaboration patterns between advisees and advisors which exist among the disciplines.