A deep graph model for the signed interaction prediction in biological network

All experiments were conducted on a server equipped with an NVIDIA GeForce RTX 3090. The model’s encoder was a two-layer graph convolutional network optimized with Chebyshev polynomials[22], and the model was optimized using the Adam optimizer, with an early stopping mechanism added during the training process. The early stopping function is based on the model’s validation loss value, with a patience of 5. The final model was trained for 50 epochs with a batch size of 8192, and the best model parameters were found at epoch 48. The model was tested using a test size of 8192, and the test results were saved. The list of parameters used in the model is shown in the following Table 1.

In this study, we first introduced a model that integrates the Relational Graph Convolutional Network with a Tensor Decomposition decoder, aimed at improving the accuracy of predicting interactions between genes and chemicals. We evaluated the model’s performance across multiple metrics, including the Area Under the Receiver Operating Characteristic curve (AUROC) and the Area Under the Precision-Recall Curve (AUPRC), as well as the average precision at the top 20 predictions (AP@20).We used three metrics to compare the prediction performances for gene-to-chemical ($P_{\mathrm{gene-chem}}$), chemical-to-gene ($P_{\mathrm{chem-gene}}$), and the average of both predictions ($P_{\mathrm{average}}$). The results (Table 2) indicate that $P_{\mathrm{average}}$ performs the best across all three metrics, with an AUROC of 0.966, AUPRC of 0.961, and AP@20 of 0.978. In contrast, $P_{\mathrm{chem-gene}}$ outperforms $P_{\mathrm{gene-chem}}$ in both AUROC and AUPRC, achieving scores of 0.939 and 0.925 respectively, compared to 0.882 and 0.841 for $P_{\mathrm{gene-chem}}$. In the AP@20 metric, $P_{\mathrm{chem-gene}}$ also shows superior predictive performance (0.824), significantly higher than that of $P_{\mathrm{gene-chem}}$ (0.689). These findings emphasize the superior overall performance of the average prediction model and also reveal potential differences in efficiency between different prediction directions.

The model’s ROC and PRC curves are shown in the following (Fig. 1). A and D shows results predicting genes to chemicals relations ($P_{\mathrm{gene-chem}}$) (Fig. 1 A,D); B and E shows the results predicting chemicals to genes relations ($P_{\mathrm{chem-gene}}$) (Fig. 1 B,E); C and F shows the average predictive performance of the $P_{\mathrm{average}}$ model on both tasks(Fig. 1 C,F).And these curves represent different polarity relationship edges respectively.

The figures reveal that the $P_{\mathrm{average}}$ model rapidly ascends to nearly perfect true positive rates on the ROC curve while maintaining a low false positive rate, indicating its high effectiveness in distinguishing non-interactions between chemicals and genes. In the PRC curve, the $P_{\mathrm{average}}$ model also displays precision and recall rates that exceed those of these two single-link prediction models, especially maintaining high precision at high recall levels, indicating the $P_{\mathrm{average}}$ model’s significant advantage in ensuring the credibility of the prediction results. Overall, these graphs confirm the good performance of our model in prediction tasks.

In our study, “edge conflicts” specifically refer to the conflicts during polarity predictions by the model, such as the same sample being predicted as both “activation” and “inhibition”. To avoid such conflicts, the model has to accurately distinguish between polar edges, such as activation and inhibition, which is crucial for understanding and predicting network behavior. By introducing the Cannot-Link (CL) conflicting edge sampling strategy, our model demonstrates enhanced discriminative capabilities when handling data with polar edges, significantly improving the accuracy in identifying polar relationships.
Although AUC and AP@k provide global view of model performances, they ignore the local intricacy when evaluating heterogenous networks. In our experiments, we explored the impact of the CL conflicting edge sampling strategy when classifying Must-Link (ML)-CL relationships using the RGCNTDDecoder model. Specifically, when applying the CL sampling strategy (YES), the AUROC value reached 0.959, the AUPRC value was 0.945, AP@20 was 0.902. In contrast, without using the CL sampling strategy (NO), the AUROC and AUPRC values were slightly higher, at 0.987 and 0.988 respectively, with AP@20 at 0.986. Further, considering the two new metrics we designed to evaluate the ability of handling polar relations, after using the CL sampling, the $AUC_{polarity}$ increased from 0.378 to 0.696 and the CP@500 also shown a considerable improvement from 0.766 to 0.862. Both metrics demonstrated that the model acquired the capability to distinguish polar relationships (Table 3). This indicates that although the CL sampling strategy may slightly reduce performance on some traditional metrics, it significantly lowers the probability of prediction conflicts when dealing with data of polar relations.

After constructing and evaluating multiple candidate models, we selected the best-performing model for further performance assessment. Our goal was to verify the model’s capability in predicting interactions between chemicals and genes and to understand the impact of different network connectivity on prediction accuracy. The best model’s ROC and PRC curves are shown in the figures (Fig. 2 A, D). In the Receiver Operating Characteristic (ROC) curves, the three lines illustrate the relationship between the true positive rate and the false positive rate at various thresholds. All three lines rapidly ascend to high true positive levels, indicating the model’s high sensitivity to positive cases. Especially, Curve 3 shows the steepest ascent before an increase in false positive rate, indicating its superior discriminative ability. The Precision-Recall (PRC) curves describe the relationship between precision and recall. Here, the closer a curve is to the upper right corner of the graph, the better the model’s performance. All curves maintain high precision at high recall rates, indicating the model’s ability to maintain high precision while keeping a high recall rate.
Additionally, we conducted tests on models employing the CL sampling strategy and those not utilizing any sampling strategy (hereafter referred to as the CL model and the No CL model, respectively) to observe their ability to identify different relational dynamics, particularly in distinguishing edges of opposite polarity. Using a test set of the same data volume, we predicted the probability of four types of relationships (Increase, Decrease, Binding, Affect) for each edge within the model. Notably, Increase and Decrease were considered relations of opposite polarity; the emergence of high probabilities simultaneously in both directions indicated a polarity conflict error, suggesting the model’s inability to differentiate edge polarity in the chemical-gene interactions.

For models before and after the implementation of the CL sampling strategy, we defined the Polarity Degree (C) as the absolute difference between the predicted probabilities of the ”Increase” and ”Decrease” directions for the same edge. We calculated the C values for both models and showed the frequency distribution of C values using histograms (Fig.2 E) and use Equation(12) to make transformation. It was observed that the model without the CL sampling strategy (represented in red) predominantly had a lower polarity degree, whereas the CL model (in blue) showed a higher polarity degree. This indicates that the CL model was more effective in managing or reducing contradictions and could more distinctly differentiate edge polarity.
Further, we marked correctly identified polar relationships with a +1 and incorrect or unrecognizable polar relationships with a -1 during the testing phase, depicted in red and blue lines, respectively. Through aggregate analysis of each line in the testing process (Fig.2 F), it was evident that in the CL model, occurrences and the extent of blue lines were significantly less frequent, indicating that instances of wrong predictions or errors were substantially fewer compared to the No CL model.
To validate the significant effect of our model on handling polar edges after employing the CL sampling strategy, we conducted a validation prediction on the model. We tested the model without the CL sampling strategy and with the CL sampling strategy on a randomly selected set of compounds, ranking the probability of each gene node across all connection scenarios. We separately tallied the top 100 nodes with the highest rankings in the polar relations of ”Increase” and ”Decrease”, converted the ranking values to log2, and plotted the log2-transformed rankings for each gene for increase and decrease scenarios (Fig.3 A-D). We sampled 900 compounds for repeated testing and plotted a boxplot of the average value of the log2 difference of the rankings of the top 100 nodes for each compound (Fig.3 E).

From the figures, it’s evident that in the absence of CL sampling (NO_CL), both the top 100 rankings for increase and decrease show little difference in log2 rank values, and overlap exists. However, after applying CL sampling, the overlap significantly reduces and disappears. Looking at the slope boxplots obtained through repeated testing, after applying CL sampling (CL), the slope values of the boxplots significantly increase, indicating that the ranking changes between the ”Increase” and ”Decrease” states of gene activity are more pronounced. The increase in slope values, especially the higher absolute values observed in the CL boxplot, clearly demonstrates that CL sampling enhances the differentiation between data points, making the distinction clearer in the direction of chemical-gene activity. Overall, CL sampling significantly enhances the differentiation effect of chemical-gene interaction data, which is crucial for understanding and interpreting gene regulatory mechanisms.

As detailed in previous sections, our prediction model and its unique features, including the sampling method and decoder structure we adopted, have been discussed. Specifically, when we attempted to introduce subgraph structures (only chemical-chemical subgraph, only gene-gene subgraph, and both subgraphs) for training to explore their potential impact on the predictive performance of chemical-gene data, the results could have met our expectations. We discovered that, in some instances, adding subgraph structures could actually have a detrimental effect on model performance. Specifically, without using subgraphs, our model’s AUROC, AUPRC, and AP@20 evaluation metrics scored 0.985, 0.985, and 1.00000, respectively. And our two new mertics shown 0.695 and 0.121. However, upon introducing subgraph structures, these metrics experienced a decline (Table 4), indicating that in our model, subgraph structures are not a beneficial addition for enhancing predictive performance and might instead lead to performance degradation due to overcomplexity. This finding poses important considerations for model design: even if specific approaches are effective in other models or datasets, they may not be universally applicable.

This result highlights the independence and efficiency of our model in processing chemical-gene interaction data. It indicates that for the specific dataset, our model has been optimized to a point where introducing subgraph structures not only fails to enhance predictive performance but may even reduce it due to the introduction of unnecessary complexity. Our study underscores the importance of developing models tailored to a dataset’s specific needs and provides practical guidance on balancing increasing model complexity with maintaining or enhancing performance. Future work will be needed to further explore the relationship between model complexity and predictive performance, as well as how to adjust the model structure for chemical-gene datasets to maximize prediction accuracy.

Our model aims to compute the probability of the types of edges of interest given a specific node combination. This model employs an encoder-decoder architecture, where the encoder adopts the Graph Convolutional Network (GCN) principles designed by Yang et al. in the development of BioNet[23], and the decoder utilizes the design of Relational Graph Convolutional Networks (RGCN). During training, a strategy of sampling conflicting edges from signed and unsigned network combinations is adopted to address the diversity of edge types. Our model consists of two main components: GCNEncoder and RGCNTDDecoder. The GCNEncoder is divided into two layers; the first layer uses a graph convolutional layer to capture the features of the original input data; the second layer continues to abstract these features through another graph convolutional layer to form a more refined representation of the nodes. Subsequently, the RGCNTDDecoder combines Relational Graph Convolutional Network (RGCN) layers with the DEDICOM tensor decomposition layer, where the former is responsible for aggregating information according to different relation types, and the latter for parsing global and local interactions, ultimately generating predictions on the interactions between entities. The model’s architecture is shown in (Fig.4), which fully considers the multi-relational nature of graph data, enabling the capture of rich structural information and thereby holding extensive application potential in fields such as bioinformatics and recommendation systems.

In this study, we constructed a network of interactions between compounds and genes for the training of graph neural networks. By delving into open-source databases, we generated a graph for graph neural network training using compound-gene records obtained from the STITCH database[20]. Considering exploring the actual significance of compounds and genes, we mapped the PubChem CID used by the STITCH database[24] to the CHEBI ID in the ChEBI database[25]. The ChEBI database excludes protein molecules encoded by genes and other insignificant small molecules, focusing only on chemically meaningful molecules. Moreover, ChEBI’s ontology provides a rich structure that enables researchers to understand the relationship between specific chemical entities and the broader chemical world[26]. Similarly, we mapped protein IDs used by the STITCH from the String database to the widely used NCBI Gene ID. After data cleaning and removing meaningless compound molecules, we obtained a heterogeneous Chemical-Gene graph containing 12,537 compound nodes, 28,432 gene nodes, and 183,3943 interaction records, existed four relations (Table 5).

Furthermore, we introduced homogeneous graphs such as the Chemical-Chemical and Gene-Gene interaction networks. On the one hand, we aimed to provide the model with more information to enhance its predictive capability; on the other hand, we wanted to observe the impact of complex network information input on model performance. The Gene-Gene interaction network was derived from 1,307,492 interaction records from the String database[27]. Chemical-Chemical interaction data was obtained from 1,770,581 records in the CHEBI database[28].

Usually chemical-gene links and gene-chemical links were predicted separately (Equation(7)).

where $e_{1}$ and $e_{2}$, respectively, represent the constructed embeddings for compounds and genes, and $F$ and $G$ denote the parameters used for tensor decomposition. We combine these two predictions in the following (Equation8):

In our approach, by integrating predictions of chemical-gene and gene-chemical interactions, we aim to overcome biases that single-direction predictions might introduce and to enhance robustness against potential imbalances in the dataset. This fusion method not only improves the model’s generalization ability for complex interactions between chemicals and genes, but also enhances the overall accuracy of the predictions, as validated in our subsequent experiments.

In the chemical-gene prediction task, our dataset presented the challenge of polar edges, where the pair of a chemical and a gene (e.g., Chem A-Gene B) might have signs of direction (i.e. increase, decrease) that are supposed to be mutually exclusive. To effectively handle these polar edges, we adopted a dual sampling strategy that includes ”Must-Link” (ML) and ”Cannot-Link” (CL) constraints[29]. This sampling strategy is defined as follows Equation(9):

The modeling of these links is captured through a custom-designed loss function, which aims to maximize the score difference between $ML$ and $CL$, penalizing the model for polarity predictions. The loss function Equation(10) is defined as follows, where $L$ represents the loss for a set of relationship types $T$ in the network:

Here, $e_{1}$ and $e_{2}$ represent the edges within the same set of relationship types $T$, and $S$ denotes the scoring function that assigns prediction scores to edges based on model output. The loss function enhances separation by subtracting the score of $ML$ from that of $CL$, adding a constant margin of 1 to ensure separation, and applying a hinge function to penalize only when the margin is not met. In our implementation, we specifically targeted the polarity information between the ”Increase” and ”Decrease” relationship types, enhancing the model’s ability to differentiate mutually exclusive interactions. By incorporating this mechanism into the training routine, our model learned more nuanced data representations that reflect the complex nature of biological systems, where some relationships are inherently contradictory. This approach has been proven to improve the model’s discriminative ability, leading to more accurate and biologically reasonable predictions

In this study, we introduced a variety of evaluation metrics to assess our model, in addition to the Area Under the Receiver Operating Characteristic Curve (AUROC) and Area Under the Precision-Recall Curve (AUPRC), as well as AP@20. We have also designed two new metrics specifically focused on the model’s ability to distinguish polar edges. These metrics not only demonstrate the performance of our model in predicting chemical-gene interactions but also offer significant reference value for future related research.