DeepE: a deep neural network for knowledge graph embedding

The structure of a DeepE building block is shown in Fig. 1 (a). x and $\mathcal{F}(\textbf{x})$ are the input and output vectors of a building block. Formally, a DeepE building block is defined as follows:

$\textbf{W}_{1}$ and $\textbf{W}_{2}$ are the weight matrices. $\sigma$ is the non-linear function, and we use ${\rm Relu}$ across the paper. For convenience, we omit the bias terms, dropout layer and Batch Normalization (BN) layer here. Generally, a BN layer Ioffe and Szegedy (2015) and a dropout layer Srivastava et al. (2014) are inserted right after each linear layer, to reduce overfitting and stabilize the training.

The proposed DeepE building block is quite similar to the one of ResNet He et al. (2016a), as shown in Fig. 1 (b). The only difference is that DeepE building block removes the final non-linear layer of ResNet building block. Hence, a DeepE building block outputs a linear feature vector x, and a non-linear feature vector $\textbf{W}_{2}\sigma(\textbf{W}_{1}\textbf{x})$. In contrast, a ResNet building block can only produce a non-linear feature vector $\sigma(\textbf{x}+\textbf{W}_{2}\sigma(\textbf{W}_{1}\textbf{x}))$.

If the dimensions of x and $\textbf{W}_{2}\sigma(\textbf{W}_{1}\textbf{x})$ are not equal, a linear projection $\textbf{W}_{s}$ will be applied to x, to make the element-wise addition available.

In the feature extraction network, the input and output dimensions of the first building block are $2d$ and $d$ respectively, as in Eqn. 2, where $\textbf{W}_{s}\in\mathbb{R}^{d*2d}$. The other building blocks share the same input and output dimensions of $d$, so the compute function is as in Eqn. 1.

The identity mapping of x can well address the problem of network degradation He et al. (2016a), which allows training a network with multiple stacked building blocks. Considering a simple case of stacking 2 building blocks. The superscript $(i)$ denotes the ith building block.

Eqn.3 contains 3 terms, whose non-linearity orders are $0,1,2$ respectively. It can be easily derived that when stacking $n$ DeepE building blocks, the output will consist $n+1$ terms, and their orders of non-linearity should be $0,1,2..n$ respectively. Therefore, deep and shallow features can be easily learned by functions with different non-linear depth. In contrast, stacking $n$ ResNet building blocks can only produce a non-linear function with $2n$th depth, which is not suitable for extracting shallow features. Our experiments compare the performance of ResNet and DeepE building blocks in section 4.3.2.

It is not recommended to add more non-linear layers in a DeepE building block, i.e. $\mathcal{F}(\textbf{x})=\textbf{x}+\textbf{W}_{n}\sigma(\textbf{W}_{n-1}...\sigma(\textbf{W}_{2}\sigma(\textbf{W}_{1}\textbf{x}))...)$. Considering a case of stacking $n$ building blocks, each with 2 inner non-linear layers. The output will consist $n+1$ terms with non-linear orders of $0,2,4,..2n$. Clearly, the more inner non-linear layers a building block contains, the more levels of non-linear order the stacked architecture will loss.

Although identity mapping is benefit for training deep network, it may cause the problem of diminishing feature reuse Srivastava et al. (2015). The gradients will prefer to go through the identity mappings rather than the weights within building blocks.

To address the problem, we propose to dropout the identity mapping with a small ratio $\alpha$. For a network of $n$ stacked building blocks, the $i$th order of non-linear feature vector has to go through the identity mappings for $n-i$ times. Hence, its total dropout probability will be $(1-\alpha)^{n-i}$. Such design will gradually dropout more shallow features than deep features. For example, for a DeepE model with 40 building blocks (the one we use for FB15k-237), $\alpha=0.01$, the final dropout probability of the $0$th, $10$th, $20$th and $30$th order of non-linear features will be 0.331, 0.260, 0.182, 0.096.

Note that He et al. (2016b) also experimented dropout on the identity mapping in ResNet, and found the network failed to converge to a good solution. One probable reason is that they used a dropout value of 0.5, which is too big. As discussed before, such a large value will impede signal propagation.

Following  Dettmers et al. (2018); Sun et al. (2019); Vashishth et al. (2020), three most common benchmark datasets are used. The summary statistics of the datasets are presented in Table 1.

• •

  FB15k-237 Toutanova and Chen (2015) is a improvement version of FB15k Bordes et al. (2013) dataset where the some inverse relations are removed, to avoid data leakage.

• •

  WN18RR Dettmers et al. (2018) is a subset of WN18 Bordes et al. (2013), where the inverse relations are also removed.

• •

  YAGO3-10 Suchanek et al. (2007) is a subset of YAGO3. Most of the triples deal with descriptive attributes of people.

Four metrics are used to measure the performance, including Mean Rank (MR), Mean Reciprocal Rank (MRR), Hit@1 and Hit@10. We follow the filtered settings in Bordes et al. (2013), that is, excluding all true entities appear in the train, valid and test sets when ranking all entities. Note that models with higher MRR, Hit@1, Hit@10 and lower MR are preferred.

We compare our method with various KGE methods, which can be classified into three categories.

• •

  Shallow methods, whose score functions contain no non-linear functions, include DistMult Yang et al. (2014) and RotatE Sun et al. (2019).

• •

  Convolutional neural network (CNN) based methods. These CNN based methods are the dominating techniques among non-linear methods, include ConvEDettmers et al. (2018), HypER Balažević et al. (2019), InteractE Vashishth et al. (2020), AcrE Ren et al. (2020) and JointE Zhou et al. (2022).

• •

  A multi-layer perception (MLP) method, ER-MLP-2d Ravishankar et al. (2017). Although ER-MLP-2d is not a very strong baseline, both the method and DeepE are based on MLP, rather than CNN. Hence, our method is closer to ER-MLP-2d than other baseline methods.

All parameters are random initialized with xavier normal distribution Glorot and Bengio (2010). We use Adam Kingma and Ba (2014) optimizer with an initial learning rate of 0.003. The learning rate will decrease with a coefficient of 0.8 when the training loss does not decrease for 5 epoches. The other training parameters for each dataset are shown in Table 2. When the MRR does not improve for 10 epochs on the valid set, training will be terminated. The best models are selected when the best MRR are obtained on the valid set. We report the average results of 5 runs with different randomly initialization. The maximum training epoch is 1000. In practice, WN18RR and YAGO will finish training in about 200 epochs, and FB15k-237 takes about 700 epochs.

The main results are presented in Table 3. Overall, DeepE achieves the best and the second best results on most metrics. In particular, our method outperforms the baseline methods on MR with a large margin, i.e. 172 -> 161 (-6%) on FB15k-237, 3340->2337 (-30%) on WN18RR, and 1671->591 (-65%) on YAGO3-10. Comparing to MRR, Hit@1 and Hit@10, MR is a metric more concerning about robustness, since a bad prediction will pull down the value a lot. We will analyze the robustness of DeepE in section 4.5.

FB15k-237 is a dataset suitable for deep methods mostly. On FB15k-237, InteractE, AcrE and JointE outperform RotatE on almost all the metrics. On YAGO3-10, the advantage is less significant. InteractE and JointE are better than RotatE on 3 out of 4 metrics. But on WN18RR, the situation changes. RotatE outperforms all other non-linear methods on MR, MRR and Hit@10. The parameter settings of our method (Table 2) also validate the phenomenon. The number of building blocks for the best DeepE models are 40, 2 and 1 on FB15k-237, YAGO3-10 and WN18RR. However, unlike other non-linear methods that suffer from the problem of inconsistent performance, DeepE can work well on all three datasets.

As a closer baseline that based on MLP, ER-MLP2d’s performance is much falling behind DeepE. Note that DeepE maybe the first MLP based method that achieves SOTA results, which validates the effectiveness of our motivation.

Some recent studies try to exploit graph structure information for link prediction on knowledge graphs. These methods often use KGE methods as tools for extracting features Dai et al. (2022); Wu et al. (2021). Since the score function of a graph based method is built on a sub-graph rather than on a single triple, it is meaningless to compare the non-linear orders between these methods and DeepE. Table 4 presents the results of some SOTA graph based methods as an additional comparison with DeepE, include MRGAT Dai et al. (2022), CompGCN Vashishth et al. (2019), ReinceptionE Xie et al. (2020) and DisenKGAT Wu et al. (2021). Although DeepE does not model graph structure information explicitly, it can still achieve first or second best results on most metrics.

The result of ablation analysis is shown in Table 5. Project network is essential for the performance. Without the project network, MRR decreases 0.016 on FB15k-237, and 0.026 on WN18RR. Without the identity dropout, MRR decreases 0.003 on FB15k-237. As discussed in section 2.4, identity dropout is designed for training very deep networks. We do not use it in WN18RR and YAGO3-10, since their best models only contain 1 or 2 DeepE building blocks.

Fig 3 gives the performance on different model depth, i.e. number of building blocks. On FB15k-237, the MRR of the ResNet building blocks drops quickly as the network going deeper. In contrast, DeepE gains precision improvement with deeper layers. The result shows that DeepE building block is essential for training deeper networks for KGE.

On WN18RR, both DeepE or ResNet buidling blocks perform worse when adding more layer, but the result of DeepE is more robust. When stacking a new building block, a network based on ResNet building blocks will replace the old learning function with a new one, while DeepE adds a new function to the original learning function without modifying it. Even if the new function is not suitable, DeepE’s performance will not decrease too much, since the original learning function still works. We believe this is why the building block of DeepE is more robust than that of ResNet on different depth.

A model with only one DeepE building block is trained on FB15k-237, to investigate the effect of the linear and non-linear functions of a building block. The results are shown in Table 6. Th 0th order non-linear function suffices for the accurate prediction on simple relations of N-1. The main contribution of the 1th order non-linear function is on the complex relations of 1-N (with an improvement of 59% on MRR).

The results validate our hypothesis. For a single DeepE building block, the linear function is responsible for learning simple relations, and the non-linear function mainly serves for complex relations.

We train several models with different number of DeepE building blocks. More building blocks result in more orders of non-linear learning functions. The results are shown in Fig. 4. DeepE has addressed the problem of training deep neural networks on KGE: deeper non-linear functions no longer degrade the learning on simple relations. Relations in all groups achieve consistent improvement. For the simplest relation group, N-1, the curve is almost steady on all layers. The results validate the advantage of using different order of non-linear learning functions on KGE.

Learning on entities with low degree is difficult, since the number of training samples are small. However, these entities make up a large proportion of the total entities. For example, 25% entities in FB15k-237 has a degree smaller than 10.

We compare the performance of low-degree entities between DeepE and InteractE on FB15k-237, as in Fig. 5. DeepE is more robust on low degree entities. Specially, the MRR of DeepE is about 3-4 times higher than InteractE when degree is 1 or 2.