Dual-Geometric Space Embedding Model for Two-View Knowledge Graphs

We propose to embed the entities from the instance-view on the surface of the spherical ball from the spherical space, $\mathbb{S}^{d}$, in order to better capture the cyclic structures present in this KG view. Entities are represented as points $\bm{h}_{e_{i}}$ that belong to the surface of the spherical ball in $\mathbb{S}^{d}$ as shown in Figure 1(b). Formally, $\mathbb{S}^{d}=\{\bm{h}_{e_{i}}^{S}\in\mathbb{R}^{d}\big{|}\lVert\bm{h}_{e_{i}}^{S}\rVert=w^{S}\},w^{S}\in[0,1)$, is the $d$-dimensional $w^{S}$-norm ball, where $\lVert\cdot\rVert$ is the Euclidean norm.

For an entity $\bm{h}_{e_{i}}^{S}$, we propose to model the relation applied to the entity as a rotation operation, and therefore represent the relation $r_{Ik}$ as a vector of angles $\bm{\theta}_{r_{Ik}}^{S}\in[0,2\pi)^{d-1}$. Below, we show our proposed vector transformation procedure to model the relation as rotation of the head entity, $f_{\mathrm{rot}}:\bm{h}_{e_{i}}^{S,new}=f_{\mathrm{rot}}(\bm{h}_{e_{i}}^{S},\bm{\theta}_{r_{Ik}}^{S})$, and prove that the operation is closed in the spherical space. $f_{\mathrm{rot}}(\cdot)$ eliminates the need to project representations to the tangent Euclidean space in order to perform standard transformations. To the best of our knowledge, we are the first to propose a closed vector transformation procedure that operates directly in the spherical space.

This section describes our proposed vector transformation procedure, $f_{\mathrm{rot}}$, for the spherical space, which directly performs operations using properties, such as positive curvature, in the spherical space. This section also outlines the proof of the closedness property of the transformation procedure. For computational efficiency, we also extend the vector transformation procedure to the batch version, the batch vector transformation procedure, which is detailed in the Supplements. $f_{\mathrm{rot}}$ takes as input an entity embedding and relation operator, which it uses to transform the entity embedding: $\bm{h}_{e_{i}}^{S,new}=f_{\mathrm{rot}}(\bm{h}_{e_{i}}^{S},\bm{\theta}_{r_{Ik}}^{S})$:

1. (1)

   Given $\bm{h}_{e_{i}}^{S},\bm{\theta}_{r_{Ik}}^{S}$, we convert $\bm{h}_{e_{i}}^{S}$ to the corresponding representation $\bm{\theta}_{e_{i}}^{S}$ in polar coordinates, with $\mathit{rad}$ denoting the radius which has the value $w^{S}$. Refer to Section A.1 for details on the conversion procedure.

   (3)

   $$\bm{h}_{e_{i}}^{S}:[x_{e_{i},1}^{S},x_{e_{i},2}^{S},...,x_{e_{i},d}^{S}]\rightarrow\bm{\theta}_{e_{i}}^{S}:[\theta_{e_{i},1}^{S},\theta_{e_{i},2}^{S},...,\theta_{e_{i},d-1}^{S}];\mathit{rad}=w^{S}$$

   (4)

   $$\bm{\theta}_{r_{Ik}}^{S}:[\theta_{r_{Ik},1},\theta_{r_{Ik},2},...,\theta_{r_{Ik},d-1}];\mathit{rad}=w^{S}$$

2. (2)

   Denote $\bm{z}[l]$ to be the $l$-th entry of $\bm{z}$ and apply the transformation:

   (5)

   $$(\bm{\theta}_{e_{i}}^{S}+\bm{\theta}_{r_{Ik}}^{S})[l]=(\bm{\theta}_{e_{i}}^{S}[l]+\bm{\theta}_{r_{Ik}}^{S}[l])\ \mathrm{mod}\ 2\pi;l\in[1,d-1]$$

   (6)

   $$\bm{\theta}_{e_{i}}^{S,new}=[(\theta_{e_{i},1}^{S}+\theta_{r_{Ik},1}^{S})\ \mathrm{mod}\ 2\pi,...,\\ (\theta_{e_{i},d-1}^{S}+\theta_{r_{Ik},d-1}^{S})\ \mathrm{mod}\ 2\pi];\mathit{rad}=w^{S}$$

3. (3)

   Convert from polar coordinates back to Cartesian coordinates. Refer to Section A.2 for details on the conversion procedure.

   (7)

   $$\bm{\theta}_{e_{i}}^{S,new}\rightarrow\bm{h}_{e_{i}}^{S,new}:[x_{e_{i},1}^{S,new},x_{e_{i},2}^{S,new},...,x_{e_{i},d}^{S,new}]$$

The vector transformation procedure, $f_{\mathrm{rot}}$, is closed in the spherical space.

The instance-view model uses a hinge loss function that is maximized for all triples in the instance space $R_{I}$, with positive triples denoted, $\mathrm{pt}_{I}=(\bm{h}_{e_{i}}^{S},\bm{\theta}_{r_{Ik}}^{S},\bm{h}_{e_{j}}^{S})$ with corresponding score $\phi^{S}_{\mathrm{obs}}(\mathrm{pt}_{I})$ and negative triples denoted, $\mathrm{nt}_{I}=(\bm{h}_{e_{i}}^{S^{\prime}},\bm{\theta}_{r_{Ik}}^{S},\bm{h}_{e_{j}}^{S^{\prime}})$ with corresponding score $\phi^{S}_{\mathrm{corr}}(\mathrm{nt}_{I})$, and $\gamma^{R_{I}}>0$ is a positive margin hyperparameter. Specifically, the instance loss function measures the distance between the predicted tail entity and the ground truth.

We calculate spherical geodesic distance (Meng et al., 2019) between points $\bm{x}^{S}$ and $\bm{y}^{S}$ on the manifold as follows:

$R_{I}$ includes links between all entities including both non-bridge entities and bridge-entities. In this way, the representation of non-bridge entities is influenced by the bridge entities.

We propose to embed concepts from the ontology-view on the Poincaré disk from the hyperbolic space, $\mathbb{H}^{d}$, in order to better capture the hierarchical structures present in this KG view. Concepts are represented as points, $\bm{h}_{c_{i}}$ that belong inside the Poincaré disk in $\mathbb{H}^{d}$ as shown in Figure 1(b). Formally, $\mathbb{H}^{d}=\{\bm{h}_{c_{i}}^{H}\in\mathbb{R}^{d}\big{|}\lVert\bm{h}_{c_{i}}^{H}\rVert=w_{c_{i}}^{H}\},w_{c_{i}}^{H}\in[0,1)$, is the $d$-dimensional $w_{c_{i}}^{H}$-norm ball, where $\lVert\cdot\rVert$ is the Euclidean norm. We assume the center of the disk is aligned with the center of the sphere, and for convenience set the last dimension $d$ to 0.

For concepts in the hyperbolic space, $\mathbb{H}^{d}$, any hyperbolic space model for KG can be applied in principle, which we denote as follows with $r_{\mathcal{O}k}\in R_{\mathcal{O}}$ to denote a relation between two concepts:

We illustrate MuRP (Balaževic´ et al., 2019) as an example scoring function, which uses the hyperbolic geodesic distance and relies on Möbius addition to model the relation, where $\mathrm{exp}_{0}(\cdot)$ and $\mathrm{log}_{0}(\cdot)$ are defined in Section 2.2, $\bm{R}\in\mathbb{R}^{d\times d}$ is a learnable diagonal relation matrix representing the stretch transformation by relation $r_{\mathcal{O}k}\in R_{\mathcal{O}}$ with representation $\bm{h}_{r_{\mathcal{O}k}}^{H}$, and scalar biases $b_{c_{i}}^{H},b_{c_{j}}^{H}$ of concepts $c_{i}$ and $c_{j}$:

We calculate hyperbolic geodesic distance (Nickel and Kiela, 2017) between points $\bm{x}^{H}$ and $\bm{y}^{H}$ on the manifold as follows:

The ontology-view model uses a hinge loss function that is maximized for all links between concepts in the ontology space and all links between bridge nodes and concepts, i.e., $R_{\mathcal{O}}\cup R_{I\mathcal{O}}$, with positive triples denoted, $\mathrm{pt}_{\mathcal{O}}=(\bm{h}_{c_{i}}^{H},\bm{\theta}_{r_{\mathcal{O}k}}^{H},\bm{h}_{c_{j}}^{H})\cup(\bm{h}_{e_{i}}^{H},\bm{h}_{r_{I\mathcal{O}k}},\bm{h}_{c_{j}}^{H})$ with corresponding score $\phi^{H}_{obs}(\mathrm{pt}_{\mathcal{O}})$ and negative triples denoted, $\mathrm{nt}_{\mathcal{O}}=(\bm{h}_{c_{i}}^{H^{\prime}},\bm{\theta}_{r_{\mathcal{O}k}}^{H},\bm{h}_{c_{j}}^{H^{\prime}})\cup(\bm{h}_{e_{i}}^{H^{\prime}},\bm{h}_{r_{I\mathcal{O}k}},\bm{h}_{c_{j}}^{H^{\prime}})$ with corresponding score $\phi^{H}_{corr}(\mathrm{nt}_{\mathcal{O}})$, and $\gamma^{R_{\mathcal{O}}\cup R_{I\mathcal{O}}}>0$ is a positive margin hyperparameter:

$R_{O}\cup R_{I\mathcal{O}}$ includes triples composed of concepts and bridge-entities. In this way, the representation of concepts is influenced by the bridge entities, which is described in Section 3.1.3.

Bridge nodes are entity nodes that bridge the communication between the instance-view and ontology-view components. Bridge nodes are connected to concepts in the graph, formed by link ontology $\bm{h}_{r_{I\mathcal{O}k}}$ but may also be connected to other entities, formed by link instance $\bm{\theta}_{r_{\mathcal{I}k}}$. As shown in Figure 1(a), nodes 5 and 6 are bridge nodes that are involved in both cyclic and hierarchical structures through their links to other entities as well as concepts. As such, we propose to embed bridge entities in the intersection space of the Poincare disk and surface of the spherical ball, in order to better capture the heterogeneous KG structures that are associated with these nodes. We refer to the intersection submanifold embedding space as the bridge space, $\mathbb{B}^{d}$, where the representation of these nodes are informed by both $\mathbb{S}^{d}$ and $\mathbb{H}^{d}$ and has one lower degree of freedom. The bridge space can therefore be derived as a sphere in general. Formally, $\mathbb{B}^{d}=\{\bm{h}_{e_{i}}^{B}\in\mathbb{R}^{d}\big{|}\lVert\bm{h}_{e_{i}}^{B}\rVert=w^{S}\},w^{S}\in[0,1)$, is the $d$-dimensional $w^{S}$-norm ball, where $\lVert\cdot\rVert$ is the Euclidean norm, and the value of last dimension $d=0$. Links associated with bridge nodes are $\bm{\theta}_{r_{Ik}},\bm{h}_{r_{I\mathcal{O}k}}\in[0,2\pi)^{d-1}$, and operations on bridge nodes, such as geodesic distance and loss functions, happen in either the spherical space or hyperbolic space.

To ensure compatibility with the connected concept nodes, we map the bridge entities, $\bm{h}_{e_{i}}^{B}$, to an embedding in the ontology space through a non-linear transformation function, $g_{\bm{h}_{r_{I\mathcal{O}k}}}(\bm{h}_{e_{i}}^{B})$, where $\mathrm{AGG}(\cdot)$ denotes an averaging over all relations $k$ in $R_{IO}$. Logarithmic and exponential mapping functions of $\mathrm{log}_{0}(\cdot)$ and $\mathrm{exp}_{0}(\cdot)$ are described in Section 2.2.

where both the weight matrix $\bm{W}_{h_{r_{I\mathcal{O}k}}}$ and bias $\bm{b}_{h_{r_{I\mathcal{O}k}}}$ are specific to each relation $k$ in $R_{IO}$ and reserved for the ontology $\bm{h}_{r_{I\mathcal{O}k}}$.

The bridge-node model uses a hinge loss function as a combination of the entity’s ontology-specific loss, $\mathrm{ontoLoss}_{\bm{h}_{r_{I\mathcal{O}k}}}$, and instance-specific loss, $\mathrm{instLoss}_{\bm{\theta}_{r_{Ik}}}$, that is maximized for $R_{I}\cup R_{IO}$ which contains all triples associated with the bridge nodes. Positive triples are denoted $\mathrm{pt}_{B,I}=(\bm{h}_{e_{i}}^{B},\bm{\theta}_{r_{Ik}},\bm{h}_{e_{j}}^{B})$ and $\mathrm{pt}_{B,IO}=(\bm{h}_{e_{i}}^{B},\bm{h}_{r_{IOk}},\bm{h}_{c_{j}}^{B})$, and negative triples are denoted $\mathrm{nt}_{B,I}=(\bm{h}_{e_{i}}^{B^{\prime}},\bm{\theta}_{r_{Ik}},\bm{h}_{e_{j}}^{B^{\prime}})$ and $\mathrm{nt}_{B,IO}=(\bm{h}_{e_{i}}^{B^{\prime}},\bm{h}_{r_{IOk}},\bm{h}_{c_{j}}^{B^{\prime}})$ with loss function defined as follows:

The combination loss function above enables bridge nodes to learn from both spaces of intersection of $\mathbb{S}^{d}$ and $\mathbb{H}^{d}$.