IMF: Interactive Multimodal Fusion Model for Link Prediction

In order to fully exploit the complicated interaction between different modalities, we propose a two-stage fusion model instead of simply considering the multimodal information separately in a unified vector space. As shown in Figure 2, IMF consists of four key components:

• 1

  The Modality-Specific Encoders are used for extracting structural, visual and textual features as the input of multimodal fusion stage.

• 2

  The Multimodal Fusion Module, which is the first fusion stage, effectively models bilinear interactions between different modalities based on Tucker decomposition and contrastive learning.

• 3

  The Contextual Relational Model calculates the similarity of contextual entity representations to formulate triple scores as modality-specific predictions for decision fusion stage.

• 4

  The Decision Fusion Module, which is the second fusion stage, takes all the similarity scores from structural, visual, textual and multimodal models into account to make the final prediction.

In this subsection, we first introduce the pre-trained encoders used for different modalities. These encoders are not fine-tuned during training and we treat them as fixed feature extractors to obtain the modality-specific entity representations. Note that IMF is a general framework and it is straightforward to replace them with other up-to-date encoders or add ones for new modalities into IMF.

The multimodal fusion stage aims to effectively get multimodal representations, which fully capture the complex interactions between different modalities. Many existing multimodal fusion methods have achieved promising results in many tasks like VQA (Visual Question Answering). However, most of them aim at finding the commonality to get more precise representations by modality projecting (Frome et al., 2013; Collell et al., 2017) or cross-modal attention (Perez et al., 2018). These types of methods will suffer from the loss of unique information in different modalities and can not achieve sufficient interaction between modalities. To this end, we propose to employ the bilinear models, which have a strong ability to realize full parameters interaction as the cornerstone to perform the fusion of multimodal information. Specifically, we extend the Tucker decomposition, which decomposes the tensor into a core tensor transformed by a matrix along with each mode to 4-mode factors as expressed in Equation (3):

where $\mathbf{M}_{s}\in\mathbb{R}^{d_{s}\times t_{s}}$, $\mathbf{M}_{v}\in\mathbb{R}^{d_{v}\times t_{v}}$, $\mathbf{M}_{t}\in\mathbb{R}^{d_{t}\times t_{t}}$, $\mathbf{M}_{d}\in\mathbb{R}^{\mathcal{D}\times t_{d}}$ denotes transformation matrix and $\mathcal{P}_{c}\in\mathbb{R}^{t_{s}\times t_{v}\times t_{t}\times t_{d}}$ denotes a smaller core tensor.

In such a situation, entity embeddings are first projected into a low-dimensional space and then fused with the core tensor $\mathcal{P}_{c}$. Following (Ben-younes et al., 2017), we further reduce the computation complexity by decomposing the core tensor $\mathcal{P}_{c}$ to merge representations of all modalities into a unified space with element-wise product. The detailed calculation process is expressed as Equation (4):

where $\tilde{\mathbf{e}}_{k}=\mathrm{ReLU}(\mathbf{e}_{k}\mathbf{M}_{k})\in\mathbb{R}^{t_{k}}$ denotes latent representations and $\mathbf{e}_{k}\in\mathbb{R}^{d_{k}}$ is the original embedding representations and $\mathbf{M}_{d}^{k}\in\mathbb{R}^{t_{k}\times t_{d}}$ is decomposed transformation matrix for each modality $k\in\{s,v,t\}$.

However, the multimodal bilinear fusion has no bound limitation while the gradient produced by the final prediction result can only implicitly guide parameter learning. To alleviate this problem, we add constraints to limit the correlation between different modality representations of the same entity to be stronger. Therefore, we further leverage contrastive learning (Chen et al., 2020; Li et al., 2021; Yuan et al., 2021) between different entities and modalities as an additional learning objective for regularization. In the settings of contrastive learning, we take the pairs of representations of the same entity of different modalities as positive samples and the pairs of representations of different entities as negative samples. As shown in Figure 3, we aim at limiting the distance of negative samples to be larger than positive samples to enhance multimodal fusion, which is:

where $d(\cdot,\cdot)$ denotes the distance measure and $f(\cdot)$ denotes the embedding function. The superscript $+,-$ represent the positive and negative samples, respectively.

Specifically, we randomly sample $N$ entities from the entity set as a minibatch and define contrastive learning loss upon it. The positive pairs are naturally obtained with the same entities while the negative pairs are constructed by negative sharing (Chen et al., 2017) of all other entities. We take the latent representations $\tilde{\mathbf{e}}_{k}=\mathrm{ReLU}(\mathbf{e}_{k}\mathbf{M}_{k})\in\mathbb{R}^{t_{k}}$ and leverage cosine similarity $d(u,v)=-\mathbf{u}^{\mathsf{T}}\mathbf{v}/||\mathbf{u}||\mathbf{v}||$ as distance measure. Then we have the following contrastive loss function for each entity $i$:

where $\mathcal{M}=\{(s,v),(s,t),(v,t)\}$ is set of modality pairs.

After obtaining representations of each modality and multimodal, we then design a contextual relational model, which takes relations in triples as contextual information for scoring, to get the predictions. Note that this relational model can be easily replaced by any scoring function like TransE.

Due to the variety and complexity of relations in KGs, we argue that improving the degree of parameter interaction (Vashishth et al., 2020) is crucial for better modeling the relational triples. The degree of parameter interaction means the calculation ratio of each parameter to all other parameters. For example, dot product could achieve $1/d$ degree while cross product could achieve $(d-1)/d$ degree. Based on this assumption, we propose to use bilinear outer product between entity and relation embeddings to incorporate contextual information into entity representations. Instead of taking relations as input as in previous studies, our contextual relational model utilizes relations to provide context in the transformation matrix of entity embeddings. Then, entity embeddings are projected using the contextual transformation matrix to get contextual embeddings, which are used for calculating similarity with all candidate entities. The learning objective is to minimize the binary cross-entropy loss. For each modality $k\in\mathcal{K}$, the computation details are shown as Equation (7) to Equation (9):

where $\mathbf{e}_{k}$ and $\hat{\mathbf{e}}_{k}$ are original and contextual entity embeddings respectively; $\mathbf{W}_{k}^{r}=\mathbf{W}_{k}\mathbf{r}$ denotes contextual transformation matrix which is obtained by matrix multiplication of weight matrix $\mathbf{W}_{k}$ and relation vectors $\mathbf{r}$ while $\mathbf{b}_{k}$ is a bias vector; $\sigma$ is sigmoid function and $\mathbf{y}_{k}=[y_{1,k},y_{2,k},...,y_{N,k}]$ is final prediction of modality $k$.

Existing multimodal approaches mainly focus on projecting different modality representations into a unified space and predicting with commonality between modalities, which will fail to preserve the modality-specific knowledge. We alleviate this problem in the decision fusion stage by joint learning and combining predictions of different modalities to further leverage the complementarity.

Under the multimodal settings, we assign different contextual relational models for each modality and utilize their own results for training in different views. Recall the contrastive learning loss in Equation (6), the overall training objective is to minimize the joint loss shown in Equation (10):

where $\mathcal{L}_{k}$ denotes binary cross entropy loss for modality $k$ as Equation (9) and $\gamma_{k}$ is a learned weight parameter.

To better illustrate the whole training process of IMF, we describe it via the pseudo-code of the optimization algorithm. As shown in Algorithm 1, we first obtain the pre-trained encoders of structural, visual and textual and utilize them for entity embeddings (line 3-5). Since the pre-trained models are much larger and more complex than IMF, they are not fine-tuned and their outputs are directly used as inputs of IMF. The multimodal embeddings are obtained by multimodal fusion while contrastive learning is applied to further enhance the fusion stage (line 9-11). During training, each modality delivers its own prediction and loss via the modality-specific scorers (line 12), and then the joint prediction and loss are computed based on all modalities including multimodal ones (line 14).

For inference, we propose to jointly consider the predictions of each modality as well as multimodal ones. Specifically, the overall predictions are shown in Equation (11):

where $\gamma_{k}$ denotes weight for modality $k$ as same as Equation (10) while the values in $\mathbf{y}$ are in [0, 1].

In this paper, we use four public datasets to evaluate our model. All the datasets consist of three modalities: structural triples, entity images and entity descriptions. DB15K, FB15K and YAGO15K datasets are obtained from MMKG⁵⁵5https://github.com/nle-ml (Liu et al., 2019), which is a collection of multimodal knowledge graph. Specifically, we utilize the relational triples as structural features, entity images as visual features and we extract the entity descriptions from Wikidata (Vrandecic and Krötzsch, 2014) as textual features. FB15K-237⁶⁶6https://www.microsoft.com/en-us/download/details.aspx?id=52312 (Toutanova and Chen, 2015) is a subset of FB15K, the visual and textual features in FB15K can be directly reused. Each dataset is split with 70%, 10% and 20% for training, validation and test. The detailed statistics are shown in Table 1.

In the process of evaluation, we consider four metrics of valid entities to measure the model performance following previous studies: (1) mean rank (MR); (2) mean reciprocal rank (MRR); (3) hits ratio (Hits@1 and Hits@10).

To demonstrate the effectiveness of our model, we choose two types of methods for comparison, which are monomodal methods and multimodal methods.

For monomodal models, we take the baselines including:

• •

  TransE (Bordes et al., 2013) defines relations as transformations between entities and designs an energy function of relational triples as scoring function.

• •

  ConvE (Dettmers et al., 2018) converts 1D entity and relation embeddings into 2D embeddings and utilizes Convolutional Neural Network (CNN) to model the interactions between entities and relations.

• •

  ConvKB (Nguyen et al., 2018) employs CNN on the concatenated embeddings of relational triples to compute the triple scores.

• •

  CapsE (Nguyen et al., 2020) utilizes Capsule Network (Sabour et al., 2017) to capture the complex interactions between entities and relations for prediction.

• •

  RotatE (Sun et al., 2019) introduces rotation operations between entities to represent relations in the complex space to infer symmetry, antisymmetry, inversion and composition relation patterns.

• •

  QuatE (Zhang et al., 2019) extends rotation of the knowledge graph embeddings in the complex space into the quaternion space to obtain more degree of freedom.

• •

  KBAT (Nathani et al., 2019) leverages Graph Attention Network (GAT) (Velickovic et al., 2018) as encoder to aggregate neighbors and employs ConvKB as decoder to compute triple scores.

• •

  TuckER (Balazevic et al., 2019) applies Tucker decomposition to capture the high-level interactions between entity and relation embeddings.

• •

  HAKE (Zhang et al., 2020a) projects entities into polar coordinate system to model hierachical structures for incorporating semantics.

For multimodal models, we take the baselines including:

• •

  IKRL (Xie et al., 2017) utilizes the TransE energy function as scoring function on each pair of modalities for joint prediction.

• •

  MKGC (Sergieh et al., 2018) extends IKRL with combination of different modalities to explicitly deliver alignment between modalities.

• •

  MKBE (Pezeshkpour et al., 2018) employs DistMult (Yang et al., 2015) as scoring function and designs Generative Adversarial Network (GAN) (Goodfellow et al., 2014) to predict missing modalities.

For the ablation study, we design three variants of IMF: IMF (w/o MF) utilizes only structural information; IMF (w/o DF) simply takes multimodal representations for training and inference without decision fusion; IMF (w/o CL) removes the contrastive learning loss.

The experiments are implemented on the server with an Intel Xeon E5-2640 CPU, a 188GB RAM and four NVIDIA GeForce RTX 2080Ti GPUs using PyTorch 1.6.0. The model parameters are initialized with Xavier initialization and are optimized using Adam (Kingma and Ba, 2015) optimizer. The evaluation is conducted under the RONDOM settings (Sun et al., 2020), where the correct triples are placed randomly in test set and the negative sampling are correctly employed without test leakage.

For DB15K, FB15K and YAGO15K, we obtain the results by running all the baselines with their released codes. For FB15K-237, we directly obtain the results of TransE, ConvE, ConvKB, CapsE, RotatE, KBAT and TuckER from the re-evaluation work (Sun et al., 2020) and run the models of QuatE, HAKE, IKRL, MKGC and MKBE with their released codes.

Note that the methods with other enhancing techniques, such as data augmentation (Bamler et al., 2019; Zheng et al., 2022b, a; Li et al., 2022) or AutoML (Zhang et al., 2020b; Zhao et al., 2021b; Zhao et al., 2021a; Lin et al., 2022; Wang et al., 2022) are orthogonal to our approach for comparison.

As shown in Table 2 and Table 3, we can observe that:

• •

  IMF significantly outperforms all the baselines. The performance gain is at most 42% for MRR on DB15K while is also more than 20% for all the evaluation metrics on average.

• •

  State-of-the-art monomodal methods employ a variety of complex models to improve the expressiveness and capture latent interactions. However, the results illustrate that the performance is highly limited by the structural bias of the nature of knowledge graph itself. Although these methods have already achieved promising results, IMF can easily outperform them by a significant margin with a much simpler model structure, which amply demonstrates the effectiveness.

• •

  In comparison with multimodal methods that treat the features of different modalities separately, our IMF jointly learning from different modalities with the two-stage fusion, which is beneficial in modeling the commonality and complementarity simultaneously.

Overall, our proposed IMF can model more comprehensive interactions between different modalities with both commonality and complementarity thanks to the effective fusion of multimodal information and thus achieve significant improvement of link prediction on KGs.

Table 4 shows the evaluation results of leveraging different modality information on FB15K-237, where $S$ denotes structural information; $V$ denotes visual information of images and $T$ denotes textual information of descriptions. We can see that by introducing visual or textual information, the performance is significantly improved. The significant performance gain brought by multimodal fusion module not only demonstrates the effectiveness of our approach, but also indicates the potential of integrating multimodal information in KG.

To verify the effectiveness of decision fusion, we choose a case of <LeBron James, playsFor > and visualize the prediction scores of each modality as Figure 4 shows. Due to biases in each modality, the prediction of monomodal is inevitable error-prone. The results in Table 2 and Table 3 also demonstrate the effectiveness of applying decision fusion to ensemble the specific latent features of each modality.

Besides, the performance comparison between IMF (w/o CL) and IMF in Table 2 and Table 3 illustrates the necessity of contrastive learning for more robust results, especially in the scenario with fewer training samples and relation types.

From the results shown above, we can see that each component in our propose IMF has a significant contribution to the overall performance and it is beneficial to capture the commonality and complementarity between different modalities.

In order to evaluate the generalization of our proposed approach, we simply replace the scoring function (contextual relational model) with existing methods such as TransE, ConvE and TuckER. The results in Figure 5 illustrate that our proposed framework of two-stage fusion is general enough to be applied to any link prediction model for further improvement.

Figure 6 shows the performance influence of embedding size for IMF. From the picture, we can see that the embedding size plays an important role in the model performance. Meanwhile, it is worthy of note that a larger embedding size not always results in better performance due to the overfitting problem, especially in the datasets with fewer relation types like YAGO15K. Considering the performance and the efficiency, the best choices of embedding size for these three datasets are 256, 256 and 128, respectively.

In order to illustrate the effectiveness of our IMF model in a more intuitive way, we apply t-SNE to reduce dimension and visualize the contextual entity representations of basketball players in five different basketball teams. We can see in Figure 7 that the representations of basketball players are messed up with monomodal information due to the biases. However, with the help of interactive multimodal fusion, IMF can effectively capture complicated interactions between different modalities.

Knowledge embedding methods have been widely used in graph representation learning tasks and have achieved great success on knowledge base completion (a.k.a link prediction). Translation-based methods aim at finding the transformation relationships from source to target. TransE (Bordes et al., 2013), the most representative translation-based model, projects entities and relations into a unified vector space and minimizes the energy function of triples. Following this route, many translation-based methods have emerged. TransH (Wang et al., 2014) formulates the translating process on relation-specific hyperplanes. TransR (Lin et al., 2015) projects entities and relations into separate spaces.

Recently, some neural network methods have shown promising results in this task. ConvE (Dettmers et al., 2018) and ConvKB (Nguyen et al., 2018) utilize Convolutional Neural Network (CNN) to increase parameter interaction between entities and relations. KBAT (Nathani et al., 2019) employ Graph Neural Networks (GNN) as the encoder to aggregate multi-hop neighborhood information.

However, all these methods above utilize only structural information, which is not sufficient for more complicated situations in real world. By incorporating multimodal information in the training process, our approach is able to improve the representations with external knowledge.

Leveraging multimodal information has yielded extraordinary results in many NLP tasks (Ben-younes et al., 2017). DeViSE (Frome et al., 2013) and Imagined (Collell et al., 2017) propose to integrate multimodal information with modality projecting which learns a mapping from one modality to another. FiLM (Perez et al., 2018) extends cross-modal attention mechanism to extract textual-attentive features in visual models. MuRel (Cadène et al., 2019) utilizes pair-wise bilinear interaction between modalities and regions to fully capture the complementarity. IKRL (Xie et al., 2017) is the first attempt at multimodal knowledge representation learning, which utilizes image data of the entities as extra information based on TransE. MKGC (Sergieh et al., 2018) combines textual and visual features extracted by domain-specific models as additional multimodal information compared to IKRL. MKBE (Pezeshkpour et al., 2018) creates multimodal knowledge graphs by adding images, descriptions and attributes, and employs DistMult (Yang et al., 2015) as scoring function.

Although these approaches did incorporate multimodal information to improve performance, they cannot take full advantage of it as they fail to effectively model interactions between modalities.