MixTEA: Semi-supervised Entity Alignment with Mixture Teaching

While the recent years have witnessed the rapid development of deep learning techniques, embedding-based EA approaches obtain promising results. Among them, some early studies Chen et al. (2017); Sun et al. (2017) are based on the knowledge embedding methods, in which entities are embedded by exploring the fine-grained relational semantics. For example, MTransE Chen et al. (2017) applies TransE Bordes et al. (2013) as the KG encoder to embed different KGs into independent vector spaces and then conducts transitions via designed alignment modules. However, they need to carefully balance the weight between the encoder and alignment module in one unified optimization problem. Due to the powerful structure learning capability, Graph Neural Networks (GNNs) like GCN Kipf and Welling (2017) and GAT Veličković et al. (2018) have been employed as the encoder with Siamese architecture (i.e., shared-parameter) for many embedding-based models. GCN-Align Wang et al. (2018) applies Graph Convolution Network (GCN) for the first time to capture neighborhood information and embed entities into a unified vector space, but it suffers from the structural heterogeneity of different KGs. To mitigate this issue and improve the structure learning, AliNet Sun et al. (2020a) adopts multi-hop aggregation with a gating mechanism to expand neighborhood ranges for better structure modeling, and KE-GCN Yu et al. (2021) combines GCN and knowledge embedding methods to jointly capture the rich structural features and relation semantics of entities. More recently, IMEA Xin et al. (2022a) designs a Transformer-like architecture to encode multiple structural contexts in a KG while capturing alignment interactions across different KGs.

In addition, some works further improve the EA performance by introducing the side information about entities, such as entity names Zhang et al. (2019), attributes Liu et al. (2020), and literal descriptions Yang et al. (2019). Afterward, a series of methods were proposed to integrate knowledge from different modalities (e.g., relational, visual, and numerical) to obtain joint entity representation for EA Chen et al. (2020); Liu et al. (2021); Lin et al. (2022). However, these discriminative features are usually hard to collect, noise polluted, and privacy sensitive Pei et al. (2022).

Since the manually labeled mappings used for training are usually insufficient, many semi-supervised EA methods have been proposed to take advantage of labeled mappings and the large amount of unlabeled data for alignment, which can provide a more practical solution in real scenarios. The mainstream solutions focus on iteratively generating pseudo mappings to compensate for the lack of training data. IPTransE Zhu et al. (2017) applies threshold filtering-based self-training to yield pseudo mappings but it fails to obtain satisfactory performance since it brings much noise data, which would misguide the subsequent training. Besides, it is also hard to determine an appropriate threshold to select “confident” pseudo mappings. KDCoE Chen et al. (2018) performs co-training of KG embedding model and literal description embedding model to gradually propose new pseudo mappings and thus enhance the supervision of alignment learning for each other. To further improve the quality of pseudo mappings, BootEA Sun et al. (2018) designs an editable strategy based on the one-to-one matching rule to deal with matching conflicts and MRAEA Mao et al. (2020) proposes a bi-directional iterative strategy which imposes a mutually nearest neighbor constraint. Inspired by the success of self-training, RANM Cai et al. (2022) proposes a relation-based adaptive neighborhood matching method for entity alignment and combines a bi-directional iterative co-training strategy, making become a natural semi-supervised model. Moreover, CycTEA Xin et al. (2022b) devises an effective ensemble framework to enable multiple alignment models (called aligners) to exchange their reliable entity mappings for more robust semi-supervised training, but it requires high complementarity among different aligners.

Additionally, other effective semi-supervised EA methods, such as SEA Pei et al. (2019), RAC Zeng et al. (2021), GAEA Xie et al. (2023), focus on introducing specific loss terms (e.g., reconstruction loss, contrastive loss) via auxiliary tasks.

We first introduce the KG encoder (denoted as $f(;\theta)$) which utilizes neighborhood structures and relation semantics to embed entities from different KGs into a unified vector space. We randomly initialize the trainable entity embeddings $\textbf{H}^{ent}\in\mathbb{R}^{(|\mathcal{E}_{s}|+|\mathcal{E}_{t}|)\times{d_{e}}}$ and relation embeddings $\textbf{H}^{rel}\in\mathbb{R}^{|\mathcal{R}_{s}\cup{\mathcal{R}_{t}}|\times{d_{r}}}$, where $d_{e}$ and $d_{r}$ are the dimension of entities and relations, respectively.

In the following, we will introduce mixture teaching, which is reached by the supervised alignment learning and probabilistic pseudo mapping learning in an end-to-end training manner.

Probabilistic pseudo mapping learning. As mentioned above, the teacher model is responsible for generating probabilistic pseudo mappings for the student model to provide more alignment signals and thus enhance the alignment performance. Benefiting from the EMA update, the predictions of the teacher model can be seen as an ensemble version of the successive student models’ predictions. Therefore it is more robust and stable for pseudo mapping generation. Moreover, bi-directional iterative strategy Mao et al. (2020) reveals the asymmetric nature of alignment directions (i.e., source-to-target and target-to-source), which can produce pseudo mappings based on the mutually nearest neighbor constraint. Inspired by this, we propose a bi-directional voting (BDV) strategy which fuses alignment decisions in each direction to yield more comprehensive pseudo mappings and model their uncertainty via the joint matching confidence score. Concretely, after encoding, we can first obtain the similarity matrix by performing pairwise similarity calculation between the unlabeled source and target entities as follows:

We evaluate our method on the 15K benchmark dataset (V1) in OpenEA Sun et al. (2020b) since the entities thereof follow the degree distribution in real-world KGs. The brief information of experimental data is shown in Table 3 (Appendix B). It contains two cross-lingual settings, i.e., EN-FR-15K (English-to-French) and EN-DE-15K (English-to-German), and two monolingual settings, i.e., D-W-15K (DBPedia-to-Wikidata) and D-Y-15K (DBPedia-to-YAGO). Following the data splits in OpenEA, we use the same split setting where 20%, 10%, and 70% pre-aligned mappings are utilized for training, validation, and testing, respectively.

Entity alignment is a typical ranking problem, where we obtain a target entity ranking list for each source entity by sorting the similarity scores in descending order. We use Hits@k ($k$=1, 5) and Mean Reciprocal Rank (MRR) as the evaluation metrics Sun et al. (2020b); Xin et al. (2022a). Hits@k is to measure the alignment accuracy, while MRR measures the average performance of ranking over all test samples. The higher the Hits@k and MRR, the better the alignment performance.

We choose the methods from the related work as baselines and divide them into two classes below:

• •

  Structure-based methods. These methods focus on capturing useful structural context to enrich entity representation, such as (1) MTransE Chen et al. (2017), (2) GCN-Align Wang et al. (2018), (3) AliNet Sun et al. (2020a), (4) KE-GCN Yu et al. (2021) and (5) IMEA Xin et al. (2022a).

• •

  Semi-supervised methods. These methods aim to explore alignment signals from unlabeled entities, such as (1) IPTransE Zhu et al. (2017), (2) SEA Pei et al. (2019), (3) KDCoE Chen et al. (2018), (4) BootEA Sun et al. (2018), (5) MRAEA Mao et al. (2020), (6) RANM Cai et al. (2022) and (7) GAEA Xie et al. (2023).

As our method and the above baselines only contain a single model and mainly rely on structural information, for a fair comparison, we do not compare with ensemble-based frameworks (e.g., CycTEA Xin et al. (2022b)) and models infusing side information from multi-modality (e.g., EVA Liu et al. (2021), RoadEA Sun et al. (2022)). For the baseline RANM, we remove the name channel to guarantee a fair comparison.

All the experiments are performed in PyTorch on an NVIDIA GeForce RTX 3090 GPU. Following OpenEA Sun et al. (2020b), we report the average results of five-fold cross-validation. The embedding dimensions of entities $d_{e}$ and relations $d_{r}$ are set to 256 and 128, respectively, the number of GAT layer $L$ is 2, the margin $\rho$ is 2.0, and the momentum $m$ is 0.9. In the EA inference phase, we use Cosine distance as the distance metric and apply Faiss²²2https://github.com/facebookresearch/faiss to perform NN search efficiently. The default alignment direction is from left to right, e.g., in D-W-15K, we regard DBpedia as the source KG and seek to find the counterparts of source entities in the target KG Wikidata. The details of hyperparameter settings are shown in Appendix C.

Table 1 reports the experimental results of all the methods on the OpenEA 15K datasets. Even utilizing only the structure information, KE-GCN and IMEA achieve inspiring performance by exploring the rich structural contexts. However, they are hard to further improve the performance because suffer from the lack of enough training data. We also observe that some semi-supervised EA methods (e.g., IPTransE and SEA) fail to outperform these structure-based EA methods, reflecting the fact that both encoder design and semi-supervised strategy are important components of facilitating high-accuracy EA. IPTransE obtains unsatisfactory alignment results since it produces many noise pseudo mappings during the self-training process but does not design an appropriate mechanism to eliminate the influence of noise. Besides, the performance of KDCoE is unstable. According to Sun et al. (2020b), this is because many entities lack textual descriptions, thus preventing the model from finding complementary mappings for co-training from the textual description embedding model. BootEA and MRAEA are competitive baselines in semi-supervised EA domain. Nevertheless, BootEA needs a carefully fine-tuned confidence threshold to filter pseudo mappings, which often leads to unstability, while MRAEA and RANM still follows the data augmentation paradigm, which ignores the uncertainty of pseudo mappings and is prone to cause error accumulation. Although GAEA learns representations of vast unseen entities via contrastive learning, its performance is unstable. The bottom part of Table 1 shows our method consistently achieves the best performance in all tasks with a small standard deviation (std.). More precisely, our model surpasses state-of-the-art baselines averagely by 3.1%, 3.3%, and 3.5% in terms of Hit@1, Hit@5, and MRR, respectively.

To verify the effectiveness of our method, we perform the ablation study with the following variant settings: (1) w/o $rel.$ removes the relation modeling. (2) w/o $\mathcal{L}_{u}$ removes probabilistic pseudo mapping learning. (3) w/o BDV only considers EA decisions in the default alignment direction to generate pseudo mappings instead of applying the bi-directional voting strategy (i.e., $\textbf{P}^{tea}=g(\textbf{M}^{tea}_{s\rightarrow{t}})$). (4) w/o MDR removes matching diversity-based rectification module in pseudo mappings learning. (5) w/o B&M denotes that the complete model deletes both BDV and MDR module.

The ablation results are shown in Table 2. We can observe that the complete model achieves the best experimental results, which indicates that each component in our model design contributes to the performance improvement. Removing relation modeling from entity representation causes performance drops, which identifies the relation semantics can help in enriching the expressiveness of entity representations. W/o $\mathcal{L}_{u}$ caused the most significant performance degradation, especially in monolingual settings, showing the crucial role of the pseudo mapping learning in general. The results of w/o BDV and w/o MDR suggest that the bi-directional voting strategy and matching diversity-based rectification module can do benefit to improving the quality of pseudo mapping learning. W/o B&M also demonstrates that the combination of BDV and MDR can further improve the alignment performance. Although w/o BDV only takes EA decisions in one direction into account, it still inevitably brings matching conflicts since NN search neglects the inter-dependency between different EA decisions. Compared to w/o B&M, the MDR in w/o BDV has a certain positive effect, which indicates that our proposed MDR can be applied to other pseudo mapping generation algorithms and help the models to train better.

Training visualization. To inspect our method comprehensively, we also plot the test Hit@1 curve throughout the training epochs in Figure 3. KG Encoder (th=0.9) represents the KG Encoder described in Sec. 4.1 applying the self-training with threshold=0.9 to generate pseudo mappings every 20 epochs. We control the same experimental settings to remove the performance perturbations induced by different parameters. From Figure 3, we observe that our method converges quickly and achieves the best and most stable alignment performance. The performance of the KG Encoder gradually decreases in the later stages since it gets stuck in overfitting to the limited training data. Although self-training brings some performance gains after data augmentation in the early stages, the performance drops dramatically in the later stages. This is because it involves many noise pseudo mappings and causes error accumulation as the self-training continues. In the later stages, self-training has difficulty in further generating new mappings while existing erroneous mappings constantly misguide the model training, thus hurting the performance.

Hyperparameter analysis. We design hyperparameter experiments to investigate the performance varies with some hyperparameters. Due to the space limitation, these experimental results and analyses are listed in Appendix D.