Harnessing Diverse Perspectives: A Multi-Agent Framework for Enhanced Error Detection in Knowledge Graphs

Out_Neighbor Subgraph: The set of triples where the entity serves as the head. For an entity $e$, the Out_Neighbor Subgraph is $\{(e,r^{\prime},t^{\prime})\mid(e,r^{\prime},t^{\prime})\in\mathcal{G}\}$, where $r^{\prime}$ is outgoing relations from $e$, and $t^{\prime}$ is the corresponding tail entity.

In_Neighbor Subgraph: The set of triples where the entity serves as the tail. For an entity $e$, the In_Neighbor Subgraph is $\{(h^{\prime},r^{\prime},e)\mid(h^{\prime},r^{\prime},e)\in\mathcal{G}\}$, where $h^{\prime}$ is the corresponding head entity, and $r^{\prime}$ represents incoming relations to $e$.

(a) Head_Forward_Subgraph: The Out_Neighbor Subgraph of the head entity $h$, excluding the current triple $(h,r,t)$. Formally:

(b) Head_Backward_Subgraph: The In_Neighbor Subgraph of the head entity $h$, capturing all incoming relations to $h$. Formally:

(c) Tail_Forward_Subgraph: The Out_Neighbor Subgraph of the tail entity $t$, capturing all outgoing relations from $t$. Formally:

(d) Tail_Backward_Subgraph: The In_Neighbor Subgraph of the tail entity $t$, excluding the current triple $(h,r,t)$. Formally:

Datasets: We use two real-world knowledge graph datasets: FB15K [17] and WN18RR [6]. We chose these two datasets because they are highly representative in the field of knowledge graph error detection, encompassing most typical scenarios and possible graph structural representations found in knowledge graph data. In each dataset, we simulate realistic errors by replacing entities and relations with similar ones selected based on cosine similarity within the dataset, resulting in approximately 30% of the data being erroneous. We split each dataset into training, validation, and test sets with a ratio of 8:1:1. The fine-tuning process used only the training set, while the test set was reserved for final evaluations. FB15K is derived from Freebase and contains a rich set of entities and relations, while WN18RR is a subset of WordNet with corrected inverse relations, increasing the complexity. Additionally, we conducted experiments on a knowledge graph dataset from China Mobile’s business scenarios, achieving the best results compared to other methods.

Baselines: We compare MAKGED against various baseline methods, including traditional knowledge graph embedding models such as TransE [2], DistMult [23], and ComplEx [18], which learn triple embeddings to compute confidence scores. Additionally, we compared recent embedding-based KG error detection methods including CAGED [25] and KGTtm [9]. We also compared KG-BERT [24] and CSProm-KG [4] models that combine pre-trained language models for error detection, as well as models that use text structure and graph structure for comparative learning, such as StAR [19], SeSICL [14], and CCA [13] models. All comparison experiments are conducted under the same experimental settings.

Implementation Details: We use the Llama2 for fine-tuning and employ LoRA [8] for instruction tuning. Experiments are conducted on V100 GPU servers. GCN Component: A three-layer GCN with 128 hidden dimensions and 64-dimensional embeddings is trained separately using the Adam optimizer (learning rate: 0.001), a batch size of 64, for 100 epochs. The resulting GCN embeddings are concatenated with Llama2’s text embeddings to create a unified representation for each agent. Fine-Tuning Llama2: During fine-tuning, the combined embeddings (GCN + Llama2) serve as model inputs. We utilize mixed precision training and gradient checkpointing to accelerate training and reduce memory usage. We use Accuracy, F1-Score, Precision, and Recall as evaluation metrics. These metrics use macro averaging for both classes. Each round of agent discussion took an average of 2.3 seconds. This demonstrates the framework’s practical feasibility for industrial applications.

Experiment Setup: To study RQ1, we conducted comprehensive experiments on two KG datasets, comparing it with the previously mentioned baseline models across four key metrics. The experimental results are presented in Table 1.

Comparison to Embedding-based Methods: MAKGED combines subgraph structural information with semantic insights from LLMs, leading to a 10-20% improvement in accuracy and a significant increase in the F1-Score.

Comparison to PLM-based Methods: MAKGED further demonstrates its strong error detection capabilities. Although models like KG-BERT show advantages in semantic understanding, their utilization of knowledge graph structure is limited. By integrating subgraph embeddings with LLMs embeddings, MAKGED improves recall by around 10% on the WN18RR compared to KG-BERT, demonstrating better coverage and precision in detecting complex errors.

Comparison to Contrastive Learning Methods: MAKGED shows stronger performance by leveraging multi-agent collaboration and integrating multiple perspectives. On the FB15K dataset, MAKGED achieved an F1-Score improvement of approximately 5-8%.

Comparison to LLM-based Methods: While LLM-based methods, such as Llama2 and GPT-3.5, perform well in knowledge graph tasks, their limited understanding of graph structures reduces their effectiveness in error detection. MAKGED overcomes this limitation by combining the semantic power of LLMs with the structural insights of GCN, allowing for more accurate error detection. Notably, its performance rivals that of GPT-3.5 and Llama3. Overall, MAKGED consistently surpasses state-of-the-art models in accuracy, F1-Score, and recall, demonstrating outstanding error detection capabilities.

Experiment Setup: To address RQ2, we conducted an ablation study evaluating several model variants under the same experimental setup as the full framework. The Head_as_Head variant refers to the results where all four agents are trained using the Head_Forward Subgraph constructed from the triple’s head entity, with other ablation versions following a similar naming convention. The Analysis variant performs only initial judgments without multi-agent discussions, while the Llama2 baseline excludes subgraph information during fine-tuning, relying solely on Llama2’s original outputs.

Full Framework Outperforms Variants: As shown in Table 2, when only specific subgraphs are used or subgraph information is excluded, performance drops notably compared to the complete MAKGED, especially in F1-Score and Recall, indicating that a single perspective or lack of structural information limits error detection. While using multiple agents without discussion offers slight improvements, it still lags behind the full framework. The complete framework achieves the best results on two datasets, maximizing four metrics.

Running Example: To study RQ3, we select an incorrect triple (Huawei Honor 10, network support, 5G) from the industrial KG of China Mobile to demonstrate the effectiveness of our framework in industrial applications. Fig 3 shows the evaluation and discussion paths for this triple under our framework.

Subgraph-Aided Error Correction: The framework effectively uses subgraph information for in-depth analysis, enabling agents to correct initial errors and reach the correct conclusion. In contrast, using the original Llama2 model without subgraph fine-tuning leads to a lower accuracy performance.

Value of Multi-Agent Collaboration: If we had relied only on the initial model output, the result would have been “correct”, conflicting with the ground truth. However, after three rounds of discussion, the agents reached the correct conclusion, demonstrating the effectiveness of multi-agent collaboration.