Heterogeneous Entity Matching with Complex Attribute Associations using BERT and Neural Networks

Entity Matching, also known as entity resolution, refers to the process of identifying pairs of records from structured datasets or text corpora that correspond to the same real-world entity. Let D be a collection of records from one or multiple sources, such as rows of relational tables, XML documents, or text paragraphs. Entity Matching typically involves two steps: blocking and matching . In this paper, we focus on the matching step of entity matching. Formally, we define the entity matching problem as follows:

Input: A set M of pairs of records to be matched.For each pair$(e_{1},e_{2})\in M$, $e=\{(attr_{i},\nu al_{i})\}_{1\leq i\leq k}$each entity is represented in the form of K key-value pairs, where Key and Value are respectively the attribute name and attribute value of the entity.

Output: A set of pairs of records$M^{*}$, where each entity in each pair $(e_{1},e_{2})$points to the same entity, indicating the same real-world entity.

In this definition, our input is sufficiently general to apply to both structured and textual data. Attribute names and attribute values can take any form, including even indices or other identifiers, even if their true semantics are not available, such as ”attr1” and ”attr2”.

the schema of an entity represents an abstracted representation of the basic information about that entity. Schema matching is often a necessary prerequisite in the context of Entity Matching (EM), as there might be differences among attributes of different entities. Traditional EM methods typically establish one-to-one mapping relationships between attributes from different entities. However, in reality, the associations between two entity attributes can be intricate, and simple one-to-one mappings may fail to capture these complex relationships. To address EM more effectively, it becomes crucial to consider the intricate associations between attributes during the entity matching process, thereby enhancing the performance of entity matching.

For entities comprising distinct attributes, an entity itself can be seen as an instance of a schema. Given two entities,$e_{1}=\{<a_{1}^{s},\nu_{1}^{s}>...<a_{m}^{s},\nu_{m}^{s}>\}$ and $e_{2}=\{<a_{1}^{t},\nu_{1}^{t}>...<a_{n}^{t},\nu_{n}^{t}>\}$ , $\{a_{1}^{s},a_{2}^{s},...,a_{m}^{s}\}$ and $\{a_{1}^{t},a_{2}^{t},...,a_{n}^{t}\}$ respectively denote the distinct schemas of the two entities, and each schema is composed of Tokens representing attribute values of the entities.

To achieve this goal, we construct a neural network based on BERT for entity matching. As illustrated in Fig.2, the left part involves a neural network that captures the complex associations between entity attributes. On the right side of the matching process, the final matching results are derived by considering the association matrix, which focuses on the degree of matching between different entity attributes. The matching process of the network mainly comprises the following steps: (1) Token Embedding: Converting Tokens within attribute values into vectors using BERT, while capturing contextual relationships between each Token. (2) Token Self-attention: Obtaining attention scores between Tokens of the same entity through self-attention mechanism. (3) Token Aggregation: Aggregating attention scores between Tokens to obtain similarity information. (4) Attribute Inter-Attention: Determining attention scores between Tokens of different entities through interactive attention. (5) Attribute Comparison: Aggregating similarity scores between Tokens within and between entities to create a similarity matrix for attributes. (6) Serialize: Serializing entities in the form of ¡Key, Value¿. (7) Sentence Embedding: Converting the serialized result into sentence vectors. (8) Linear: Focusing on the matching degree of similar attributes within sentence vectors. (9) Softmax: Normalizing the output of the Linear layer, resulting in a match (0/1) output.

Each token of both attribute values and attribute names needs to be embedded into a low-dimensional vector for subsequent calculations. Since attribute values are composed of sequences of different tokens, in the embedding process, the same token may have different vector representations in different contexts. Therefore, contextual semantic information should be integrated into the vectors. BERT is a pre-trained deep bidirectional Transformer model that effectively captures the semantic information of each token in its context through unsupervised learning on large-scale corpora. This implies that the token vectors generated by BERT can better represent the meaning of each token. Therefore, we choose to use the pre-trained BERT model for token embedding.

In the process of obtaining the attribute similarity matrix, the vector representation of entities is obtained by concatenating different tokens.$H^{S}=[H_{1}^{s},H_{2}^{s},...,H_{m}^{s}]$, $H^{T}=[H_{1}^{s},H_{2}^{s},...,H_{n}^{t}]$.

During the matching process, it is necessary to compare the similarity of each attribute, so we need to determine the significance of attributes within an entity. This is achieved through Token Self-attention, which computes the weights of tokens to aggregate the importance of attributes. Its role is to establish relationships between each token and other tokens within the sequence, capturing significant interdependencies to derive the importance of attributes. Through Self-attention, each token can be weighted and combined based on the importance of other tokens in the sequence, thereby better reflecting contextual information and semantic dependencies. Such an attention mechanism enables the model to dynamically focus on important tokens while disregarding less significant ones. Consequently, token-level self-attention is employed to weight the tokens within an entity. For attributes, their self-attention scores are computed using trainable matrices, as shown in the equation.

To better harness token information for token-level comparisons, we employ Token Aggregation to merge the representations obtained after the Self-Attention operation, creating a comprehensive representation of the entire entity. This aggregation fuses all token information from the sequence into a single vector, enhancing the overall representation of the entity’s information. This process involves an attention matrix, but we require a token weight vector for token aggregation. Therefore, we utilize a transformation function $m2\nu()$ to convert it into a token weight vector $W_{i}$ . $m2\nu()$ By summing the aggregation of each row of $\alpha_{i}^{s}$, a vector is derived, and subsequently, each element of the vector is normalized by dividing it by the maximum element.

The aforementioned operations yield attribute relationships within individual entities. However, our objective is to capture the complex relationships between attributes in heterogeneous data. Therefore, it is necessary to perform matching across different entities. Through Attribute Inter-Attention in entity matching tasks, interactive attention calculation is applied to different entity attributes. This process yields correspondences between attributes of different entities, enabling the learning of correlations among various attributes. As a result, a better grasp of the associations between entities is achieved. This attention mechanism aids in focusing on attributes relevant to matching while disregarding irrelevant ones. Each entity is considered as a sequence concatenated by tokens, inter-entity interaction attention is leveraged to obtain interaction representations between ${H_{i}^{s}}$ and ${H_{t}^{j}}$. Here, $W_{i\to T}$denotes the inter-entity interaction attention.

After obtaining the cross-entity correspondences, we need to use these correspondences to calculate the similarity between entity attributes, thus deriving the relationships between attributes. Attribute Comparison involves comparing different attributes of distinct entities during the matching process, computing their similarity or dissimilarity, and thereby gauging the level of association between different attributes. This attribute comparison mechanism aids the model in capturing essential features among attributes in entity matching tasks, further assisting the model in entity-level matching or classification. We apply element-wise absolute difference and Hadamard product to ${H_{i}^{s}}$ and$\dot{H}_{i}^{s}$ , and incorporate the intermediate representation into a highway network. Subsequently, the token-level similarity $\overset{s}{\operatorname*{C}}$ from $e_{1}$ to $e_{2}$ is the output of the highway network.

Lastly, the aggregation of token similarities obtained through the interaction attention mechanism of self-attention yields the similarity between entity attributes.

We employ the methods from Ditto [2] to serialize the data and generate sentence embeddings. For each entity pair, we serialize it as follows: $serialize(e)=[COL]attr_{1}[VAL]val_{1}[COL]attr_{2}[VAL]val_{2}...[COL]attr_{k}[VAL]val_{k}$ , Where [COL] and [VAL] are special tokens used to indicate the start of attribute names and values, respectively. For example, the first entry in the second table is serialized as: For each candidate entity pair, $serialize(e_{1},e_{2})=[CLS]serialize(e_{1})[SEP]serialize(e_{2})[CLS]$, where [SEP] is a special token that separates the two sequences, and [CLS] is a special token required by BERT to encode the sequence pair into a 768-dimensional vector.

In entity matching tasks, a linear layer is employed to perform a linear transformation on the vectors that have undergone feature extraction and encoding. The formula representing the linear layer for input feature vector X is as follows:

Here, W represents the weight matrix to be learned, and b is the bias vector. In this context, the vector X is obtained through previous serialization and sentence embedding, resulting in the embedded vector E for entity pairs. The matrix W corresponds to the obtained entity attribute similarity matrix

Further applying a softmax function yields the output vector. This vector represents a probability distribution, where each element signifies the probability for the respective category. In this context, the output of 0 signifies a non-match, while 1 signifies a match.

When evaluating our approach, we utilized various types of datasets, including: Two isomorphic public datasets[9] (simple 1:1 attribute associations). Two heterogeneous public datasets (complex associations of 1:m and m:n). A hydraulic heterogeneous dataset (complex associations of 1:m and m:n). The information for the public datasets is presented in the following table.

Homogeneous Data: The patterns of homogeneous data involve simple associations (1:1). iTunes-Amazon (iA) and DBLP-Schoral1 (DS1) correspond respectively to iTunes-Amazon1 and DBLP-choral1[5].

Heterogeneous Data: The complex data SM involves complex associations (1:m or m:n). We implemented a variant of the Synthetic Data Generator UIS to generate the UIS1-UIS2 (UU) dataset[10]. The initial five attributes are name, address, city, state, and zip code. Address and city are combined into a new attribute in UIS1 records, while city and state are integrated into a new attribute in UIS2 records. Therefore, the attribute numbering for UU is 4–4. Walmart-Amazon1 (WA1) is a variant of Walmart-Amazon (5-5)[12] . Brand and model are merged into a new attribute in Walmart records, while category and model are integrated into a new attribute in Amazon records. Hence, the attribute numbering for WA1 is 4–4.

Next, we introduce the composition of the hydraulic dataset, which we crawled separately from Wikipedia and Baidu Baike. We collected data about the Songhua River Basin and the Liao River Basin, including 4039 reservoirs and 6576 river records. Reservoirs include attributes such as reservoir name, location, area, region, normal water level, watershed area, normal storage level, etc. Rivers include attributes such as river name, region, river grade, river length, basin area, etc. As shown in Figure 1, there exist complex correspondences among these attributes. During data processing, we labeled data belonging to the same entity as matching and labeled data pointing to different entities as non-matching. Since this data was crawled from web pages based on names, most of the non-matching entities are entities with the same name, and the proportion of same-name entities in the data is not high. Therefore, there are not enough negative examples in the data. We created some negative examples by replacing attribute values with synonyms. Finally, our dataset has a size of 21230, including 5000 positive instances, with 2000 positive instances for reservoirs and 3000 for rivers.

We implemented our model using PyTorch and the Transformers library. In all experiments, we used the base uncased variants of each model. We further applied mixed-precision training (fp16) optimization to accelerate both training and inference speed. For all experiments, we fixed the maximum sequence length to 256, set the learning rate to 3e-5, and employed a linear decay learning rate schedule. The training process runs for a fixed number of epochs (10, 15, or 40 depending on the dataset size) and returns the checkpoint with the highest F1 score on the validation set.

Comparison Methods. We compare EM-CAR with state-of-the-art EM solutions such as Ditto, the attribute correspondence-aware method DEM-SSR, and the classical method DeepMatcher. Here’s a summary of these methods. We report the average F1 score over 6 repeated runs in all settings.

DeepMatcher: DeepMatcher is a state-of-the-art classical method , customizes an RNN architecture to aggregate attribute values and then compare/align the aggregated representations of attributes.

Ditto: Ditto is a state-of-the-art matching solution that employs all three optimizations, Domain Knowledge (DK), TF-IDF summarization (SU), and Data Augmentation (DA).

DER-SSM: In comparison to Ditto, DER-SSM defines and implements soft pattern matching, obtaining context relations between tokens through BiGRU. It considers soft pattern matching by aggregating token similarity during entity matching based on the context relationships between tokens.

The F1 score is used to measure the precision of entity matching (EM) and is the harmonic mean of precision (P) and recall (R). Precision (P) represents the score of correct matching predictions, while recall (R) represents the score of true matches predicted as matches.

Typically, in EM, there are two phases: blocking and matching[2] . Our focus is on the matching phase in entity matching (EM), assuming that blocking has already been performed. We follow the same blocking setup[2] , where blocking is applied to generate a candidate set for the dataset. All pairs in the candidate set are labeled. The dataset is then divided into a 3:1:1 ratio for training, validation, and testing.

To further demonstrate the performance of EM-CAR, we conducted a case study comparing it with Ditto. First, it should be noted that Ditto directly utilizes context-based embeddings obtained from pre-trained language models (LM) for classification, making it not entirely suited for entity matching (EM) tasks. Specifically,Ditto’s embeddings might not be fully optimized for the specific task of entity matching, as they are derived from a broader range of language modeling objectives. This could potentially limit Ditto’s ability to capture the nuanced and complex relationships between attributes required for accurate entity matching.

In contrast, our model aims to address this issue by placing greater emphasis on attributes with higher similarity. As depicted in the figure, our model takes into account the similarity of attributes, particularly those that are more closely related, This approach allows our model to better capture the nuanced relationships between attributes and improve the overall matching accuracy.

As shown in the Fig.3, when performing matching, Ditto’s utilization of pre-trained models can lead to erroneous judgments. This is attributed to the fact that, while determining whether these two entities match, the top two attention scores are placed on the tokens ”YichangCity” and ”YilingDistrict” while the score for ”SandoupingTown” is not as high. Consequently, more attention is directed towards the correspondence between ”YichangCity” and ”YilingDistrict” as well as ”YilingDistrict” and ”SandoupingTown” .

In contrast, we aim to prioritize the matching probability between ”YilingDistrict” and ”SandoupingTown” As illustrated in Fig.4, our model focuses more on the matching degree of similar attributes, denoted by $(e_{1},e_{2})$. This enables our model to appropriately emphasize the similarity between attributes and achieve accurate results.