DAME: Domain Adaptation for Matching Entities

Each record pair $x=(e_{1},e_{2})$ is composed of two data entries $e_{1}\in D_{1}$ and $e_{2}\in D_{2}$ that correspond to candidate rows from two collections of data entries $D_{1}$ and $D_{2}$. Both $D_{1}$ and $D_{2}$ are from the same source domain. Each data entry $e_{i}=\left\{\left(\mathrm{attr}_{j},\mathrm{val}_{j}\right)\right\}_{1\leq j\leq C}$ is a set of attribute-value pairs denoted by $(\mathrm{att}_{j},\mathrm{val}_{j})$, where $C$ is the number of attributes in each record. We follow the encoding of Ditto (Li et al., 2020) for serializing data entries to produce a sequence for each record from the attribute-value pairs:

where $[\mathrm{COL}]$ and $[\mathrm{VAL}]$ are special tokens that denote the start of attributes and values, respectively. The input of EM is a pair of records $x=(e_{1},e_{2})$. So, $Rep$ takes as input a pair of records, and produces a sequence pair of serialized entries that is given by:

where [SEP] and [CLS] are BERT special tokens that are added into the sequence similar to the sentence pair classification setting.

We have $K+1$ DistilBERT models: $K$ expert models $\left\{f_{S_{i}}\right\}_{i=1}^{K}$ and a global shared model $g$. We use $Rep(x)$ as input to the $K+1$ models to extract $K$ source domain-based embeddings denoted by $f_{S_{i}}(Rep(x)),i=1,\ldots,K$, and a global model-based embedding denoted by $g(Rep(x))$. The embeddings from the source domain models and the global model are extracted using the hidden state of the [CLS] token from the last Transformer block in each DistilBERT model. In conclusion, the output of $F$ is given by:

When aggregating the embeddings that are extracted using $F$, the embeddings from the source domains and the global model should not be treated equally as there are domains that are more relevant to a given record pair $x$ than others. We use a parameterized attention model that attends to all domains using a dot product-based attention where three parametric matrices are introduced: a query matrix $Q\in\mathbb{R}^{d\times d}$, a key matrix $K_{e}\in\mathbb{R}^{d\times d}$, and a value matrix $V\in\mathbb{R}^{d\times d}$, where $d$ is the dimension of the embedding. We first concatenate all the expert embeddings from $F(Rep(x))$ to form an embedding matrix denoted by $E\in\mathbb{R}^{K\times d}$. The attention operations are defined by:

An important design choice in our attention module $Att$ is the use of the global representation $g(Rep(x))$ to map the query matrix $Q$ to a query vector $\alpha$. Given that the global model is trained on all the source domains, we expect the global model’s embedding to transfer to the target domain, and by consequence we obtain more accurate attention weights in the target domain to aggregate the source domains, mainly in the zero-shot learning case. The output of the attention module is used as input to the classification layer $N$ to predict the matching score of the input record pair $x$.

$f_{S_{i}}$ represents the expert model of $S_{i}$, for all $i\in 1,2,\ldots,K$. To optimize each expert model $f_{S_{i}}$, we add a classification layer $N_{S_{i}}$ that predicts the probabilities of matches and non-matches for each domain $S_{i}$. So, in total we add $K$ classification layers. Given that $X_{j}$ is sampled from the $j$-th domain, the domain expert loss $\mathcal{L}_{1}$ is given by:

where $CrossEnt$ denotes the cross entropy loss function.

The global model is trained on all the source domains in order to learn a universal embedding for the EM task that supports transfer to the target domain while maintaining important matching signals for each source domain. In addition, the embedding of the global model is multiplied with the query matrix $Q$ in the attention module $Att$ to compute the contribution of each source domain to the final representation. After learning how to aggregate features in the training phase on source domains, the global model guides the attention module $Att$ to pick the most important source domains for the target domain during the testing phase. To optimize the global model $g$, we add a classification layer $N_{g}$ that predicts the probabilities of matches and non-matches for all source domains. The global model loss $\mathcal{L}_{2}$ is given by:

In DA, the objective is to incorporate multiple source domains to predict labels for samples from the target domain during the testing phase. In order to simulate the process of DA during the training phase, we use the meta-target and meta-sources similar to Guo et al. (Guo et al., 2018). Given that $X_{j}$ is sampled from the $j$-th domain, the meta-target is the $j$-th source domain and the meta-sources are $\left\{\mathcal{S}_{i}\right\}_{i=1,i\neq j}^{K}$. The meta-model $M_{S_{j}}$ differs only on the feature extractor part $F_{S_{j}}$ compared to $M$. $M_{S_{j}}$ is given by:

where:

The same attention module $Att$ is applicable to the output of the meta-feature extractor $F_{S_{j}}$ where the query matrix based on the global model attends to all the expert embeddings in the key matrix regardless of the number of expert models. Finally, the meta-target loss $\mathcal{L}_{3}$ for the batch $X_{j}$ is given by:

The global model $g$ plays an important role in the attention module $Att$. Learning a domain invariant embedding from the global model makes the transfer to the target domain smoother as the attention weights should be more accurate. To obtain a domain invariant representation from $g$, we adapt the domain adversarial training for EM. Similar to the generative adversarial network (GAN), a min-max objective function is introduced to optimize the parameters of the generator which is the global model $g$ and the discriminator denoted by $D$. The parameters of $D$ are optimized to predict the domain of a sample $x$ using $g(Rep(x))$, and the parameters of $g$ are optimized to produce a confusing representation $g(Rep(x))$ for $D$. We alternate between updating $D$ and $g$. Given that $X_{j}$ is sampled from the $j$-th domain, in order to update $D$, we minimize $\mathcal{L}_{D}$ which is given by:

$\mathcal{L}_{D}$ is minimized with respect to only the parameters of $D$. Then, we set $\mathcal{L}_{4}(X_{j})=-\mathcal{L}_{D}(X_{j})$ to update the parameters of $g$ when minimizing $\mathcal{L}$ ($D$ parameters are fixed). Unlabeled samples $\mathcal{T}=\left\{x_{j}^{\mathcal{T}}\right\}_{j=1}^{|\mathcal{T}|}$ from the target domain can also be considered as an additional domain when updating the parameters of $D$ and $g$ by alternating between minimizing $\mathcal{L}_{D}$ and $-\mathcal{L}_{D}$, respectively. In this case, the total number of labels that are used in $\mathcal{L}_{D}$ is equal to $K+1$.

Figure 2 shows the comparison of DAME results against Ditto for Shoes, Computers, Watches, and Cameras. The caption of each subfigure represents the target domain, and the remaining 3 domains represent the source domains. Each data point represents the mean of 5 trials, and the vertical line in each data point represents the standard deviation (std). The plots report two evaluation metrics: F1 score and accuracy. In all figures, the light blue plot represents the F1 score of DAME, and is compared against the green plot that represents the F1 score of Ditto; the red plot represents the accuracy of DAME, and is compared against the blue plot that represents the accuracy of Ditto. DAME and Ditto outperform DeepMatcher for all evaluation metrics by a large margin, so that we only include DAME and Ditto results to avoid clutter in the figures. The magenta color represents the F1 score of the DAME (ZSL) for the target domain, which is equivalent to 0% of supervised training data from the target domain. We achieve high F1 scores for DAME (ZSL) for both Shoes and Cameras datasets, where the F1 score for DAME (ZSL) is equivalent to training Ditto on 72% and 85% of training data for the Shoes and Cameras, respectively. The results are lower for Computers and Watches where the F1 score of DAME (ZSL) is equivalent to Ditto trained on around 25% of training data. Figure 2 shows the results of fine-tuning DAME using different percentages of training data. Fine-tuning DAME leads to a better and more stable (smaller std in most fractions of the training data) performance than Ditto for all datasets which means that DAME generalizes better than existing methods in EM for datasets with similar structures. This can be explained by the important role of DA in learning the task so that the weights are better warmed up for EM.

Table 2 summarizes the performance of different approaches on the second set of datasets with the same structure which is composed of DBLP-GoogleScholar and DBLP-ACM. In this case, we have one target dataset and one source dataset. We achieve high results for DAME (ZSL) for both datasets. In addition, fine-tuning DAME slightly increases the F1 and accuracy for both datasets. So, consistent with the first set of experiments, we conclude that DAME transfers the task knowledge from the source domains to a target domain in the case of datasets with similar structures.

We study the case of transferring knowledge between datasets with different domains and structures. We call this setting DA in the wild which simulates real-world scenarios. Table 4 (end of the paper) shows extensive experiments on 12 datasets reporting evaluation metrics for multiple methods. DAME (ZSL) achieves a better F1 score than DeepMatcher trained with 50% of training data from the target domain for 7 out of 12 datasets. The difference between the F1 score of Ditto trained on $50\%$ of data and DAME (ZSL) is less than 0.1, and 0.05 for 83% and 41% of datasets, respectively. By comparing the F1 score of fine-tuning all methods using 50% of training data from the target domain, we achieve SOTA results for 10 out of 12 datasets. By comparing the F1 score of fine-tuning all methods using all training data from the target domain, we achieve SOTA results for 10 out of 12 datasets. This means that DAME generalizes better than existing methods for datasets in the wild.

Figure 3 shows the comparison of F1 score results with different numbers of expert domains against using the global model representation during the testing phase on the target domain in the case of ZSL. The x-axis represents the number of experts that we use for predictions. For example, if the number of experts is equal to 6, it means that we randomly choose 6 experts and we drop the remaining 5 experts. Each data point in Figure 3 represents an average of 5 trials. The dashed line represents the F1 score for the global model. For 10 out of 12 datasets, combining multiple experts using the attention network $Att$ leads to better results than the global model. Figure 3 shows that the fewest number of experts needed to outperform the global model was 5 (DBLP-ACM); the most required was 11 (Cameras). Overall, we obtain better F1 scores for the mixture of experts when we increase the number of experts. This means that the experts help to better understand the EM task, and therefore transfer the learned task knowledge to the unseen target domain.

So far, we have discussed the performance of fine-tuning DAME using randomly selected samples from the target domain. To improve the results of fine-tuning our model, we investigate multiple AL selection techniques given a limited budget of labeled instances. Table 3 shows the results of multiple AL selection methods applied to the DAME (ZSL) model. The starting point is our DA-based model which is not fine-tuned on the target domain, and the best performance corresponds to DAME fine-tuned on all training data from the target domain. We report results using two budget levels: 5% and 25% of the training data from the target domain. The simplest baseline is Random Sampling. The remaining baselines can be categorized into two groups: the confidence-based baselines which are: Least Confidence (Wang and Shang, 2014), Entropy Sampling (Wang and Shang, 2014), Uncertainty Sampling with Dropout Estimation (USDE) (Gal et al., 2017), and Bayesian Active Learning Disagreement (BALD) (Gal et al., 2017); and the embedding-based baselines which are K-Centers Greedy (Sener and Savarese, 2018), K-Means (Sener and Savarese, 2018), and Core-Set (Sener and Savarese, 2018). The selection of samples in the first group is based on the confidence scores of the training data from the target domain that are computed using the DAME (ZSL) model. For example, for a budget of $b$ samples, Least Confidence corresponds to the top $b$ samples with the lowest confidence level. Multiple predictions for a given sample are needed for USDE and BALD to compute the uncertainty functions, and we obtain these different predictions by activating the dropout layers during the inference phase on the target domain. The second group is based on the embeddings of samples from the target domain that are obtained using the DAME (ZSL) model. Clustering of the input space is then applied to determine centers of clusters or core sets. Table 3 shows that the confidence-based methods lead to better results than the embedding-based methods. In particular, when we select 25% of samples using the BALD method for the Cameras dataset, we achieve the same F1 score of a fully fine-tuned DAME model using all training data from the target domain. This indicates that the predictions from the classification layer $N$ of our model $M$ accurately reflect the data points where DA was unsuccessful. Therefore, by fine-tuning on these samples from the training data, our model generalizes better on the testing set of the target domain.

We show the embedding of DAME in the case of ZSL. Figure 4 shows the t-SNE visualization of the final embeddings for the target and source domains after DA in the wild (ZSL case). We only show the embeddings of four domains due to the space limitation in the paper, but we notice similar patterns for all the datasets. The gray and blue colors represent randomly selected data points from the source domains with a label 0 and label 1, respectively; the green and red colors represent randomly selected data points from the testing set of the target domain with a label 0 and label 1, respectively. 12 domains are used in each experiment, where the caption of each subfigure in Figure 4 represents the target domain, and the 11 remaining datasets represent the source domains. The best case is to have a mixture of blue and red dots which represent the matching class for the source and target domains, respectively, and a mixture of gray and green dots which represent the non-matching class for the source and target domains, respectively. This means that we transfer the task knowledge from sources to the target domain for both labels. For example, for Computers and DBLP-ACM, we obtain embeddings that respect the matching and non-matching classes as shown in Figure 4 (a) and (b), respectively. On the other hand, for Amazon-Google and Walmart-Amazon, there are green dots that are closer to the blue dots than the gray dots as shown in Figure 4 (c) and (d), respectively, and this leads to incorrect predictions for DAME (ZSL).