Multifile Partitioning for Record Linkage and Duplicate Detection

In the setting of multifile record linkage and duplicate detection, the datafiles are the product of $K$ data collection processes, which possibly introduce different patterns of duplication, measurement error, and missingness. This indicates that records coming from different datafiles should be treated differently. To take this into account, we introduce the concept of a multifile coreference partition by endowing a coreference partition $\mathcal{C}$ with additional structure to preserve the information on where records come from. Each cluster $c\in\mathcal{C}$ can be decomposed as $c=c^{1}\cup\cdots\cup c^{k}\cup\cdots\cup c^{K}$, where $c^{k}$ is the subset of records in cluster $c$ that belong to datafile $\bm{X}_{k}$, which leads us to the following definition.

For simplicity we will continue using the notation $\mathcal{C}$ to denote a multifile coreference partition, although technically this new structure is richer and therefore different from a coreference partition that does not preserve the datafile membership of the records. The multifile representation of partitions is useful for decoupling the features that are important for within-file duplicate detection or for across-files record linkage.

For duplicate detection, the goal is to identify coreferent records within each datafile. This can be phrased as estimating the within-file coreference partition $\mathcal{C}_{k}$ of each datafile $\bm{X}_{k}$. Clearly, these $\mathcal{C}_{k}$ can be obtained from the multifile partition $\mathcal{C}$ by extracting the $k$th entry of each cluster $c=(c^{1},\dots,c^{K})\in\mathcal{C}$. Two useful summaries of a given within-file partition $\mathcal{C}_{k}$ are the number of within-file clusters $n_{k}=|\mathcal{C}_{k}|$, which is the number of unique entities represented in datafile $\bm{X}_{k}$, and the within-file cluster sizes $\bm{d}^{k}=\{|c^{k}|:c^{k}\in\mathcal{C}_{k}\}$, which represent the number of records associated with each entity in datafile $\bm{X}_{k}$.

On the other hand, in record linkage the goal is to identify coreferent records across datafiles. Given the within-file partitions, $\mathcal{C}_{1},\cdots,\mathcal{C}_{K}$, the goal can be phrased as identifying which clusters across these partitions represent the same entities. This across-datafiles structure can be formally represented by a $K$-partite matching. Given $K$ sets $V_{1},\cdots,V_{K}$, a $K$-partite matching $\mathcal{M}$ is a collection of subsets from $\cup_{k=1}^{K}V_{k}$ such that each $m\in\mathcal{M}$ contains maximum one element from each $V_{k}$. If we think of each $V_{k}$ as the set of clusters $\mathcal{C}_{k}$ representing the entities in datafile $\bm{X}_{k}$, then it is clear that the goal is to identify the $K$-partite matching $\mathcal{M}$ that puts together the clusters that refer to the same entities across datafiles. This structure can be extracted from a multifile coreference partition $\mathcal{C}$, given that each element $c=(c^{1},\dots,c^{K})\in\mathcal{C}$ contains the coreferent clusters across all within-file partitions. Indeed, a multifile coreference partition can be thought of as a $K$-partite matching of within-file coreference partitions.

A useful summary of the across-datafile structure is the amount of entity-overlap between datafiles, represented by the number of clusters $c=(c^{1},\dots,c^{K})\in\mathcal{C}$ with records in specific subsets of the files. We can concisely summarize the entity-overlap of the datafiles through a contingency table. In particular, consider a $2^{K}$ contingency table with cells indexed by ${\bm{h}}\in\{0,1\}^{K}$ and corresponding cell counts $n_{\bm{h}}$. Here, $\bm{h}$ represents a pattern of inclusion of an entity in the datafiles, where a 1 indicates inclusion and a 0 exclusion. For instance, if $K=3$, $n_{011}$ is the number of clusters $c=(c^{1},c^{2},c^{3})\in\mathcal{C}$ representing entities with records in datafiles $2$ and $3$ but without records in datafile $1$. We let $\mathcal{H}=\{0,1\}^{K}\setminus\{0\}^{K}$ and denote the (incomplete) contingency table of counts as $\bm{n}=\{n_{\bm{h}}\}_{\bm{h}\in\mathcal{H}}$, which we refer to as the overlap table. We ignore the cell $\{0\}^{K}$ which would represent entities that are not recorded in any of the $K$ files. This cell is not of interest in this article, although it is the parameter of interest in population size estimation (see e.g. Bird & King, 2018).

Example. To illustrate the concept of a multifile partition, consider two files with five and seven records respectively, so that $\bm{X}_{1}$ contains records $1-5$ and $\bm{X}_{2}$ contains records $6-12$. Suppose the coreference partition is $\{\{1,9\},\{2\},\{3,8,10,11\},\{4,5,7\},\{6\},\{12\}\}$. The corresponding multifile partition is $\mathcal{C}=\{(\{1\},\{9\}),(\{2\},\emptyset),(\{3\},\{8,10,11\}),(\{4,5\},\{7\}),(\emptyset,\{6\}),(\emptyset,\{12\})\}$. As illustrated in Figure 2, the within-file partitions can be extracted as $\mathcal{C}_{1}=\{\{1\},\{2\},\{3\},\{4,5\}\}$ and $\mathcal{C}_{2}=\{\{6\},\{7\},\{9\},\{8,10,11\},\{12\}\}$, and the within-file cluster sizes are $\bm{d}^{1}=(1,1,1,2)$ and $\bm{d}^{2}=(1,1,1,3,1)$. The overlap table in this case is $\{n_{11},n_{10},n_{01}\}$, indicating that $n_{11}=3$ entities are represented in both datafiles, $n_{10}=1$ entity is represented only in the first datafile, and $n_{01}=2$ entities are represented only in the second datafile. In total, there are $n_{1}=|\mathcal{C}_{1}|=n_{11}+n_{10}=4$ unique entities represented in $\bm{X}_{1}$, $n_{2}=|\mathcal{C}_{2}|=n_{11}+n_{01}=5$ unique entities represented in $\bm{X}_{2}$, and $n=|\mathcal{C}|=n_{11}+n_{10}+n_{01}=6$ entities among both datafiles.

Prior for the Number of Entities or Clusters. In the absence of substantial prior information, we follow a simple choice for the prior on the number of clusters, by taking a uniform distribution over the integers less than some upper bound, $U$, i.e. $\mathbb{P}(n)=U^{-1}I(n\in\{1,\cdots,U\})$. In practice, we set $U$ to be the actual number of records across all datafiles $r$, which is observed. More informative specifications are discussed in Appendix A.

The structured prior for multifile partitions allows us to incorporate prior information about the total number of clusters, the overlap between files, and the amount of duplication in each file. If we restrict the prior for the within-file cluster sizes to be $p_{k}(d^{k}_{i})=I(d^{k}_{i}=1)$ for a given datafile $\bm{X}_{k}$, then we enforce the assumption that there are no duplicates in that file. Imposing this restriction for all datafiles leads to the special case of a prior for $K$-partite matchings, which is of independent interest, as we are not aware of any such constructions outside of the bipartite case (Fortini et al., 2001; Larsen, 2005; Sadinle, 2017).

Our prior construction, where priors are first placed on interpretable summaries of a partition and then a uniform prior is placed on partitions which have those summaries, mimics the construction of the priors on bipartite matchings of Fortini et al. (2001); Larsen (2005) and Sadinle (2017), and the Kolchin and allelic partition priors of Zanella et al. (2016) and Betancourt, Sosa & Rodríguez (2020). While the Kolchin and allelic partition priors could both be used as priors for multifile partitions, these do not incorporate the datafile membership of records. Using these priors in the multifile setting would imply that the sizes of clusters containing records from only one file have the same prior distribution as the sizes of clusters containing records from two files, which should not be true in general.

Miller et al. (2015) and Zanella et al. (2016) proposed the microclustering property as a desirable requirement for partition priors in the context of duplicate detection: denoting the size of the largest cluster in a partition of $[r]$ by $M_{r}$, a prior satisfies the microclustering property if $M_{r}/r\to 0$ in probability as $r\to\infty$. A downside of priors with this property is that they can still allow the size of the largest cluster to go to $\infty$ as $r$ increases. For this reason Betancourt, Sosa & Rodríguez (2020) introduced the stronger bounded microclustering property, which we believe is more practically important: for any $r$, $M_{r}$ is finite with probability $1$. Our prior satisfies the bounded microclustering property as $M_{r}\leq\sum_{k=1}^{K}U^{k}$.

While our parameter of interest is a partition $\mathcal{C}$ of $r$ records, the prior developed in this section is a prior for a partition of a random number of records. In practice we condition on the file sizes, $\{r_{k}\}_{k=1}^{K}$, and use the prior $\mathbb{P}(\mathcal{C}\mid\{r_{k}\}_{k=1}^{K})\propto\mathbb{P}(\mathcal{C})I(r^{\prime}_{k}(\mathcal{C})=r_{k}(\mathcal{C})\text{ for all }k)$, which alters the interpretation of the prior. A similar problem occurs for the Kolchin partition priors of Zanella et al. (2016). This motivated the exchangeable sequences of clusters priors of Betancourt, Zanella & Steorts (2020), which are similar to Kolchin partition priors, but lead to a directly interpretable prior specification. It would be interesting in future work to see if an analogous prior could be developed for our structured prior for multifile partitions. Despite this limitation, we demonstrate in simulations in Section 6 that incorporating strong prior information into our structured prior for multifile partitions can lead to improved frequentist performance over a default specification.

We construct comparison vectors for pairs of records to provide evidence for whether they correspond to the same entity. For $k\leq k^{\prime}$, let $\mathcal{P}_{kk^{\prime}}=\{(i,j):i<j,i\in{\bm{X}}_{k},j\in{\bm{X}}_{k^{\prime}}\}$ denote the set of all record pairs between files ${\bm{X}}_{k}$ and ${\bm{X}}_{k^{\prime}}$, and let $F$ be the total number of different fields available from the $K$ files. For each file pair $(k,k^{\prime})$, $k\leq k^{\prime}$, and record pair $(i,j)\in\mathcal{P}_{kk^{\prime}}$, we compare each field $f=1,\dots,F$ using a similarity measure $\mathcal{S}_{f}(i,j)$, which will depend on the data type of field $f$. For unstructured categorical fields such as race, $\mathcal{S}_{f}$ can be a binary comparison which checks for agreement. For more structured fields containing strings or numbers, $\mathcal{S}_{f}$ should be able to capture partial agreements. For example, string fields can be compared using a string metric like the Levenshtein edit distance (see e.g. Bilenko et al., 2003), and numeric fields can be compared using absolute differences. Comparison $\mathcal{S}_{f}(i,j)$ will be missing if field $f$ is not recorded in record $i$ or record $j$, which includes the case where field $f$ is not recorded in datafiles $\bm{X}_{k}$ or $\bm{X}_{k^{\prime}}.$

While we could directly model the similarity measures $\mathcal{S}_{f}(i,j)$, this would require a custom model for each type of comparison, which inherits similar problems to the direct modeling of the fields themselves. Instead, we follow Winkler (1990) and Sadinle (2014, 2017) in dividing the range of $\mathcal{S}_{f}$ into $L_{f}+1$ intervals $I_{f0},I_{f1},\cdots,I_{fL_{f}}$ that represent varying levels of agreement, with $I_{f0}$ representing the highest level of agreement, and $I_{fL_{f}}$ representing the lowest level of agreement. We then let $\gamma_{ij}^{f}=l$ if $\mathcal{S}_{f}(i,j)\in I_{fl}$, where larger values of $\gamma_{ij}^{f}$ represent larger disagreements between records $i$ and $j$ in field $f$. Finally, we form the comparison vector $\bm{\gamma}_{ij}=(\gamma_{ij}^{1},\cdots,\gamma_{ij}^{F})$. Constructing the comparison data this way allows us to build a generic modeling approach. In particular, extending Fortini et al. (2001); Larsen (2005) and Sadinle (2014, 2017), our model for the comparison data is

where $\mathcal{C}$ is a multifile partition, $\mathcal{C}(i)$ denotes record $i$’s cluster in $\mathcal{C}$, $\mathcal{C}(i)=\mathcal{C}(j)$ indicates that records $i$ and $j$ are coreferent, $\textsf{M}_{kk^{\prime}}(\bm{m}_{kk^{\prime}})$ is a model for the comparison data among coreferent record pairs from the file pair ${\bm{X}}_{k}$ and ${\bm{X}}_{k^{\prime}}$, $\textsf{U}_{kk^{\prime}}(\bm{u}_{kk^{\prime}})$ is a model for the comparison data among non-coreferent record pairs from datafile pair ${\bm{X}}_{k}$ and ${\bm{X}}_{k^{\prime}}$, and $\bm{m}_{kk^{\prime}}$ and $\bm{u}_{kk^{\prime}}$ are vectors of parameters.

In the next section we make two further assumptions that simplify the model parameterization. Before doing so, we note a few limitations of our comparison-based model. First, computing comparison vectors scales quadratically in the number of records. Second, comparison vectors for different record pairs are not actually independent conditional on the partition (see Section 2 of Tancredi & Liseo, 2011). Third, modeling discretized comparisons of record fields represents a loss of information. While the first limitation is computational and unavoidable in the absence of blocking (see Appendix B), the other two limitations are inferential. Despite these limitations, we find in Section 6 that the combination of our structured prior for multifile partitions and our comparison-based model can produce linkage estimates with satisfactory frequentist performance.

Under the assumptions that the fields in the comparison vectors are conditionally independent given the multifile partition of the records and that missing comparisons are ignorable (Sadinle, 2014, 2017), the likelihood of the observed comparison data, $\bm{\gamma}^{obs}$, becomes

Here $m_{kk^{\prime}}^{fl}=\mathbb{P}(\gamma_{ij}^{f}=l\mid\mathcal{C}(i)=\mathcal{C}(j),(i,j)\in\mathcal{P}_{kk^{\prime}})$, $u_{kk^{\prime}}^{fl}=\mathbb{P}(\gamma_{ij}^{f}=l\mid\mathcal{C}(i)\neq\mathcal{C}(j),(i,j)\in\mathcal{P}_{kk^{\prime}})$, $a_{kk^{\prime}}^{fl}(\mathcal{C})=\sum_{(i,j)\in\mathcal{P}_{kk^{\prime}}}I_{obs}(\gamma_{ij}^{f})I(\gamma_{ij}^{f}=l)I(\mathcal{C}(i)=\mathcal{C}(j)),$ $b_{kk^{\prime}}^{fl}(\mathcal{C})=\sum_{(i,j)\in\mathcal{P}_{kk^{\prime}}}I_{obs}(\gamma_{ij}^{f})I(\gamma_{ij}^{f}=l)I(\mathcal{C}(i)\neq\mathcal{C}(j)),$ $I_{obs}(\cdot)$ is an indicator of whether its argument was observed, and $\Phi=(\bm{m},\bm{u})$ where $\bm{m}$ collects all of the $\bm{m}_{kk^{\prime}}^{f}=(m_{kk^{\prime}}^{f0},\cdots,m_{kk^{\prime}}^{fL_{f}})$ and $\bm{u}$ collects all of the $\bm{u}_{kk^{\prime}}^{f}=(u_{kk^{\prime}}^{f0},\cdots,u_{kk^{\prime}}^{fL_{f}})$. For a given multifile partition $\mathcal{C}$, $a_{kk^{\prime}}^{fl}(\mathcal{C})$ represents the number of record pairs in $\mathcal{P}_{kk^{\prime}}$ that belong to the same cluster with observed agreement at level $l$ in field $f$, and $b_{kk^{\prime}}^{fl}(\mathcal{C})$ represents the number of record pairs in $\mathcal{P}_{kk^{\prime}}$ that do not belong to the same cluster with observed agreement at level $l$ in field $f$.

We will use independent, conditionally conjugate priors for $\bm{m}_{kk^{\prime}}^{f}$ and $\bm{u}_{kk^{\prime}}^{f}$, namely $\bm{m}_{kk^{\prime}}^{f}\sim\textsf{Dirichlet}(\mu_{kk^{\prime}}^{f0},\cdots,\mu_{kk^{\prime}}^{fL_{f}})$ and $\bm{u}_{kk^{\prime}}^{f}\sim\textsf{Dirichlet}(\nu_{kk^{\prime}}^{f0},\cdots,\nu_{kk^{\prime}}^{fL_{f}})$. In this article we will use a default specification of $(\mu_{kk^{\prime}}^{f0},\cdots,\mu_{kk^{\prime}}^{fL_{f}})=(\nu_{kk^{\prime}}^{f0},\cdots,\nu_{kk^{\prime}}^{fL_{f}})=(1,\cdots,1)$. We believe this prior specification is sensible for the $\bm{u}$ parameters, following the discussion in Section 3.2 of Sadinle (2014), as comparisons amongst non-coreferent records are likely to be highly variable and it is more likely than not that eliciting meaningful priors for them is too difficult. For the $\bm{m}$ parameters, it might be desirable in certain applications to introduce more information into the prior. For example, one could set $\mu_{kk^{\prime}}^{f0}>\cdots>\mu_{kk^{\prime}}^{fL_{f}}$ to incorporate the prior belief that higher levels of agreement should have larger prior probability than lower levels of agreement. Another route would be to use the sequential parameterization of the $\bm{m}$ parameters, and the associated prior recommendations, described in Sadinle (2014).

In Appendix B we outline a Gibbs sampler that produces a sequence of samples $\{\mathcal{C}^{[t]},\Phi^{[t]}\}_{t=1}^{T}$ from the posterior distribution $p(\mathcal{C},\Phi\mid\bm{\gamma}^{obs})$, which we will use to obtain Monte Carlo approximations of posterior expectations involved in the derivation of point estimates $\hat{\mathcal{C}}$, as presented in the next section. In Appendix B, we discuss the computational complexity of the Gibbs sampler, how computational performance can be improved through the usage of indexing techniques, and the initialization of the Gibbs sampler.

In a Bayesian setting, one can obtain a point estimate $\hat{\mathcal{C}}$ of the multifile partition using the posterior $\mathbb{P}(\mathcal{C}\mid\bm{\gamma}^{obs})=\int p(\mathcal{C},\Phi\mid\bm{\gamma}^{obs})d\Phi$ and a loss function $L(\mathcal{C},\hat{\mathcal{C}})$. The Bayes estimate is the multifile partition $\hat{\mathcal{C}}$ that minimizes the expected posterior loss $\mathbb{E}[L(\mathcal{C},\hat{\mathcal{C}})\mid\bm{\gamma}^{obs}]=\sum_{\mathcal{C}}L(\mathcal{C},\hat{\mathcal{C}})\mathbb{P}(\mathcal{C}\mid\bm{\gamma}^{obs})$, although in practice such expectations are approximated using posterior samples. Previous examples of loss functions for partitions included Binder’s loss (Binder, 1978) and the variation of information (Meilă, 2007), both recently surveyed in Wade & Ghahramani (2018). The quadratic and absolute losses presented in Tancredi & Liseo (2011) are special cases of Binder’s loss, and Steorts et al. (2016) drew connections between their proposed maximal matching sets and the losses of Tancredi & Liseo (2011).

In many applications there may be much uncertainty on the linkage decision for some records in the datafiles. For example, in Figure 1 it is unclear which of the records with last name “Smith” are coreferent. It is thus desirable to leave decisions for some records unresolved, so that the records can be hand-checked during a clerical review, which is common in practice (see e.g. Ball & Price, 2019). In the classification literature, leaving some decisions unresolved is done through a reject option (see e.g. Herbei & Wegkamp, 2006), which here we will refer to as an abstain option. We will refer to point estimates with and without abstain option as partial estimates and full estimates, respectively. We now present a family of loss functions for multifile partitions which incorporate an abstain option, building upon the family of loss functions for bipartite matchings presented in Sadinle (2017).

We start by describing the general characteristics of the simulations. For each of the simulation scenarios we conduct 100 replications, for each of which we generate three files as follows. For each of $n=500$ entities, $\bm{h}\in\mathcal{H}$ is drawn from a categorical distribution with probabilities $\{p_{\bm{h}}\}_{\bm{h}\in\mathcal{H}}$, where $\bm{h}$ represents the subset of files the entity appears in, and so we change the values of $\{p_{\bm{h}}\}_{\bm{h}\in\mathcal{H}}$ across simulation scenarios to represent varying amounts of file overlap. Files are then created by generating the implied number of records for each entity. In the additional simulations considered in Appendix D, the generated number of records for each entity depends not only on $\bm{h}$, but also on the duplication mechanism.

All records are generated using a synthetic data generator developed in Tran et al. (2013), which allows for the incorporation of different forms of measurement error in individual fields, along with dependencies between fields we would expect in applications. The data generator first generates clean records before distorting them to create the observed records. In particular, each observed record will have a fixed number of erroneous fields, where errors selected uniformly at random from a set of field dependent errors displayed in Table 3 of Sadinle (2014) (reproduced in Appendix D), with a maximum of two errors per field. We generate records with seven fields of information: sex, given name, family name, age, occupation, postal code, and phone number.

For each simulation replicate, we construct comparison vectors as given in Table 4 of Sadinle (2014) (reproduced in Appendix D). We use the model for comparison data proposed in Section 4 with flat priors on $\bm{m}$ and $\bm{u}$ as discussed in Section 5.1, and the structured prior proposed in Section 3 with a uniform prior on the number of clusters and $\bm{\alpha}=(1,\cdots,1)$ as described in Section 3.1. Using the Gibbs sampler presented in Appendix B we obtain $1,000$ samples from the posterior distribution of multifile partitions, and discard the first $100$ as burn-in. In Appendix D we discuss convergence of the Gibbs sampler, present running times of the proposed approach, and present an extra simulation exploring the running time of the approach with a larger number of records. We then approximate the Bayes estimate $\hat{\bm{Z}}$ for multifile partitions using the loss function described in Section 5.3.1 as described in Appendix C. For full estimates, we use the default values of $\lambda_{\scriptscriptstyle\mathrm{FNM}}=\lambda_{\scriptscriptstyle\mathrm{FM1}}=1$ and $\lambda_{\scriptscriptstyle\mathrm{FM2}}=2$ recommended by Sadinle (2017). In Appendix D we explore alternative specifications of the loss function.

We will assess the performance of the Bayes estimate using precision and recall with respect to the true coreference partition $\bm{Z}$. Let $\mathcal{P}$ be the set of all record pairs. Using notation from Section 5.3.1, let $TM(\bm{Z},\hat{\bm{Z}})=\sum_{(i,j)\in\mathcal{P}}\Delta_{ij}\hat{\Delta}_{ij}$ be the number of true matches (record pairs correctly declared coreferent), $FM(\bm{Z},\hat{\bm{Z}})=\sum_{(i,j)\in\mathcal{P}}(1-\Delta_{ij})\hat{\Delta}_{ij}$ be the number of false matches (record pairs incorrectly declared coreferent), and $FNM(\bm{Z},\hat{\bm{Z}})=\sum_{(i,j)\in\mathcal{P}}\Delta_{ij}(1-\hat{\Delta}_{ij})$ the number of false non-matches (record pairs incorrectly declared non-coreferent). Then precision is $TM(\bm{Z},\hat{\bm{Z}})/[TM(\bm{Z},\hat{\bm{Z}})+FM(\bm{Z},\hat{\bm{Z}})]$, the proportion of record pairs declared as coreferent that were truly coreferent, and recall is $TM(\bm{Z},\hat{\bm{Z}})/[TM(\bm{Z},\hat{\bm{Z}})+FNM(\bm{Z},\hat{\bm{Z}})]$, the proportion of record pairs that were truly coreferent that were correctly declared as coreferent. Perfect performance corresponds to precision and recall both being 1. In the simulations, we computed the median, $2$nd, and $98$th percentiles of these measures over the 100 replicate data sets. Additionally, in Appendix D, we assess the performance of the Bayes estimate when estimating the number of entities, $n$.

In this simulation study we explore the performance of our methodology by varying the number of erroneous fields per record over $\{1,2,3,5\}$, and also varying $\{p_{\bm{h}}\}_{\bm{h}\in\mathcal{H}}$, which determines the amount of overlap, over four scenarios:

• •

  High Overlap: $p_{001}=p_{010}=p_{100}=0.4/3,p_{011}=p_{101}=p_{110}=0.15,p_{111}=0.15$,

• •

  Medium Overlap: $p_{001}=p_{010}=p_{100}=0.7/3,p_{011}=p_{101}=p_{110}=0.05,p_{111}=0.15$,

• •

  Low Overlap: $p_{001}=p_{010}=p_{100}=0.8/3,p_{011}=p_{101}=p_{110}=0.05/3,p_{111}=0.15$,

• •

  No-Three-File Overlap: $p_{001}=p_{010}=p_{100}=0.55/3,p_{011}=p_{101}=p_{110}=0.15,p_{111}=~{}0$.

These are intended to represent a range of scenarios that could occur in practice. In the high overlap scenario 60% of the entities are expected to be in more than one datafile, in the low overlap scenario 80% of the entities are expected to be represented in a single datafile, and in the no-three-file overlap scenario no entities are represented in all datafiles.

To implement our methodology, in addition to the general set-up described in Section 6.1, we restrict the prior for the within-file cluster sizes so that they have size one with probability one, incorporating the assumption of no-duplication within files (see Section 3.2). Imposing this restriction for all datafiles leads to a prior for tripartite matchings. To illustrate the impact of using our structured prior, we compare with the results obtained using our model for comparison data with a flat prior on tripartite matchings.

The results of the simulation are seen in Figure 3. We see that our proposed approach performs consistently well across different settings, with the exception of the no-three-file overlap setting under high measurement error, where the precision decreases dramatically. The approach using a flat prior on tripartite matchings has poor precision in comparison, and it is particularly low when the amount of overlap is low. This suggests that our structured prior improves upon the flat prior by protecting against over-matching (declaring noncoreferent record pairs as coreferent). In Appendix D we demonstrate how the performance in the no-three-file overlap setting can be improved through the incorporation of an informative prior for the overlap table through $\bm{\alpha}$.

In this simulation study we have different patterns of measurement error across the three files. Rather than each field in each record having a chance of being erroneous according to Table 3 of Sadinle (2014), we will use the following measurement error mechanism to generate the data. For each record in the first file, age is missing, given name has up to seven errors, and all other fields are error free. For each record in the second file, sex and occupation are missing, last name has up to seven errors, and all other fields are error free. For each record in the third file, phone number and postal code have up to seven errors and all other fields are error free. Under this measurement error mechanism, there is enough information in the error free fields to inform pairwise linkage of the files. We further vary $\{p_{\bm{h}}\}_{\bm{h}\in\mathcal{H}}$ over the no-three-file and high overlap settings from Section 6.2.

Our goal in this study is to demonstrate that having both the structured prior for partitions and the separate models for comparison data from each file-pair can lead to better performance than not having these components. We will compare our model as described in Section 6.2 to both our model for comparison data with a flat prior on tripartite matchings (as in Section 6.2) and a simplification of our model for comparison data using a single model for all file-pairs but with our structured prior for partitions.

The results of the simulation are given in Figure 4. We see that our proposed approach outperforms both alternative approaches in both precision and recall in both overlap settings. This suggests that both the structured prior for tripartite matchings and the separate models for comparison data from each file-pair can help improve performance over alternative approaches. We note that in the no-three-file (high) overlap setting the precision of the proposed approach is greater than or equal to the precision of the approach using a single model for all file-pairs in $98$ ($100$) of the $100$ replications.