Combining Data-driven Supervision with Human-in-the-loop Feedback for Entity Resolution

The distribution discrepancy between training datasets and data encountered in production is widely known [2, 12, 17]. A common workflow for collecting training datasets is to curate data over a fixed period of time and then label them using annotators. This limits the number and kinds of variations that are present in a training dataset. Models relying only on training datasets may lead to underwhelming performance in production when confronted with perturbed or out-of-distribution samples. This training-production performance gap has been addressed more generally from several closely related perspectives including out-of-distribution learning [16], model robustness [8], and data augmentation [7]. Confronted with the training-production performance gap in a business setting, we found that an application-specific approach was needed.

We were tasked with building an entity resolution system - a model that identifies and consolidates data points that represent the same person [3]. For example, the model intends to recognize that a Jane Doe with address 123 Main St, San Francisco, CA is the same person as J. V. Doe living on 123 Main St. in Zipcode 94158. Circumstantially, more information such as phone number or email address might be available too. Our model would be deployed in production and used as an automated data pre-processing step to potentially benefit many downstream analytics and modelling tasks. For this initial development phase, our model intends to classify identities based on one field - an individual’s name and leaves the incorporation of other contextual fields in future phases. For training data, we were given a highly curated set of labeled name pairs consisting of examples such as (‘‘Joe’’, ‘‘Joseph’’, match) or (‘‘Joe’’, ‘‘Jane’’, mismatch). For evaluation, we had a diverse set of unlabeled name pairs that included new instances of real-world name variations. Our first model exhibited the training-production performance gap, inspiring us to create a human-in-the-loop enabled, data-centric approach for this application.

In this case study, we discuss the details of our approach, a mix of data augmentation and data-derived rules, and the key takeaways applicable to data-centric learning more generally.

Across documents and records, the name of an individual can have different surface forms including capitalization, spellings, accents, ordering and others. We summarize some of the complex variances below. (a) Missing spaces/hyphens: “Mary Ellen”, “MaryEllen”, and “Mary-Ellen” may represent the same person; (b) Initials: a name, such as “James Earl Smith”, could also be represented with initials such as “JE Smith”, “J.E. Smith”, “JE. Smith” or “J.E Smith”; (c) Out-of-order components: such as “Diaz Carlos Alfonzo” vs. “Carlos Alfonzo Diaz”; (d) Truncated name: such as “Livingston Charles” vs. “Living Charles”; (e) Nicknames: a name’s nickname is often a shorter string (e.g., “Mike” vs. “Michael”) or can even be totally different by metaphone (e.g., “William” vs. “Bill”); (f) Multilingual: the users from different language backgrounds are allowed to type their names in native languages so that our system is expected to match multilingual names. For example, “Schütze” (German) matches with “Schuetze” (English); (g) Junk words: when the system requires a user to type in her or his name, the user sometimes just puts a random string as the name, such as “too”, “API”, “such”, etc; (h) Gender mismatch: we should clearly reject the match of two names if they are typically used in different genders, such as “Daniel” (M) vs. “Daniela” (F).

To provide supervision for developing a system, we manually annotated some data in different languages, including English, Arabic, French, German, Russian, Spanish and Portuguese. The data consisted of the most popular names within our consumer data. Also included were the most popular names by country (both male and female), as well as most popular nicknames. We then created multiple permutations of these names to annotate different matching patterns. The detailed statistics are presented in Table 1.

More matched pairs were included than mismatched pairs. To generate more mismatched instances, we used rules, such as deletion, insertion, and n-gram similarity, to collect about 20,000 mismatched pairs automatically. This labeled data is randomly split into train (70%), dev (20%) and test (10%).

The baseline is a system already in production. It is based on name transformation given predefined patterns: (i) Longest common substring pattern, e.g., (“Jörn” vs. “Jorn”) $\rightarrow$ (“jö2”, “jo2”), where digit “2” indicates the length of the common substring; (ii) Position unigrams metaphone pattern, e.g., (“aab” vs. “abbc”) $\rightarrow$ (“a1” vs. “b1c3”). So, given the input name pair (“aab”, “abbc”), the following transformed pairs will be created: [(‘‘aab’’, ‘‘abbc’’), (‘‘a2’’, ‘‘2bc’’), (‘‘a1’’, ‘‘b1c3’’)]. Then, in the training phase, each transformed pair will be mapped to the frequencies of belonging to the match and mismatch classes. For a test name pair, apply the same transformation extraction, and then feed into model, use the feature with highest probability to decide whether the input is a match or not.

Our goal was to build a model that outperforms this baseline on the held-out test set and real consumer data. Next, we describe our human-in-the-loop enabled process for a more advanced system.

Our system was developed with the help of multiple rounds of human evaluations and feedback. We summarize the four versions in the following subsections.

Most prior works, such as [1, 10, 9, 13, 15, 14, 11], tried to collect adversarial training data by human annotators to address the error patterns found in human-in-the-loop. We tried too but finally gave up due to two reasons: (i) collecting large-scale adversarial data is even more costly than enlarging the training data with data augmentation; (ii) it performed worse than rules for some error patterns in which a simple rule can get high precision, such as gender-aware predictions.

In this case study, we shared our human-in-the-loop-enabled, data-centric approach for building a production quality entity resolution model. In contrast to some research that work on clean labeled data and claim state of the art performance, we are trying to handle a real-world problem which inspires us to rethink the following questions:

What is data? By “data”, the community usually refer to labeled or unlabeled data. Here, we advocate that rules are also a sort of precise expression of data. More specifically, If the issue can be exactly expressed by concise rules, that means rules are actually the objective status of model training on data. Therefore, we think “data” should be extended as “data+rules”.

How much data is needed? Labeled data is always limited. However, the challenges in the real world are often beyond the imagination. Data-centric learning can play a bigger role once combined with the human-in-the-loop interactions.