Named Entity Linking with Entity Representation by Multiple Embeddings

A named entity can be described in very different contexts Ma et al. (2021); FitzGerald et al. (2021). The same person can be a scientist and a dissident, the same location can be described by its nature and by its social events and so on. This is the motivation to represent an entity by multiple embeddings - at least if each embedding is created from a mention in specific context.

Our representation of KB entity $E_{a}$ is composed of multiple embeddings, and consists of:

1. 1.

   Normalized embeddings $e_{i}$, their norms $|e_{i}|$ and assigned thresholds $t_{i}$, initially set $t_{i}=-1$. The number of the embeddings is restricted by agglomerative clustering (Appendix B), $i<=N_{E}$.

2. 2.

   Entity threshold $T$.

3. 3.

   Entity surface names $s_{k}$.

4. 4.

   References to similar entities $E_{b}$.

In this and the next subsections we explain the details of this representation, and our NEL procedure that uses it.

A fundamental element used in building an entity is an embedding of a mention of this entity in some context. We tuned a pretrained BERT Devlin et al. (2019) language model ’bert-base-uncased’, accessed via the transformers library ((Wolf et al., 2020)), on generic random news, with named entities located in the texts. The only goal of tuning is to enhance LM performance on the mentions of named entities, without changing LM goal or making LM specialized on any particular set of named entities. The tuning and inference are as following:

1. 1.

   At tuning only the named entities - all the located mentions within the input window in the text - serve as the labels for prediction. In input the mentions are being either left unchanged (probability 0.5) or replaced by another random mention from the same text.

2. 2.

   At inference the text is kept as it is, and the model is run only once on each input-size chunk of the text. The embeddings are picked up from the first token of the entity surface form - for each named entity that happened to occur in the input-size chunk.

Through all the paper, except Section 4.3, we use the model tuned on random generic news. In Section 4.3 we use the model tuned on texts of more uniform style: the texts from XSum Narayan et al. (2018) and CNN / Daily Mail Hermann et al. (2015); Nallapati et al. (2016) datasets. This allowed us to observe effect of using embeddings from a model exposed to a lesser variety of styles. For more details of the tuning see Appendix A.

There can be many mentions available for building a KB entity, even if only reliable verified mentions are used. If the number of the embeddings obtained from the mentions is higher than $N_{E}$, the embeddings are clustered, and only $N_{E}$ ’central’ embeddings (closest to the centers of the clusters) are stored. We use agglomerative clustering (Appendix B). The surface names $s_{k}$ of all the mentions used for creating an entity are stored in the entity.

In linking a mention to KB we use only the normalized embedding $e$ of the mention. We define similarity $S$ of the mention to an entity by the scalar products with its embeddings $e_{i}$:

Here all the thresholds $t_{i}$ are set to $-1$, and are irrelevant unless adjusted as described in Section 2.3; the entity’s threshold $T$ is defined further below.

The KB entity with the highest similarity $S$ is the candidate for linking the mention to. We set a linking threshold $T_{L}$: The mention is linked to the candidate-entity only if

Otherwise the mention is left unlinked (unassociated with any KB entity). It is natural to assume $T_{L}=1$, but our results will show that we had to lower it.

The entity’s threshold $T$ is defined from the assumption that any embedding of the entity would have to successfully link to the entity (with $T_{L}=1$):

This definition makes sense only if there are at least two embeddings in the entity, hence we create a KB entity only if there are at least two mentions available (our observations in Section 4 suggest a more strict requirement).

When linking a mention to KB we find the similarities not to all KB entities, but only to the entities that have a surface name at least somewhat similar to the mention. For this purpose, KB stores a map of all words from all the surface names to the entities that have any such word in their surface names:

When linking a mention to KB, all KB entities that are mapped from at least one word of the mention’s surface name are considered as the candidates for linking. The similarities of the mention’s embedding to these candidates are calculated then by Eq.1; the mention is linked to the candidate with the strongest similarity if exceeds the threshold Eq.2. Generally, KB entities with similar (by some measure) embeddings can be considered in selecting the candidates, but we focus here on the namesakes.

Can we improve KB right after it is created, even without using any knowledge about the texts and mentions on which NEL will be used or evaluated? We suggest adjusting the thresholds $t_{i}$ in each entity $E_{a}$ by considering the relation of $E_{a}$ with its similar entities $E_{b}$.

Each entity $E_{a}$ stores references to its most ’similar’ entities $E_{b}$. For an entity with surface names $s_{k}$ we select its similar entities using a (non-symmetric) surface-similarity, which we define as

where $s_{m}$ are the surface names of another KB entity, w is any word in $s_{k}$ also existing in $s_{m}$, and $l(w)$ is the number of characters in the word $w$. For each KB entity $E_{a}$ we find $N_{S}=10$ entities $E_{b}$ having the highest $L>0$ (may be less than 10 because of the requirement $L>0$). References to these ’surface-similar’ entities $E_{b}$ are stored in the entity $E_{a}$.

In order to adjust a KB entity $E_{a}$, we impose a requirement on each embedding $e$ from each similar entity $E_{b}$: the embedding $e$ should not be able to link to the entity $E_{a}$ (because $E_{a}$ and $E_{b}$ are different entities).

KB is adjusted to satisfy this requirement, by iterating through all KB entities $E_{a}$; for each entity $E_{a}$ iterating through its similar entities $E_{b}$; for each pair $E_{a}$ and $E_{b}$ iterating through the embeddings $e_{i}$ of $E_{a}$ and $e_{j}$ of $E_{b}$. The threshold $t_{i}$ for $e_{i}$ is adjusted as:

Here $T$ is the threshold of the entity $E_{a}$. We use $c=1.01$, just enough to make the linking impossible. For clarity, the adjustment procedure is also explained by pseudo-code in Figure 1.

For definiteness, we iterate through entities $E_{a}$ in KB in order from larger to smaller dissimilarity of the entity; where we defined ’dissimilarity’ of $E_{a}$ as the sum

with summation over all the pairs of embeddings from $E_{a}$. The motivation for this order is to start with entities that might be more prone to the conflicts and adjustments in KB. However, in the experiments described in this paper there were no any noticeable difference between this version and versions with somewhat different definitions of ’dissimilarity’, and even with the opposite order of the iteration.

We do not consider here an alternative possibility: instead of increasing an embedding’s threshold, we can rotate the embeddings from each other with the purpose of decreasing their product below the threshold; for more detail see Appendix C.

We present here evaluation results in terms of the three indicators introduced in the previous section. In Figure 2 we show the level of NEL errors for evaluating all the mentions from News.

We observe that the minimal number of mentions allowed to create a KB entity plays an important role in reducing the amount of errors, even though the entity mentions were clustered into only 4 embeddings.

The role of the linking threshold $T_{L}$, as expected, is the trade-off between the performance for familiar mentions and for stranger mentions. A higher threshold decreases the fraction of stranger mentions linked to KB, while increasing the fraction of familiar mentions not linked to KB.

In Figure 3 we show again the dependency of NEL errors on the minimal number of mentions per KB entity, but now we evaluate linking of Backlinks mentions to KB (this is the only difference between the settings for the figures 3 and 2).

We observe a comparable level of errors in linking a familiar mention to wrong KB entity and in wrongly linking a stranger mention to KB, but there is a much higher fraction of unlinked familiar mentions. We speculate that the reason is in the less context usually given for a named entity mention in a Wikipedia backlink, as opposed to a mention in the news.

The number of samples participating in the evaluation is presented in left panes of Tables 1 and 2. Relaxed requirement on minimal number of mentions per entity allows for larger KB, and makes more familiar mentions. Backlinks part of Namesakes provides more mentions for evaluation.

In Figure 4 we show that the limit on the number of stored embeddings can be as low as 4 - at least judging by evaluation on Namesakes News. Evaluation on Namesakes Backlinks - in Figure 5 - show only a very weak dependency on the maximal number of embeddings. We suggest that it may be helpful to store more embeddings, depending on the type and cleanness of the data involved in creating KB. We observe more evidence for this in Section 4.2.

The number of samples involved in the evaluations in Figures 4 and 5 are presented by the last rows in the left panes in Tables 1 and 2 correspondingly.

Increase of the linking threshold $T_{L}$, as expected, suppresses wrong linking, but increases the fraction of unlinked mentions. We show an example of such dependency for wide range of threshold values in Figure 6 for NEL applied to the mentions from News, and in Figure 7 for NEL applied to the mentions from Backlinks. The main difference between the figures is in higher fraction of unlinked familiar mentions from Backlinks; this again suggests that too many mentions in Namesakes Backlinks must have a limited context. The choice of $T_{L}$ should be guided by a required trade-off between the errors $F_{SL}$ vs $F_{FW}$ and $F_{FN}$.

We consider here the effect of lowering KB quality on NEL. When creating KB entities, we add erroneous mentions, imitating the real life situation of not fully reliable sources. Such polluted KB should increase NEL errors. We also expect that KB adjustment described in Section 2.3 can alleviate the effect of the pollution.

We use for pollution the "Other"-tagged mentions from Namesakes Entities Vasilyev et al. (2021a): these mentions have the surface names of the considered entity (Wikipedia entry) but represent some different entity mentioned in the same entry. The pollution can be characterized by the fraction of the polluted KB entities, and by the average fraction of the wrong mentions used in creating a polluted KB entity. We set both these pollution levels to 0.5, meaning that we pollute each second entity, and that we are creating each polluted entity by making it with up to 50% added wrong mentions (subject to availability of ’Other’ mentions in the corresponding document in Namesakes Entities).

In Figure 8 we show how much KB pollution affects NEL, and how much our KB adjustment, described in Section 2.3, can alleviate the effect of pollution.

The pollution somewhat increases the fraction of familiar mentions linked to wrong KB entities. Pollution increases even more the fraction of stranger mentions wrongly linked to KB. The KB adjustment reduces this effect.

Figure 9 shows the evaluation for linking Backlinks mentions to the clean, polluted and polluted-and-adjusted KB. Here we observe that the effect of pollution and adjustment on unlinked familiar mentions $F_{SL}$ is opposite: polluted KB entities encourage linking, wrong or not. Of course the effects on wrongly linked familiar mentions $F_{FW}$ and wrongly linked stranger mentions $F_{SL}$ are more important.

The number of samples participating in the evaluation is presented in Tables 1 and 2.

Pollution changes dependency of NEL performance on max number of entity embeddings $N_{E}$: in Figure 10 the stranger mentions errors still decrease by $N_{E}=7$.

Figure 11 show even more change for Backlinks, as compared to almost no dependency in Figure 5.

In previous subsections we observed NEL where the evaluation mentions are quite different from the mentions used in creating KB entities: KB entities were created from mentions of the entity on its own Wikipedia page; Backlinks mentions come from mentioned entities on Wikipedia backlinks; News mentions come from generic news texts. The difference between KB entities and evaluation mentions added to the difficulty of NEL.

In this subsection we consider one more variation that may cause additional difficulty for NEL: we will use a model tuned not on generic news (see Section 2.1), but on texts of more uniform style - a mix of texts from XSum Narayan et al. (2018) and CNN / Daily Mail Hermann et al. (2015); Nallapati et al. (2016) datasets (we take the texts of the documents, not the summaries.) The details of the data and tuning are in Appendix A.

From Figure 12 we observe that the optimal value of threshold is now different, and that NEL is more sensitive to threshold.

From Figure 13 we observe that effect of KB adjustment can be stronger. The adjustment, while strongly reducing the most serious errors - $F_{FW}$ and $F_{SL}$ - increases the fraction of unlinked familiar mentions $F_{FN}$. The latter is understandable as a conseqeunce of increasing individual embedding thresholds in KB entities.

From Figure 14 we observe that choosing the KB entity’s max number of embeddings $N_{E}$ may also be affected by the type of model used for embeddings (compare with Figure 4).