CLaCLab at SocialDisNER: Using Medical Gazetteers for Named-Entity Recognition of Disease Mentions in Spanish Tweets

Sequence analysis classifying each token into BIO tags requires good tokenization. We pre-process the data using ANNIE [Cunningham et al., 2002] and the Twitter English Tokenizer [Bontcheva et al., 2013].

Hashtags and usernames usually include several words strung together without separating characters, e.g. #InvestigaEpilepsia. Since disease mentions can occur in these word composites, their individual word components have to be separated. Camel casing makes splitting this example into Investiga and Epilepsia easy, but for hashtags like #nosolohaycovid only knowledge of words in Spanish leads to the desired split into no, solo, hay and covid. Because disease names are not always part of the general vocabulary of a language, disease gazetteers are a lightweight means to inject such domain knowledge into a general machine learning system.

A gazetteer lists entities of a particular type. For example, a disease gazetteer would contain a list of disease names like COVID, Hepatitis, Epilepsia etc. Ad hoc gazetteer lists that are written for a specific task often suffer from limited coverage. We circumvent our own limitations by using four gazetteer lists compiled from extant resources:

GoldGaz:

disease names compiled from the gold annotations of SocialDisNER training and validation data

DistemistGaz:

[Miranda-Escalada et al., 2022] Spanish disease gazetteer compiled from Snomed-CT [Donnelly and others, 2006]

SilverGaz:

disease gazetteer compiled from silver standard data made available for SocialDisNER [Gasco et al., 2022]

UmlsGaz:

disease gazetteer of Spanish and English disease terms we compiled from UMLS [Bodenreider, 2004]. Semantic type T047 represents Disease or Syndrome, identifying all UMLS concepts representing diseases.

The GATE Twitter English Tokenizer [Bontcheva et al., 2013] identifies hashtags and usernames, but doesn’t split them. To split usernames and hashtags, we identify the longest substring in a hashtag/username that matches an entity in our gazetteer, then treat that substring as a separate token. The extension matches first against GoldGaz, then DistemistGaz, followed by UmlsGaz, and finally SilverGaz.

We frame the named entity recognition (NER) task as a sequence labeling task with the BIO (beginning, inside, outside) tagging scheme. In other words, we classify each token as either B (token is the beginning of a disease name), I (token lies inside a disease name) or O (token lies outside a disease name).

Preprocessing produces $S$, a list of $t$ tokens. $S$ is fed into a pretrained XLM RoBERTa Large [Conneau et al., 2020] model, which returns the matrix $\textbf{H}\in\mathbb{R}^{t\times d}$, with $d=1024$. The $i^{th}$ row of H is a vector of size $d$ and corresponds to the contextualized embedding of the $i^{th}$ token.

We use the GATE ANNIE Gazetteer plugin to find tokens that match terms in UmlsGaz exactly (case-insensitively), creating $\textbf{G}_{\textit{umls}}\in\mathbb{R}^{t\times 2}$, a matrix where each row is a one hot vector indicating whether the $i^{th}$ token matches with UmlsGaz or not. Similarly, we create $\textbf{G}_{\textit{silver}}$ and $\textbf{G}_{\textit{distemist}}$ corresponding to SilverGaz and DistemistGaz respectively. Then, H is row-wise concatenated with $\textbf{G}_{\textit{umls}}$, $\textbf{G}_{\textit{distemist}}$, and $\textbf{G}_{\textit{silver}}$ to produce $\textbf{Z}\in\mathbb{R}^{t\times 1030}$. Z is then fed into a 6-layer Transformer Encoder [Vaswani et al., 2017] with positional encoding and 10 attention heads per layer. The Transformer Encoder outputs $\textbf{Y}\in\mathbb{R}^{t\times 1030}$ where the $i^{th}$ row represents the encoded representation of the $i^{th}$ token. Y is fed into a linear classifier which produces $\textbf{I}\in\mathbb{R}^{t\times 3}$, classifying each token into one of the 3 classes corresponding to the BIO tagging scheme. This represents our submission system $M_{\textit{sub}}$.

XLM RoBERTa large is fine-tuned on the training data using the Adam optimizer [Kingma and Ba, 2015] with a learning rate of 1e-5 and early stopping. Training for 10 epochs takes 2.5 hours on one Nvidia RTX 3090 gpu.