Assessment of contextualised representations in detecting outcome phrases in clinical trials

Outcome detection has previously been simultaneously achieved along with Participant and Intervention detection, where researchers aim to classify sentences (extracted from RCT abstracts) into one of P, I and O labels [?; ?; ?]. Despite being restrained by shortage of expertly labeled datasets, few attempts to create EBM-oriented datasets have been made. Bryon et al., [?] use distant supervision to annotate sentences within clinical trial articles with PICO elements. Dina et al., [?] use an experienced Nurse and a medical student to annotate outcomes by identifying and labelling sentences that best summarise the consequence of an intervention. Similarly, other attempts have precisely partitioned PubMed abstracts into sentences that they label one of P, I, and O [?; ?]. Given the sentence-level annotation adopted in these datasets, it becomes difficult to use them for tasks that require extraction of individual PICO elements [?; ?] such as outcome phrase detection. Nye et al., [?] recently released EBM-NLP corpus that they built using a mixture of crowd workers (non-experts) and expert workers (with the non-experts being exceedingly more) to annotate individual spans of P, I, O elements within clinical trial articles. This dataset has however been discovered with annotation flaws [?] and uses arbitrary outcome classification labels as discussed in section 2. Cognizant of the growing body of research to standardise classifications of outcomes, we are motivated to annotate a dataset with outcome types drawn from a standardised taxonomy.

TL is a machine learning (ML) approach that enables usage of a model to achieve a task that it was not initially built and trained for [?]. Usually, the assumption is that, train and test data for a specific task exists, however this is never the case, therefore, TL allows learning across different task domains i.e. the term pre-trained, implies a model was previously trained on a task different from the current target task. Context-dependent embeddings such as context2vec [?], ELMo [?] and BERT [?] have emerged and outperformed context-independent embeddings [?; ?] in various downstream NLP tasks.

Bert variants, SciBERT [?] and ClinicalBERT [?] yielded performance improvements in the BNER tasks on the BC5DR dataset [?; ?], text-classification tasks like Relation extraction on the ChemProt [?] and on PICO extraction. Despite being pre-trained on English biomedical text, BioBERT [?] outperformed generic BERT model ( pre-trained on Spanish biomedical text) in Pharma-CoNER, a multi-classification task for detecting mentions of chemical names and drugs from Spanish biomedical text [?]. Recently Qiao et al., [?] discovered that, in comparison to BioBERT, BioELMo (Biomedical ELMo) better clustered entities of the same type such as, an acronym having multiple meanings or a homonym. For example, unlike BioBERT, BioELMo clearly differentiated between ER referring to “Estrogen Receptor” and ER referring to “Emergency Room” in their work.

We leverage the datasets to investigate the ODP task performance of 6 different biomedical CLMs (table 2) derived from 3 main architectures. 1) BERT [?], a CLM built by learning deep bidirectional representations of input words by jointly incorporating left and right context in all its layers. It works by masking a portion of the input words and thereby predicting missing words in each sentence. BERT encodes a word by incorporating information about words around it within a given input sentence using a self-attention mechanism [?] 2) ELMo [?] is a CLM that learns deep bidirectional representations of input words by jointly maximizing the probability of forward and backward directions in a sentence, and 3) FLAIR [?], a character-level bidirectional LM which learns representations of each character by incorporating character information around it within a sequence of words.

We begin by further training the pre-trained CLMs in table 2 in a fine-tuning approach [?], where the CLMs learn to (1) encode each word $w_{i}$ into a hidden state $\boldsymbol{h}_{i}$ and (2) predict the correct label given $\boldsymbol{h}_{i}$. Similar to Sun et al.. [?], we introduce a non-linear softmax layer to predict a label for each $\boldsymbol{h}_{i}$ corresponding to word $w_{i}$, as shown in figure 2, where $\boldsymbol{h}_{i}=\mathrm{CLM}(w_{i})$, {BERT-variants, BioELMo, BioFLAIR} $\in\mathrm{CLM}$. (see Appendix A.1 (Fine-tuning) for more details).

We build ODP-tagger to not only assess context-independent (W2V) representations, but also assess the performance of frozen context-dependent representations for the ODP task. Demonstrated by the dotted line from Fine-tuning to input tokens in figure 2, is a feature extraction [?] approach, where the tagger’s embedding layer takes as input, a sequence of tokens (sentence) and a sequence of POS terms corresponding to the tokens. We add a POS feature for each token to enrich the model in a manner similar to how prior neural classifiers are enhanced with character and n-gram features [?]. Each word/token is therefore represented by concatenating either a pre-trained CLM or a W2V embedding $\boldsymbol{w}$ and a randomly initialised embedding for the corresponding POS term $\boldsymbol{p}$. The token embeddings are then encoded to obtain hidden-states for each sequence position,

$\displaystyle\boldsymbol{h}_{i}=\alpha(\text{\bf W}[\boldsymbol{w}_{i};\boldsymbol{p}_{i}]+b)$

(1)

where $\boldsymbol{w}_{i}\in\text{\bf E}^{w}$ and $\boldsymbol{p}_{i}\in\text{\bf E}^{p}$, $\{\text{\bf E}^{w},\text{\bf E}^{p}\}\in\mathbb{R}^{n\times d}$ denote Word and POS matrices, each containing d-dimensional embeddings for $n$ words and $n$ corresponding POS terms. $\boldsymbol{w}_{i}$ and $\boldsymbol{p}_{i}$ are the word and POS embeddings representing the $i^{th}$ word and its POS term, ; implies a concatenation operation and then $\alpha$ is a linear activation function that generates hidden states for the input words. We then use a condition random field (CRF) layer for classification given the hidden state $\boldsymbol{h}_{i}$. A CRF is an undirected graphical model which defines a conditional probability distribution over possible labels [?].

All the models are each trained to maximize the probability of the labels given each word $w_{i}\in s$.

$\displaystyle\underset{\theta}{\mathop{argmax}}P(y_{i}|\boldsymbol{w}_{i};\theta)$

(2)

The training loss objective.

$\displaystyle loss=-\beta\underset{(S,L)\in\mathcal{T}}{\sum}\sum_{i}^{n}p(y_{i}|\boldsymbol{w}_{i})$

(3)

where $\beta$ is a scaling factor that empirically sets each labels weights to be inversely proportional to the square root of the label frequency i.e. $\beta=\frac{1}{\sqrt{N_{y}}}$ and $N_{y}$ is the number of training samples with ground-truth label $y$. $\mathcal{T}$ is the training set containing sentences, $\boldsymbol{w}_{i}\in S$ and $y\in L$.

All models are evaluated on the two datasets discussed in section 3.1. These datasets are each partitioned as follows, 75% for training (train), 15% for development (dev) and 10% for testing (test). We exploit the large size of EBM-NLP${}_{\textbf{rev}}$ (as shown in table 4) and use its dev set to tune hyperparameters for the ODP-tagger and fine-tuned models (Parameter settings in Appendix B). Each model is trained on a train split of a particular dataset and evaluated on the corresponding test split culminating into results shown in table 3. We use a simple powerful NLP python framework called flair²²2https://github.com/flairNLP/flair to extract word embeddings from all the BERT and FLAIR variants, and AllenAI³³3https://github.com/allenai/bilm-tf for BioELMO. Dimensions of the extracted BioFLAIR and BioELMO embeddings are very large, i.e. 7672 and 3072 respectively, which would most likely overwhelm our memory and power-constrained devices during training. Therefore, we apply Principal component Analysis (PCA) dimensionality reduction technique to reduce their dimensions to half their original sizes while preserving semantic information [?]. Alongside these embeddings, we evaluate context-independent embeddings which we obtain by training word2vec (W2V) embedding algorithm [?] on 5.5B tokens of PubMed and PMC abstracts. Python and PyTorch [?] deep learning framework are used for implementation, which together with the datasets are made publicly available here https://github.com/MichealAbaho/ODP-tagger.

Results shown in table 3 firstly reveal the superiority of fine-tuning the CLMs in comparison to the ODP-tagger. The best performance across both set-ups is obtained when BioBERT is fine-tuned on the EBM-COMET dataset. However, we observe SciBERT outperform it in the ODP-tagger set-up on the EBM-COMET dataset. Secondly, we observe CLM embeddings produce stronger performances in comparison to context-independent (W2V) embeddings especially with the EBM-COMET dataset. BioFLAIR and ClinicalBERT were the least performing models. For BioFLAIR, we hypothesize that, (1) pre-training on a relatively smaller corpus, (2) it being of much less depth (1-layered BiLSTM) compared to multi-layered BERT and ELMo and (3) downsizing its embeddings using PCA dimensionality reduction are reasons that led to its low performance. For ClinicalBERT, we attributed its struggles to the nature of the corpora on which it is pre-trained. Unlike BioBERT, SciBERT and BioELMo which are pre-trained on PubMed text which is mostly clinical trial abstracts that more often report health outcomes, ClinicalBERT is pre-trained on clinical notes associated with patient hospital admissions [?]. An additional insight we drew was, performance on the EBM-NLP${}_{\textbf{rev}}$ dataset is lower compared to that achieved on EBM-COMET. This was attributed to the annotation inconsistencies in the original EBM-NLP, some of which were resolved in [?]. Another aspect we closely observed was the runtime. Using a TITAN RTX 24GB GPU, the average runtime for the fine-tuning experiments on EBM-COMET and EBM-NLP${}_{\textbf{rev}}$ respectively was 7 and 12 hrs. On the other-hand, feature extraction (ODP-tagger) experiments were much longer consuming 20 and 36 hours respectively on the same datasets. Overall, we recommend fine-tuning as a preferred approach for outcome detection, more saw using BioBERT and SciBERT as ideal embedding models.

Motivated by the need to detect accurate fine-grained information in the medical domain [?], we examine the extent to which our models detect precise mentions of full outcome phrases. To achieve this, we investigate how well the best performing models (Fine-tuned+BioBERT+EBM-COMET and Fine-tuned+BioBERT+EBM-NLP${}_{\textbf{rev}}$ from Table 3) can detect full mentions of outcome phrases or otherwise exact matches of outcome phrases in prediction results. We use a strict criteria to evaluate full mention of outcomes, where a classification error FN (False Negative) accounts for the number of full outcome phrases the model fails to detect, which includes partially correctly detected phrases i.e. some of their tokens were misclassified. In table 6, we observe the F1 of the best models drop from 53.1 to 52.4 for EBM-NLP${}_{\textbf{rev}}$ and 81.5 to 69.6 for EBM-COMET. This implies that the model struggles to identify full outcome phrases, especially with the EBM-NLP${}_{\textbf{rev}}$ dataset. Specificity on the other hand is very high for both datasets simply because it is calculated as a True Negative Rate (TNR), in which case True Negatives (non-outcomes) are certainly so many because they are precisely individual words and therefore are counted word by word as opposed to True positives (actual outcome phrases) that can consist of multiple words.

We further investigate the errors from the best performing models BioBERT+EBM-COMET (Fine-tuned) and ODP-tagger+SciBERT+EBM-COMET. In table 5, we show examples of outputs of both models for the ODP task given an input sentence with known actual outcome phrases (underlined). Fine-tuned model correctly detects (blue-coded) all full outcome phrase in the first example sentence i.e. Precision (P), Recall/Sensitivity (R) are 100%, whereas tagger only detects 3/4 outcomes, hence P is 100%, R is 75%. Neither of the models correctly capture full mention of the outcome phrase in the second example, they incorrectly predict some words (red-coded) to not belong to the outcome phrase. While traditionally, results of fine-tuned model would be a P of 100% and R of 50% for correct prediction of 2/4 tokens, in our strict full name evaluation, P and R are 0%, because some tokens in the full outcome phrase are mis-classified in both models i.e. True positives = 0. Similarly, in the third example, fine-tuned model achieves P of 100% and R of 60% for correct prediction of 3/5 tokens in the traditional evaluation, whereas for the strict full name evaluation, R is 50% because only 1/2 full outcome phrases are detected. We attribute these errors to the length of some outcome phrases with some containing extremely common words such as prepositions (“of”). Additionally, we note that the contiguous outcome span annotations (containing several outcomes sharing terms e.g. “right heart size and function” in the third example) are rare.

We additionally fine-tune our best model for the task of detection of all PIO elements in the original EBM-NLP dataset. To be consistent with the original EBM-NLP paper, we consider the token-level detection of the PIO elements task in their work, comparing their evaluation results for hierarchical labels with those we obtain by fine-tuning our best model. Using their published training (4670) and test (190) sets of the starting spans, we see fine-tuned BioBERT model outperform the current leader board results ⁴⁴4https://ebm-nlp.herokuapp.com/ and the SOTA results published by Brockmeier et al [?] (table 7). We attribute this improvement to the fact, unlike the LSTM-CRF and Logreg models in previous SOTA scores, BioBERT’s has an internal capability to encode information using self-attention mechanisms to generate context-sensitive representations of words.

To further understand our results, we investigated how well the best models BioBERT+EBM-COMET (Fine-tuned) and ODP-tagger+SciBERT+EBM-COMET (Feature-extraction) detected outcome phrases of varying lengths. We calculate a prediction accuracy as number of correctly predicted outcome-phrases of length x/number of all outcome-phrases of length x, where x ranged from 1-10. As observed in figure 3, the fine-tuned model slightly outperforms the ODP-tagger especially for outcome phrases having 3-6 words (i.e. 3-6 entity span length). However, it is also clear that both models struggled to accurately detect outcome phrases containing 7 or more words.