Participation in TREC 2020 COVID Track Using Continuous Active Learning

CAL is a method for finding virtually all relevant information on a particular subject within a vast sea of electronically stored information (ESI): it repeatedly refines its understanding about which of the remaining documents are most likely to be of interest, based on the users’ feedback regarding the documents already judged [4]. This protocol is most famously used in technology-assisted review (TAR) for electronic discovery in legal matters, achieving the best results reported in scientific literature to date [2]. Building on the CAL protocol, many implementations, such as BMI, have been highly successful at performing ad-hoc retrieval tasks, such as in the TREC 2015/2016 Total Recall Track [7, 5] and the TREC 2019 Decision Track [1].

BMI is an augmented version of CAL. It is autonomous and was initially made available to participants of the TREC 2015/2016 Total Recall Tracks [7, 5], as well as the TREC 2019 Decision Track [1] to provide a baseline for comparison. However, BMI turned out to be highly competitive, with none of the manual participants achieving consistently superior results to this fully automated method [6].

While BMI has been shown to generally outperform human-in-the-loop CAL implementations [6], it requires labelled data, which was very limited, if available at all, for TREC-COVID; thus, we chose to insert a human back into the loop to make judgements. All other components, such as creating feature vectors, the learner, etc., were taken directly from the BMI tool kits.

The document set used in the TREC-COVID Challenge is the COVID-19 Open Research Data-set (CORD-19). Our team opted to judge a document’s relevancy using strictly the information available in the metadata file (year, authors, publisher, title, abstract) based on the work of Zhang et al. [9] which show that participants achieve higher recall using CAL when presented with only a single short excerpt rather than an entire document.

The following shows an outline of our specific implementation of CAL.

• •

  STEP 1: Create a hypothetical relevant document, known as a synthetic document.
  To create the synthetic documents, we concatenated the query, question, and narrative components of the topics file provided by TREC-COVID, as shown in Figures 2 and 2.

  

  

• •

  STEP 2: Use a machine-learning algorithm to suggest the next most-likely relevant document.
  The machine-learning algorithm we chose is Sofia-ML which Roegiest and Cormack used in their participation in the 2015 Total Recall Track [6].

• •

  STEP 3: Review the suggested documents and provide relevance feedback to the learning algorithm, indicating whether each suggested document is actually relevant or not.
  To do this, we sorted the results given by Sofia-ML in decreasing order of confidence, presenting the top most result to the human assessor using a text based user interface. The judgement made by our human assessors is one of {0-not relevant, 1-partially relevant, 2-relevant}. This corresponds to the annotations made by biomedical experts as part of TREC-COVID following each round. As Sofia-ML does not distinguish between relevant judgements and partially relevant judgments, both were designated to be relevant in training.

• •

  STEP 4: Repeat Step 2 and 3 until very few, if any, of the suggested documents are relevant.
  Using the same stopping condition as in [5], we aimed to stop when the following criterion was met:

  $$n\geq a*m+b$$

  where m is the number of relevant documents reviewed, n is the number of irrelevant documents reviewed, a is a constant which determines how many non-relevant documents are to be reviewed in the course of finding each relevant document, and b is a constant which represents a fixed overhead for the number of irrelevant documents that must be reviewed.

One of the major drawbacks of the CAL method outlined above is the impractical number of documents that must be reviewed when the number of relevant documents is large. Scalable Continuous Active Learning (S-CAL) [3] addresses this issue by

1. 1.

   Segmenting the corpus into batches and allowing assessors to label only a small finite sample of documents from each successive batch.

2. 2.

   Temporarily augmenting each training set by adding a set of 100 random documents from the corpus - which is, with high probability, not relevant for a large corpus - labelled not relevant.

However, the stopping condition for S-CAL outlined in [3] is still infeasible to achieve with CORD-19 and our team size; thus, we exchange the initial dynamic stopping condition for a static goal of assessing 300 documents per topic.

Given the availability of labelled data after the first round, we performed hyper-parameter tuning on both the loop_type and the lambda value to better fit CORD-19. Finding no significant differences in our tests, we decided to continue with our initial values taken from [6], which were decided upon discussion with the author of Sofia-ML as well as their internal experiments.

To generate the results for our runs, we created lists of 1000 documents ordered as shown in Figure 3.

Key-term highlighting is a feature commonly provided by IR systems, such as Google, to assist human readers in processing information. Following the online sample of CAL, as show in Figure 5, given as a supplement to [4], we chose to highlight the top five highest-scoring words from a document, according to Sofia-ML, in our UI for assessors, as show in Figure 5.