CliniQG4QA: Generating Diverse Questions for Domain Adaptation of Clinical Question Answering

We first give an overview of our CliniQG4QA framework (Fig 2). CliniQG4QA improves clinical QA on new contexts by automatically synthesizing QA pairs for new clinical contexts. To approach this, we first leverage an answer evidence extractor to extract meaningful text spans from unlabeled documents, based on which a QG model can be applied to generate questions.

In order to encourage diverse questions, we reformulate the question generation process as two-stage. In the first stage, we propose a question phrase prediction module to predict a set of question phrases, which represent the types of questions humans would ask, given an answer evidence. In the second stage, following a specific question phrase predicted by our QPP, a QG model is used to complete the rest of the question.

Therefore, our framework CliniQG4QA is able to produce questions of more diverse types. The generated QA pairs by QG models are finally used to train QA models on new contexts.

When human annotators create questions, they first read a document and then select a text span to ask questions about. To imitate this process, we implement an answer evidence extractor to extract possible text spans from a document. Following [3, 8], we focus on longer text spans (as answer evidences) instead of short answers (e.g., a single named entity), since longer text spans often contain richer information compared with short ones, which are very important for the clinical QA task.

More formally, given a document (context) $\mathbf{p}=\{p_{1},p_{2},...,p_{m}\}$, where $p_{i}$ is the $i$-th token of the document and $m$ is the total number of tokens, we aim to extract potential evidence sequences. Since the answer evidence is not always a single sentence (sometimes could be multiple sentences), instead of treating it as a sentence selection task, we formulate it as a sequence labeling (or tagging) task. We follow the BIO tagging (short for beginning, inside, outside), a commonly used sequence labeling scheme [30], to label answer evidences.

Firstly, we adopt the ClinicalBERT model [31] to encode the document:

where $\mathbf{U}\in\mathbb{R}^{m\times d}$, and $d$ is size of the dimension.

Following the same paradigm of the BERT model for the sequence labeling task [7], we use a linear layer on top of the hidden states output by ClinicalBERT followed by a softmax function to do the classification:

where $a_{j}$ is the predicted BIO tag.

After prediction, we observe that the extracted answer evidences sometimes are broken sentences due to the noisy nature and uninformative language (e.g., acronyms) of clinical texts. To make sure that the extracted evidences are meaningful, we designed a “merge-and-drop” heuristic rule to further improve the extractor’s accuracy. Specifically, for each extracted evidence candidate, we first examine the length (number of tokens) of the extracted evidence. If the length is larger than the threshold $\eta$, we keep this evidence; otherwise, we compute the distance, i.e., the number of tokens between the current candidate span and another closest candidate span If the distance is smaller than the threshold $\gamma$, we merge these two “close-sitting” spans; otherwise, we drop this overly-short evidence span. In our experiments, we set $\eta$ and $\gamma$ to be 3 and 3, respectively, since they help achieve the best performance on the dev set.

Existing QG models are often biased to generate limited types of questions. To address this problem, we introduce our question phrase prediction module that can be used to diversify the generation of existing QG models.

Formally, denote $V_{l}=\{s_{1},...,s_{L}\}$ as the vocabulary of all available question phrases of length $l$ in the training data and $L=|V_{l}|$ as its size. $V_{l}$ can be obtained by collecting the first $n$-gram words in the questions. We set $n=2$ in our experiment as it achieves the best performance on the dev set. Given an answer evidence $\mathbf{a}$, the goal of QPP is to map $\mathbf{a}\rightarrow\mathbf{y}=(y_{1},...,y_{L})\in\{0,1\}^{L}$, where $y_{i}=1$ indicates predicting $s_{i}$ in $V_{l}$ as a question phrase for the evidence $\mathbf{a}$. Instead of treating it as a common multi-label classification problem, we formulate the task as a sequence prediction problem and adopt a commonly used seq2seq model with an attention mechanism [32] to predict a sequence of question phrases $\mathbf{s}=(s_{j_{1}},...,{s_{j_{|\mathbf{s}|}}})$ (e.g., “What treatment” ($s_{j_{1}}$) $\rightarrow$ “How often” ($s_{j_{2}}$) $\rightarrow$ “What dosage” ($s_{j_{3}}$), with $|\mathbf{s}|=3$).

During training, we assume that the set of question phrases is arranged in a pre-defined order. Such orderings can be obtained with some heuristic methods, e.g., using a descending order based on question phrase frequency in the corpus³³3In our dataset, each answer evidence is tied with multiple questions, which allows the training for QPP.. In the inference stage, QPP can dynamically decide the number of question phrases for each answer evidence by predicting a special [STOP] token. By decomposing QG into two steps (diversification followed by generation), the implemented QPP can increase the diversity in a more controllable way.

Algorithm 1 illustrates the pretraining and training procedure of our CliniQG4QA.

During the pretraining stage, we first train the answer evidence extractor (AEE) module on the source contexts by minimizing the negative log-likelihood loss:

where $\phi$ represents all the parameters of the answer evidence extractor. For the supervision signals, we identify all evidences in the source data as ground-truth chunks which are marked using the BIO scheme.

Moving to the Question Phrase Prediction (QPP) module, given an answer evidence $\mathbf{a}$, we aim to predict a question phrase sequence $\mathbf{y}$ and minimize:

where $\theta$ denotes all the parameters of QPP.

Then we can train any QG model (e.g, NQG [10]) on source data by minimizing:

where $\mu$ denotes all parameters of the QG model.

During the training stage, given unlabeled target clinical documents, we first extract answer evidences, based on which QPP can be “plugged” into the QG model to generate diverse questions. Finally, a QA model (e.g., DocReader [6]) can be trained on the generated QA pairs of the target documents:

where $\delta$ denotes all parameters of the QA model.

We instantiate our CliniQG4QA framework using three base QG models:

It has been studied that beam search and sampling strategies show competitive performance in diversifying generations [36, 37]. We thus include Top-k [38] and Nucleus samplings [39] as representative sampling strategies in our experiments.

As such, to investigate the effectiveness of diverse QG for QA, we consider the following variants of each base QG model: (1) Base Model: Inference with greedy search; (2) Base Model + Beam Search: Inference with Beam Search of beam size $K$ and keep top $K$ beams ($K=3$); (3) Base Model + Top-k sampling: Inference with sampling from top-k tokens ($k=20$); (4) Base Model + Nucleus sampling: Inference with sampling from top-p tokens ($p=0.95$); (5) Base Model + QPP: Inference with greedy search for both QPP module and Base model.

For QA, we instantiate CliniQG4QA with two base models, DocReader [6] and ClinicalBERT [31]. When training a QA model, we only use the synthetic data on the target contexts and do not combine the synthetic data with the source data since the combination does not help in our preliminary studies.

Note that more complex QG/QA models and training strategies can also be used in our framework. As this work focuses on exploring how diverse questions help QA on target contexts, we adopt fundamental QG/QA models and training strategies, and leave more advanced ones that are complementary to our framework as future work.

For QA evaluation, we report exact match (EM) (percentage of predictions that match the ground truth answers exactly) and F1 (average overlap between the predictions and ground truth answers) as in [40]. Since our main goal is to evaluate whether the generated questions are useful to improve the QA performance on the target contexts, the common language generation metrics such as BLEU [41] and ROUGE-L [42] are not suitable to reflect the quality of the generated questions, and thus we do not adopt these metrics in our experiments.

Table II summarizes the performance of two widely used QA models, DocReader [6] and ClinicalBERT [31], on the MIMIC-III testing set. The QA models are trained based on different corpora, including the emrQA dataset as well as QA pairs generated by different models. For a fair comparison, we keep the total number of generated QA pairs roughly the same as emrQA. As can be seen from the table, the QA models based on the corpora that are generated using the three base QG models can only achieve roughly the same or even worse performance compared with the QA models trained on the emrQA dataset. Though the Beam Search and sampling strategies could boost the diversity of generated questions to some extent, and thus lead to the improvement of QA models, our proposed QPP module can improve the QA performance by a larger margin. For example, training DocReader using questions generated by NQG++ with our QPP module outperforms that using the emrQA dataset by around 8% under EM and 4% under F1 on the overall test set. Moreover, the results on human-generated portion are consistently better than that on human-verified. It’s attributed to the fact that human-created questions are more readable and sensible while human-verified questions are a bit of less natural though correctness is ensured.

All these results indicate that generating a diverse QA corpus is useful for downstream QA on new contexts, and our simple QPP module can help existing QG models achieve such a goal.

To further explore why QG can boost QA, we consider three major factors when generating a QA corpus: the number of documents, the number of answer evidences per document, and the number of generated questions per answer evidence. When we test one factor, we fix the other two. For example, we fix the number of answer evidences and questions at 20 and 6 when we test the influence of the number of documents. We use NQG++ and DocReader as our base QG and QA models to instantiate our CliniQG4QA framework and report the performance on the Dev set.

As can seen from Fig 4, the performance steadily increases when we use more documents and more answer evidences during QA corpus generation. This can demonstrate the first hypothesis: The generated corpus enables a QA model to see more new contexts during training, which can help the QA model get a better understanding of similar contexts during testing. The more contexts it sees, the more benefits it could obtain. We can also see that with the increase of the number of generated questions per evidence, the performance generally rises up. This indicates that multiple diverse questions are essential for boosting QA performance.

We observe that questions generated by base NQG and NQG+BeamSearch are limited in terms of the question types. However, more types of questions (e.g., “How”, “Why”) can be generated when enabling sampling strategies. Furthermore, when being equipped with our QPP module, the NQG model can even generate questions of an extremely rare type, i.e., ”When” questions. Though Top-k and Nucleus sampling methods also generate questions of less frequent types, our QPP module could cover even more types.

In summary, we think seeing many new contexts and diverse questions are the two main reasons why QA models are boosted.

In Fig 5, we present a QA example and a QG example from MIMIC-III for qualitative analysis.

In the QA example, this “why” question can be correctly answered by the QA model (DocReader) trained on the “NQG+QPP” generated corpus while the QA models trained on other generated corpora fail. This is because, as shown in Fig 3, the NQG model and “NQG+BeamSearch” cannot generate any “why” questions and sampling strategies could only help generate a limited number of “why” questions. Thus QA models trained on such corpora cannot answer questions of less frequent types. Though the emrQA dataset contains diverse questions (including “why” questions), its contexts might be different from MIMIC-III in terms of topic, note structures, writing styles, etc. So the model trained on emrQA struggles to answer some questions as well.

In the QG example, the base model NQG can only generate one question. Though utilizing the Beam Search enables the model to explore multiple candidates, the generated questions are quite similar and are less likely to help improve QA. Sampling strategies, though further diversifying the generation during decoding, suffer from generating irrelevant contents (e.g., “NQG+Nucleus” generates a irrelevant “morphine” token). Enabling our QPP module helps generate relevant and diverse questions including “Why”, “What”, “How”, etc.

From Table IV, we can see: (1) The seq2seq design in our QPP module performs better overall and especially in terms of Recall, which is particularly important since we aim for generating diverse question types; (2) A simple seq2seq model achieves great performance across all metrics, which renders developing more complex models for this task less necessary.