Domain Adaptation for Question Answering via Question Classification

In the proposed QC4QA, we design a self-supervised framework that facilitates question classification for QA domain adaptation. QC4QA can be divided into three stages: (1) Question classification; (2) Pseudo labeling & sampling and (3) Self-supervised adaptation. In the first stage, we perform classification for all input questions, which provides additional question class information for the adaptation stage. In the next stage, we label and filter all target samples and perform distribution-aware sampling to build mini-batches that resemble the target data distribution. Finally, we perform self-supervised adaptation on the QA system to minimize inter-domain and intra-class discrepancies. Once input questions are classified, we iteratively perform stage 2 and stage 3 in each epoch. The QA system is trained with both source and target data, where we encourage domain-invariant features and minimize intra-class discrepancies of data samples from the same question class.

Our approach leverages question classification for fine-grained domain adaptation. Here, QC is designed for evaluating intra-class discrepancies and distributional changes by introducing the additional question classes instead of using QA labels. The idea behind it is that QA labels are defined by subspans in input contexts, if we treat every combination of start and end position as a class, the corresponding label space would be too large and sparse for any meaningful discrepancy estimation. Therefore, we proposes the question classification stage to introduce additional semantic knowledge for intra-class discrepancy estimation. Moreover, by performing pseudo labeling and distribution-aware sampling, we resemble the target question distribution in the adaptation stage to correct the potential bias in the pretrained QA system. In other words, QC4QA simulates the target data distribution and ‘pulls together’ source and target samples of the same question class to encourage domain invariance.

For question classification, we adopt the commonly used question taxonomy in TREC and categorize all questions into 6 coarse classes $Q$: ABBR: Abbreviation, DESC: Description, ENTY: Entity, HUM: Human, LOC: Location and NUM: Numeric Value. Each class indicates the potential answer type to the question Li and Roth (2002). In practice, we rarely find ABBR questions.

The proposed QC model leverages pretrained sentence embedding methods to generate vectorized feature for input questions. We then build a multilayer perceptron (MLP) to perform classification on the encoded questions, see Figure 2. Specifically, we adopt InferSent and Universal Sentence Encoder to encode the input questions separately Conneau et al. (2017); Cer et al. (2018). The encodings are concatenated and used as an input feature for the MLP classifier. With the trained QC model, inference can be performed on all training questions for the later adaptation stage.

To further examine the effectiveness of question classification without additional model and annotation, we introduce an unsupervised clustering method, where we refrain from using an additional dataset or classifier to perform question classification. In particular, we feed the input data within the transformer encoder (part of the QA system) and utilize the output from the [CLS] token position as features Devlin et al. (2019). We sample a fixed number of source features (10k in our experiments) and perform KMeans clustering with a predefined number of clusters $k$ (Similar to TREC, we use 5 as default). Then, cluster centroids are preserved to classify source and target QA datasets.

Provided with the access to labeled source data, we pretrain the QA system $\bm{f}$ to learn to answer questions. After pretraining, we can use $\bm{f}$ to predict target answers for self-supervised adaptation. The pseudo labels are filtered according to the answer confidence, we preserve the target samples above confidence threshold $\lambda_{con}$. The pseudo labeling and confidence thresholding steps are repeated in each epoch to dynamically adjust the target distribution used for training.

For mini-batch training, we sample the same amounts of QA pairs from both domains to minimize the inter-domain and intra-class discrepancies. However, with randomly sampled data, training is less efficient as source and target questions in each batch can be entirely different (e.g., source samples are all Human questions and target samples are all Description questions). To solve this problem, we design a distribution-aware sampling strategy: we first sample target QA pairs from $\bm{X}_{t}$ and within the same question classes, we sample from $\bm{X}_{s}$ such that the source and target question classes in each batch are identical. Consequently, the QA system can be trained on a data distribution similar to the target dataset. Moreover, the estimation of intra-class discrepancy between both domains can be performed more efficiently.

The sampled batches are used to adapt the pretrained QA system, where we optimize the model to reduce the negative log likelihood loss, as in Equation 1. Meanwhile, we encourage the domain invariance by computing the discrepancies and incorporate them in the training objective.

To measure the discrepancy between samples from different domains, we adopt the maximum mean discrepancy (MMD) distance Gretton et al. (2012). MMD estimates the distance between two distributions with samples drawn from them, with $f$ and $\mathcal{H}$ representing the feature mapping and the reproducing kernel Hilbert space:

To simplify the computation, we adopt the Gaussian kernel as feature mapping, i.e., $k(\bm{x}_{s}^{(i)},\bm{x}_{t}^{(j)})=\mathrm{exp}(-\frac{\|\bm{x}_{s}^{(i)}-\bm{x}_{t}^{(j)}\|^{2}}{\gamma})$. We further leverage the kernel trick and empirical kernel mean embeddings Long et al. (2015) to estimate the squared MMD distance between samples from $\bm{X}_{s}$ and $\bm{X}_{t}$:

where $\phi$ represents the transformer encoder in the QA system. With $\mathcal{D}^{\mathrm{MMD}}$, it is possible to measure the discrepancies between different domains and question classes. The discrepancy values are used to guide the self-supervised adaptation and encourage domain-invariant features.

Among all tokens in each QA sample $\bm{x}$, we distinguish two types of features $\bm{x}_{\mathrm{a}}$ and $\bm{x}_{\mathrm{o}}$. $\bm{x}_{\mathrm{a}}$ stands for the mean vector of answer token representations, while $\bm{x}_{\mathrm{o}}$ is the mean vector of all other tokens in the representation space Yue et al. (2021). The QA system extracts the answer when the answer tokens in the representation space are separated from $\bm{x}_{\mathrm{o}}$ van Aken et al. (2019). Therefore, we adopt $\bm{x}_{\mathrm{a}}$ as feature representation to compute MMD distances. By introducing question classification, we introduce an additional term w.r.t. the intra-class discrepancy in the objective function:

$\bm{X}_{s}$ refers to all answer features in source samples and $\bm{X}_{t}$ represents answer features in target samples, while $\bm{X}^{(q)}$ denotes the set of samples that belong to question class $q\in Q$. $\mathcal{L}_{\text{{QC4QA}}}$ ‘pulls together’ features from the same question class across domains by minimizing their MMD distances.

To encourage domain-invariant features, we incorporate Equation 4 into the training objective. Using the NLL loss and the contrastive adaptaion loss Yue et al. (2021, 2022c), the overall objective function can be formulated as follows:

in which $\mathcal{L}_{\mathrm{CAQA}}$ is the same as in Yue et al. (2021), while $\lambda$ is a scaling factor we choose empirically. Although we introduce $\mathcal{L}_{\mathrm{CAQA}}$ in our training objective function, QC4QA is largely different from CAQA as: (1) $\mathcal{L}_{\mathrm{CAQA}}$ only reduces the inter-domain discrepancy, we incorporate question classification to additionally reduce intra-class discrepancy via $\mathcal{L}_{\mathrm{{QC4QA}}}$ for fine-grained adaptation; (2) We perform pseudo labeling and distribution-aware sampling to account for the distribution shifts between the source and target dataset; and (3) QC4QA leverages an efficient self-supervised adaptation framework instead of the computationally expensive question generation in Yue et al. (2021). As such, the proposed QC4QA efficiently reduces the domain discrepancy and effectively transfers learnt knowledge from the source domain to the target domain.

The overall framework is illustrated in Figure 3. We first generate question classes for all data samples. Next, the source-pretrained QA model generates pseudo labels for target data and we select target samples above the confidence threshold $\lambda_{con}$ for training. Pseudo labeling and self-supervised adaptation are performed iteratively to refine the pseudo labels and improve the performance in the target domain using Equation 5. Unlike previous works Lee et al. (2019); Cao et al. (2020); Shakeri et al. (2020); Yue et al. (2021), we discard domain-adversarial training or question generation and introduce a self-supervised adaptation framework based on question classification for improved efficiency and adaptation performance. We also design a distribution-aware sampling strategy to resemble the target data distribution and correct the potential bias in the pretrained QA system. Additionally, a fine-grained adaptation loss based on question classification is introduced in training to minimize both the inter-domain and intra-class discrepancies across the source and target domain.

We first study the benefits of question classification. The performance gains can be achieved by comparing the results between CAQA^* and QC4QA in Table 1 and Table 2. This is because CAQA^* is adapted to the same self-supervised adaptation framework as in QC4QA. CAQA^* models are trained without minimizing the intra-class discrepancies in Equation 5. The improvements from QC4QA suggest that both supervised and unsupervised question classification can consistently improve QA systems in answering questions from unseen domains. Moreover, QC4QA is particularly effective on target datasets with different question formats (e.g., CNN and Daily Mail).

To study the influence of distribution-aware sampling in QC4QA, we replace the distribution-aware sampling strategy with random sampling. Then we perform unsupervised adaptation with QC4QA TREC on CNN, Daily Mail and NewsQA to verify the merits of the sampling strategy. Results are presented in Table 3.

In all target datasets, we see performance drops when we replace the proposed strategy with random sampling. In particular, we find relatively large performance deterioration on CNN without distribution-aware sampling. We believe the reason is that the question distribution in CNN is less similar to SQuAD (see Table 5), the resulting inconsistency in sampled batches reduces the effectiveness in discrepancy estimation.

Now we evaluate the influence of $\lambda$ to study the robustness of the proposed objective function. We select different values ranging from 0 to 5e-2 and perform adaptation with QC4QA TREC. Experiments on CNN, Daily Mail and NewsQA are presented to estimate the influence of $\lambda$.

Figure 4 visualizes EM / F1 with increasing $\lambda$. Despite certain variations, we observe the results first go up and then decrease. Although CNN and Daily are more sensitive to $\lambda$, we observe greater improvements by reducing inter-domain and intra-class discrepancies. Overall, QC4QA consistently improves adaptation performance.

We study the influence of $\lambda_{con}$ to understand how the performance varies with different confidence thresholds in pseudo labeling. We select different threshold values ranging from 0.2 to 0.8 to filter pseudo labels and train with QC4QA TREC. Experiments are performed on HotpotQA and SearchQA to estimate the influence of $\lambda_{con}$.

Table 4 shows adaptation performance with different $\lambda_{con}$. The best performance can be reached with $\lambda_{con}$ ranging from 0.4 to 0.8. For large datasets like SearchQA (with over 100k QA pairs), a higher confidence threshold yields a better adaptation performance since we avoid noisy pseudo labels. In sum, carefully selected $\lambda_{con}$ yields comparatively large improvements for QC4QA.