Contrastive Domain Adaptation for Early Misinformation Detection: A Case Study on COVID-19

Existing methods in misinformation detection can be mainly divided into three coarse categories: (1) content-based misinformation detection: content-based models perform classification upon statements (i.e., claim) or multimodal input. For example, language models are used for misinformation detection based on linguistic features or structure-related properties (Wang, 2017b; Karimi and Tang, 2019; Das et al., 2021). Multimodal architectures are introduced to learn features from both textual and visual information for improved performance (Jin et al., 2017; Wang et al., 2018; Khattar et al., 2019); (2) social context-aware misinformation detection: Jin et al. (2016) leverage online interactions to evaluate the credibility of post contents. The propagation paths in the spreading stage can be used to detect fake news on social media platforms (Liu and Wu, 2018). Other context like user dynamics and publishers are introduced to enhance social context-aware misinformation detection (Guo et al., 2018; Shu et al., 2019b); (3) knowledge guided misinformation detection: Vo and Lee (2020) and Popat et al. (2018) propose to search for external evidence to derive informative features for content verification. Multiple evidence pieces are used to build graphs for fine-grained fact verification (Liu et al., 2020). Knowledge graphs introduce additional information to provide useful explanations for the detection results (Cui et al., 2020; Kou et al., 2022a, b). Nevertheless, the generalization and adaptability of such misinformation detection systems are not well studied. Therefore, we focus on domain adaptation of content-based language models for misinformation detection.

Domain adaptation methods are primarily studied on the image classification task (Long et al., 2013; Tzeng et al., 2014; Long et al., 2015, 2017; Kang et al., 2019). Existing approaches minimize the representation discrepancy between the source and target domains to encourage domain-invariant features, and thereby improving the model performance on shifted input distributions (i.e., covariate shifts) (Long et al., 2013; Tzeng et al., 2014). Early domain adaptation methods utilize an additional distance term in the training objective to minimize the distance between the input marginal distributions (Tzeng et al., 2014; Long et al., 2015). Long et al. (2013) and Long et al. (2017) adapt source-trained models by optimizing the distance between the joint distributions across both domains. Similarly, a feature discriminator can be introduced to perform minimax optimization, where the regularization on domain generalization can be implicitly imposed when source and target features are distinguishable by the discriminator (Ganin et al., 2016; Tzeng et al., 2017). Class-aware and contrastive domain adaptation are proposed for fine-grained domain alignment, such methods adapt the conditional distributions by regularizing the inter-class and intra-class distances via an additional domain critic (Pei et al., 2018; Kang et al., 2019). Recently, calibration and label shifts are studied for domain adaptation problems when the label distribution changes between the domains (Guo et al., 2017; Lipton et al., 2018; Azizzadenesheli et al., 2019; Alexandari et al., 2020). To the best of our knowledge, domain adaptation that considers both label correction and conditional distribution adaptation is not studied in current literature. Yet label shift is often observed in real-world scenarios (e.g., misinformation detection). As such, we propose an adaptation method that corrects the label shift and optimizes the conditional distribution to improve the out-of-domain performance.

Currently, few methods are tailored for domain adaptation in misinformation detection, therefore, we review the related work in both text classification and misinformation detection. A domain-distinguishing task is proposed to post-train models for domain awareness and improved adversarial adaptation performance (Du et al., 2020). Energy-based adaptation is proposed via an additional antoencoder that generates source-like target features (Zou et al., 2021). Xu et al. (2019), Zhang et al. (2020) and Zhou et al. (2022) utilize domain adversarial training and multi-task learning to promote domain-invariant features in multimodal misinformation detection. Pseudo labeling is used to integrate domain knowledge via a weak labeling function for multi-source domain adaptation (Li et al., 2021). However, domain adaptation is not well studied for content-based misinformation detection, let alone the adaptation for COVID-related early misinformation. We develop a domain adaptation method tailored for misinformation detection CANMD. Our method significantly improves the adaptation performance in COVID misinformation detection by: (1) correcting the label shift between the source and target data distributions; and (2) exploiting knowledge from the source domain by minimizing the distance between the source and target conditional distributions.

Data: Given labeled source data and unlabeled target data, our setting aims at improving the misinformation detection performance in an unseen target domain. We define misinformation detection as a binary text classification task. For this purpose, labeled source data and unlabeled target data are available, we denote the source data with $\bm{\mathcal{X}}_{s}$ and the target data with $\bm{\mathcal{X}}_{t}$. Formally, each input example is defined by a tuple consisting of an input text $\bm{x}$ and a label $y$ with $y\in\{0,1\}$. Input $\bm{x}$ is considered as misinformation (i.e., $y=0$) if it contains primarily or entirely false information. Otherwise, the label is considered as non-misleading (i.e., $y=1$):

• •

  Source data: Labeled source data from $\bm{\mathcal{X}}_{s}$. Each example $(\bm{x}_{s},y_{s})\in\bm{\mathcal{X}}_{s}$ is defined by an input text $\bm{x}_{s}$ and a label $y_{s}$. We denote the joint distribution $(\bm{x}_{s},y_{s})$ in $\bm{\mathcal{X}}_{s}$ with $\mathcal{P}$.

• •

  Target data: Unlabeled target data from $\bm{\mathcal{X}}_{t}$. For target example $(\bm{x}_{t},y_{t})\in\bm{\mathcal{X}}_{t}$, we only have access to the input text $\bm{x}_{t}$. Ground truth label $y_{t}$ is not given for training. Similarly, we denote the joint distribution in $\bm{\mathcal{X}}_{t}$ with $\mathcal{Q}$.

Model: The classification model can be represented with function $\bm{f}$. $\bm{f}$ takes text $\bm{x}$ as input and yields output logits. The output probability distribution is obtained by using the softmax function $\sigma$. Ideally, the model $\bm{f}$ can correctly predict the ground truth $y$ with the maximum logit value, namely $y=\arg\max\bm{f}(\bm{x})$.

Objective: The objective is to adapt a classifier $\bm{f}$ trained on source distribution $\mathcal{P}$ to the target distribution $\mathcal{Q}$, such that the performance can be maximized in the target data $\bm{\mathcal{X}}_{t}$. In other words, we minimize the negative log likelihood (NLL) loss between model output distribution $\sigma(\bm{f}(\bm{x}_{t}))$ and ground truth $y_{t}$ for $\bm{\mathcal{X}}_{t}$:

To adapt the trained classifier to the target domain, we propose contrastive adaptation network for early misinformation detection (CANMD) by approximating the target joint distribution. Unlike previous works with the assumption of covariate shift (Long et al., 2013; Tzeng et al., 2014; Long et al., 2015; Ganin et al., 2016), we consider label shift (i.e., $p(y)\neq q(y)$) and conditional shift (i.e., $p(\bm{x}|y)\neq q(\bm{x}|y)$) between the source domain and target domain. Since $p(\bm{x},y)=p(\bm{x}|y)p(y)$ (Similarly, $q(\bm{x},y)=q(\bm{x}|y)q(y)$), our solution to adapt the misinformation detection model $\bm{f}$ to the target domain is two-fold: (1) label shift estimation and correction: we generate pseudo labels for target examples in $\bm{\mathcal{X}}_{t}$ and rescale the labels to correct the shift in the joint distribution caused by a shift in label proportion (Lipton et al., 2018; Azizzadenesheli et al., 2019; Alexandari et al., 2020). In other words, we estimate and correct $q(y)$. (2) Conditional distribution adaptation: using the pseudo labels generated from the previous step, we minimize the distance between the conditional distributions $p(\bm{x}|y)$ and $q(\bm{x}|y)$ with contrastive domain adaptation. As we reduce the distance between both conditional distributions (i.e, $p(\bm{x}|y)\approx q(\bm{x}|y)$), we improved the estimation of $q(\bm{x},y)=p(\bm{x}|y)q(y)$ using the corrected $q(y)$.

The reason for label shift correction lies in the rapid changing dynamics of COVID-19 misinformation. For instance, the proportion of misinformation on social media platforms changes rapidly in different stages (i.e., label shift) (Cui and Lee, 2020; Patwa et al., 2021; Hayawi et al., 2022). Additionally, conventional misinformation detection systems often fail to distinguish COVID misinformation due to large domain discrepancies, as we demonstrate in Section 4. Therefore, the proposed CANMD comprises of two stages: (1) pseudo labeling and label correction, where we perform pseudo labeling and correct the label distribution $q(y)$ in the target domain; and (2) contrastive adaptation, in which we reduce intra-class discrepancy and enlarge inter-class discrepancy among examples from both domains. By leveraging pseudo labels from the first stage, we estimate and minimize the discrepancy between conditional distributions $p(\bm{x}|y)$ and $q(\bm{x}|y)$. The illustration of CANMD is provided in Figure 2, unlike existing methods, we propose to correct the label shift and adapt the conditional distributions instead of solely reducing the feature discrepancy.

Given access to labeled source data $\bm{\mathcal{X}}_{s}$, we first pretrain the misinformation classification system. In practice, we can skip this step if the model is already trained on the source data. Once the source-trained model is ready, we generate the pseudo label $\hat{y}_{t}$ of the unlabeled target example $\bm{x}_{t}$ with $\arg\max\bm{f}(\bm{x}_{t})$.

However, the generated pseudo labels are noisy, which often cause deterioration of domain adaptation performance, especially when there exists a large domain gap (Yue et al., 2021). Additionally, different label proportions between both domains (i.e., label shift) can result in biased pseudo labels and potential negative transfer results (Alexandari et al., 2020). Inspired by calibration methods (Guo et al., 2017), we propose a label correction component to correct the distribution shift between the pseudo labels and the target output distribution. Specifically for input $\bm{x}$, we design vector rescaling to introduce learnable parameter $\bm{w}$ and $\bm{b}$ and compute the corrected output as follows:

where $\odot$ represents element-wise product and $\sigma$ represents the softmax function. In rare cases when the the source and target output distributions result in large bias values in $\bm{b}$, we discard the bias to avoid generating constant output probability (i.e., $\sigma(\bm{w}\odot\bm{f}(\bm{x})+\bm{b})\approx\sigma(\bm{b})$ when $\bm{b}$ consists of values much greater than $\bm{w}\odot\bm{f}(\bm{x})$). Thus the correction method reduces to:

To obtain the optimal $\bm{w}$ and $\bm{b}$, we optimize the parameter $\bm{w}$ and $\bm{b}$ in training to rescale the output probabilities and fit the distribution $q(y)$. Given pretrained misinformation detection function $\bm{f}$ and input pair $(\bm{x}_{t},y_{t})$, we follow (Guo et al., 2017) and minimize the NLL loss w.r.t. the parameters using the gradient descent algorithm, i.e.:

where $\bm{b}$ is discarded in the case of constant output probability or when the optimization fails. Similar to (Guo et al., 2017; Lipton et al., 2018; Alexandari et al., 2020), the validation set is used to optimize the vector rescaling parameters.

In sum, the pseudo labeling and label correction stage via the label correction component can be formulated as:

After label correction, we filter the pseudo labels to select a subset of the target examples for contrastive adaptation. The pseudo labels are filtered according to the label confidence (i.e., $\max\sigma(\bm{w}\odot\bm{f}(\bm{x})+\bm{b})$), we preserve the target sample if the confidence value is above confidence threshold $\tau$. In the following, we denote the filtered target data with pseudo labels using $\bm{\mathcal{X}}^{\prime}_{t}$. By introducing the label correction component, we reduce the bias from $\bm{f}$ when the source and target label distributions shift. For example, COVID misinformation in the CoAID dataset consists of less than $10\%$ false claims (Cui and Lee, 2020). If $\bm{f}$ is pretrained on a balanced dataset, the pseudo labels would demonstrate a similar label distribution regardless of the label distribution in CoAID and result in noisy labels. CANMD rescales the model output using learnable parameters to adjust the pseudo labels, followed by confidence thresholding that selects high-confidence examples from a target-similar distribution.

We now describe the contrastive adaptation stage of CANMD. We perform mini-batch training and sample identical amounts of input pair $(\bm{x},y)$ from both the source data $\bm{\mathcal{X}}_{s}$ and labeled target data $\bm{\mathcal{X}}^{\prime}_{t}$. To approximate the target data distribution and efficiently estimate the discrepancy, we adopt a class-aware sampling strategy to guarantee the same amount of examples within each class. Specifically, a target batch is first sampled from the target data $\bm{\mathcal{X}}^{\prime}_{t}$. By counting the examples in the target batch, we sample from the source data $\bm{\mathcal{X}}_{s}$ with the same number of examples from each class respectively. Consequently, we build mini-batches that comprise of the same classes and identical number of examples in both domains. We also resemble the target data distribution and incorporate the source domain knowledge during adaptation (Kang et al., 2019).

To estimate the domain discrepancy, we revisit the maximum mean discrepancy (MMD) distance. MMD estimates the distance between two distributions using samples drawn from them (Gretton et al., 2012). Given $\mathcal{P}$ and $\mathcal{Q}$, MMD distance $\mathcal{D}$ is defined as:

where $f$ is a function (kernel) in reproducing the kernel Hilbert space $\mathcal{H}$. By introducing the kernel trick and empirical kernel mean embeddings (Long et al., 2015; Yue et al., 2021), we further simplify the estimation of the squared MMD distance between $\bm{\mathcal{X}}_{s}$ and $\bm{\mathcal{X}}^{\prime}_{t}$ as follows:

where we adopt the output from the [CLS] position in the transformer model as $\phi$. Here, the expectation is simplified by using mean embeddings of the drawn samples, $k$ refers to the Gaussian kernel, i.e., $k(\bm{x}_{i},\bm{x}_{j})=\mathrm{exp}(-\frac{\|\bm{x}_{i}-\bm{x}_{j}\|^{2}}{\gamma})$. The estimated MMD distance is used to compute a contrastive adaptation loss. In particular, we leverage the Equation 7 to regularize the distances among examples within the same class and examples from different classes. For this purpose, we define the class-aware MMD as:

with $\mathbbm{1}_{c_{1}c_{2}}(y_{1},y_{2})=\left\{\begin{aligned} 1,&\,\mathrm{if}\,y_{1}=c_{1},y_{2}=c_{2},\\ 0,&\,\mathrm{else}.\end{aligned}\right.$ and $c_{1}$ and $c_{2}$ representing two classes. If $c_{1}$ and $c_{2}$ are the same class, then $\mathcal{D}_{\mathrm{c_{1}c_{2}}}^{\mathrm{MMD}}$ estimates the intra-class discrepancy between the source and target domain. If $c_{1}$ and $c_{2}$ represent two different classes, then $\mathcal{D}_{\mathrm{c_{1}c_{2}}}^{\mathrm{MMD}}$ evaluates the inter-class discrepancy between both domains.

Using the class-aware MMD distance, we define the contrastive loss $\mathcal{L}_{\mathrm{contrasive}}$ as a part of the optimization objective:

As we consider binary classification in misinformation detection, $\mathcal{D}_{00}^{\mathrm{MMD}}$ and $\mathcal{D}_{11}^{\mathrm{MMD}}$ refers to the intra-class discrepancy of misinformation examples and credible examples in the representation space. $\mathcal{D}_{01}^{\mathrm{MMD}}+\mathcal{D}_{10}^{\mathrm{MMD}}$ represents inter-class discrepancy, the distance between different classes is enlarged to avoid misclassification (by taking the negative value). Based on the combination of intra-class and inter-class discrepancy, the misinformation detection system $\bm{f}$ learns a feature representation that separates the input from different classes and confuses input from the same class. In other words, the loss pulls together the same-class samples and pushes apart samples from different classes. As such, we minimize the distance between $p(\bm{x}|y)$ and $q(\bm{x}|y)$ based on the pseudo labels generated in the previous stage, we illustrate the adaptation process in Figure 3. Existing methods (left) that adapts marginal distributions can cause performance drops when there exist label shifts between the source and target domains (see green triangles). As opposed to such methods, CANMD (right) corrects the label shift and samples from the target distribution to adapt the conditional distributions, which yields an improved decision boundary for the target domain.

By combining both the NLL loss for classification and the above contrastive loss, we formulate the overall optimization objective in our contrastive adaptation stage as follows:

where $\mathcal{L}_{\mathrm{nll}}$ incorporates knowledge from both the source and target domains (via pseudo labels), while $\mathcal{L}_{\mathrm{contrasive}}$ adapts the conditional distribution to the target domain. $\lambda$ can be chosen empirically.

The overall framework of CANMD can be found in Figure 2. In the pseudo labeling and label correction stage, pseudo labels are first generated for all target examples using the pretrained misinformation detection system $\bm{f}$. Next, we optimize the learnable correction parameters and correct the generated pseudo labels, we also filter the target data with a minimum confidence threshold $\tau$ (chosen empirically). In the contrastive adaptation stage, we perform class-aware sampling to resemble the target data distribution. Then, we compute a contrastive adaptation loss to reduce the intra-class discrepancy and enlarge the inter-class discrepancy. For each epoch, the optimization process is described in Algorithm 1.

Different from previous domain adaptation work in text classification (Du et al., 2020; Li et al., 2021; Zou et al., 2021), we discard domain-adversarial training for efficient training and approximate the joint distribution of the target domain. In particular, we propose a label correction component to correct the estimation of $q(y)$ during pseudo labeling. This corresponds the correction of pseudo labels in COVID misinformation, where we remove the potential bias in the pretrained model to reduce noisy labels in adaptation. Next, the model is adapted by minimizing the distance between the conditional distributions via MMD distance, in which the model learns domain-confusing yet class-separating features. Put simply, we transfer the knowledge from the source misinformation data to the COVID domain via fine-grained alignment in the feature space. By estimating $q(y)$ and adapting $p(\bm{x}|y)\approx q(\bm{x}|y)$, we improve the estimation of $q(\bm{x},y)$ with $q(\bm{x},y)=p(\bm{x}|y)q(y)$ and thereby improving the adaptation performance for misinformation detection in COVID-19.

Datasets: In our experiments, we adopt five source datasets released before the COVID outbreak and three COVID misinformation datasets to validate the proposed CANMD. For source datasets, we use FEVER (Thorne et al., 2018), GettingReal (Risdal, 2016), GossipCop (Shu et al., 2020), LIAR (Wang, 2017a) and PHEME (Buntain and Golbeck, 2017). For target datasets, we adopt CoAID (Cui and Lee, 2020), Constraint (Patwa et al., 2021) and ANTiVax (Hayawi et al., 2022), which are released in 2020, 2021 and 2022 respectively. Dataset details are provided in Table 1, where Negative and Positive are the proportion of misinformation and valid information in the dataset. Avg. Len represents the average token length of text and Type denotes the source type of the text (e.g., news, claim or social media).

Model: Following (Li et al., 2021; Zou et al., 2021; Liu et al., 2019), we select the commonly used RoBERTa as the misinformation detection model to perform the proposed CANMD. RoBERTa is a transformer-based language model with modified pretraining method that improves the performance on various NLP downstream tasks.

Baselines: The pretrained misinformation detection model is first directly evaluated on the target test data as naïve baseline. We additionally select two state-of-the-art baselines: DAAT and EADA (Du et al., 2020; Zou et al., 2021). DAAT leverages post training on BERT to improve the domain-adversarial adaptation. EADA performs energy-based adversarial domain adaptation with an additional autoencoder.

Evaluation: Similar to (Kou et al., 2022a; Li et al., 2021; Zou et al., 2021), we split the data into training, validation, and test sets with the ratio of 7:1:2. We adopt accuracy and F1 score to perform evaluation. We additionally introduce balanced accuracy (BA) to equivalently evaluate the adaptation performance in both classes, balanced accuracy is defined as the average value of sensitivity and specificity:

where TPR represents sensitivity and TNR represents specificity. TP, TN represent true positive and true negative, FP and FN represent false positive and false negative. In other words, balanced accuracy equally considers the accuracy of both classes, and thus yields a better estimation of adaptation performance under label shift.

Implementation: The input text is preprocessed as follows: (1) special symbols like emojis are back translated into English words; (2) hashtags, mentions and URLs are tokenized for posts on social media; (3) special characters are removed from the input text. For training, we first pretrain the misinformation detection model on the source dataset using Adam optimizer without linear warm-up. We adopt the learning rate of 1e-5, batch size of 24 and the maximum epoch of 5. We validate the model after each epoch with the validation set and report metric scores on the test set. In the adaptation step of CANMD, we adopt the same training and evaluation methods. For batching, we sample 24 target examples and perform class-aware sampling to sample another 24 source examples. We select confidence threshold $\tau$ from $0.6$ to $0.8$ and scaling factor $\lambda$ from $0.001$ to $0.05$. For baseline methods, we adopt the identical training conditions as CANMD and follow the default hyperparameter selections in the original papers (Du et al., 2020; Zou et al., 2021).

We first present the supervised training results of all datasets in Table 2, the evaluation results on the target datasets provide an upper bound for the adaptation performance. We observe the following: (1) the performance varies for different data types. For example, RoBERTa does not perform well when we train on claims without additional knowledge (e.g., LIAR provides short claims). Predicting misinformation on social media posts achieves better performance with average BA of 0.9094 (i.e., PHEME, Constraint and ANTiVax), potentially due to the syntactic patterns of misinformation (e.g., second-person pronouns, swear words and adverbs (Li et al., 2021)). (2) For disproportionate label distributions, balanced accuracy (BA) better reflects the classification performance by weighting both classes equally. For example, the classification accuracy is 0.9587 on GettingReal, while the BA score drops drastically to 0.8459. This is because the dataset contains overwhelmingly true examples and by only predicting one class, the model still achieves over 0.9 accuracy. Overall, the supervised performance depends heavily on the input data type and label distributions, suggesting the potential benefits of domain similarity in adapting misinformation detection systems.

Now we study the adaptation performance and report the results in Table 3. Each row represents a combination of source datasets and domain adaptation methods, and each column includes metric scores on one target dataset. We report BA, accuracy and F1 metrics for all source-target combinations, the best results are marked in bold and the second best results are underlined. We report our observations: (1) due to the large domain discrepancies, domain adaptation for COVID misinformation detection is a non-trivial task. For instance, the naïve baseline (i.e., None) trained on FEVER achieves 0.5633 BA and 0.5818 accuracy in Constraint; (2) in rare cases, we observe negative transfer. For example, consistent performance drops can be found on the BA metric in FEVER $\rightarrow$ CoAID experiments due to the discrepancy and the label shift across both domains. (3) Baseline adaptation methods achieve superior results than the naïve baseline (i.e., None) in most cases. For example, the DAAT baseline achieves 4.7% and 6.4% relative improvements on BA and accuracy, similar improvements can be found using EADA. (4) On average, CANMD performs the best by achieving the highest performance in most scenarios and outperforming the best baseline method by 11.5% and 7.9% on BA and accuracy. (5) CANMD performs well even when the source and target datasets demonstrate significant label shifts. For instance, GettingReal consists of over 90% true examples while Constraint is a balanced dataset in the COVID domain. CANMD achieves 0.7440 BA and 0.7416 accuracy in GettingReal $\rightarrow$ Constraint, with 34.5% and 29.3% improvements compared to the strongest baseline (i.e., DAAT). Altogether, the results suggest that the proposed CANMD is effective in improving misinformation detection performance on out-of-domain data. By comparing the performance in source-target combinations with dissimilar label distributions (e.g., GossipCop $\rightarrow$ CoAID), CANMD is particularly effective in adapting models to target domains under significant label shifts due to label correction.

We also present qualitative examples of the learnt domain-invariant features generated by the baseline methods and CANMD. In particular, we first train the misinformation detection models with different domain adaptation methods. The trained models are evaluated on the test set and we visualize the [CLS] position output from the transformer encoder. We select the correct predictions and perform T-SNE to generate the plots. The plots can be found in Figure 4. Constraint is a widely used COVID misinformation dataset, we thus perform the experiments with DAAT, EADA and CANMD using Constraint as the target dataset (Const.), valid examples are in orange while misinformation examples are in green. For source datasets, we denote the FEVER with F, GettingReal with GR, GossipCop with GC, LIAR with L and PHEME with P.

From the qualitative examples we observe the following: (1) all domain adaptation methods generate domain-invariant features in the representation space, as we do not observe clustering of examples in the plots. Nevertheless, different classes of examples are not clearly separated in most cases. (2) The proposed CANMD reduces intra-class distances while separating the true samples from the misinformation samples. For example, we observe a clear decision boundary between both classes in Figure 4(k) and Figure 4(l). (3) The label correction component in CANMD effectively corrects the feature distribution in the representation space. For instance, the target dataset Constraint has a balanced label distribution, while FEVER, GettingReal and GossipCop comprise of over 70% credible examples (orange). Yet we observe comparable area consisting of credible and fake examples in Figure 4(k), Figure 4(l) and Figure 4(m), whereas the baseline methods demonstrate overwhelming orange area (as of the source data distribution). In sum, CANMD exploits the source domain knowledge by enlarging inter-class margins and learning compact intra-class features, CANMD also corrects the label shifts to remove potential bias from the source data.

We now discuss the potential reasons for the performance variations in the adaptation results. For this purpose, we refer to the adaptation results in Table 3 and qualitative examples in Figure 4. In particular, we discuss how does the adaptation performance vary across different source-target combinations?

First, the different adaptation performance can be traced back to the similarity between the source domain and the target domain. When the source and target datasets are alike, knowledge can be transferred to the target domain with less efforts. An example can be found in the LIAR $\rightarrow$ Constraint dataset in Table 3, where both datasets have similar input types and text lengths. In this case, the naïve baseline achieves 0.7255 BA while CANMD can reach up to 0.8147 in BA. In contrast, GettingReal $\rightarrow$ Constraint demonstrates a larger domain gap with different input types (News $\rightarrow$ Social Media) and length (738.9 $\rightarrow$ 32.7). Thus, we observe less improvements using CANMD, with 0.7440 and 0.7416 in BA and accuracy respectively. The variations in the improvements can be further attributed the to the label shift between the source and target domain. When the source domain has a significantly different label distribution from the target domain, the bias may be carried over to the target domain during adaptation. An example can be found in GossipCop $\rightarrow$ CoAID, where the target label distribution is 10:90. As such, the baseline methods perform poorly in the balanced accuracy metric. Compared to the baseline methods, CANMD achieves 0.6487 in BA in contrast to only circa 0.5 of baseline methods, suggesting the effectiveness of the proposed label correction component. Therefore, the domain discrepancy and the label shift between the source and target domains are observed to be crucial for the expected adaptation performance in COVID-19 misinformation detection.

Label Correction and Contrastive Adaptation: We evaluate the effectiveness of the label correction component and contrastive adaptation by comparing our results from CANMD to the results trained without label correction and contrastive adaptation. We report the performance on target datasets (averaged over all source datasets) in Table 4. Note that removing the contrastive adaptation component reduces CANMD to the naïve baseline, as the pseudo labeling and label correction are designed for the following contrastive adaptation stage. As expected, we observe performance drops by removing the label correction component in CANMD. On average, the performance reduces 6.1% and 3.6% in BA and accuracy when we remove the label correction component in CANMD. Surprisingly, removing label correction slightly improves the F1 score on Constraint, this is because the model tends to predict true more frequently without label correction. As F1 does not considers true negatives, the value can be lower even if the model predicts with an increased fairness between both classes. Additionally, we observe that the majority of the improvements in CoAID are from the label correction component rather than the contrastive adaptation stage. A potential reason is the disproportionate label distribution in CoAID, where over 90% data examples are credible information. Thus, the performance improves significantly by correcting the label shift between CoAID and the source datasets.

Sensitivity Analysis of Hyperparameters: We study the sensitivity of the hyperparameters $\lambda$ and confidence threshold $\tau$. Similarly, we vary the input hyperparameter in adaptation and report the results on all target datasets. The reported numbers are BA values and are averaged across source datasets, see Figure 5. The results on hyperparameter $\lambda$ are comparatively stable and the BA scores are less sensitive to the changes of $\lambda$. For confidence threshold $\tau$ in pseudo labeling, the results vary across datasets. For example, the performance peaks for Constraint and CoAID at $\tau=0.6$, while the results improve steadily with increasing $\tau$ for AnTiVax, suggesting further room for improvements on ANTiVax. Overall, the proposed CANMD is robust to the changes of $\lambda$ and $\tau$ and consistently outperforms baselines even with sub-optimal hyperparameters.