From Fake to Hyperpartisan News Detection Using Domain Adaptation

Machine learning techniques for detecting fake news include various feature-based methods, ranging from text to visual features Zhang and Ghorbani (2020). For example, linguistic features Choudhary and Arora (2021); Pérez-Rosas et al. (2018) capture aspects related to conveyed information, document organization, and vocabulary used in news. In contrast, style-based features Potthast et al. (2018); Zhou and Zafarani (2020) are related to the writing style, such as redaction objectivity and deception Shu et al. (2017). In recent years, Transformer-based models Vaswani et al. (2017) emerged in the fake news detection literature Jwa et al. (2019); Zhang et al. (2020); Kaliyar et al. (2021); Szczepański et al. (2021). Other techniques for detecting fake news use social aspects, such as the profiles of the users who spread the news on social media platforms Shu et al. (2017); Onose et al. (2019); Zhou and Zafarani (2020); Sahoo and Gupta (2021). Techniques successfully employed for these scenarios rely on custom embeddings and linear classifiers Shu et al. (2019), classic supervised machine learning techniques Reis et al. (2019), and deep learning networks, such as recurrent Wu and Liu (2018) and graph neural networks Monti et al. (2019); Hamid et al. (2020); Paraschiv et al. (2021).

Task 4 of SemEval-2019 Kiesel et al. (2019) introduced hyperpartisan detection from news articles as a binary classification task. The organizers created two balanced datasets by crawling data from various online publishers. Participants were asked to detect whether the news articles were hyperpartisan or mainstream. The winning team Jiang et al. (2019) of the shared task proposed an architecture based on multiple pre-trained ELMo embeddings Peters et al. (2019) averaged in the embedding space, followed by convolutional layers Kim (2014) and batch normalization Ioffe and Szegedy (2015). They achieved 84.04% accuracy on the training set and 82.16% accuracy on the test set, suggesting the challenging setting. Other works for the SemEval-2019 Task 4 were based on lexical and semantic handcrafted features via Universal Sentence Encoder Cer et al. (2018) or BERT, and a linear classifier Srivastava et al. (2019); Hanawa et al. (2019). Furthermore, Potthast et al. (2018) showed that hyperpartisan news detection could be analyzed using fake news approaches. They argued that the writing style for hyperpartisan news is similar to fake news, despite their political orientation.

The core objective of unsupervised domain adaptation is to enforce a feature representation invariant to the domain of the examples with the same labels. One of the most effective techniques is the work of Ganin and Lempitsky (2015), which treated the problem as a minimax optimization. Wang et al. (2018) utilized domain adaptation techniques via adversarial training for fake news detection by employing an event discriminator to learn event-invariant features in a multi-modal setting. Deng et al. (2019) relied on the similarity in the feature space by enforcing a clustered structure among similar features. In this case, the training procedure optimizes clustering loss alongside the domain adaptation loss. For the target dataset, a teacher model consisting of an ensemble of students generates pseudo-labels (i.e., estimates of the true labels). Also, contrastive learning Chen et al. (2020) was used to achieve unsupervised domain adaptation. It aims to have closer representations of the examples from the same class, while representations from different classes should stay far apart. In addition, Wang et al. (2022) proposed the cross-domain contrastive loss to minimize the $l_{2}$-norm distance between features from the same category, and employed K-Means to compute pseudo-labels.

In our current work, we utilize the pre-trained RoBERTa language model, which shares the same architectural design as BERT, the only difference being the pre-training objectives. The RoBERTa architecture stacks multiple Transformer encoders, each based on the multi-head self-attention mechanism Vaswani et al. (2017). On top of the RoBERTa model, we add a label predictor containing fully connected layers. RoBERTA uses the Byte-Pair Encoding (BPE) tokenizer Sennrich et al. (2015). In what follows, we present the settings in which RoBERTa is employed in our work (see Figure 1).

Given two datasets $D_{s}=\{(x^{i}_{s},y^{i}_{s})\}_{i=1}^{N_{s}}$ and $D_{t}=\{x^{i}_{t}\}_{i=1}^{N_{t}}$ from different domains, the UDA setting reduces the shift between them Ganin and Lempitsky (2015); Ganin et al. (2016). This approach comprises a feature encoder $G_{f}$, a label predictor $G_{y}$, and a domain discriminator $G_{d}$. The feature encoder maps the input space into a latent space. Then, the label predictor computes the labels of the underlying examples. Simultaneously, the domain classifier uses the latent space to predict the domain of the features (i.e., the source or target domain).

To obtain domain-invariant features, the optimization is two-fold. First, we minimize the prediction loss concerning $G_{f}$’s parameters $\theta_{f}$ and $G_{y}$’s parameters $\theta_{y}$. Second, we maximize the domain classification loss until $G_{d}$ cannot distinguish the domains of the features. Formally, the loss function $L$ (see Eq. 1) depends on the prediction loss $L_{y}$ between $G_{y}$’s outputs and source labels, and the domain adaptation loss $L_{d}$ between $G_{d}$’s outputs and domains $d^{i}$ (i.e., hyperpartisan and fake news). The trade-off between $L_{y}$ and $L_{d}$ is controlled by $\lambda$. Note that we omitted the model’s parameters for clarity.

The optimization problem associated with this formulation is described below:

where the parameters with hat are fixed during the optimization step. This problem can be solved with an implementation trick, namely gradient reversal layer (GRL) Ganin and Lempitsky (2015), which acts as the identity function during feed-forward and negates the gradients during back-propagation. The GRL layer is inserted between the feature encoder and the domain discriminator.

In our setting, we use the RoBERTa’s encoders for feature extraction and fully connected layers for both the label predictor and domain discriminator.

As an extension to UDA, Deng et al. (2019) exploited the class-conditional structure of the feature space by cluster alignment in the teacher-student paradigm. A teacher model trained on the labeled source examples estimates pseudo-labels for the unlabeled target dataset. To reduce the error amplification caused by label estimation, the teacher model is built as an ensemble of previous student classifiers. In addition, a student classifier minimizes the prediction loss $L_{y}$ on the source examples in the supervised setting. The optimization involves minimizing both the prediction loss $L_{y}$ and the sum of clustering losses $L_{c}$ (i.e., for both the source and the target domains) and the cluster-base alignment loss $L_{a}$:

where the hyperparameter $\alpha$ controls the trade-off between the supervised and semi-supervised losses.

Considering the labeled samples $X_{s}=\{x_{s}^{i},y_{s}^{i}\}_{i=1}^{N_{s}}$, the unlabeled samples $X_{t}=\{x_{t}^{i}\}_{i=1}^{N_{t}}$, the feature extractor $f(\cdot)$, and the distance metric $d$ between features, the total clustering loss is:

where $L_{c}$ is as follows for each $X_{*}$:

The intuition is to enforce class-conditional structure at the feature representation by grouping the classes into clusters, i.e., by minimizing the distance between features $x^{i}$ and $x^{j}$ that have the same label when the indicator function $\delta_{ij}=1$, whereas pushing different clusters away from at least a margin $m$ by maximizing the feature distance when $\delta_{ij}=0$. The classifier trained on the source features may not be able to differentiate between the same class from different domains, and therefore, an alignment loss $L_{a}$ is imposed between the domains as follows:

In this case, given the number $K$ of classes to be predicted, and the samples $X_{*,k}$ from either source or target whose labels are equal to $k$, the cluster centroids $\lambda_{*,k}$ are computed using:

The loss $L_{a}$ tries to match the source and target statistics by aligning the clusters for each class $k$ in the feature space. Additionally, the performance can be further improved by aligning the marginal distributions, i.e., adding a confidence threshold that ignores the data points likely to be included in the wrong class.

Self-supervised contrastive learning Chen et al. (2020) aims to learn representations such that, given a pair of examples, closely related examples should behave similarly, while dissimilar examples should stay far apart from each other. This can be achieved by employing various techniques such as data augmentation and custom losses (e.g., NT-Xent Chen et al. (2020), InfoNCE Oord et al. (2018)). Since there is no clear way to construct positive and negative pairs in an unsupervised domain adaptation framework, Wang et al. (2022) argued that samples from the same category should be similar. In contrast, samples from different categories should have other feature representations, regardless of the domain from which they come. Based on this hypothesis, they proposed the cross-domain contrastive (CDC) loss to reduce the domain shift between source and target labels. We assume $z^{a}_{t}$ and $z^{p}_{s}$ are the $l_{2}$-normalized features for the anchor sample from the target domain $x^{a}_{t}$ and the positive sample from the source domain $x^{p}_{s}$, respectively. In this case, the loss function is described by:

where $P_{s}(\hat{y}^{a}_{t})$ denotes the set of positive samples from the source domain having the same label as the anchor point, and $I_{s}$ is the set of all source samples from the mini-batch. Similar to Eq. 9, we compute $L^{s,a}_{CDC}$, for which we consider the positive samples from the target domain instead. The CDC loss with alignment at the feature level is¹¹1Note that we included the normalization terms compared to the original formulation.:

The objective function is given by the sum of the prediction loss $L_{y}$ and the loss $L_{CDC}$ scaled by $\gamma$:

We generate pseudo-labels using the K-Means algorithm since we require them when creating positive pairs. We initialize K-Means with the centroids of the source domain and predict on the target domain. The pseudo-labels are chosen to minimize the similarity distance between the feature representation and the centroid. K-Means is performed at the beginning of each epoch.

We propose an addition to the UDA approach, considering the supervised setting (i.e., we have access to the labeled source dataset). First, we represent the input text using TF-IDF or a pre-trained RoBERTa model. We employ a clustering/topic modeling algorithm in this feature space to identify $k$ clusters or topics, which will be assigned as domain labels. For clustering, we employ four algorithms, namely K-Means, K-Medoids, Gaussian Mixture, and HDBSCAN. Also, we use four topic modeling algorithms, namely LDA, NMF, LSA, and pLSA. The motivation is to compact the latent representation, given estimates of latent domains under a topic model (i.e., a dataset split). During training, it is minimized the loss given by Eq. 1 while using the proposed domain labels. For the target examples, we do not include labels during training. We choose the number of clusters using the elbow method²²2https://www.scikit-yb.org/en/latest/api/cluster/elbow.html. After training on each pair of domain labels, the best-performing model is selected for the inference stage.

We perform experiments on three datasets related to fake (i.e., ISOT and BuzzFeed) and hyperpartisan (i.e., BuzzFeed and Hyperpartisan Kiesel et al. (2019)) news detection.

The ISOT fake news dataset contains news articles collected from reuters.com, and other websites, which were validated by Politifact³³3An organization that checks the veracity of the news.. The dataset comprises 44,898 articles, of which 21,417 contain truthful information, and 23,481 are fake news. All collected articles are related to politics and have at least 200 characters.

The BuzzFeed dataset contains 1,627 articles in three categories: mainstream, left-wing, and right-wing. The mainstream and hyperpartisan data are evenly distributed, and the length of the articles ranges between 400 and 800 words. This dataset is annotated for both fake and hyperpartisan news detection.

The Hyperpartisan dataset which contains hyperpartisan news was released under the SemEval-2019 Task 4 shared task Kiesel et al. (2019). The dataset was crawled from news publishers listed by BuzzFeed⁴⁴4https://github.com/BuzzFeedNews/2017-08-partisan-sites-and-facebook-pages and Media Bias Fact Check⁵⁵5https://mediabiasfactcheck.com. From these sources, 754,000 news articles were extracted and semi-automated labeled using distant supervision Mintz et al. (2009) at the publisher level, provided in the HTML format. It was split into 600,000 articles for training, 150,000 articles for validation, and 4,000 articles for testing. Half of the dataset consists of non-hyperpartisan articles, and the other half is split equally among left-wing and right-wing articles. Since the authors also released a smaller version of the dataset (645 examples for training and 628 examples for testing), in what follows, we will refer to the larger dataset as Hyperpartisan-L and the smaller dataset as Hyperpartisan-S.

We perform data cleaning on all three corpora, ignoring non-ASCII characters and removing HTML-specific symbols and constructions that do not provide any information about the actual content, such as multiple chains of dots in a line. BPE was utilized for tokenization, setting to output a maximum of 128 tokens per text sample.

Since the ISOT and BuzzFeed datasets are not provided with separate splits for validation and testing, we use the following split: 70% for training, 10% for validation, and 20% for testing. In addition, due to limited computational resources and the large size of the Hyperpartisan dataset, we select a random 5% of the data from the training set (i.e., 30,000 examples) and 5% of the data for the validation set (i.e., 7,500 examples). Also, we use the entire Hyperpartisan test set since it contains only 4,000 examples.

We utilize the pre-trained RoBERTa base version (123M parameters), which consists of a stack of 12 Transformer blocks. For all experiments, the Adam optimizer Kingma and Ba (2015) with a linear scheduler is used with a warm-up (it is set with 5% of the gradient steps) for the learning rate. The learning rate varies among experiments, between $1e-4$ and $1e-5$. We employ a dropout set between 0.1 and 0.5. We also set the optimizer’s weight decay parameter to $1e-3$, and clip the gradients between -1 and 1 to increase training stability and reduce overfitting.

We start with the most straightforward approach for training a neural network. That is, we take a pre-trained model on similar tasks and transfer some of the acquired knowledge to the downstream task via fine-tuning. The baseline model consists of the RoBERTa model followed by a stack of fully connected layers. We employ two fully connected layers, with 256 hidden units and two output neurons. The models are trained for 3 epochs, with a learning rate of $1e-4$ and batch size between 32 and 64.

First, we evaluate the model on all four datasets for baseline results. Table 1 presents the final results obtained during experiments. We observe that ISOT achieves the highest scores, followed by BuzzFeed and Hyperpartisan-S. We note that humans annotated these datasets, whereas the Hyperpartisan-L dataset was annotated with a semi-supervised approach.

By comparing three fine-tuning methods (see Table 2), we observe that freezing the model’s encoders yields poor performance. This increases the number of false positives and decreases the number of true negatives because of the domain shift between the datasets and training with fewer parameters. On the other hand, fine-tuning improves the results since the model’s parameters are adapted to the new domain.

We consider the encoders from the RoBERTa model as feature generators. We also use a stack of fully connected layers, with 256 hidden neurons and two outputs for both the label predictor and the domain discriminator. The domain discriminator is linked to the output of the RoBERTa encoder via a gradient reversal layer. We tested three values for $\lambda\in\{0.1,1,5\}$.

Furthermore, we perform larger-to-smaller and smaller-to-larger dataset adaptations between Hyperpartisan-L and BuzzFeed. The model is trained for 3 epochs (i.e., the steps required to pass through all examples from the larger dataset). The batch size is set to 64, from which half are labeled and the other half are unlabeled examples. The results are shown in Table 3. We observe that if $\lambda$ is set too large, the model does not learn the data distribution but predicts only one class. Conversely, UDA performs better when $\lambda=0.1$, achieving higher accuracy on the Hyperpartisan-L target dataset. This adaptation may have helped because of the inherent similarities between domains and improved performance on out-of-distribution points.

Moreover, we employ different ways of linking the GRL layer with the RoBERTa encoders. Since the RoBERTa-base model uses 12 encoders, we utilized the 4th, 6th, and 10th, besides the previous experiments. While the encoder returns a feature representation for each element in the sequence, we take the representation of the [CLS] token. Table 4 shows the results. The 12th layer performs best, while similar performances are achieved using the 4th or 6th layer. The results are supported by the fact that more layers for the encoder mean more representational power for the feature encoder that needs to be adapted among domains.

In addition to the previous experimental setup, we set the parameter $\alpha\in\{0.1,1\}$ for the clustering loss in the CAT configuration. We also consider a lower learning rate (i.e., $1e-5$) to improve convergence. We consider an epoch is a complete pass through the smaller dataset to update the pseudo-labels for the entire target domain using the teacher model. As such, we trained the models for 10-30 epochs. We set the margin $m=2$, the ensemble size to 3, and the ensemble accumulation to 0.8.

We performed domain adaptation from BuzzFeed to Hyperpartisan-L. The results are shown in Table 5. The model obtains over 90% accuracy on the source domain and is bounded by 66.4% on the target domain. This approach generally achieves a smaller accuracy than previous techniques, the best score being when $\lambda=\alpha=0.1$. Also, we can observe that the difference between $\lambda$ and $\alpha$ affects the performances. Analyzing the model predictions, we notice that using smaller values for $\lambda$ and $\alpha$ yields a high number of false positives, while larger values increase the number of false negatives. Using $\lambda=1$ and $\alpha=0.1$ resulted in a biased model towards mainstream examples.

For the CDCL method, the experimental setup is similar to the one used for the CAT. We varied the temperature $\tau\in\{0.1,0.5,1\}$ and the coefficient $\gamma\in\{0,0.1,1,5\}$. Table 6 provides the results of our analysis. We observe that both $\tau$ and $\gamma$ affect the performance. The best results were attained when $\tau=1$, and $\gamma=5$, achieving 63.9% accuracy on the target domain, while $\tau=0.5$ generates the best values on the source dataset. It proves that $L_{CDC}$ performs some regularization on the source domain. We noticed that the models often produce a high false positive rate, affecting the recall more than the precision. In addition, training for more epochs, the model starts overfitting on both source and target domains while degrading the performance of the validation set.

We explore a data augmentation technique based on TF-IDF as proposed by Oord et al. (2018) for consistency training. Thus, we compute the TF-IDF score for every token from the corpus and associate it with the probability of it being changed. The words with the higher probability are replaced with non-keywords from the vocabulary to avoid changing the meaning of the text. The TF-IDF-based word replacement depends on a hyperparameter $p$ that controls the level of augmentation enabled on the dataset. We vary $p$ for our experiments to augment the BuzzFeed dataset with multiple augmentation levels. Table 7 shows the results for all training configurations, where two or three values per augmentation type indicate that we applied each value of $p$ and concatenated the augmented examples over the original dataset. Also, zero suggests that only the unaltered dataset was used. Using more augmentations (e.g., $p\in\{0.1,0.2,0.3\}$) on the CDCL and CAT frameworks yields better overall results, while on UDA, using a much stronger augmentation (i.e., $p=0.5$) leads to better results.

One problem with this data augmentation technique is that it may alter the text in a way that is not coherent anymore, specifically when many tokens are changed. The most frequent words may not always have the same meaning, so their contextualized representation is affected. Since the context defines the meaning of a word in language models, this augmentation changes the representation, especially on unlabelled data. Table 7 illustrates the issue on the target dataset. However, on the source dataset, the performance is not affected but generally improved.

In the topic-based UDA approach, we follow the same experimental setup as in classical UDA. For training, the only difference is that we train all models for 10 epochs. We explore both, the clustering on RoBERTa features (i.e., K-Means with Euclidean or cosine distance, K-Medoids, Gaussian Mixture, and HDBSCAN) and the topic modeling algorithms on TF-IDF features (i.e., LDA, NMF, LSA, and pLSA) to split the representation. We evaluate the experiments on the Hyperpartisan-L test set and present the results in Table 8. Using clustering algorithms for domain labels provides the best overall results compared to Table 3. The best-performing models outperform the UDA approach by over 3% in accuracy and are obtained when we adapted from a larger to a smaller split. It is noteworthy that for the HDBSCAN, the cluster 2 contains very few annotated examples (i.e., 332) compared with the other two (i.e., 17,092 and 12,576), resulting in adaptation failure. When using the topic modeling, we see a degradation in performance, especially in the case of NMF. Compared with the RoBERTa baseline (see Table 2), the model achieves similar F1-scores.

We use t-SNE Van der Maaten and Hinton (2008) to visualize the feature representation learned by the best models we obtained for each category. In Figure 2, we present the plots for the baseline, the UDA, the CAT, and the CDCL. Using different approaches to domain adaptation may reduce the domain gap in the feature space between the two domains. Still, many examples cluster together far apart from their counterparts. UDA obtains better representations than the other methods. When considering the topic-based adaptation (see Figure 3), we notice a better separation when employing topic models. Also, we achieve poor separation among classes for K-Means and K-Medoids.