Articles that can potentially mislead or deceive readers by providing fabricated information are known as fake news. Usually fake news are written and published with the intent to damage the reputation of an agency, entity, or person¹¹1https://en.wikipedia.org/w/index.php?title=Fake_news&oldid=921640983. The popularity of social media (e.g. Facebook), easy access to online advertisement revenue, increased political divergence have been the reasons for the spread of fake news. Hostile government actors have also been implicated in generating and propagating fake news, particularly during elections and protests [TUFEKCI, 2018] to manipulate people for personal and organizational gains.

Impact of Fake news is creating havoc worldwide. During the 2016 US election, 25 percent of the Americans visited a fake news website in a six-week period of election which has been hypothesized as one of the issues that influenced the final results [Grave et al., 2018]. In Bangladesh, the 2012 Ramu incident is an exemplary event where almost 25 thousand people participated in destroying the Buddhist temples on the basis of a Facebook post from a fake account [Manik, 2012]. About 12 Buddhist temples and monasteries and 50 houses were destroyed by the angry mob. Fake news that contains blasphemy can easily repeat these types of incidents where people are very sentimental to their religions.

To tackle fake news there are some dedicated websites like www.politifact.com, www.factcheck.org, www.jaachai.com where they manually update potential fake news stories published in online media with logical and factual explanations behind the news being false. But these websites are not capable enough as they cannot respond quickly to any fake news event. Recently computational approaches are also being used to fight against the menace of fake news. ?) tried multi-perspective speaker profiles to detect fake news, where ?) has used linguistic features to detect satirical news. Besides ?) has proposed a fully automated fact-checking model using external sources to check the claim of news stories. To detect fake news in social media, ?) has used deep two-path semi-supervised learning. However, these works have been done only on news published in the English. As of now, around 341 millions of people in the world speak Bangla and it is the fifth language in the world in terms of number of speakers ²²2https://en.wikipedia.org/wiki/List_of_languages_by_number_of_native_speakers. But to the best of our knowledge, there is no resource or computational approach to tackle the risk of fake news written in Bangla, which can negatively affect this large group of population.

In this paper, we aim at bridging the gap by creating resource for detecting fake news in Bangla. Our contributions can be summarized as follows:

• •

  We publicly release an annotated dataset of $\approx 50K$ Bangla news that can be a key resource for building automated fake news detection systems.

• •

  We develop a benchmark system for classifying fake news written in Bangla by investigating a wide rage of linguistic features. Additionally, we explore the feasibility of different neural architectures and pre-trained Transformers models in this problem.

• •

  We present a thorough analysis of the methods and results and provide a comparison with human performance for detecting fake news in Bangla.

We expect this work will play a vital role in the development of fake news detection systems. In the rest of the paper, we briefly describe our dataset preparation methods, human performance & observation, and the development of fake news detection systems along with their performance.

Acknowledging the impact of fake news, researchers are trying different methodologies to find a quick and automatic solution to detect fake news in the recent past years. Previous works on satirical news detection mostly use Support Vector Machine (SVM) [Rubin et al., 2016, Burfoot and Baldwin, 2009, Ahmad et al., 2014]. Focusing on predictive features an SVM model is proposed by ?) with 360 news articles collected from 2 satirical news sites (The Onion and The Beaverton) and 2 legitimate news sources (The Toronto Star and The New York Times) in 2015 and showed absurdity, grammar and punctuation marks are best for identifying satirical news. Leveraging neural networks, ?) built a 4-level hierarchical network and utilized attention mechanism by using $\approx 16K$ satirical data (collected from 14 satirical news websites) and $\approx 160K$ true data and showed paragraph-level features are better than document level features in terms of the stylistic features. Some approaches focused on deception detection and utilised traditional machine learning models such as Naive Bayesian models [Oraby et al., 2017] and SVM [Ren and Zhang, 2016] to work with linguistic cues.
To learn the fake news patterns ?) also used a linear SVM classifier but to build the supervised learning model they used only linguistic features such as N-grams [Scholkopf and Smola, 2001], Punctuations, LIWC [Pennebaker et al., 2015] and conduct their evaluations using five-fold cross-validation. First they collected a dataset of 240 legitimate news from different mainstream news websites in the US then made another dataset containing fake versions of those legitimate news. To generate fake versions of the legitimate news items, they used the crowdsourcing via Amazon Mechanical Turk (AMT). However to find out the underlying features of fake news, neural networks are more useful [Shu et al., 2017] and for textual feature extraction, word embedding technique with deep neural networks are producing quality results. But it is noticeable that neural network based models requires large dataset. Liar [Wang, 2017] is a comparatively large dataset containing 12.8K human-labeled short statements collected from POLITIFACT.COM’s API. They used both surface-level linguistic realization and deep neural network architecture.
Existing research towards clickbait detection involves hand-crafted linguistic features [Chakraborty et al., 2016, Biyani et al., 2016] and deep neural networks [Gairola et al., 2017, Rony et al., 2017] with datasets mostly containing clickbaits (headline and article of clickbait news) from different newspapers. Besides [Ciampaglia et al., 2015, Vlachos and Riedel, 2014] have proposed a fact-checking method through knowledge base. And ?) has proposed a fully automated fact-checking using external sources where their dataset contains 761 claims from snopes.com, which span various domains including politics, local news, and fun facts. Each of the claims is labeled as factually true (34%) or as a false rumor (66%).

Though researches focused on the English language have achieved significant advancement, very few works are available for different low resource languages like Indonesian, Bangla, Hindi. ?) created a dataset containing 250 pages of hoax and valid news articles and proposed a hoax detection model using the Naive Bayes classifier for the Indonesian language. In Chinese, a dataset of 50K news is used by ?) for their real-time news certification system on the Chinese micro-blogging website, Sina Weibo³³3https://s.weibo.com/.

To the best of our knowledge, our work is the first publicly available news dataset in the Bangla for fake news detection. Throughout our literature review, we found that most of the works introduce a dataset suitable for their research approach and there is some dataset only focused on particular research topics like stance detection⁴⁴4http://www.fakenewschallenge.org/. Since fake news related research for the Bangla are still in its early stage, we design our dataset in a diverse way so that it can be used in multiple lines of research. So we enrich our dataset with clickbaits, satirical, fake and true news with their headline, article, domain and other metadata which is explained briefly in the next section.

To collect a set of authentic news, we select 22 most popular⁵⁵5We used the Alexa rankings to determine the popularity (www.alexa.com) and mainstream trusted news portals in Bangladesh. For collecting fake news we include the following types of news in our dataset.

• •

  Misleading/False Context: Any news with unreliable information or contains facts that can mislead audiences.

• •

  Clickbait: News that uses sensitive headlines to grab attention and drive click-throughs to the publisher’s website.

• •

  Satire/Parody: News stories that are intended for entertainment and parody.

we have collected news from popular websites that publish satire news in Bangla. While collecting satirical news from these sites we found that most of the sites have the exact same news. So after scraping news from these sites, we discarded the duplicates. We have collected the misleading or false context type of news from www.jaachai.com and www.bdfactcheck.com. These two websites provide a logical and informative explanation of fake news that is already published on other sites. So we have also collected the news that is mentioned on those two sites from the actual publishing sites and make sure that we avoid the duplicates. Clickbait is used to grab attention and drive click which eventually increases site visitors and generates revenue for them⁶⁶6www.webwise.ie/teachers/what-is-fake-news. And we have found that most of the local or less popular sites usually do this. To collect clickbaits, we have gone through some of these sites and manually collect potential clickbait news from there. We call satire, clickbait, and false informative news all commonly as fake news throughout the paper to avoid ambiguity. We have also collected the following meta-data along with the headlines and content:

• •

  The domain of the published news site

• •

  Publication time

• •

  Category

From our dataset, we got 242 different categories as different publishers categorize the news in their own way. To generalize it, we took similar categories from different news to map into a single one. Finally, we categorize all news from the dataset into the 12 categories (Table 1).

Human observation suggests that the source plays a key role in an article’s credibility. Note that, by source here we mean one or more person or organization capable of providing verification of the claimed news. If there is no such source, then reporters or journalists are taken as the source of news. Besides, ?) has shown that adding speaker profiles along with document-level features improved the performance of fake news detection. So to make our dataset more resourceful, we include the source information as a meta-data for each news. Besides the source, we have also included the headline article relation in meta-data. “Related” and “Unrelated” tags are provided upon checking the relationships of the headline with the article. Since we have to go through each of the news to find out the source and headline-article relation so far we have managed to annotate only $\approx 8.5K$ data. All of the members of our data annotator team are undergraduate students of Computer Science and Engineering and Software Engineering department. Figure 1 is a sample from our dataset.

Detecting fake news is a tough task. To see how good humans perform, we have conducted an experiment where we took 60 fake news and mixed them randomly with 90 authentic news and gave them to 5 human annotators who are undergraduate students(one Industrial and Production Engineering, one Chemistry and three Computer Science and Engineering students). They were told to read 150 news one by one and answer 2 questions for each news. Note that we only gave them the headlines and the articles. The first question is if the news is fake or authentic. Based on the answer of the first question they were provided another question with four options to select why they think the news is fake or authentic. Here the set of four options are different for fake and authentic. For creating these four options, we first took feedback from 10 people regarding what properties or methods they use to find out if any news is fake or authentic. Then we generalized their feedback into four options. The process of the human baseline experiment is shown in Fig. 2 using a flowchart.
From the first question, we got an estimate of how accurately humans can detect fake news. The F1-score for the fake class of the five annotators is 58%, 65%, 70%, 68%, and 63% respectively. The inter-annotator agreement, measured as Fleiss’ Kappa[Fleiss, 1971] is 38.83% and mean pairwise Cohen’s Kappa[Cohen, 1960] is 39.05% which indicates that our human annotators have given the same answer on nearly 39% percent of the questions. The second question helped us to find out what factors are crucial for humans to find the difference between fake and authentic news. If the news is fake, on an average 44% answer is ‘The content is unrealistic’ and 42% answer is ‘Has no trustworthy source’. When news is true, on an average 62% answer is ‘The Content is believable’ and 21% answer is ‘Source is reliable’. Here the source is a person or an organization who/that can provide the validation of the claimed news. From the feedbacks of annotators on overall answers, we found that they choose the option ‘The Content is believable’ because they read or hear similar news on their daily life and the content has nothing to disbelieve it. If the content looks fishy to them then they look for the source of the news. So the experiment shows that source of the news is important. After the experiment we had a follow-up interview with the human participants to find out why they make mistakes in detecting fake news. Most significant reasons are as follows:

• •

  Disguise: If any fake news is represented just like true news that is the fake news contains enriched information such as strong references, scientific facts and statistics in detail then human mistake it as true news.

• •

  Trending: People tend to believe any news when they notice a lot of newspapers are reporting/have reported the same news.

• •

  Source: When a dubious news doesn’t contain any reference (an entity who provided the information) then humans take it as false news.

• •

  Satire: Humans can detect almost all types of satire news but sometimes some true news also sounds like satire and humans take them as fake news.

These analysis indicates that the source plays an influential role in fake news detection. So In our dataset, we manually annotated the source of the news so that in future we can use the source as a momentous feature for detecting fake news.

In this section, we describe the systems we develop to classify fake news written in Bangla. Our approaches include traditional linguistic features and multiple neural networks based models.

Data Pre-processing: We perform several pre-processing techniques like text normalization and stop words, punctuation removal from the data. Here we use Bangla stop words from Stopwords ISO ¹¹¹¹11https://github.com/stopwords-iso. We observe better validation performance by such pre-processing of the data.

Evaluation Metric: We use Micro-F1 scores to evaluate different methods. As the dataset is imbalanced, we also report the precision (P), recall (R), and F1 score for the minority class (fake).

Baselines: We compare our experimental results with a majority baseline and a random baseline. The majority baseline assigns the most frequent class label (authentic news) to every article, where the other baseline randomly tags an article as authentic or fake. We report the mean of precision, recall, and F1-score of 10 random baseline experiments in Table 3. The Standard Deviation(SD) of precision, recall, and F1-score in both overall and fake class is less than $10^{-2}$ except the recall of Fake class which is 0.027.

Experiments: With the linguistic features, we experiment on training a Linear Support Vector Machine (SVM) [Hearst, 1998], Random Forest (RF) [Liaw and Wiener, 2002] and a Logistic Regression (LR) [McCullagh and Nelder, 1989] model. We split our dataset for training and testing in a 70:30 train-to-test ratio. We tune the penalty parameter ($C$) based on the validation results.

For BiLSTM, CNN, and BERT based experiments, the hyper-parameters are Optimizer: Adam Optimizer [Kingma and Ba, 2015], Learning rate: 0.00002, Batch size: 32. Hidden size for BERT model is 756 while in CNN and BiLSTM it is 256. For CNN, we use the kernel lengths of 1 to 4 and zero right paddings in the experiment. The train and test dataset is kept at a 70:30 ratio. And In training, we use 10% of the test data as validation data. We use 50 epochs for each experiment and put a checkpoint on F1 score of fake class using validation data. And finally we report the result using our test dataset on the best scoring model from training.

In BERT, For fine tuning our dataset we use the sequence classification model by HuggingFace’s Transformers. And we use the BERT’s pre-trained multilingual cased model which is trained on 104 different languages¹²¹²12https://github.com/google-research/bert.

We report our results in Table 3. Overall performance of every experiment is almost the same. Most of the cases we achieve almost perfect Precision, Recall and F1. But the results of Precision, Recall, and F1-Score of fake class vary in experiments to experiments. In our dataset for experiments, the number of authentic news is 37.47 times higher than the number of fake news which could be the reason behind such variance in results of the overall and fake class. To evaluate the performance of different models we will use the precision, recall, F1-Score of the fake class in the rest of the section.

Fig. 3 shows that experiment with linguistic features with SVM, RF and LR. Here SVM outperforms LR and RF by a quite margin except result of the news embedding. In the case of news embedding RF scores 55% of F1-score and here SVM, LR scores 46%, 53% of F1-score respectively. For most of the features RF performs better than the LR model. We report the result of SVM on Table 3 since it has performed better than the others. Table 3 shows that lexical features perform better than other linguistic features, majority & random baselines, and neural network models as well. It is also observed that the F1-score of fake class in the SVM model decreases while increasing the number of grams in both word and character n-grams.

The result of POS tag does not improve over the random baselines and the F1-score of this feature indicates that it cannot separate fake news from authentic news. Again F1-Score of MP, Word Embedding(Fasttext), Word Embedding(News) are better than the POS tags but fall behind the lexical features and neural network models. However, these features outperform the majority, and random baselines. We got our best result when incorporating all linguistic features with SVM. It scores 91% F1-score.

Neural Network models have shown better results in different text classification problems [Lai et al., 2015, Joulin et al., 2016]. But in our experiments, we found that F1-Score of fake class in neural networks cannot outperform the linear classifiers. In CNN, experiment with average pooling, global max technique scores 59% and 54% F1-Score respectively. Generally an attention model with LSTM performs better than CNN [Yang et al., 2016, Chen et al., 2016]. In our case, the F1-Score of BiLSTM with attention cannot improve over the CNN model. The best results of neural networks based experiments came from BERT, whose F1-Score is 68%. Though neural network models perform better than the majority and random baselines still it falls behind the performance of the SVM model.

To compare the performance of our fake news detection system with humans, We tested the same dataset that we used for the human baseline with our two best models. The model with SVM and character 3-gram scores 89% F1-score where it scores 90% with SVM and all linguistic features. Here, the performance of our best models is almost similar to the test set. In our human baseline dataset, there is 90 authentic news along with 60 fake news. And the model with linguistic features has successfully made correct assumptions on 50 fake news. On the other hand, the character 3-gram model has picked out 49 correct assumptions. Compared to human performance which 64.8% of F1-score these two models have performed much better. These results suggest that linear classifiers could separate the slightest margin of true and authentic news which humans usually overlooked.

In this paper, we present the first labeled dataset on Bangla Fake news. Here the evaluation of linear classifiers and neural network based models suggest that linear classifiers with traditional linguistic features can perform better than the neural network based models. Another key finding is that character-level features are more significant than the word-level features. In addition to that, it is also found that the use of punctuations in fake news is more frequent than authentic news, and most of the time fake news is found on the least popular sites.

However, since character level features have shown better results so we will incorporate the character level features in neural network models such as [Kim et al., 2016, Hwang and Sung, 2017]. From human observation, we found that the source can also play a key role in fake news detection. We will also include this feature in our future experiments. Besides we will also continue to expand our dataset. We have manually annotated around 8.5K news. We will continue this process to reach the 50K mark. We hope our dataset will provide opportunities for other researchers to use computational approaches to fake news detection.

We would like to thank data annotators and resource team who helped us during human baseline experiment. Also we thank Natural Language Processing Group, Dept. of CSE, SUST for their valuable comments and discussions with us.