Is it Fake? News Disinformation Detection on South African News Websites

Modern technology makes it easy for anyone with access to a cell phone or computer to publish anything online, regardless of expertise on the topics. For humans, it is getting harder to distinguish between factual news and fake news [1]. It is not feasible for online companies to have a team of people read everything that is being portrayed as news for validity. Although research has been done and tools, e.g. machine learning, are in place at some social media companies to identify and warn users [2], there is still a lot of fake news on these platforms. This could be due to localisation or national differences in the way people write and/or conceal fake news. For example, the way South Africans and Americans speak and write English is not the same. South Africa further may have instances of code-switching [3].

In this work, we investigate developing a machine learning model to detect South African fake news web-sites/content. We investigate the impact of using US fake news datasets against using a new dataset that is focused on South African fake news. For people to trust such a model to analyse news for them, the model should not be a black box where they cannot see how it classifies news as fake or factual. Thus, interpretable machine learning is used, highlighting on the articles’ text used for evaluation and testing how much which parts contributed to the prediction [4]. The stylistic differences or behavioural characteristics between writing fake news and real news are also investigated in the interpretable articles when the model predicts on an article although it is not very obvious to spot these differences. We further release, with this paper, the South African fake news website dataset.

A survey from 2016 by BuzzFeed [1] found that 75% of American adults are fooled by the fake news headlines. If this study should reflect a wider part of the world’s population to a reasonable extent, the damage that fake news can cause becomes serious. It should be clear that fake news has the potential to ruin people’s lives and businesses. It should also be clear that fake news can be used for other agendas such as misleading people to vote for another party amongst other things. This is well documented as Information Disorder [5]. South Africa is not immune and has had its own challenges with disinformation [6]. Because of increased demand in fact-checking for fake news detection, multiple researchers have tried multiple Natural Language Processing (NLP) techniques to automate the task because of the large amount of data available. Fact-checking by a person requires researching and identifying evidence, understanding the information and what could be inferred from it which is a tedious task for everything that has to be checked. Thus a human has to be in the loop.

Natural language processing is a range of computerised computational techniques used to analyse text linguistically such that a computer can achieve a human-like language processing for real world application. The text used to train, evaluate and use the model has to be in a language used by humans to communicate and not constructed for analytical purposes [7]. There are multiple approaches to NLP, the two popular ones are statistical (mathematical) and machine learning (artificial intelligence) based approaches. Statistical approaches use mathematical techniques to model linguistic phenomena and only use observable data as a source of evidence. As part of machine learning we have neural network based approaches that take large amounts of text data into consideration when creating a model, taking into account the purpose of the model resulting in a more robust model [7].

Supervised learning, an artificial intelligence learning approach, learns a function that maps input (n-dimensional vectors) to their respective desired output [8]. Typically, the training data is given to the model as example input-output pairs without any specific rules. To create the mapping function, neurons are created with some arbitrary values called weights. These weights are changed in the training process to ensure each example input maps to its desired output, or as close to it as possible. In most fake news detection models, supervised learning is the training method of choice. Using existing articles and or statements that has been classified by means of fact-checking [9]. A typical text classification problem, even more so for fake news detection, is that the evaluation of a claim requires more context than just the fact being evaluated. Another problem is that well-constructed sentences may also convey false information [9], thus the model would need to be trained with more context in mind.

Another approach that has been tested is using information surrounding the article like the author, origin, and credit history of the author [10]. The problem with this approach is that it can introduce bias to the model, thus we will only be using the text content of the articles. When the model does not have knowledge of the author, the model would not be able to successfully classify the article. Most research on fake news detection has been done using very specific attributes and only focusing on certain aspects of the problem. Various combinations of the following were used:

• •

  Short political facts [11], [12].

• •

  Social media statements [11], [13].

• •

  Authors, location, affiliation, speaker title and credit history [12], [10].

• •

  Number of followers, likes and comments (media platform dependent) [13].

• •

  Authenticity [14], [15].

• •

  Tone, emotion [14], [15].

• •

  Lexical, syntactic, semantic and readability properties [10], [11].

Interpret means to “present in understandable terms” [16]. Which is an important aspect of machine learning as some countries require algorithms that make decisions on interpretations of humans to provide an explanation [17]. Interpretable machine learning can be divided into two broad categories, namely intrinsic interpretability and posthoc interpretability. Intrinsic interpretability [18] is the art of constructing the model itself to explain what it does. For intrinsic interpretability there are some trade-offs that one needs to be aware of, between prediction accuracy and interpretability. Post-hoc interpretability [18] is the art of creating a second model that can explain the main model by modelling what the main model has done then attempt to explain what the main model has done. This method only estimates what the main model had done but there are no trade-offs in the main model’s performance or accuracy.

We investigate the stylistic differences in writing between nations with machine learning in areas where it could play an important role, such as fake news detection.

There are three aspects to this research that build on top of each other:

• •

  fake news classification,

• •

  stylistic understanding,

• •

  and making the results interpretable such that a person can understand the model’s prediction.

To evaluate the stylistic understanding of the models, two models of each neural network type were trained, one with SA corpus and the other with US corpus then each model’s test data was used to evaluate the other model trained with the different nation’s data.

The data used to train the Logistic Regression models consisted of balanced classes. Training with the American corpus produced a better accuracy score than the model trained with the South African corpus. Predicting classes for the US corpus using the SA corpus trained model produced poor results. Refer to Table II. The F1 score for this experiment is 0.564 with a per class accuracy of 0.431 for fake news articles and 0.696 for true news articles. The drop in accuracy was expected but turned out to be more severe than the expectation. Predicting classes for the SA corpus using the US corpus trained model also produced poor results but even more so for classifying the fake news itself. With an F1 score of 0.473 and a per class accuracy of 0.260 for the fake news articles and 0.686 for the true news articles.

To preserve the context of word sequences, LSTM neural networks were constructed. To create an LSTM model baseline, equal numbers of articles of fake news and real news were used to train the models. The balanced classes trained and tested LSTM model trained with SA corpus achieved 0,520 accuracy and the US corpus trained model achieved 0,895 accuracy.

To increase the accuracy of the first LSTM models, a data imbalanced model was trained with the SA corpus, since there are more real than fake news in the real world, reaching an accuracy of 0,755. No training data imbalance was introduced to the American trained model as the data source only contained half of each class.

To increase the accuracy of the SA corpus trained LSTM model, Wordnet based augmentation [26] was used to expand the dataset. The increased dataset size improved the model’s accuracy score from 0,755 to an average of 0,845. Augmenting the US corpus also increased the American trained models accuracy from 0,895 to an average of 0,968.

Combining the SA corpus and the US corpus (US 1) for training and testing another model, the model reached an average accuracy score of 0.884 and a peak of 0.906 which is better than testing different nations’ data on different models. The results were harder to analyse for behavioural characteristics and the accuracy a bit lower, but it performed better than training models with one nations data and testing with another nation’s data. The final LSTM models were trained 5 times to get an average accuracy score. Refer to Table II for results..

The logistic regression models’ results were made interpretable using Local interpretable model-agnostic explanations (LIME) [28]. Analysing the articles with the LIME explanations showed that the model relies on the presence and counts of specific keywords to make its prediction, not taking enough context into account. The word based reliability can be seen in the Fig 4.

Intrinsic interpretability was used to make these LSTM models interpretable. Two permutation layers were added and the weights of the second permutation layer was extracted from the model. These weights were used as values to indicate how much which words in the text contributed to the classification. When a piece of text was processed by the model, a breakdown of how much each word contributed to the classification was given as a bar graph with only the contributing words and the text itself. The words are highlighted lighter or darker according to the weight that it contributed to the classification. See Fig 5.

In this work we have investigated the creation of fake news website detection models using a new South African dataset as well as two US datasets. We further provide insight into the behavioural factors that may lead the models to classify the websites as fake or not. This is just the beginning of providing models that are better suited for South Africa. We also release the South African fake news dataset as an online dataset for others to use and also expand on. There is still more work to be done. Investigations into changes into how fake news spread in South Africa is still to be done. What happens when the websites themselves change how they write to avoid detection. The archiving of past fake news websites is important as they tend to disappear after a few years. As such there is a need for archiving tools for such websites as they provide good data for future models.

The authors would like to thank Media Monitoring Africa for their support. Further, we would like to thank the reviewers for their feedback and Tshephisho Sefara for proof reading.