Machine Learning Methods for Evaluating Public Crisis: Meta-Analysis

Over the last decade, the scientific community like IEEE and the Information Systems for Crisis Response and Management (ISCRAM) have contributed many studies that utilize real-time information sources to support situation awareness during large-scale events [1, 2]. The Machine learning field has advanced on how they automate processes to filter large volumes of data [3]. This study explores a variety of machine learning solutions utilized in scholarly articles to understand human actions towards crises. It was also informed by studies that addressed disaster management, health, politics, and other forms of crisis that utilized data beyond social media with evidential proof from many scholarly articles available in major academic databases that focused on analyzing human actions. While the first interactive medium for an individual that has no control over the mainstream media is the social media platform, local sources for tracking crises exist. People tend to report incidents, or debate about the ongoing incident via a social network that is familiar to them, or verifiable local reporting agencies and news media [4, 5]. Scientific researchers have taken advantage of the mass surge of social media data [6], and local reports to carry out machine learning procedures like predictions. The objective of this paper is to examine prevalent machine learning methods utilized by academics for managing crises, how those methods are implemented, and the source of the data.

As noted earlier, historical and real-time data play an important role in managing crisis, or in our case, evaluating crisis [7]. In order to plan for, mitigate, and avoid future crises, it is recommended to investigate historical or existing solutions. The concept of human action is predicated on some causative factors i.e. before someone acts, there is a cause [8]. The human environment is the main factor that drives crisis, and how humans manage environmental resources plays a huge role in crisis occurrence. Some of the concepts that aid in determining human actions from a data source are sentiments, perceptions, and/or attitudes [9]. While perception uniquely identifies opinion-based thoughts and impressions [10, 11], and can simply be distinguished from sentiments and attitudes, sentiments emphasizes emotions [12, 13], and attitudes leads to actions [7]. This study seeks to demonstrate with the help of peer-reviewed articles, the machine learning methods that are most prevalent in today’s world for evaluating or predicting crisis. The focus would be on public or human actions towards crisis with key concepts like attitudes and/or perceptions.

Statistical methods were among the first tools used to evaluate crises [14]. In 1932, when Patrick [15] completed a multivariate analysis on some organizations, financial stakeholders began developing models to assess the likelihood of a crisis in their organization. Since then, academics have devised a number of quantitative ways to detect and evaluate crises. Some quantitative analysis like the t-test has been successful in quantifying ratios [16]. Altman [17] devised a score that was used to categorize observations into good and bad. Multiple Discriminant Analysis (MDA) also played a role in some advanced analyses to compress variance between datasets [18]. Despite their widespread use in both academia and industry, these types of models have proven to be about numbers and quantities, necessitating the need for improvements that span beyond numbers [19]. To address the constraint of these models, various research that employs pattern matching approaches has been substantially researched in the field of machine learning [20]. Several of which have proved machine learning models’ ability to deal with unbalanced datasets [21], pictorial data and text data [22]. Even the difference between parametric and non-parametric methods for analyzing risks can be detected [23].

In addressing the research questions, the authors explored some background literature related to auto-coding, pattern matching, and text analysis. Most articles tackled issues of crisis management using machine learning, data transformation/scaling, and natural language processing. Machine learning has helped IT practitioners perform tasks in a very short amount of time [24]. It appears to be a quick option for identifying disruption events, getting authentic feeds or detecting periodic incidents in real-time [25]. Learning such patterns was also a major game-changer, as the approach tries to map patterns of interest or similarities in a given dataset [26], while also showing the capacity to learn and produce accurate results [27]. The pattern that is key to understanding human actions are cues demonstrating preference. This preference can be in the form of emotions, opinions, viewpoints, or specific annotations that help in explaining why people act the way they do [28]. When evaluating or predicting actions, some key concepts to take note of are perceptions, attitudes, sentiments, etc. The term ”perception” can be misconstrued to mean the same thing as attitudes or even sentiments. In simpler terms, sentiments are concerned with people’s feelings about an event, i.e., positive and negative events [13], perception and attitudes are concerned with people’s perspective towards an event [28, 11]. Emotions can aid in understanding perception, but the distinction is that perceptions or attitudes can be formed on the basis of facts and not only emotions [29]. While attitudes are reactionary (they can produce immediate action), perceptions are internal cues suggesting future actions or attitudes [30]. Focusing on perceptions, the formal definition is the process of organizing and interpreting sensory inputs to make sense of events [11].

To detect or predict a crisis, there is a need to make sure that the dataset used for analysis is actionable. Actionable data is characterized by information that can be acted upon or data that provides sufficient insight about the future [31] i.e., the data gives insight into actions that informs valuable decisions. In other words, it is more than just data kept in data warehouses. They have undergone analytical and data manipulation and are presented in a clear, intelligible, and frequently visually appealing manner [32]. It enables researchers to spot mistakes or potential crises and capitalize on new opportunities, improve future actions, and make faster and more informed decisions for the future [33].

The methodology employed in this study is the meta-review technique. It identifies the feasible variables in a cluster of articles with a common interest. This approach is often applied to literature published in a particular language; in our case, the review focused on academic articles written in English and published in ISCRAM, ScienceDirect, and the IEEEXplore databases. These databases were selected because they have subsections that addressed crisis management using machine learning. That is not to say that some other databases with an emphasis on crisis do not exist; we mainly focused on three for this study. The search and selection criterion illustrated in Figure 1 shows the various building blocks and the flow of data across them. The search terms are essentially the keywords needed to be used in a query for specific databases. More emphasis on the search term was made in the search technique section. The AND and OR operators were used in the query structure because they help to sieve out the articles that do not fall within our query parameter. Figure 2 expanded the inclusion and exclusion criteria shown in Figure 1 to visualize the different components of the inclusion and exclusion mechanisms, and how data were truncated or reduced in the process to get a final result.

From the analysis, majority of the articles reviewed supported the classification machine learning task more than the regression or clustering tasks. 3(b) shows that 41% of the reviewed papers made use of the classification task, while regression and clustering accounted for 18% and 16%, respectively. As shown in 3(a), the supervised machine learning method garnered 69% when mapped together with unsupervised and active learning, with 27% and 3%, respectively. This does not imply that one method is superior to the other, rather it demonstrates a preference across scientific communities. The method preference of supervised learning can be attributed to the availability of training data peculiar to crisis from CrisisNLP [43], https://crisisLex.org - CrisisLexT26, SoSItalyT4, and BlackLivesMatterU/T1 [44]. Sometimes, the format in which crisis data is communicated may not be easily suited as a corpus for training a model. In this case, the unsupervised learning approach comes in handy [45]. Some of the studies examined had hybrid approaches where supervised and unsupervised methods were used depending on the availability of training data for some subset of their analysis. Active learning seems not to be largely applied in crisis research. One factor that could have influenced this is that active learning is a form of semi-supervised learning that engages outside sources to label a dataset. Because of that, the practice of semi-supervised learning was not noticed on a full scale. Following the distribution timeline of methods, it is clear that there is an upsurge in the use of supervised machine learning as the years go by across the three databases, as shown in Figure 3.

The methods in 3(a) were further broken down into classification, clustering, regression, NLP, and sentiment analysis in 3(b). The NLP and sentiment analysis were identified separately in this figure because they can be a subclass of either classification, clustering, or regression tasks. It should be noted that sometimes they exist on their own (as transformers), not following the algorithmic process in the three classes mentioned above.

The consistency in the usage of classification further strengthens the earlier statement about supervised learning being predominant amongst researchers. The next in line was clustering where k-nearest neighbors (kNN), K-Means, and some tasks with random forest and decision trees algorithms conform as shown in Figure 5 and Figure 6. The Regression tasks like linear and logistic regression shows that they are still relevant in today’s research gaining 4% and 10% respectively. The analysis also shows that the implementation of machine learning methods for crisis management or evaluation reached its highest in the years 2020 and 2021. Again, the increase in the usage of machine learning classification techniques can be linked to the availability of training datasets, the ease with which the algorithms can be implemented, and the understandability of training data labels. The use of complex technologies can also be a contributing factor since such analysis may take human agents a considerable amount of time to analyze. The need for automated data processing, predictions, or analysis is something that has come to stay to aid humans.

The distribution of keywords in Figure 4 yielded some noteworthy results. The terms ”COVID, disaster management, and social media” appeared more frequently in the reviewed publications, and they peaked in the year 2020. There was a clear pattern in the frequent mentions of social media and how they are used to aggregate crisis data. Our analysis also identified ”social media” as the key term that received the most attention in the years 2019, 2020, and 2021, indicating that social media is a useful tool for gathering information about crises in the modern era. We can link the mention of a health crisis and COVID in 2021 to the recent COVID-19 pandemic and the way researchers are deploying intelligent systems to aggregate data across various media. Disaster management was also used as a keyword in several articles but was scarce in the year 2021. The mention of text classification, domain adaptation, and topic modeling was noted as well, and these relate to sub-methods in analyzing crises. The topic modeling which was scarcely present but important in the machine learning community was notably used in some research. This can be connected to our earlier statement that an NLP task can act as a transformer in an unsupervised environment (e.g., top modeling makes use of unsupervised methods).

Another significant discovery from this study is that when we further break down the methods to their algorithmic standards or the terminologies that indicate how they function, our earlier resolve could be strengthened. The SVM algorithm appears to dominate the reviewed articles, followed by the neural network. This also strengthens our earlier statement that the classification task and the supervised method were predominant. The strength of SVM can be uncovered in structural risk minimization, efficient memory management during training, and high dimensional spaces needed in datasets. Memory management is an issue for deep learning algorithms, which have more accuracy than SVM depending on the volume of data, but it seems the benefits of SVM appeal more to researchers in the crisis domain. The neural network was also prominent among the articles studied. Amongst the variations of neural networks present in crisis research are convolutional neural networks and long-term memory (LSTM) had more traction. The Naive Bayes, random forest, decision trees, and logistic regression also show promise as identified in Figure 6.

We conducted this evaluative review on scientific articles published between 2010 and 2021, with an emphasis on crisis management and machine learning. Crises can emerge in different forms, e.g., health crises, natural disasters, economic crises, food crises, and political events, among others. These variations introduce complexity in developing automated methods to manage crises as well as track emerging events to allow for fast decision-making. The methodology was explicit enough in describing the meta-review processes and how we collected and evaluated different publications that addressed machine learning for crisis management. The concept of machine learning from the literature review and result sections is demonstrative of how many ML techniques and algorithms can be used to manage crises. It is evident from our results that all the algorithms had their fair representation in terms of the value they added. The machine learning field is continuously evolving as it tries to help in the analysis of large chunks of data, easing the tasks of data scientists in an automated process and changing the way data extraction and interpretation work.

All the reviewed articles produced results based on the structure of the problem they addressed, the type of crisis tackled, the source of the data, and the volume of the data. The percentage distribution, as shown in 3(a), describes the preferred machine learning tasks. NLP tasks such as sentiment analysis, which may be classified as supervised or unsupervised learning, were highlighted as critical in automating crisis management. The classification, clustering, and regression tasks were highly preferred, with classification topping the list due to the availability of training datasets and easy implementation of the algorithms through pre-packaged libraries in popular programming languages like Python, R, and Java. Furthermore, there is a general recognition of a reproducibility crisis in science right now. Machine learning techniques are often simpler to perform analysis with. The reproducibility crisis is the increasing number of study findings that aren’t replicated when a different group of researchers performs the same experiment. This problem has ramifications in a variety of sectors where machine learning is utilized to make discoveries.

Our findings show that a significant proportion of articles (41%) used classification over regression or clustering, owing to the availability of training data/corpus and pre-packaged machine learning libraries. 69% of the articles made use of the supervised machine learning method (RQ1), showing preference across scientific communities in dealing with crises. Consequently, 27% of the studies made use of the unsupervised learning technique, while the remaining 4% used active learning methods. To address RQ2, our analysis revealed the machine learning methods and prevalent keywords used in the reviewed articles. It suggests that the SVM, Neural Networks, Naive Bayes, and Random Forest algorithms, amongst others, are popular among researchers in the crisis management domain (RQ2). Also on RQ2, the keyword crisis informatics garnered great interest in the scientific literature explored. Some interesting projections like health and disaster management rose by the year 2020 and social media received the most attention in 2019, 2020, and 2021, implying that social media is beneficial to gathering crisis-related data in modern times.