Measuring Happiness Around the World Through Artificial Intelligence

Emotions are a distinctive part of human nature. It is an essential element of life, and it contributes to our interactions with other people. Our intelligence attaches great importance to how we feel and distinguishes between different emotions. It is not possible to define or measure emotions in an exact manner, since they are abstract and subjective. However, social scientists have used different indicators about how one feels and what affects one’s feelings. Thus, measuring happiness is possible to some extent.

Happiness is a multidimensional complex fact [1]. It is a driving force and natural goal for most people. Researchers in social sciences have different methods on measuring happiness and variables that affect happiness.

Analyzing happiness levels or life satisfaction of countries attracted wide interest in social sciences and media. Generally, social sciences use polls to measure the level of happiness of society. Acquiring accurate results with happiness levels is a desired goal in different research areas. Machine learning can help with this problem by adding unbiased approach on measuring happiness and add an extra dimension to researchers in social sciences. In this mindset, our work examines the emotions from facial images collected from publicly available street footages of eight different cities around the world. Facial expressions from raw city footage are categorized into seven basic emotions; Anger, Sad, Neutral, Disgust, Surprise, Fear, Happy.

Our approach is fundamentally different than that of indicators used by social scientists. It is relatively unbiased, because it relies on machine learning to determine the emotions of people. Machine learning has been a powerful tool for solving problems in various domains such as medicine, economics and robotics. It can accurately detect emotions by analyzing the facial images.

The organization of the paper is as follows; related works in this area are discussed in Section II. Section III describes the proposed methods in detail. Results are given in Section IV and Section V concludes the paper.

Understanding the factors that affect happiness have been studied extensively. Veenhoven, Ruut and Ehrhardt [2] investigated if the data available on happiness are in accordance with the three theories of happiness; comparison, folklore and livability. In a previous study, the researchers concluded that happiness is adversely affected for the people when their neighboring countries becomes wealthier [3]. In another study [1], it is investigated whether social capital has a significant effect on happiness. In [4], researchers updated the Easterlin Paradox, the idea that happiness is proportional to income, is valid for only developed countries. They concluded that this paradox also holds for less developed countries. In [5], the effects of different variables on the happiness (migration, health issues, income, etc.) are investigated on both national and global scale. HSBC’s expat survey [6] measured the satisfaction of expats in different aspects for various countries.

There are varying approaches on the subject in literature for emotion recognition ranging from examining brain signals to hybrid approaches like analyzing audio and text input together with videos. Machine learning algorithms such as Support Vector Machines (SVM) [7], Neural Networks and K-nearest Neighbors (k-NN) can be used to predict emotions from face images.

Some of the notable works in this field are as follows; Ekman and Friesen [8] developed a system which is called FACS (Facial Action Coding System), where basic universal facial expressions are represented as combinations of different action units. Black and Yacoob [9] used polynomials to represent optical flow and extract the motion of the facial features. After that, they used a rule-based classifier to classify basic emotions. Yacoob and Davis [10] used optical flow of the detected facial features within rectangular regions with rule-based classifier for recognition. Essa and Pentland [11] transformed face images into mesh models, then calculated the optical flow from them to recognize facial expressions. Donato et al. [12] compared different techniques to extract and classify facial expressions. Cowie et al. [13] published a survey about emotion recognition containing detailed information about both audio and visual approaches. Cohen et al. [14] in their work used Naive Bayes Classifiers with a Cauchy distribution together with HMM for facial expression recognition. Ioannou et al. [15] used a combination of SVM, morphological operators, neural networks and neuro-fuzzy system to classify facial expressions. Gunes and Piccardi [16] used face and hand gestures in combination to determine emotional expression. Mansoorizadeh and Charkari [17] introduced a hybrid feature space from extracted features of speech and video and performed emotion recognition. Dahmane and Meunier [18] used HOG and SVM to classify emotions from videos containing face. Halder et al. [19] used different combinations of fuzzy sets to classify emotions from faces. Poria et al. [20] used Convolutional (CNN) and Recurrent Neural Networks (RNN) with Multiple Kernel Learning to analyze sentiment from audio, video and text. Zhang et al. [21] introduced a fusion approach using a Part-based Hierarchical Bidirectional RNN (PHRNN) and a Multi-Signal CNN Network (MSCNN) on images to recognize emotions. Jain et al. [22] also proposed a CNN-RNN hybrid model on face images for recognition. Hossein and Muhammad [23] worked on audio and image inputs together using CNN and SVM for emotion recognition.

In our work, we present an unbiased emotion detector through machine learning using publicly available videos of the crowded streets in various countries. We aimed to gather unbiased data from different cities around the world to analyze the psychology of the general population. Footages from different cities are converted to frames to be used as dataset. All methods mentioned here are performed with Python language.

To detect facial emotion recognition, first we need to detect faces in each frame in the dataset. For this purpose, we used Adam Geitgey’s [24] face recognition model. With this model, it is possible to locate face in the frame and crop the face region for further processing.

For each city, we traverse through all frames and detected faces are cropped, turned into grayscale and resized to 48x48 pixels using OpenCV library. Then, these images are sent to pre-trained deep learning model of Priya Dwivedi [25] as inputs. Classification result belongs to one of these emotion categories; Anger, Sad, Neutral, Disgust, Surprise, Fear, Happy. Finally, classification results are exported to a CSV file categorized by city, emotion and value.

Our results showed that most common emotions detected with the algorithm were; Surprise, Fear, Happy and Anger. Sad and Neutral emotions were not detected in any of the cities. In Table-I detailed results containing all emotions are given. We notice that "surprise" emotion is very common. One explanation can be due to the recording nature of video (i.e. a youtuber recording the video) people might be surprised to notice a camera.

In Fig. 2, 95% confidence intervals for happiness proportion by cities are shown. The intervals show that there is no statistically significant difference between cities in terms of proportion of happy people. There can be several explanations for that. First, even though we collected street footage, the recordings used were not taken in an obscure manner and this might have biased emotions. Another explanation could be that the proportion of happy people in street is indeed similar across the world. Results are discussed in Conclusion section.

In our work, we proposed the use of public footage as a resource to determine happiness levels in society. We collected footage from various cities around the world together, then used artificial intelligence to recognize emotions from facial expressions.

For the reason why surprise and fear are prevalent in results, we think that it could be due to sun light directly hitting people outdoors, thus making them grimace, or maybe they were in a rush to catch up something, or maybe they were surprised when they noticed the camera recording. Another reason may be different races have different facial structures which can give a way to unintended classifications. It could be due to the neural network architecture, or the learning method altogether, it may need more training with different datasets or update in architecture.

For a future direction, further work needed in creating a more specialized dataset for the purposed idea and different machine learning algorithms needs to be explored in terms of performance and accuracy.