On Predicting Personal Values of Social Media Users using Community-Specific Language Features and Personal Value Correlation^††thanks: To be appear in the proceedings of ICWSM 2021

Motivation. Personal values (or simply, values) express what is most important to people in life. Every individual holds personal values (e.g., achievement, security, benevolence) with varying degrees of importance. A particular personal value may be very important to one person but unimportant to another. Personal values influence a person’s attitude and behavior. Although there are various approaches (?; ?; ?) to measure personal values, the one proposed by Schwartz in (?) is widely used by psychology researchers. Schwartz’s theory of basic human values (?) defines 10 specific personal values which can be measured by a specially designed questionnaire. These personal values are: Power, Achievement, Hedonism, Stimulation, Self-Direction, Universalism, Benevolence, Tradition, Conformity, and Security (see Fig. 1). These personal values are further grouped into five higher order personal values, namely: Self-enhancement, Hedonism, Openness, Self-transcendence, and Conservation. The questionnaire includes 56 items (?) to measure the value dimensions¹¹1To the best of our knowledge, other works also focus on analyzing and predicting high-order personal values (?; ?).. However, it is costly and privacy-intrusive to get people to complete the questionnaire. Researchers thus seek other more scalable alternative approaches to obtain personal values. Among these approaches, prediction of personal values from users’ social media content is very promising and has been studied in only a few works (?; ?). Because, such descriptors related personality could be subsequently served as features for the downstream applications like personalized recommendations (?).

There also exists some software products to predict personal values based on content²²2https://www.ibm.com/watson/services/personality-insights/, they are based on models trained on labeled users from specific geographical region and culture. These models may not predict accurately for another community of users due to: (a) different personal value profile distribution among new users compared with those used in training the models; and (b) some word use patterns of new users very different from that of training data. While the above limitations are well understood, there has not been much research to illustrate the impact of community specific language and personal value profile distribution to personal values prediction using user-generated content.

Research Objectives. This work aims to show that accurate value prediction has to leverage on word usage patterns specific to the region the users come from. A value prediction model trained using the content from another region may yield poorer performance when applied to the target region even the two regions have large lexicon overlap. At times, the prediction accuracy may not be high enough for data science studies on a large social media user population. The important research questions to address in this work are therefore: (a) how to cope with regional differences in word usage patterns? (b) how to develop accurate models to leverage features and task knowledge? (c) how to deploy value prediction models to gain insights about a large social media user population?

To answer the above questions with concrete illustration, we focus on personal values of users from Singapore which has an ethnic composition of 76% Chinese, 15% Malays, 7% Indians and others. Most Singapore users are asians who may have personal value profiles different from users from the US and European communities. The languages used in the Singapore community has major content differences which may render the previous models less ineffective. To study the personal value profiles of Singapore users and to develop personal value prediction models using their social media data, we have collected the Facebook content generated by a group of Singapore users. According to Alexa at the time of this study, Facebook is the most visited social media site in Singapore. The users in this group also completed the Schwartz’s personal value questionnaire so as to have their personal values determined.

In developing the prediction models, we explore several novel ideas. The first idea is to use a Singapore variant of Linguistic Inquiry and Word Count (LIWC) instead of the original LIWC to derive features adapted to the linguistic characteristics of Singapore users. LIWC consists of word categories that characterize the linguistics profiles of users. The second idea leverages on the positive and negative correlations between personal values according to the circular structure of Schwartz’s Personal Values (?) as shown in Fig. 1. This work proposes a method to exploit the circular structure of personal values to boost their prediction accuracy. Both idea have not been studied in any previous works on personal value prediction.

In this work, we further illustrate a scalable way to analyse online behavior of a large group of Twitter users using our personal value prediction model trained using Facebook data. Despite the analysis involves different social media data, we empirically show that the predicted labels of our model uphold the inter-relationships among personal values (?), highlight some hypotheses about user online behaviors in Twitter, and compare them with those reported in related works (?; ?; ?).

Contributions. In summary, the following are the novel contributions of this work:

• •

  We introduce a novel dataset collected for Singapore Facebook users with the ground truth labels of Schwartz’s personal values³³3The anonymous dataset is shared via shorturl.at/fhjS9. Singapore users represent a unique user community with its own linguistic characteristics. To our best knowledge, we are the first to analyze and predict personal values using a community-specific LIWC incorporating these linguistic characteristics. This is known as the Singapore-LIWC or S-LIWC.

• •

  We propose a new personal value prediction model known as Stack Model to improve the prediction accuracy of the personal values. The key idea is to exploit the circular structure of Schwartz’s personal values. This Stack Model is a two-layer model which supports features derived from both LIWC and S-LIWC. The model yields accuracy higher than most of the earlier state-of-the-art prediction models.

• •

  Instead of using social media post content only, we investigate user representation using features from Facebook profile information (i.e., interests and groups). Empirically, we show that the profile features are strong features to predict personal values.

• •

  We conduct a data science study of the personal values of a large community of Singapore Twitter users (85,000 users) who share their tweet content publicly, and correlate the personal values with their online behavior. In this study, our proposed stack model discovers interesting online user behavior with specific personal values. This is a major breakthrough extending the study of personal values to a large user community.

We divide the related works into two categories, namely: (a) personal value prediction; and (b) personality prediction.

Personal Value Analysis and Prediction. Personal values have been studied in the context of decision making and personal interests. In (?), the effect of individual’s personal values over their decision process is analyzed. Hsieh et al. studied the relationship between personal values and personal interests (?). There are a few previous efforts (?; ?; ?) to analyze the relationship between personal values and word usage. In (?), Chen et al. uses Reddit as the social media platform to collect user generated online context while Boyd explicitly asks users to produce content (?). LIWC word categories and modeled topics are used in (?) and (?) respectively to capture content features. Both works show the correlation analysis between word usage and personal values. Chen et al. further analyzes the prediction potential of personal values using simple binary classification models such as Logistics Regression, SVM, and Naive Bayes. Nevertheless, there has been little work on personal value prediction for user communities using region/culture-specific languages.

Personal value prediction using Facebook data is new. In addition to the word content in posts, Facebook profiles consist of other user generated profile data such as interests, group affiliations, and activities. These profile data however have not yet be used as features for personal values prediction in similar previous works (?; ?). There also has not been any other previous works on large-scale data science study of personal values and online behavior.

Personality Prediction. Prediction of other psychological attributes such as personality (?), and dark triad personality traits (?) based on word usage have been studied in recent years. There were several efforts (?; ?) to analyze the relationship between personal values and personality traits. In addition, researchers have studied specific human behavior, thinking pattern or preference using personal values or personality traits (?; ?; ?). Mukta et al. analyzed the prediction potential of individuals’ movie genre preferences using personality and personal values (?). Most of the previous efforts generally use content features with simple classification models to perform prediction (?; ?). To the best of our knowledge, none of them exploits the circular structure among personal values to improve the prediction potential of personal values. Golbeck et al. proposed a few discrete features extracted from Facebook profile information for predicting personality (?).

In this paper, we use two datasets covering users from the same community. The first dataset is a small Facebook dataset with ground truth personal value labels for training prediction models. The second dataset is a large Twitter dataset covering more than 80K users and their tweet and social network data. The Twitter dataset allows us to predict the personal values of the target Twitter user community and to conduct a large-scale study of user behavior for users of different personal values.

Facebook Dataset Construction. Social media datasets with ground truth personal value labels for research are generally not available. We therefore construct our own dataset by recruiting 125 undergraduate students (42 males and 83 females) to contribute their Facebook data and personal values labels by completing the Schwartz Value Survey (SVS) (?). Each participant received a small monetary reward for the participation.

The Schwartz Value Survey (SVS) is currently the most widely used personal value assessment instrument (?; ?; ?; ?). This survey requires participants to rate the importance of 56 value items as guiding principles in their life within the scale from -1 to 7. To remove individual differences in rating, we subtract every user’s original ratings across all personal values by the average of all ratings of the user. In this way, all users after the above adjustment share the same average rating of 0. This is the same technique used in previous works (?) to preprocess the ratings, thus we follow the same for the comparison purposes.

Along with the survey, participants shared with the research team their Facebook profile archives, which consist of Facebook posts and other profile information such as gender, preferences, and network details. The 125 users together have a total of 383,335 posts which were posted within the time span from the 4th quarter of 2007 to 1st quarter of 2018. They are required to remove sensitive posts before handing their data to the research team. The least and most prolific users have 12 posts and 20971 posts respectively (Avg. number of posts per user = 3067). All the users had been members of Facebook for at least 18 months at the end of year 2017. Personal values are generally considered as rather stable broad psychological attribute (?; ?) compared to other similar attributes such personality traits. Hence it is still valid to analyze individuals’ personal values using their older posts.

To check the internal consistency of survey results, we calculated Cronbach’s alpha (?) for each higher-order value dimension. The alpha values range from 0.73 to 0.86 (even higher) for the 5 higher-level personal values. These results are considered quite high and far better than that derived from the dataset used in (?).

To further ensure that the dataset is well collected, we compare the statistics of our dataset with that of a dataset used in (?). According to Table 1, most of the statistical figures of our dataset are similar to that of dataset in (?). In both datasets, the mean value for Self-Enhancement and Conservative are negative and it is positive for the rest. Highest variance between users is observed for the dimension of Hedonism in both datasets. On the other hand, Table 1 also reveals that our dataset has low mean values for all the personal value dimensions except for Conservative compared to the previous dataset (?). This suggests that Singapore users are highly driven by the goals like acceptance of tradition and customs, safety and stability of society, and relationship. This also highlights the importance of this research, which is done for a specific user community to yield more accurate prediction results and findings.

Most of the correlations between value dimensions are also similar between the two datasets in terms of sign and magnitude (e.g., Self-Transcendence vs Self-Enhancement, Conservative vs Openness to Change, and Conservative vs Hedonism). The correlation values, which are different between our Facebook dataset and the (?)’s dataset can also be explained using the circular structure of the Schwartz’s personal values. For example, our dataset shows a significant positive correlation between self-enhancement and hedonism while it is a weak negative correlation in (?)’s dataset. Between the two, positive correlation is more consistent with the personal values theory as Self-Enhancement and Hedonism are closer to each other in the circumplex structure. Hence, we can conclude that our dataset is reasonably well-collected and it complies with the circular structure of Schwartz’s personal values, which will be exploited in our subsequently proposed model for accurate prediction.

Twitter Dataset Construction. To conduct a larger-scale study of personal values for users from the Singapore community, we collected public tweets posted by 85,308 Twitter users based in Singapore during the 6 months period from January 2017 to June 2017. The dataset was constructed by first identifying a seed set of well known Twitter users based in Singapore. By crawling their following links, we reached out to other users who are also based in Singapore. We repeated the steps until the user set does not increase by size. In addition to the tweets, the Twitter network between these users (i.e., followers, friends) was also extracted. All these post and social network data are subsequently used in our user behavioral study. The descriptive statistics of this dataset are shown in Table 2.

Experiment Setup. We evaluate our proposed prediction models against two state-of-the-art baselines using our Facebook dataset, the only dataset we can use in this research. The two baselines are:

• •

  IBM Watson Personality Insight API⁶⁶6https://www.ibm.com/watson/services/personality-insights/: A well known personal value prediction package which has been used in many commercial settings. We report the predicted Schwartz’s personal values returned by the API when it receives the post content from us.

• •

  LR (all): This follows basically the Chen’s paper (?) and trains a logistic regression (LR) classifier (empirically observed that LR outperforms other classifiers such as naive Bayes, support vector machines, and decision tree) for each value dimension. All LIWC features are used as features in this model. The model however does not consider regional language differences and correlation between personal values.

We implemented our proposed base and stack models using Scikit-learn and tensorflow⁷⁷7https://www.tensorflow.org/ respectively. We measure the prediction accuracy by Area Under ROC Curve (AUC ROC) as it allows the results to be comparable to that in (?). For the base models for the different personal value dimensions, we further select top $n$ LIWC word categories as features to represent each user. We compare the accuracy of the top $K$% prediction task for $n\in\{5,10,15,20,30,40,85\}$. To select the top $n$ LIWC word category features, we considered different feature selection strategies like univariate correlation methods and, recursive feature elimination (RFE)⁸⁸8RFE recursively prunes away the least important features obtained using regression coefficients until the desired number of features is reached. and found RFE performs better in most of the cases. We empirically found out that $n=15$ yields the best performance for all the value dimensions. Hence, RFE and $n=15$ are used in all the subsequent experiments. For the stack model, we have used base models using S-LIWC word categories as these features show better results than base models using LIWC word categories.

Summary of Results. Note that our baseline methods can only yield results based on post features only. As shown in Table 4, our proposed models outperform the baselines across different value dimensions for both $K=50$ and $K=40$. Our base and stack model results are far better than the random baseline, which returns exactly 0.5 for AUC.

Surprisingly, IBM Watson Personality API does not even outperform the random guess for Hedonism (0.470) and Self-Enhancement (0.480) value dimensions. Another interesting observation is that Chen’s model (i.e., LR (all)) also perform poorly for this dataset, possibly due to its inability to generalize with limited data. This observation shows the effectiveness of our feature selection strategy.

S-LIWC vs LIWC Features. Feature wise, our base models using S-LIWC outperform those using LIWC in most of the value dimensions. The S-LIWC profile features perform particularly well for predicting Self-Transcendence, while the S-LIWC post features perform well for the rest. The concatenation of both post and profile feature sets further improves the prediction results considerably for most of the value dimensions.

Our stack model using a two-layer neural architecture (involving base models using S-LIWC based features) gives the overall best performance. This result is encouraging as it shows that both S-LIWC and personal values correlation contribute to the accuracy of personal values prediction.

Interestingly, profile features alone could often yield prediction accuracy comparable to post features when using S-LIWC (e.g., Base model using S-LIWC for predicting SE). This is not observed when using LIWC. This is due to most of user profiles involving community-specific features.

Stack vs Base Models. As shown in Table 4, our proposed stack model outperforms even the best-performing base models for most of the personal values. The improvements are especially significant in the Openness to Change (6.7% better than Base-S-LIWC(Post+Profile)) and Hedonism (3.3% than Base-S-LIWC(Post only)) value dimensions. To further analyze the results of the stack model, we report the feature weights derived from the final layer of our stack model in Table 5. It shows the inter-correlations between value dimensions learned by the stack model consistent with what we know from Schwartz’s theory (?). The circular structure also implies that opposing values in the structure should have significant negative correlation. For example, Table 5 shows significant negative weight assigned to the base model predicted conservative value $\hat{y}_{u,`conserv.^{\prime}}$ which allows predicted conservative value in the first layer to contribute negatively to the final prediction of openness to change value. Only 73.9% ($=\frac{4.28}{4.28+1.42+0.09}$) of the predicted score of base model for openness to change contributes to the final prediction. 24.5% ($=\frac{1.42}{5.79}$) of the predicted openness to change score is determined by the predicted conservative value by the base model. Note that conservative value is opposite to openness to change in the circular structure of personal values.

Top Feature Analysis. Table 6 lists the top 15 features of the base models using LIWC and S-LIWC for each value dimension. A positive (or negative) correlation between a word category and a value dimensions means that users high in the value dimension use words from the word category frequently (or rarely). We thus attempt to interpret some additional relationships captured by word categories with respect to the underlying goals of the value dimensions.

For Self-transcendence, the top positively correlated S-LIWC word category features such as “feel”, “sadness”, and “1st person plural” are highly reasonable. Users high in self-transcendence adopt a people-oriented sensitive persona, which is inlined with the defining goals of benevolence and universalism values. The top negatively correlated S-LIWC features are word categories such as “achievment” and “anger”, which are top positively correlated features for Self-Enhancement. This result further demonstrates the correlation between personal values, i.e., a negative correlation between Self-transcendence and Self-Enhancement. In contrast, top LIWC word category features includes a few function word categories such as adverbs and punctuation marks.

For Conservative, we observe that top 15 features extracted from posts using S-LIWC and LIWC are similar. Only the profile features shows some differences between the top features from S-LIWC and LIWC. This shows that profile features contribute to identifying different kinds of word usage patterns of individuals, which are not always found in posts. According to S-LIWC, Singaporeans who are high in conservative are likely to use words from the categories such as “health”, “home”, “religion”, and “family”. Over here, the conservative’s positive association with self-transcendence but negative association with self-enhancement may be the reason.

On the whole, our stack model shows the best results almost in all the cases and our models achieve significant improvements, 39.9% in Conservative, 44.2% in Hedonism, 54.1% in Openness to Change, 47.5% in Self-Transcendence, and 55.4% in Self-Enhancement, compared to the best of IBM personality and random baselines (when $K$= 50% for the classification task). Our model clearly benefits from using user community specific features. There could be differences in the personal value profile distribution for different user communities. Without considering the distribution differences and word use patterns, the prediction accuracy of personal values can suffer significantly.

In this paper, we first study how personal values prediction using a user’s social media content can be significantly improved by considering geographic differences in word usage and profile information.

In addition, we proposed a new stack model to predict individuals’ personal values by exploiting the correlations between personal values. Through our experiments, we show a significant boost in prediction accuracy for our proposed stack model compared to the state-of-the-art models proposed in previous works (e.g., IBM Personality Insight API). We finally showed that our model predicts personal values of a large set of Twitter users and derived interesting findings linking their personal values to their behavior on Twitter. These findings are largely consistent with previous research using traditional survey based studies. With reasonably accurate personal values prediction models, we envisage that many interesting research studies on the impact of personal values to opinion polarization, community formation, and others can be carried out at scale. Although we have considered the individual’s word usage in user profiles, i.e., groups, likes, and interests, there are other behavioral features that can be used to enhance the prediction accuracy of personal values which should be studied in future work. In addition, incorporating network structure and dynamic nature in social media to predict personal values may also be another promising future direction.