Two-Faced Humans on Twitter and Facebook:
Harvesting Social Multimedia for Human Personality Profiling

To represent human personality, in this work we use the Myers-Briggs Type Indicator (MBTI) (Myers, 1998), which has been widely adopted by the research community (Buraya et al., 2018, 2017) and splits one’s personality into 16 types, each formed by the following four binary dimensions:

• •

  Extroversion and Introversion (EI): this dimension determines how an individual focuses her energies and interest, whether she is influenced externally by the opinion and interpretation of others (Extroverts) or motivated by her inner thoughts (Introverts).

• •

  Sensing and iNtuition (SN): this aspect demonstrates how people interpret knowledge. Sensing personalities make decisions based on their five senses and solid observation, whereas Intuitive individuals favour imagination to constancy.

• •

  Thinking and Feeling (TF): a person with Thinking aspect, always exhibits logical behaviour in their decisions, while Feeling individuals are empathic and give priority to emotions over logic.

• •

  Judging and Perceiving (JP): this dichotomy describes an individual approach towards work, decision-making, and planning. Judging individuals are highly organized in their thoughts, while Perceivers behave more spontaneously.

The data was collected from Twitter, Facebook, and PersonalityCafe³³3https://www.personalitycafe.com/ social networks, during the time interval of $1$st Jan $2018$ to $1$st Jan $2021$, and via the two following steps:

1) Ground truth collection. To obtain personality ground truth from Twitter, we have downloaded all of the tweets which contain self-reported personality-related keywords/phrases such as “I’m an ENTP” or ”I am an ENTP” and extract the personality trait from those phrases to be the ground truth for each user (see Figure  1(a) for example). To harvest Facebook ground truth, we have monitored Facebook comments under personality test results released on 16personalities portal (see Figure 1(b) for example). Likewise, to obtain the personality-related ground truth from the PersonalityCafe forum, we downloaded user’s publicly-available self-reported personality traits on their profile pages (see Figure 1(c) for example).

2) User-generated content (UGC) collection. To establish UGC collection from Twitter and Facebook, we have downloaded user timelines through Twitter REST API⁴⁴4https://developer.twitter.com/ and Facebook GRAPH API⁵⁵5https://developers.facebook.com/, respectively. At the same time, to collect UGC from the PersonalityCafe forum, we downloaded posts from the MBTI forum thread.

3) Data pre-processing. Since social network data might exhibit significant noise levels and often contain grammatical errors, it’s necessary to perform data prepossessing prior to the data modelling stage. At the same time, it is necessary to remove the direct personality mentions from the text content so that the model might not be able to use personality abbreviations from the post content at the inference stage. To mitigate the above two problems, we have pre-processed our dataset via the following three steps: 1) Data Filtering To ensure the sufficient amount of data per user for training and inference, we have filtered out the users with less than 10 tweets available; 2) In-line label replacement For all personality traits, the personality type name was then replaced with the ”¡type¿” placeholder (e.g. ”ENTJ” might have been replaced with ”¡type¿”); 3) Social indicator replacement: similarly to Nguyen et al. (Nguyen et al., 2020), we have further converted emojis into the corresponding descriptive textual strings, removed all non-ASCII words, and normalized the text by replacing user mentions, URLs, hashtags, date-time by the corresponding placeholders as follows: @USER, HTTPURL, HASHTAG, DATETIME.

Table 1 highlights the statistics of our dataset across binary personality labels while Figure 2 visually reflects the personality label distributions. From the Table 3 and the Figure 2, it can be seen that INFP, INFJ, INTJ, ENTP, INTP are the Top-5 most popular personality labels found in both Facebook and Twitter datasets, showing the consistency of the data distributions across general social networks, reducing the risk of falling into source-dependent bias during the data modeling stage. At the same time, it is important to note that in the PersonalityCafe forum data, the ENFP, INFP, INFJ, ENFJ labels dominate the rest. The latter observation shows that the personality-related data sources might have a distribution shift towards individuals of certain personality types (different from a general distribution) that tend to participate in such specific personality-related discussions. Therefore, the evaluation based on PersonalityCafe dataset must be accomplished independently.

The MBTI personality categorization scheme defines each of the $4$ binary MBTI categories to represent a different aspect of human personality. However, when being combined into $16$ personality types, it is known to be associated with a major shortcoming of the overlap between the ”neighbour categories” (e.g. INTJ and INTP). Given the noisy nature of the content from Social Media, it might be a good idea to predict individual binary MBTI personality traits, instead of modelling the overlapping 16-category scenario. Therefore, in this work, we have adopted such binary personality categorization scheme.

To facilitate an effective data modelling process, the data needs to be properly represented in the form of feature vectors. Following the best practices described in the literature on user profiling (Khan et al., 2020) (Rangel Pardo et al., 2018) (Farseev and Chua, 2017c) we have chosen the following data representation approaches:

1) Textual Features: First, to represent the textual data at a user level, for each user all posts were concatenated into a corresponding user-specific ”documents”. Second, the term frequency-inverse document frequency (TF-IDF) has been extracted to form the document-term matrix. Finally, we have applied the Latent Semantic Analysis (LSA (Halko et al., 2011)), as such transformation has previously shown sizable performance uplift (Daneshvar and Inkpen, 2018) when being applied for user profiling. The final dimension of the compressed textual feature vector was set to $100$, where the new number of dimensions has been found empirically during a grid search.

2) Visual Features: To represent visual data, we have automatically mapped each photo into the distribution of $1000$ ImageNet (Deng et al., 2009) image concepts via the pre-trained ResNet-101 model (He et al., 2016). We then summed up the predicted concept occurrence likelihoods for each user and element-vice normalized the obtained vector by the total number of images available from the user. In such a way, for each user, we have obtained a $1000$-sized image concept distribution vector. Similarly to the text modality, Principal Component Analysis (PCA (Jolliffe, 2011)) has been further applied to reduce the dimensionality of the visual feature space to $200$.

Given a user i, we denote the multi-view data associated with the user as a set X:

where $l$ represents the number of samples in the dataset, $T_{I}$ is the collection of the $i$th user text content represented as Textual Features, $I_{i}$ is the collection of $i$th user image content represented as Image features, and $y_{i}$ represents one of the four $i$th binary personality trait ground truth labels.

In such a way, we can now formulate the personality profiling as a user-level multi-view document classification task, where users play the role of ”documents” when being rephrased in standard terms.

Below, we now can define the PERS framework as a two-step stacked generalized ensemble approach. The architecture of our PERS framework is illustrated in Figure 3.

First-Step: Given the training set $X=\{(y_{i},x_{i}),i=1,2,\ldots,I\}$ , where $y_{i}$ is the personality label and $x_{i}$ represents the feature vector, we shuffle and uniformly split $X$ into $K$ equal sub-sets. In such a way, $X_{k}$ and $X^{(-k)}=X-X_{k}$ are the definitions of the test and training sets, respectively, for the $k$-th fold in K-Fold cross-validation.

Now, we can specify a J-sized list of ”base” classifiers and train the $j$-th base classifier based on the training set $X^{(-k)}$. We denote the $Z_{ki}$ as the prediction of $j$-th classifier on $x_{i}$ for each sample in the test set for the $k$-th cross-validation fold. In such a way, immediately after the cross-validation process, we can define a new dataset based on J ”base” classifier outputs:

For each data modality, we compute $Z_{Tfirst}$ and $Z_{Ifirst}$ separately and then we form the input for the next stage of data processing $H$ via column-wise concatenation of $Z_{Tfirst}$ and $Z_{Ifirst}$. The corresponding dimension of $H$ is therefore $n\times 2J$.

Second-Step: Same as the first step, we perform K-fold cross-validation on each of $j$ base classifier results with $H$ coming from First-Step to obtain the output $Z_{second}$ formally defined as follows:

where, $w_{Kn}$ represent the $i$-th prediction inferred by the $k$-th base classifier. The corresponding dimension of $Z_{second}$ is therefore $n\times J$.

Therefore, to get the final model prediction, we trained a meta classifier $f$ on $Z_{second}$, which can be formulated as :

Specifically in this study, we have chosen Support Vector Machine with linear kernel(LinearSVM) as our meta classifier.

To maximize the performance of the PERS framework, it is of a crucial importance to choose a set of suitable machine learning algorithms as base classifiers. The previous studies  (Amirhosseini and Kazemian, 2020; Qi et al., 2020; Farseev et al., 2015) suggest XGBoost (Chen and Guestrin, 2016), LightGBM (Ke et al., 2017), and Random Forest to be the top-choice base models for user profiling, as their performance on social media data has been reported to beat state-of-the-art baselines, often including those baselines coming from Deep Learning community.

Due to the imbalanced distribution of personality labels in our Datasets (see Section 3.1 for details on the Data Distributions), for performance evaluation, we have adopted the ”$F_{1macro}$” metric (Farseev et al., 2015), which is the harmonic mean between precision and recall, and the average is calculated per label across all labels. The $F_{1macro}$ metric is formally defined as:

where $p_{j}$ and $r_{j}$ are the precision and recall for $\lambda_{j}$ $\in$ h($x_{i}$) from $\lambda_{j}$ $\in$ $y_{i}$.

We have further adopted the  Matthews correlation coefficient metric  (Matthews, 1975) (Mcor), as it incorporates both true and false positives and negatives and generally regarded as a ”balancing” measure that can be used even if the classes are of a very different size. The Mcor metric is formally defined as:

where TP is the number of true positives, TN the number of true negatives, FP the number of false positives and FN the number of false negatives.

We prioritize the $F_{1macro}$ score as our main evaluation metric, while the Mcor score plays an auxiliary role for making decisions regarding performance when the $F_{1macro}$ values are marginal.

To judge the applicability of PERS framework in a real-world scenario, we have evaluated the limits of PERS’s performance across Twitter, Facebook, and PersonalityCafe datasets. The evaluation results are presented in the Table 4.

From the table, it can be seen that, being trained on the multi-view data from Twitter, PERS framework is able to achieve an industry-level performance of $0.82$ $F1_{macro}$ score when predicting the Extroversion-Introversion (EI) personality trait. While the performance obtained for the other three personality categories is significantly lower (ranging from -0.18 $F1_{macro}$ score for Judging-Perceiving(JP) to -0.28 $F1_{macro}$ score for Sensing-Intuition(SN)), we believe that such high promising performance for the EI label could testify the tremendous potential of multi-view social media data for psycho-graphic discovery and personality profiling. The superiority for the EI label, at the same time, can be explained by the natural difference of these two human personality categories when it comes to user communication on social platforms: Extroverts are known to be much more open to others, while Introverts - opposite, being more selective and making decisions at a slightly more conservative pace. Such an inspiring results allow us to give a positive answer to our RQ1 and possibly gives birth to a wide range of new research directions related to Personality Profiling and Multi-View learning.

But what about other two data sets? An interesting finding comes from the results presented in Table 5, where PERS demonstrates a breakthrough performance based on PersonalityCafe dataset showing the best overall $F1_{macro}$ scores when predicting all $4$ binary MBTI categories. Such a result can be explained by the specific nature of the PersonalityCafe dataset, where users purposely reveal their behavioural differences and therefore often biased towards particular social behaviour concepts. Such results also Reassure our positive answer to the Q1 and allow us to conclude that indeed the nature of a data source and the social network use patterns are of the crucial importance when solving the multi-view cross-media personality profiling problem.

First, we have investigated the contribution of different data modalities towards personality profiling performance and its integration ability. An interesting observation comes from the cross-modal experimental results presented in Table 4: the PERS framework have performed 2% better than other single-source baselines for all but SN binary labels being trained on Twitter and Facebook datasets. Another interesting observation can be made from the modality combination results, where being trained based on both textual and image data, PERS is able to outperform by more than 1% not just other single-modal classifiers but also the early-fused baselines.

The above findings suggest that the introduction of multi-modality into user profiling could serve as a powerful booster of model performance. Such observation could be explained by the richness of visual data when reflecting user preferences, which serves as a greatly beneficial supplement of the textual data modality at the data modeling stage. The latter finding confidently positively answers our RQ2 by emphasizing the important role of multi-modal data learning for personality profiling application.

Finally, let us also highlight an interesting observation that comes from single-modal evaluation results (see Table 4). It is important to note that, in the cases of learning from single-modal source, ”PERS” being trained on text-modality performs best across all personality labels, ranging from $0.02$ to $0.28$ $F1_{macro}$ score superiority level. The latter can be easily explained by the quantitative domination of textual data over visual modality (see Table 1). Another potential reason behind such trend could be the high level of noise in the user-generated visual data, where the images are less strict in terms of perspective and object positioning as compared to professional photos. Moreover, such visual content often includes objects that might not directly reflect the semantics of the data and therefore might be not accurate in representing an author’s personality. To this end, such hypothesis also aligns well with our-chosen visual data representation approach, where ImageNet concept distribution might be simply too general for personality profiling tasks, as opposed to, for example, demographic profiling (Farseev et al., 2015).

At last, let’s examine the impact of the social media data origin on personality user profiling performance, so that an industry guideline can be established for future research.

As the textual modality has participated in all three data sources, let’s describe the PERS performance on textual data first. From the Table 4 and Table 5 it can be noticed that PERS framework being trained on Twitter dataset outperformed the performance on Facebook dataset and PerosnalityCafe dataset by more than $0.19$ $F1_{macro}$ score in predicting EI label. In the contrast, when it comes to the SN label, Twitter-trained PERS was not able to outperform Facebook and PersonalityCafe data, staying behind by $0.2$ and $0.11$ $F1_{macro}$ score, respectively. Finally, the PERS performance of TF and JP labels based on Twitter textual data was found to be better than Facebook by $0.03$ and $0.05$ $F1_{macro}$ score, respectively, but considerably worse than PersonalityCafe by $0.19$ and $0.11$ $F1_{macro}$ score, respectively.

The superiority of Twitter in predicting the EI label could be explained by the differences of the ”energy” source for Extroverted and Introverted personality types. Precisely, according to Martin (Martin, 1997), Extroverts prefer to source their life energy from active involvement in events and engaging into different activities, while Introverts often prefer doing things alone obtaining their energy from dealing with the ideas, pictures, memories, and reactions that are inside on their mind. Similarly, from the digital world, it can be seen that on Twitter both personality types are able to express themselves fulfilling both their enjoyment (ENJ) and observation/learning (LEN) needs, while for Facebook ENJ factor got fulfilled proportionally for a smaller number of individuals, affecting the overall user base distribution (Syn and Oh, 2015). Correspondingly Twitter and PersonalityCafe data are diverse enough to differentiate the EI personalities achieving a higher prediction scores, as compared to Facebook-based prediction. The observation is also supported by our data distribution (see Section 3.1), where Twitter and PersonalityCafe datasets are clearly skewed towards Introverts, proving more data for PERS to learn on how the personality type direct their energy and make decisions. The latter aspect is important as it is known that Extroverts might generate substantially more UGC as compared to Introverts (Syn and Oh, 2015) and therefore Introvert-crafted content sufficiency is crucial for mutually-consistent and comprehensive learning from the data.

At the same time, an inverse picture can be noticed for the SN label results, and there is a ”low hanging fruit” explanation of the phenomena: for both Twitter and Facebook the SN personality is distributed with a clear shift towards Intuitive personality type, while being short on Sensing individuals in the data. Despite reflecting the real life distribution, this data property also entails a possible technical issue of variance insufficiency limiting the model differentiation when it comes to learning the Sensing and Intuitive user personas. Considering that on Facebook and PersonalityCafe there were more Sensing personality types identified, it is reasonable to assume that this is also the reason why PERS have performed better on these latter two sources as compared to the former one. To the end, a more ”sensing” Facebook can also be explained by the fact that Facebook is mainly treated nowadays as a communication tool so that people land there for fulfilling their daily communication needs, while Twitter more often serves as a source of Inspiration attracting even more Intuitive individuals into its nets (Syn and Oh, 2015).

Now, its time to compare the visual modality performance and the first thing that might capture a reader’s attention is that the image data modality has performed similarly for the cases of TF and JP labels for both Twitter and Facebook sources, however, at the same time Facebook performed better for EI and SN labels with $0.06$ and $0.03$ $F1_{macro}$ score performance uplift, respectively. As it has been described earlier, both the personality categories are very different in the way they direct the energy and perceive the external world (Martin, 1997) and therefore the data diversity introduced by incorporation of the visual modality is of crucial importance for the PERS performance. As Twitter is a ”less visual” data source as compared to Facebook and also its data distributions are less balanced (as discussed above) for both personality labels, it is reasonably to assume that these two factors might entail the superiority of Facebook over Twitter on visual data in our particular case.

Finally, it is worth noting that PERS trained only based on the textual data from PersonalityCafe forum outperforms results obtained from both Twitter and Facebook data by at least 0.1 $F1_{macro}$ score. Such a finding can be easily explained by the precise focus of the PersonalityCafe forum on the personality topic, which provides additional meaningful data descriptors which can be utilized by PERS for improving its personality inference score.

Backed up by all the observations above, we now can give an answer to the RQ3 by highlighting the drastic difference of Social Media data sources when being used in automated personality profiling, which is dictated by the way different personalities engaged into social network activities.