Detecting Personality and Emotion Traits in Crowds from Video Sequences^††thanks: Thanks to Office of Naval Research Global (USA) and Brazilian agencies: CAPES, CNPQ and FAPERGS.

Initially, the information about people from real videos is obtained using a tracker Bins2013 to recover people trajectories. In order to transform image in world coordinates, we assume that the head position is on the ground place ($z=0$), and then we used a planar homography to rectify a perspective image and generate world coordinates. In computer vision, planar homography is defined as a projective mapping from one plane to other. Then we use the homography to rectify a perspective image, in this case, to generate a “plan” view of trajectories from a “perspective” photo. Figure 3 illustrates the mapping of the trajectories points from image coordinates to a orthogonal plan view (world coordinates). The coordinates of trajectories are used to calculate the motion parameters (speed of travel, direction) for each agent.

We compute information for each person $i$ at each timestep: i) 2D position $\vec{x}_{i}$ (meters); ii) speed $s_{i}$ (meters/frame); iii) angular variation $\alpha_{i}$ (degrees) w.r.t. a reference vector $\vec{r}=(1,0)$; iv) isolation level $\varphi_{i}$; v) socialization level $\vartheta_{i}$; and vi) collectivity $\phi_{i}$. The isolation level $\varphi$ is computed as shown in Equation 1:

For groups detection, we use the computed parameters $s$, $o$ and $d$ which respectively state for speed and orientation variation and distance of each pair of agents $i$ and $j$. We use the notion of distances based on the proxemics described by Hall Hall:1990 to define that two agents belong to the same group according with three tests empirically defined: If $(d(x_{i},x_{j})<=1.2\text{meter})$ and $(o(\alpha_{i},\alpha_{j})<=15^{\circ})$ and ($s(s_{i},s_{j})<\beta$), where $\beta=5\%$ of higher speed. Based on this set of rules, agents are grouped in pairs. In a next step, we check which pairs have one individual in common, and merge them into larger groups. This process is performed until the group formation does not share individuals, i.e. they are disjoint. For each group $g$ detected in video sequence, we define a vector $\vec{G_{g}}=\left[n_{g},\vec{I}_{g},\bar{s}_{g},\bar{\alpha}_{g},\bar{d}_{g}\right]$ of extracted data, where $n_{g}$ is the number of individuals in $g$, $\vec{I}_{g}$ states for a vector containing the id of each individual in $g$ and $\bar{s}_{g}$, $\bar{\alpha}_{g}$ and $\bar{d}_{g}$ present the average values of speed, orientation and distance applied by individuals in group $g$ during the video sequence.

At the end of this step, we have $\vec{V}$ for all individuals per frame and averaged for the video sequence and $\vec{G}$ for all groups averaged for the video sequence. In next section we describe how these data is used to find out the personality traits in our work.

We used the empirically equations proposed by Favaretto et al. favaretto:sib2017 in a previous work which goal is to map individual and group characteristics in OCEAN cultural dimensions. Basically, the method proposes to “answer” 25 items from NEO PI-R inventory for each individual in the video sequence. Although the complete version of NEO PI-R has $240$ items, 25 of them were selected since they have a direct relationship with crowd behavior. Each question is mapped to an equation using only information contained in $V$ for each $i$. For example, in order to represent the item “1 - Have clear goals, work to them in orderly way”, we consider that the individual $i$ should have a high velocity $s$ and low angular variation $\alpha$ to have answer compatible with “Strong”. So the equation for this item is $Q_{1}=s_{i}+\frac{1}{\alpha_{i}}$. Table 1 shows the equations used for each considered question.

Once all questions $k$ (in the interval $[1;25]$) have been answered for all individuals $i$, we have $\vec{Q_{i,k}^{f}}$ for each frame $f$. In addition, we computed the average values to have one vector $\vec{Q_{i,k}}$ per video. According to NEO PI-R definition, each of the questions $\vec{Q^{\prime}_{k}}$ are associated to one of the Big Five dimensions and some questions should invert the values, because an item score 4 (Strongly Agree) can represent a high or low value of a certain personality trait. So, to get the correct values, we applied a factor to the questions which score should be inverted: $\vec{Q^{*}_{i,k}}=4-\vec{Q^{\prime}_{i,k}}$, as shown in next equations:

where $\varrho$ represents the percentage of questions from the total, in each dimension (O, C, E, A and N), respectively 4%, 4%, 72%, 8% and 12%.

In this paper we connect personality and emotion traits in order to detect data in the individuals in video sequences. As mentioned by revelle2009personality :

”A helpful analogy is to consider that personality is to emotion as climate is to weather. That is, what one expects is personality, what one observes at any particular moment is emotion.” That is the reason why, in our work, we correlated emotion traits with visual behaviors that will be perceived in pedestrian motion.

Our work proposes an emotion mapping based on personality traits (i.e. OCEAN) found for each individual present in the video sequence. First, we selected four emotions from OCC ortony1990cognitive model: Fear, Happiness, Sadness and Anger. It is important to notice that we chose only four emotions that, in our opinion, are the most visible when relating with motion behavior, which is our case in video sequences with pedestrians. Any other from the total of 22 emotions proposed in OCC model could also be mapped.

Once we have computed the personality traits for an individual, we propose a way to map these personality traits into the four considered emotions. In addition to vector $\vec{V}_{i,f}$ for individual $i$ per frame $f$, we included personality and emotion parameters as follows: $\vec{V}_{i,f}=\left[s_{i,f},\alpha_{i,f},\varphi_{i,f},\vartheta_{i,f},\phi_{i,f},\vec{P}_{i,f},\vec{E}_{i,f}\right]$. $\vec{P}_{i,f}$ states for values of OCEAN personality $\left[O_{i,f},C_{i,f},E_{i,f},A_{i,f},N_{i,f},\right]$, while $\vec{E}_{i,f}$ states for values of emotion based on OCC model: $\left[F_{i,f},H_{i,f},S_{i,f},An_{i,f}\right]$. Values of $\vec{P}$ are in the interval $[0;1]$ while values of $\vec{E}$ are from $[-3;3]$. For groups analysis, we only considered parameters of personality and emotion in the detected groups $g$ in the video sequence. In addition to previously defined $\vec{G_{g}}=\left[n_{g},\vec{I}_{g},\bar{s}_{g},\bar{\alpha}_{g},\bar{d}_{g}\right]$, we include $\vec{P}_{g}$ and $\vec{E}_{g}$ containing the average values of $n_{g}$ individuals in $g$.

In order to map from OCEAN to emotion parameters we observe some aspects in literature  costa07 :

• •

  O- : person is close to interact with others;

• •

  O+ : person is aware of his/her feelings;

• •

  C+ : person is optimistic;

• •

  C- : person is pessimist;

• •

  E+ : person has a strong relationship with positive emotions;

• •

  E- : person presents relationship with negative emotions;

• •

  A+ : person has a strong relationship with positive reactions;

• •

  A- : person presents relationship with negative reactions;

• •

  N-: known by the emotional stability;

• •

  N+ : person feels negative emotions;

Such data resulted in empirical definitions included in Table 2, that shows the mapping from OCEAN traits to the chosen emotions. In fact, we were not the first one to propose this type of mapping. Saifi et al. saifi2016approach proposes similar data for a different approach where authors were interested in providing critical emotions in crowd simulators. Davis and Panksepp davis2011brain also proposed a similar approach unifying basic emotions with personality.

In Table 2, the plus/minus signals along each factor represent the positive/negative value of each one. For example, O+ stands for positive values (i.e. O $\geq$ 0.5) and O- stands for negative values (i.e. O $<$ 0.5)). A positive value for a given factor (i.e. 1) means the stronger the OCEAN trait is, the stronger is the emotion too. A negative value (i.e. -1) does the opposite, therefore, the stronger the factor’s value, the weaker is a given emotion. A zero value means that a given emotion is not affected at all by the given factor. To better illustrate, a hypothetical example is given: if an individual has a high value for Extraversion (for example, E = 0.9), following the mapping in Table 2, this individual can present signals of happiness (i.e. If E+ then Happiness= 1) and should not be angry (i.e. If E+ then Anger= -1).

Finally, emotions are normalized in the interval $[0;1]$ considering the lowest and highest achieved values in the video, in order to keep the obtained scale.

In this section, we present OCEAN analysis involving spontaneous videos (crowds in public spaces). Firstly, we calculate the OCEAN of each individual in the video at each frame. Once we get the individuals OCEAN, the group OCEAN is computed by the average of the individuals’ OCEAN that are part of the group. Figure 4 shows a representation of each individual OCEAN in a determined frame. We used five color box that represent the five dimensions, where blue is related to Openness, cyan indicates Conscientiousness, green indicates Extraversion, yellow means Agreeableness and red, Neuroticism.

The group of individuals highlighted in Figure 4 is composed of two people. The OCEAN of this group (named $G1$) over the time is illustrated in Figure 5(a), and obtained by the average OCEAN of its individuals ($P9$ and $P10$), presented in Figures 5 (b) and (c). As can be seen in such figures, the two individuals present more motion variation at the beginning of the video and then they keep the same motion characteristics until the end of the short movie (which duration is 100 frames). Their Openness is high because they keep low angular variation along their trajectories. On the other hand, their Conscientiousness dimension is lower than other dimensions because they keep low speeds in comparison to other groups.

The group OCEAN reflects the individuals OCEAN. An analysis of OCEAN in the country level is presented later in this section. Once we have the OCEAN values, individuals emotions are detected in the analyzed videos, as discussed in next section.

This section presents the emotion analysis in spontaneous videos. Figure 6 shows an example of the emotion detection in a video from Austria. A filled square represents that the person has a positive value from that emotion, a half filled square means that the emotion is neutral and a not filled square means that the person has a negative value from the emotion. For example, the highlighted person, with the blue arrow, got a negative value regarding the Anger emotion (the red square is not filled). It happens because this individual is interacting with the other (walking in the same direction and with similar angular variation), so collectivity and socialization levels are high and isolation level is low, consequently Anger receives a negative score.

Still in Figure 6, the highlighted person with the orange arrow gets a zero value for the Happiness emotion (the green square is half filled). It happens because she/he is alone, changing orientation (angular variation), with high isolation. The person highlighted with the red arrow gets a positive value for the Fear (the yellow square is completely filled). It happens because this individual is moving slower and with high angular variation, so his/her dimension C is low, generating a high value for Fear. The legend of colors is the following: red is related to Anger, yellow means Fear, green indicates Happiness and blue, Sadness.

In another example, showed in Figure 7(a), we highlight two different situations, a group (green circle) and an individual alone (red circle). It is interesting to notice that individuals who are part of a bigger group or have a high collectivity tend to be happy, as we can see in the highlighted group in Figure 7(b). On the other hand, individuals who are alone and distant from others tend to experience negative emotions (see an example in Figure 7(c)).

In the next experiment we compare our results with available literature costa07 about OCEAN values of different Countries. In the case of the present work we focus on Germany and Brazil, since in next section we also present a comparative about these two Countries. It is important to emphasize that the data registered in costa07 was acquired based on subject answers on surveys and not based on their visual behavior in video sequences. We use OCEAN dimensions of the two analyzed Countries (Brazil and Germany) as presented by Costa et al. costa07 as ground-truth in our approach. We evaluated our method in a set of $10$ available videos, where 2 videos were from Germany and 8 from Brazil.

We evaluate the accuracy of our approach to detect the OCEAN values of the Countries based on percentage difference when compared with the literature results, considering all dimensions among all videos. Figure 8 shows the results obtained with our approach for two countries in all OCEAN dimensions, in comparison with the literature costa07 . It is interesting to highlight that results achieved for Brazil showed a higher accuracy, when compared to Germany. In the same time, this was the country with more videos to be analyzed.

In the same way, we compute the emotions in the country level. For this, similar to the OCEAN approach, the emotion of each country is computed as the mean emotion from the videos from that country. Figure 9 shows the mean emotion from the analyzed countries in this experiment. As far as we know, there is no data to be compared in this matter.

In last sections we were interested about detecting personality and emotion traits in videos from different Countries, in public spaces. However, since the contexts are not the same, e.g. people can be close or faraway from others not just because their personality but because the contexts they are evolving on or the relationship with spaces are not the same. Consequently, this research presents such difficult challenge to solve in order to evaluate/validate results. The ideal is to discard the spacial context in order to analyze only the people behavior, from different Countries, while executing the same task. Therefore, we found the Fundamental Diagram applied to pedestrians, as proposed in Chattaraj:2009 . We detect personality and emotion traits in video sequences from two countries (Brazil and Germany ²²2We have access to such videos thanks to the authors of database of PED experiments, available at http://ped.fz-juelich.de/db/.), where all the individuals are performing exactly the same task. i.e. walking in a controlled space. The experiments have been performed in an environment setup as illustrated in Figure 10.

This experiment was conducted as described in Chattaraj:2009 , being the same in all the two countries, with the same populations ($N=15$, $25$ and $34$). The corridor was built up with markers and tape on the ground. Its size and shape is presented in Figure 10. The length of the corridor is $17.3m$, while the width of the passageway is $0.8m$, which is sufficient for a single person walk. In addition, we can observe on the bottom of Figure 10 a rectangle of 2 x 0.8 meters which illustrates the Region of Interest (ROI) where the populations were captured to be analyzed, as proposed in Chattaraj:2009 .

Figure 11 shows examples of emotion detection performed by individuals in the experiment of Fundamental Diagram. Pictures show the performed tests with different sizes of population in both Countries.

Figure 12 shows the OCEAN values from Brazil and Germany in the experiments with extreme sizes of populations, i.e. $N=15$ and $N=34$. Based only on a visual inspection, we can easily perceive that OCEAN values from the both Countries are more similar when the density of people is higher in the experiment. It can indicate that people assumes group-level behavior instead of individual-level behavior caused by the higher density and the lack of free space. It agrees with several theories about mass behavior as discussed in vilanova2017 and  LE_BON_THE_CROWD .

In this analysis, we also compute Pearson’s correlation to find out the similarity between the two countries in both OCEAN and emotion aspects. Figure 13 shows the Pearson’s correlation of OCEAN values among the countries, while Figure 14 shows the Pearson’s correlation of the emotion values.

Considering the both Figures 13 and 14, it is possible to see that, except from the population $n=25$ in OCEAN experiment, as the density of people in the experiment increases, the correlation among the countries also increases. We also included an investigation regarding the emotion and OCEAN impacted by the density of people. This was possible because the performed task is the same (i.e. individuals are walking in the same predefined environment) while only the number of people increases. Figure 15(a) shows how the emotion values vary according to the density of people in videos from Germany. It is interesting to see how the emotions Anger, Fear and Sadness increases proportionally as the density increases too. On the other hand, Happiness emotion decreases proportionally as a function of observed density. Indeed, the data observed in Germany was in accordance to what was empirically expected in our hypothesis, i.e. the only positive emotion (H) decreases as the density increases. However, the computed emotion for Brazil was not so well behaved as in Germany, and certainly it could be better investigated in a future work. One possible explanation is that Brazilian people were colleagues/friends while in Germany, they were related in Chattaraj:2009 as volunteers to the experiment.

Although we did not find any information about emotion and personality detection in videos from different Countries related in literature, these results seem promising in order to understand people personality, emotion and cultural differences in video sequences.

In this experiment we performed a comparison among the preferred distance people keep from others, as evaluated in a study performed by sorokowska:2017 . Results obtained with the experiment performed in our approach are compared with results of sorokowska:2017 , as shown in Figure 16. In the Sorokowska work, the answers were given on a distance (0-220 cm) scale anchored by two human-like figures, labeled A and B. Participants were asked to imagine that he or she is Person A. The participant was asked to rate how close a Person B could approach, so that he or she would feel comfortable in a conversation with Person B.

In our approach we measure the distances a person A keeps from a person B right in front of he or she, in the video sequences. For the comparison, in our approach, we use the distances from the Fundamental Diagram video sequences varying the people number ($N=15$, $N=20$, $N=25$ and $N=30$) and from the Sorokowska’s approach we select the evaluation from acquaintance people, where the people are not close neither strangers, similar to people in our experiment. As we can see in Figure 16, in spite of the fact that distances from our approach are higher than the ones from Sorokowska, when there are few people ($N=15$ and $N=20$), the proportion is similar in both scenarios. In addition, in the experiment $N=30$, the distances observed in our approach are close to those obtained in the Sorokowska study. In all scenarios, people from Brazil keeps higher distances from others than people from Germany. The same happens in Sorokowska’s research. Although they are different experiments (videos and surveys), our method indicates that there is a correlation between people feeling in abstracted environment (as in surveys) and as they behave in video sequences.