SOLVER: Scene-Object Interrelated Visual
Emotion Reasoning Network

Earlier works on VEA mainly focused on designing hand-crafted features to mine emotions from affective images. Inspired by psychology and art theory, Machajdik et al. [13] extracted specific image features and combined them to predict emotions, which consisted of color, texture, composition and content. Borth et al. [17] introduced Adjective Noun Pairs (ANPs) and proposed a visual concept detector, i.e., Sentibank, to filter out visual concepts strongly related to emotions from a semantic level. Considering both bag-of-visual word representations and color distributions, Siersdorfer et al. [15] proposed an emotion analysis method based on information theory. By leveraging object detection and concept modeling methods, Chen et al. [45] first recognized the top six frequent objects, i.e., car, dog, dress, face, flower, food, and then modeled the concept similarity between those ANPs. In order to understand the relationship between artistic principles and emotions, Zhao et al. [14] proposed a method for both classification and regression tasks in VEA by extracting principle-of-art-based emotional features, including balance, emphasis, harmony, variety, gradation, and movement. Besides, Zhao et al. [16] extracted low-level generic and elements-of-art features, mid-level attributes and principles-of-art features, high-level semantic and facial features, followed by a multi-graph learning framework. While these methods have been proven to be effective on several small-scale datasets, the hand-crafted features are still limited in covering all important factors in visual emotions.

Recently, with the great success of deep learning networks, researchers in VEA have adopted Convolutional Neural Network (CNN) to predict emotions and have achieved significant progress. Based on their previous SentiBank [17], Chen et al. [18] implemented deep networks to construct a visual sentiment concept classification method named DeepSentiBank. Leveraging half a million images labeled with website meta data, You et al. [24] proposed a novel progressive CNN architecture (PCNN) to predict emotions. Rao et al. [19] constructed a multi-level deep representation network (MldrNet), which extracted emotional features from image semantics, aesthetics and low-level visual features simultaneously through multiple instance learning framework. Earlier attempts simply implemented a general CNN to extract holistic features from affective images, neglecting the fact that visual emotions can also be evoked by local regions. Different from previous methods, You et al. [46] utilized attention mechanism to discover emotion-relevant regions, which serves as a prior attempt to focus on local regions in VEA. Similarly, Yang et al. [20] constructed a local branch to discover affective regions by implementing the off-the-shelf detection tools. A weakly supervised coupled network (WSCNet) [21] was further proposed by Yang et al., which discovers emotion regions through attention mechanism and leverages both holistic and regional features to predict emotions in an end-to-end manner. Besides, by employing deep metric learning, Yang et al. [22] proposed a multi-task deep framework for tackling both retrieval and classification tasks in VEA. Zhang et al. [47] proposed a novel CNN model to extract and integrate content information as well as style information to infer visual emotions. Considering different kinds of emotional stimuli, Yang et al. [32] proposed a stimuli-aware VEA network together with a hierarchical cross-entropy loss. Most of the existing deep learning methods directly used holistic feature or regional features to predict visual emotions. However, considering the complexity and abstractness involved in the cognitive process of human emotions, we argue that a direct mapping may underestimate the wide affective gap between low-level pixels and high-level emotions.

Psychological studies have demonstrated that visual emotions are evoked by the interactions between objects and objects as well as objects and scenes [29, 30, 31]. Based on these observations, we propose a novel Scene-Object interreLated Visual Emotion Reasoning network (SOLVER) to predict visual emotions. To mine the emotional relationships between different objects, we first construct an Emotion Graph and then conduct GCN reasoning on it to yield emotion-enhanced object features. Besides, a Scene-Object Fusion Module is further proposed to effectively interrelate objects and scenes with a novel scene-based attention mechanism.

We construct our Emotion Graph by setting objects as nodes and building up emotional relationships between them based on previous detection results as shown in Fig. 4. For word embedding, following the recent work [67, 64], we employ Global Vectors for Word Representation (GloVe) [74], an unsupervised learning algorithm to obtain vector representations for words. Thus, semantic concepts $S=\left\{{{s}_{1}},{{s}_{2}},\dots,{{s}_{N}}\right\}$ are embedded to semantic features $\mathbf{O}=\left\{{\mathbf{o}_{1}},{\mathbf{o}_{2}},\dots,{\mathbf{o}_{N}}\right\}$, where ${\mathbf{o}_{i}}\in{{\mathbb{R}}^{d_{2}}}$ and $d_{2}=300$. Notably, towards a specific object (e.g., cat, tree, balloon), visual features may vary from instance to instance while semantic features always remain unchanged. Thus, instead of visual features $\mathbf{V}$, we adopt semantic features $\mathbf{O}$ as nodes to construct the Emotion Graph, aiming to map objects to emotions with one-to-one relationships. Oppositely, we build up edges based on visual features $\mathbf{V}$, so as to depict diverse emotional relationships between distinct objects. Considering the existing gap between semantics and emotions, we first embed visual features $\mathbf{V}$ from semantic space to emotional space using a linear function followed by a non-linear $\ell_{2}$-norm function:

which may bring a richer description towards emotional space. In Eq. (1), $i\in\{1,2,...,N\}$, ${\mathbf{W}_{e}}\in{{\mathbb{R}}^{N\times N}}$ is a learnable embedding matrix, and ${\mathbf{b}_{e}}\in{{\mathbb{R}}^{N}}$ is a learnable embedding bias.

Therefore, the emotional visual features are denoted as ${\mathbf{V}^{e}}=\left\{{\mathbf{v}_{1}^{e}},{\mathbf{v}_{2}^{e}},\dots,{\mathbf{v}_{N}^{e}}\right\}$, in which ${\mathbf{v}_{i}^{e}}\in{{\mathbb{R}}^{d_{1}}}$. Following [63, 75], we build up adjacency matrix by calculating affinity matrix to construct pairwise emotional relationships between objects for the Emotion Graph:

where $i,j\in\{1,2,...,N\}$, $\mathbf{v}_{i}^{e}$, $\mathbf{v}_{j}^{e}$ represents two emotional visual features, and ${{r}_{i,j}^{e}}$ denotes the emotional relationships between them. Notably, $\phi\left(\cdot\right)$ and $\varphi\left(\cdot\right)$ are two embedding functions with different parameters. Since visual features ${\mathbf{V}^{e}}$ may vary greatly under the same emotion, we further embed them with different parameters to eliminate the bias of distinct features, hoping that they would be more comparable in the emotional space. The settings in Eq. (2), i.e., two functions with different parameters, is further ablated in Section IV-D1.

Our Emotion Graph is eventually constructed as ${\mathcal{G}_{E}}\!=\!\left(\mathcal{V},\mathcal{E}\right)$ where nodes $\mathcal{V}$ denote the objects with their semantic features:

and edges $\mathcal{E}$ represent the emotional relationships between different objects, which are described by affinity matrix:

which means there will be an edge with high affinity score connecting two objects if they have strong emotional relationships and are thus highly correlated. In order to remove redundancy of object nodes, we set a confidence threshold of 0.3 for $P=\left\{{{p}_{1}},{{p}_{2}},\dots,{{p}_{N}}\right\}$ to filter out less confident ones from the $N$ nodes, resulting in filtered nodes:

where ${\mathbf{o}_{i}^{\prime}}\in{{\mathbb{R}}^{d_{2}}}$ and $d_{2}=300$. Notably, not only do we remove redundant nodes, but we also delete adjacent edges of these nodes through masking operations as shown in Eq. (8), where $\mathrm{sgn(\cdot)}$ denotes sign function and $\mathrm{abs(\cdot)}$ denotes absolute value function. After applying the above two functions, ${m}_{i,j}$ outputs either 1 or 0, suggesting whether or not edge ${r}_{i,j}^{e}$ still exists after the masking operation:

where $i,j\in\{1,2,...,N\}$ and $\mathrm{max(\cdot)}$ takes the maximum element of ${\mathbf{o}_{i}^{\prime}}$ and ${\mathbf{o}_{j}^{\prime}}$. We then apply mask matrix $\mathbf{M}$ to affinity matrix $\mathbf{R}^{e}$ and obtain the masked affinity matrix ${\mathbf{R}^{e}}^{\prime}$ as

where $\odot$ denotes the element-wise multiplication. The validity of mask operation is further proved in Sec. IV-D1. In order to depict emotional relationships between objects, the Emotion Graph ${\mathcal{G}_{E}}\!=\!\left(\mathcal{V},\mathcal{E}\right)$ is eventually built up with $\mathbf{O}^{\prime}$ as nodes and ${\mathbf{R}^{e}}^{\prime}$ as edges and further propagates information to yield emotion-enhanced features in Sec. III-B2.

In order to mine the emotional relationships between different objects, we apply Graph Convolutional Network (GCN) [70] to perform reasoning on the Emotion Graph, which is capable of exchanging and propagating information through structured graph. In traditional GCN, as shown in Eq. (10), adjacency matrix is denoted as $\mathbf{A}$ while degree matrix is denoted as $\mathbf{D}$, where symmetric normalized Laplacian matrix ${{\widehat{\mathbf{D}}}^{-\frac{1}{2}}}\widehat{\mathbf{A}}{{\widehat{\mathbf{D}}}^{-\frac{1}{2}}}$ depicts the relationships between different nodes:

where ${{\mathbf{H}}^{\left(l\right)}}$ denotes the node features of the l-th layer and $\sigma(\cdot)$ denotes a nonlinear activation function. Different from the supervised graph learning tasks, we need to establish a set of rules to calculate edges of the Emotion Graph, which is described in Eqs. (1) (2) (9). Following similar update mechanism with Eq. (10), our GCN layer is defined as

where ${{\mathbf{O}^{\prime}}^{\left(l\right)}}$ denotes the input node features of the l-th layer and ${{\mathbf{R}}^{e}}^{\prime}$ denotes the input edge features. Notably, we add a residual block in our GCN following [63] to better maintain the original node features, drawing lessons from residual block in ResNet. For the l-th layer, $\mathbf{W}_{g}^{(l)}\in{{\mathbb{R}}^{N\times N}}$ is the weight matrix of GCN and $\mathbf{W}_{r}^{(l)}\in{{\mathbb{R}}^{N\times N}}$ is the weight matrix of the residual block. It can be inferred from Eq. (11) that each node is updated based on both its neighbors and itself, from which object features propagate throughout the whole Emotion Graph under their emotional relationships. We conduct reasoning on our Emotion Graph by applying several GCN layers to update object features iteratively:

where $l\in\{1,2,...,L\}$. The output of the final GCN layer ${{\mathbf{O}^{\prime}}^{\left(L\right)}}\in{{\mathbb{R}}^{N\times d_{2}}}$ is regarded as emotion-enhanced object features, as object features are iteratively updated under their emotional relationships. In our experiment, L is set to 4, which is further ablated in Sec. IV-D2. Therefore, we successfully construct the Emotion Graph and conduct reasoning on it to correlate objects with their emotional relationships.

As shown in TABLE III, we conduct ablation study on FI dataset, aiming to verify the effectiveness of each proposed module. Our SOLVER mainly consists of two modules: Emotion Graph (i.e., object-object interaction) and Scene-Object Fusion Module (i.e., scene-object interaction), as shown in the first column. In each module, there are some detailed network designs, which are depicted as ablated combinations in the second column in TABLE III. In the Emotion Graph, we first conduct ablation studies concerning a single object and multiple objects, which indicates that multiple objects bring a performance boost compared with a single one. After that, we introduce our GCN reasoning mechanism with detailed designs into comparisons, i.e., one/two embedding(s), w/wo mask, suggesting that the object-object interactions indeed improve the performance in predicting emotions. From the above experiments, it is obvious that two embedding functions with different parameters achieve better performance than one embedding function and mask operation can further bring a performance boost. In the Scene-Object Fusion Module, it is obvious that both scene branch and scene-based attention mechanism make a great contribution to emotion classification, which suggests that it is the scene that guide object fusion process. From the above ablation studies, we can conclude that each detailed design of the proposed method is complementary and indispensable, which jointly contributes to the final result.

We conduct experiments to validate the choice of node number $N=10$ and the layer number $L=4$ in our Emotion Graph, as shown in Fig. 7 and TABLE IV. In general settings of Visual Genome dataset [73], top-36 RoIs are selected from the pre-trained object detector. However, we believe that such a large number of nodes can be redundant in our Emotion Graph and thus further conduct experiments to figure out the optimal $N$, balancing both accuracy and computational cost. To be specific, we design experiments on both scene-based attention mechanism and the GAP layer, aiming to reduce the randomness in a single experiment. Setting $N=2$ as the step value, $N$ is varied from 0 to 20. We find that the accuracy constantly grows as $N$ varies from 0 to 10, while it slightly drops after $N=10$, due to the redundancy of RoIs in object detection. It is worth mentioning that the growth of accuracy from 0 to 10 further proves that visual emotion analysis is not about a single object, but the interactions between multiple objects. From the above analysis, we choose $N=10$ as the node number in our Emotion Graph. For GCN layers, we ablate the number of GCN layers for better illustration. Setting $L=1$ as the step value, L is varied from 1 to 8. It is obvious that the accuracy consistently grows as the GCN layers increase, contributing to the excellent reasoning ability GCN owns. However, the performance meets a bottleneck when the number of GCN layers is too large, which may be caused by over fitting of too many parameters.

Considering that there exist relationships between objects, scenes and emotions, we visualize the top-10 emotional object concepts for each category. After implementing Faster R-CNN on FI dataset, we first calculate the frequency ${{f}_{c,i}}$ for each object ${i\in\{1,2,...,I\}}$ in each emotion category ${c\in\{1,2,...,C\}}$, where $I$ denotes the overall object number and $C$ denotes the emotion categories in the whole dataset:

where ${{f}_{c,i}}\in\left[0,1\right]$ and ${{N}_{c,i}}$ represents the count of object ${i}$ in category ${c}$. Subsequently, we derive the scene-based attention coefficient ${{a}_{c,i,j}}$ from the trained model and calculate the averaged attention coefficient ${{a}_{c,i}}$ for each object ${i}$ in each emotion category ${c}$:

where ${j\in\{1,2,...,{N}_{c,i}\}}$ denotes the j-th instance of object $i$ in category $c$. While object frequency ${{f}_{c,i}}$ represents the objects distribution of each emotion in the dataset, scene-based attention coefficient ${{a}_{c,i}}$ represents the correlations between objects and emotions learned by our SOLVER. Taking both object frequency and scene-based attention coefficient into account, we obtain the weighted frequency ${{w}_{c,i}}$ for each object ${i}$ in each emotion category ${c}$:

It is obvious that not all the objects are emotional, some non-emotional objects appear in every emotion category, e.g., man, woman, people, etc. Thus, we adopt the TF-IDF [86] technique to separate the emotion-specific objects from the non-emotional ones, where the importance of an object increases as it appears in a specific emotion category and decreases inversely with its appearance in the whole dataset. As shown in Fig. 8, we list the top-10 emotional object concepts for each emotion category with their corresponding weighted frequencies. Besides, we present three typical images for each emotion to further illustrate these emotional object concepts in a concrete and vivid form. Taking awe as an example, there are mountains, castle, ocean in the first image, cliff, horizon, sunset in the second one, and ocean, wave, shore in the third one, which indicates that interactions between these emotional objects may evoke awe to a large extent. The most relevant concepts towards excitement include raft, surfboard, plane, microphone, player, etc., which implies sports events or entertainments and consequently evokes excitement.

Besides emotional object concepts, we also visualize emotional object regions by presenting one image for each emotion category as a representative. As shown in Fig. 9, after implemented the proposed SOLVER, an affective image (i.e., the upper one) is turned into a set of emotional object regions (i.e., the lower one) with corresponding semantic concepts and attention weights. Notably, we first adopt object detectors to refine emotions to object level and then mine the interrelationships between objects and emotions with scene-based attention mechanism, which takes the scene feature as a guidance to fuse objects. Attention weights reflect the relevance between objects, scenes and emotions. The more an object is related to a specific emotion, the higher the attention weight will gain. For example, in amusement, tower (0.957), building (0.938) get higher attention scores while cloud (0.764) gets a lower score, as when tower and building comes together, they often indicate castle and amusement park, which surely bring people joy. When people or other animals are angry, the most striking part of their facial expression is an opened mouth. Thus, angry is represented by an furious leopard with a distinguished opened mouth, and accordingly our SOLVER focuses more on its mouth (0.851) than its head (0.684), eyes (0.654).

LUCFER [87] contains over 3.6M images with 3-dimensional labels including emotion, context, and valence, which is currently the largest emotion recognition dataset. With a sum of 18,316 images, EMOTIC [88] serves as a pioneer dataset in emotion state recognition (ESR) field, which is labeled with both discrete categories (i.e., 26 defined emotions) and continuous dimensions (i.e., valence, arousal, and dominance (VAD)). Containing 500,000 images in total, Flickr CC [17] is one of the largest visual sentiment datasets labeled with 1,553 fine-grained ANPs (Adjective Noun Pairs).

Since SOLVER is oriented to single-label emotion classification tasks, we preprocess the three potential datasets in distinct and reasonable ways to fit our task. According to the rules in LUCFER, we degenerate the 275 fine-grained emotion-context pairs into eight basic emotions (i.e., Anger, Anticipation, Disgust, Fear, Joy, Sadness, Surprise, and Trust) in Plutchik’s wheel. Besides, we group the eight emotions into two sentiments (i.e., positive, and negative) according to their polarities. Following the same setting in [87], LUCFER is split into training set (80%) and testing set (20%). Each image in EMOTIC is annotated with multiple emotion labels, which is different from our single-label classification task. Besides, it is irrational to degrade a multi-label task to a single-label one. Fortunately, valence in VAD measures the positive degree of an emotion, where $V\in[5,10]$ corresponds to positive and $V\in[0,5]$ negative. Thus, we degenerate valence to sentiment labels (i.e., positive, and negative) according to psychological model. EMOTIC is split into training set (70%), validation set (10%), and testing set (20%) [88]. Based on Visual Sentiment Ontology (VSO) [17], we degenerate its 1,553 ANPs into sentiment labels (i.e., positive, and negative). The Flickr CC dataset is split into training set (80%) and testing set (20%), which follows the same setting in [17].

We conduct experiments in classification accuracy on three potential datasets in TABLE V and TABLE VI. Specifically, results using models trained on FI dataset are reported in TABLE V, and results using models trained on different datasets themselves are shown in TABLE VI. In particular, LUCFER is conducted with two experimental settings, eight emotions (i.e., LUCFER-8) and two sentiments (i.e., LUCFER-2), for richer comparisons and analyses. In order to validate the effectiveness of our method and to explore the emotional diversity in different datasets, we ablated our experiments to three settings including Scene branch, Object branch, and SOLVER.

In TABLE V, it can be noticed that the FI trained model achieves relatively good results on LUCFER-2 (69.38%), while meets unsatisfactory results on LUCFER-8, EMOTIC-2 and Flickr CC-2 (14.69%, 36.13%, and 36.86%). Dataset shift is a common problem in machine learning. Since emotions are complex and ambiguous to be described and labeled, this gap becomes even larger. Besides, VEA aims at finding out how people feel emotionally towards different visual stimuli, while ESR focuses on recognizing peoples’ emotional states from their frames. The mismatches between these two areas may lead to a performance degradation in TABLE V. Compared with the other 2-sentiment tasks, LUCFER-2 achieves quite a good result, which attributes to the well-constructed dataset with more emotional stimuli and less noise. It is obvious that Object branch performs generally better than Scene branch on LUCFER. The possible reason may be that there are distinct and obvious objects conveying emotions, while scene features are not clearly distinguishable to separate different emotions apart. The three results on EMOTIC dataset are close to each other, which indicates that our SOLVER failed to find key features related to ESR. Moreover, we guess that the key features concerning ESR may be facial expressions or body language, which our method does not involve. Similarly, SOLVER performs poorly on Flickr CC, resulting from its very large data scale compared with FI. Besides, most images in Flickr CC contain a distinct object or a simple scene, which our SOLVER may fail to match with.

For fair comparisons with datasets in Sec IV-C, we also report the results using models trained on three potential datasets and tested on themselves in TABLE VI. Compared with TABLE V, it is obvious that the corresponding results in TABLE VI have significant performance improvements, which are comparable to the results in TABLE II. In TABLE VI, it is undoubted that LUCFER is a well-labeled dataset, whose 8-emotion accuracy (75.59%) is higher than others’ 2-sentiment ones (65.07%, and 71.68%). TABLE VI shows that SOLVER constantly outperforms the other ablated branches on all experimental settings, which further validates the effectiveness and the robustness of each proposed branch.

From TABLE V and TABLE VI, we can conclude that SOLVER achieves competitive results on three potential datasets. Meanwhile, we also noticed some limitations of our method. Firstly, when facing ESR tasks, SOLVER fails to leverage the facial expressions as well as body language of people within the images, resulting in degraded performance. Secondly, SOLVER is proposed to mine the emotional correlations between objects and scenes, which fails to deal with the situation of a single object or a simple scene.

In addition to classification accuracy, we also visualize emotional object regions on potential datasets from different trained models (i.e., FI and themselves). In Fig. 10, we present the visualizations of emotional object regions on LUCFER. While the first row represents the input images, the second row uses model trained on FI and the third row is trained on LUCFER itself. By analyzing the difference in attention weights, we discover some possible reasons for the performance improvements in TABLE VI compared with TABLE V. Take the first image as an example, while the second row focuses more on grass, tree, and other objects, the third row attends more weights on man, shirt, and hair. It is obvious that FI trained model concentrate more on objects unrelated to people while LUCFER trained model emphasizes more on human-centered ones, which further shows the robustness of our method. As ESR datasets are constructed by images all containing people, it is understandable that increased attention weights on people and their related objects may bring a performance boost in such datasets. As is mentioned above, there are some similarities in VEA and ESR tasks yet much more differences in details. The visualizations also prove that emotions are always evoked by people within the images in ESR datasets, especially their facial expressions and body language, which is different from our VEA task.

Fig. 11 shows three types of failure cases on LUCFER. In Fig. 11 (a), two images in the first column share the same scene (i.e., stadium), and two images in the second column share the same object (i.e., tennis). However, under the same scene/object, emotions are varying according to different facial expressions and body language. Take first row as an example, by observing the waving fists and the confident smile, the coach is angry with his players while the player is satisfied with herself Since SOLVER mines emotions from the interactions between objects and scenes, facial expressions and body language are not considered in our method. This failure is mainly caused by the mismatch between VEA and ESR, which will be considered in our future work. In Fig. 11 (b), we can see that people in an image do not necessarily share the same emotion. Take the second row as an example, the woman seems happy while the man is angry. While ESR concentrates on the emotions of characters within the image, our VEA focuses on the emotions of viewers outside the image. This failure also results from the mismatch between two tasks. Besides, there are also some images where emotions may be weak and obscure as in Fig. 11 (c). In other words, these images hardly arouse any of our emotions. This failure is universal in both tasks. We may find a reasonable way to separate emotional images and emotionless images in our future work.