Label quality in AffectNet: results of crowd-based re-annotation

As our deep neural network (DNN) backbone, we used a standard, ImageNet pre-trained ResNet-50, which is one of the most widely used standard architectures in image classification tasks [20]. The architecture is based on 50 blocks of convolutional filters of size 3 X 3 with residual connections that allow the optimizer to skip a whole convolutional layer, thereby increasing the effectiveness of the total network.

Implementation details for training the ResNet50 architecture on AffectNet were as follows: for augmentation, we added intensity normalization of the color data into 0 to 1, as well as clockwise and anti-clockwise rotations of up to 10 degrees, affine shearing up to 0.1, as well as horizontal flips. Class weights (given the imbalance in label categories) were used for a weighted cross-entropy loss that was optimized with Adam at an initial learning rate of 0.0001.

This model was trained with 256x256px color images of the AffectNet training set of 287,561 images for 50 epochs and a batch size of 64, after which it reached a validation accuracy of 53.15% (ResNet50 50 epoch) - its best validation accuracy throughout the run, however, was 59.25% (ResNet50 Best), which is only less than two percent worse than the state-of-the-art in AffectNet on eight classes with a much more involved architecture[11]. To look at the early performance of this model, we also saved its snapshot after the first epoch (ResNet50 1 epoch).

Finally, we set up another training scheme that would perhaps be more akin to human learning, in which one epoch used only 512 randomly-chosen images, but we trained for much longer (1000 epochs). Interestingly, this very weakly-trained model achieved also a reasonable accuracy of around 53.5%. All models are compared in Table 1.

For valence and arousal, we followed the same model structure and created two regression models for each rating based on mean squared error (MSE) loss. These models were trained for 10 epochs (again, relatively weakly-trained), and resulted in comparable validation losses of 0.0177 (valence) and 0.0167 (arousal).

In total, there are 420,299 images in AffectNet. For our pilot exploration of label quality, we first pre-screened 100 images from each class to have a more manageable dataset size. To pick these images, we designed a HTML-based GUI that allowed us to quickly loop through all images for a class and select those exemplars that the trained annotator deemed ”confusing”.

For the experiment, we designed a software application based on HTML shown in Figure 2. For each image, participants were asked to make three annotations: in the first row, they were to click on one of the eight categories that best described the expression content of the image; the second and third row showed self-assessment manikins (SAM, [21]) for rating of valence and arousal at 9 different rating levels.

The experiment was conducted in a quiet room and supervised in real-time by the experimenter. Participants were first instructed about the experimental procedure and were familiarized with expression labels and valence and arousal ratings. The experimenter also demoed the experiment with one dummy picture to instruct participants about how to pick the expression, to rate valence and arousal, and to move on to the next picture. Unlike AffectNet, we did not constrain valence and arousal values to pre-defined ranges (see [15]), so as not to affect participants’ intuitive evaluations of the affective content.

We recruited a total of N=13 people. (7 female, 6 male, mean age (STD) = 33.23 (12.39) years) from the population of Korea University. Participants were naive as to the purpose of the experiment, and reported no neurological or psychological issues that would interfere with emotion processing.

The total number of votes added up to 10,400—as a first, omnibus result, we observed that of these votes, 8,658 (83.25%) did not agree with the original AffectNet labels, indicating that these labels may have issues.

Figure 3(a) shows the overall results for participants’ votes¹¹1In determining the final vote to be counted, we also experimented with different kinds of maximum (modal) voting, such as maximum vote across all votes, maximum votes for each image with different ways of breaking ties (counting the first tie, the second, or the third). We found that the average voting pattern was not affected by these methods, and hence used this pattern in the remainder of this paper. in more detail (as box plots). The original AffectNet data was uniformly distributed across the eight classes (Figure 3(a) orange line)—this is clearly not the case for the median votes, however: On average, participants chose a much higher proportion of neutral and happy labels and much lower proportions of contempt, fear, sad, and disgust. We also observed variability across expression categories in the voting patterns, especially for the neutral, and to lesser degrees for the happy and sad categories, indicating that there was less agreement among participants for these category labels.

The overall consistency of participants on voting was evaluated by taking each participant’s individual voting pattern per expression and correlating this with all other participants’ voting patterns. The average value of the upper triangular part of the resulting correlation matrix is a measure of relative consistency and was determined to be $r_{expr}=.550$, indicating medium to good consistency levels.

Figure 3(a) also shows the original number of images in each category in AffectNet (blue line, normalized). We can see that this distribution—as an extremely weak measure of a ”real-life” distribution of these facial expression categories—fits somewhat better to the median voting pattern (the correlation value for the human voting data to this data is $r=0.71$, see also Figure 8(b) below). This may be an indication that participants implicitly fitted their overall voting strategy to some sort of ”hidden distribution”; for example, one would expect that most of the time, faces display no strong facial emotion and hence will be neutral. The next most common expression may be happy followed by the other categories with contempt and disgust being perhaps the least commonly-encountered expressions.

Figure 3(b) displays a type of confusion matrix, which shows how the original AffectNet labels and the maximum vote labels are connected. As can be seen, the matrix does not exhibit any diagonal structure owing to the large number of differing votes we received. Many contempt images, for example, got relabelled as happy or neutral. At the same time, and in line with the results in Figure 3(a), many images originally annotated with an emotional label received votes for the neutral expression.

Overall, our results so far on the 800, pre-screened images suggest that the original expression labels in AffectNet for these difficult images are far from ideal.

To better compare our data with those from AffectNet, we mapped the 1-9 scale into an -1.0 to 1.0 range. Figure 4a) shows the full data of our experiment with our arousal and valence ratings colored by the maximum vote labels. First of all, we see a clear ”rotated U”-shape that is reminiscent of the data from affective ratings obtained for the International Affective Picture System (IAPS, [22])—in other words, as valence increases positively and negatively arousal increases into the positive direction in both cases. This pattern is a well-established result for affective ratings of pictures and videos [22, 17].

In this context, it is important to mention that the AffectNet annotations for valence and arousal were collected in a different way compared to ours: their annotation scheme used a software application in which the annotator clicked into a pre-defined region within a two-dimensional coordinate system (see [15]). In our case, the two annotations were obtained in a more ”traditional” fashion using the SAM method and without restricting participants to certain value ranges, depending on the label, which may explain some of the differences in rating pattern.

One of the reasons for the different annotation scheme for the original AffectNet ratings may have been to ensure a better consistency for valence and arousal evaluations. To look more closely into this matter, Figure 4 also shows coloring based on the expression categories obtained from the maximum votes, which clearly fit this valence/arousal space well: the green neutral expression stays near the center, whereas red anger and grey happiness clearly move to the upper positive and bottom negative quadrants of the space. If we change labels to the original AffectNet labels, however, the space becomes much more scattered and less consistent as shown in Figure 4b. Hence, the overall pattern of ratings seems consistent with the voted labels in our experiment.

To reinforce this point, in Figure 5 we show the overall distribution of valence and arousal votes when labeled with the chosen, maximum vote. Here, the distributions also have peaks at expected values (neutral being at 0, happy peaking at positive valence values, and anger peaking at positive arousal values).

We next analyzed the standard deviation of valence and arousal ratings within each category selected by participants—these results are shown in Figure 6a,b. As can be seen, overall standard deviations were in the range from 0.2 to 1.0 and were lowest for neutral and contempt for both valence and arousal. Valence seemed to have similar variability for each expression compared to arousal ratings. Overall, absolute agreement on the rating values was relatively high at these low levels of variability.

We again analyzed the overall correlations among participants in the same manner as done for the expressions. These matrices are shown in Figure 7. The average correlations measuring consistency in relative voting pattern among human participants was $r_{val}=.865$ for valence and $r_{aro}=.468$ for arousal. Relative agreement was very high for valence, but only at medium levels for arousal due to some participants choosing a different rating profile (see Figure 7b, Participant 10, for example). Our results show that participants can agree much better on valence ratings of the expressions than on the label—arousal is at similar levels of relative consistency as the labels.

Figure 8 compares voting patterns of the different DNNs we trained to the human re-annotation results. Figure 8(a) shows the different voting patterns across categories (line-plots were chosen to highlight the overall pattern similarity). At first glance, none of the ResNets’ predictions matches the human pattern perfectly—however, when looking at Figure 8(b), which plots the correlation values among all voting patterns, we can see that the most similar voting pattern to human performance is obtained for the ResNet trained very weakly with only 8 images for 1000 epochs. This is at similar levels to the correlation for human votes to the AffectNet label distribution (blue line).

Overall, we find that ResNets trained on the AffectNet original label distribution cannot capture the human voting pattern well, except for a model that was only weakly-trained (yet still achieved a relatively high performance on the AffectNet validation set).

We found that agreement with human valence ratings was very high for the simple ResNet we trained for only 10 epochs as shown in the scatter plot in Figure 9(a). The coloring in this figure was done according to the human maximum voting label and shows the expected distribution of expression categories along the valence range (from angry in red to happy in grey). Conversely, if we color the same data according to the original AffectNet labels, the plot looks much less well-structured (Figure 9(b)).

Results for arousal also show good agreement of the ResNet model with human data (Figure 10(a)). Again, the human maximum vote coloring produces the desired relationship of arousal to expression category label (cf. neutral at the bottom part to anger and surprised expressions towards the top). Compared to the valence data, however, arousal seems to have more spread—this may be due in part to the fact that arousal values did not span the whole scale, which could have resulted in worse generalization performance for the network. Again, Figure 10(b), which uses the original AffectNet labels does not produce consistent relationships between ratings and expression categories.

The last rows in Figure 7 show the correlations of the trained ResNet models with the human ratings. The overall, average correlation of the DNN with valence is $r_{val,DNN}=.866$ and for arousal $r_{aro,DNN}=.504$. These results confirm the previous analysis and show that, indeed, the ResNet with minimal training of 10 epochs is capable of capturing the valence ratings on our newly-annotated images well. The relative agreement with the arousal ratings is at medium levels—similar to human-human consistency values obtained above.