Uncertain Label Correction via Auxiliary Action Unit Graphs for Facial Expression Recognition

An overview of ULC-AG is illustrated in Fig. 2. The ULC-AG contains: 1) a target branch that takes facial features extracted by a pre-trained DNN and computes the annotation confidence using a self-attention layer. These confidence weights will affect the importance of the sample when calculating the classification loss. In addition, the class-oriented weight is computed to deal with category imbalance in the current batch; 2) an auxiliary branch that follows the idea of multi-task learning exploits the same backbone to get AU features of each facial image and feed them into a two-layer graph convolutional network (GCN) [17] for semantic feature extraction. The re-labeling strategy corrects suspicious labels according to the semantic similarity between the low confidence sample and the templates. The whole ULC-AG is an end-to-end framework and the auxiliary branch will not participate in the testing process.

Fig. 3 illustrates the pipeline of the ULC-AG target branch. To identify the ambiguous samples and estimate their uncertainties, inspired by [8, 18], a self-attention module is employed that consists of a fully connected (FC) layer and the sigmoid function. For a batch of $N$ images, $\bm{F}=[\bm{f}_{1},\bm{f}_{2},...,\bm{f}_{N}]\in\mathbb{R}^{D\times N}$ indicates the facial features extracted by the pre-trained DNN, $D$ is the dimension for each facial feature. The confidence weight of the $i$-th sample can be calculated as:

$$\alpha_{i}=Sigmoid(\bm{W}_{a}^{\top}\bm{f}_{i}),$$

(1)

where $\bm{W}_{a}^{\top}$ denotes the parameters of the self-attention layer. In addition, to prevent the uncertainty caused by category imbalance, we introduce class-oriented weights that are computed as:

$$\gamma_{j}=1-\frac{N_{j}}{N},j\in\{1,2,...,C\},$$

(2)

where $N_{j}$ is the number of images belonging to class $j$, and $C$ is the number of classes.

During the model training, it is expected that samples with lower confidence weights should impose less impact, while categories with fewer samples should receive more attention in the current batch. Therefore, we improve the weighted Cross-Entropy (CE) loss proposed in [10]. Specifically, the loss function for facial expression classifier is formulated as:

$$L_{wce}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{e^{\alpha_{i}\gamma_{y_{i}}\bm{W}_{y_{i}}^{\top}\bm{f}_{i}}}{\sum_{j=1}^{C}e^{\alpha_{i}\gamma_{y_{i}}\bm{W}_{j}^{\top}\bm{f}_{i}}},$$

(3)

where $\bm{W}_{j}^{\top}$ denotes the parameters of the $j$-th classifier, $f_{i}$ is the facial feature, $\alpha_{i}$ is the confidence weight, $y_{i}$ and $\gamma_{y_{i}}$ are the original label and its corresponding class-oriented weight, respectively. According to [19], $L_{wce}$ and $\alpha$ are positively correlated.

After we obtain the confidence weights, the high and low confidence samples in the current batch are divided with a ratio $\varphi$ by using a simple rank regularization approach [8], which is formulated as:

$$L_{rr}=max(0,\theta-(Avg_{h}-Avg_{l})),$$

(4)

where $\theta$ is a margin threshold that can be a fixed hyperparameter or updated with training, $Avg_{h}$ and $Avg_{l}$ are mean values of confidence weights in high and low groups, respectively.

Recent studies reveal that introducing knowledge of the multi-label space can alleviate the effect of ambiguous facial expressions [13, 20]. In this work, we choose AU detection as our auxiliary task and construct semantically representative AU graphs because the Facial Action Coding System is an affect description model that has latent mappings with expression categories [21, 22, 23]. The AU graph takes individual AU features as graph nodes and the co-occurring AU dependency as graph edges. Fig. 4 illustrates the pipeline of the ULC-AG auxiliary branch.

Following the multi-task learning idea, we can get a set of AU features of each image with the same backbone network, $\bm{X}^{i}=[\bm{x}_{1}^{i},\bm{x}_{2}^{i},...,\bm{x}_{M}^{i}]\in\mathbb{R}^{B\times M}$, $B$ and $M$ denote feature dimension and AU number, respectively. Considering the consistency of predefined mappings between expression categories and AUs in large-scale datasets is difficult to guarantee[24, 25], we exploit a data-driven approach based on the conditional probability to obtain co-occurring AU dependencies from the training set as graph edges, which can be calculated as:

$$\bm{A}_{p,q}=P(AU_{p}|AU_{q})=\frac{OCC_{p\cap q}}{OCC_{q}},$$

(5)

where $OCC_{p\cap q}$ denotes the number of co-occurrences of $AU_{p}$ and $AU_{q}$, and $OCC_{q}$ is the total number of occurrences of $AU_{q}$. Since the AU co-occurring relationship is actually asymmetry, so $P(AU_{p}|AU_{q})\neq P(AU_{q}|AU_{p})$.

Then, the two are input in a two-layer GCN with each AU as a graph node to extract the semantic feature. Specifically, each graph convolution layer is formulated as:

$$\bm{X}^{\prime}=g(\bm{X},\bm{A})=LeakyRELU(\bar{\bm{A}}\bm{X}\bm{W}_{g}),$$

(6)

where $\bar{\bm{A}}$ denotes the normalized $\bm{A}$ with all rows sum to one, $\bm{W}_{g}$ is the weight matrix to be learned in the current layer.

All the node features outputted by the GCN are fed into a FC layer with sigmoid functions to predict multiple AUs. The binary CE loss is used to train every AU classifier, and the total balanced group loss is defined for the two-layer GCN as:

$$L_{au}=-\sum_{m=1}^{M}{\alpha(z_{m}\log{p_{m}}+(1-z_{m})\log{(1-p_{m})})},$$

(7)

where $\alpha$ is the confidence weight, $z_{m}$ and $p_{m}$ are the pseudo label and the prediction of $m$-th AU, respectively. The feature $\bm{s}_{i}\in\mathbb{R}^{1\times M}$ before AU classifiers are treated as the semantic feature of the sample.

To determine which labels need to be corrected and which new classes should be assigned, we design a semantic preserving strategy (see Fig. 4). For the divided two sample sets, a weighted semantic template set $\bm{T}=[\bm{t}_{1},\bm{t}_{2},...,\bm{t}_{C}]\in\mathbb{R}^{M\times C}$ for every facial expression category is first generated with the semantic features and the confidence weights of valid samples, which can be formulated as:

$$\bm{T}_{j}=\frac{1}{K_{j}}\sum_{k_{j}=1}^{K_{j}}\alpha_{k_{j}}\bm{s}_{k_{j}},$$

(8)

where $K_{j}$ is the number of the samples with the $j$-th label in the high confidence set. The semantic templates will be dynamically updated throughout the whole training process.

For the ambiguous samples in the low confidence set, we calculate the cosine distance between $\bm{s}_{l}$ ($l\in\{1,2,...,N-K\}$) and each of $\bm{t}_{j}$ in the semantic template $\bm{T}$, which can be formulated as:

$$SP_{s_{l},j}=1-\frac{\bm{t}_{j}\times\bm{s}_{l}}{\|\bm{t}_{j}\|\|\bm{s}_{l}\|},$$

(9)

where $\times$ denotes the dot product operation, $K$ is the total number of high confidence samples in each batch.

Next, for every ambiguous sample, we compare its semantic feature with each semantic template in $T$. The template class with the highest semantic similarity will be assigned to this sample as a new label. Formally, the re-labeling strategy can be defined as:

$$y^{\prime}_{i}=\begin{cases}j,&\mbox{if }SP_{s_{l},org}-min(SP_{s_{l},j})>0\\ y_{i},&\mbox{otherwise}\end{cases}$$

(10)

where $y^{\prime}_{i}$ indicates the corrected label, and $j\neq org$, $org$ is the original labeled class.

Finally, the total loss function of the whole network can be written as:

$$L_{total}=\frac{\lambda_{1}}{2}(L_{wce}+L_{rr})+\lambda_{2}L_{au},$$

(11)

where $\lambda_{1}$ and $\lambda_{2}$ are the weighted ramp functions that will change with epoch rounds [26], which can be computed as follows:

$$\lambda_{1}=\begin{cases}\exp(-(1-\frac{epoch}{\beta})^{2}),&epoch\leq\beta\\ 1,&epoch>\beta\end{cases},$$

(12)

$$\lambda_{2}=\begin{cases}1,&epoch\leq\beta\\ \exp(-(1-\frac{\beta}{epoch})^{2}),&epoch>\beta\end{cases}.$$

(13)

The weighted ramp functions allow ULC-AG to pay more attention to auxiliary branches in the initial training stage. Since the number of samples accumulated at the beginning is insufficient, it is unable to generate robust semantic features. After a certain number of training rounds, the model will focus more on the target branch to extract discriminative features for final predictions.

To evaluate the performance of ULC-AG in tackling label uncertainties, we conduct experiments on two popular FER benchmarks, RAF-DB [6] and AffectNet [7]. Both datasets have unconstrained conditions and large-scale samples.

RAF-DB has $15339$ face images with annotations of six basic emotions and neutral. In our experiments, $12271$ and $3368$ samples are used for training and test, respectively.

AffectNet contains close to one million expression images. To ensure a fair comparison, we select samples manually labeled as six basic emotions and neutral for evaluation. The number of images in the training set and the test set is $283,901$ and $3500$, respectively. In addition, automatically labeled samples in AffectNet are used as a set of real noisy data, denoted as AffectNet_Auto, to verify the ability of ULC-AG in handling uncertain expressions.

Since the AU annotation requires specially trained experts and is time-consuming, it is natural that no AU labels are provided in RAF-DB and AffectNet. To account for this issue, we applied Openface 2.0 [27] to automatically generate pseudo AU labels, similar to [12, 14]. Note that our ULC-AG utilizes feature-level semantic similarity preserving, which can reduce the negative impact of incorrect pseudo AU labels.

The ULC-AG is implemented with the Pytorch platform and trained using two Nvidia Volta V100 GPUs. Face images are obtained using MTCNN [28] and further resized to $224\times 224$ pixels as inputs. For the target branch, we choose the ResNet-18 as the backbone DNN which is pre-trained on the MS-Celeb-1M [29] dataset as previous methods [8, 10, 30]. In every iteration, $\varphi$ and $\theta$ are set as $0.8$ and $0.15$, respectively. The initial learning rate is $0.01$, which is updated to $10^{-3}$ and $10^{-4}$ at the $10$-th and $20$-th epoch, respectively. For the auxiliary branch, each GCN layer has 64 channels, and the decayed learning rate is set as 0.005. The auxiliary branch will not participate in the network optimization until $10$ epochs to obtain initial templates. Thus, when the relabeling starts afterward, there is no missing template in the current batch. We choose a batch size $512$ to ensure that every template can be effectively updated during the whole training process.

We conduct the ablation experiments to demonstrate the contributions of the proposed modules in this paper.

To test our re-labeling strategy, we set up comparative experiments under synthetic uncertainty and real uncertainty, respectively. The ResNet-18 baseline and the SCN [8] also using label repair are selected for comparison.

Table V shows the performance comparisons with the state-of-the-art approaches and Fig. 7 presents the confusion matrices of ULC-AG. To summarize, our method obtains competitive results on both RAF-DB and AffectNet datasets. Although LDL-ALSG [31] and SEIIL [20] introduce different auxiliary tasks to train the network, they only consider the label-level distribution and cannot repair the uncertain samples. In addition, IPA2LT [31], SCN [8], and WSND [11] explicitly deal with ambiguous images, but the label uncertainties can still mislead feature learning without extra knowledge in side-space and cause performance limitations. Benefiting from the confidence estimation, the data-driven AU graph, and the feature-level constrained label correction, the proposed ULC-AG outperforms all the comparison methods, including NMA [13] that is similar but lacks effective facial representation.