Recursive Joint Attention for Audio-Visual Fusion
in Regression based Emotion Recognition

A) Problem Formulation: Given an input video sub-sequence $S$, $L$ non-overlapping video clips are uniformly sampled and deep feature vectors ${\boldsymbol{X}}_{\mathbf{a}}$ and ${\boldsymbol{X}}_{\mathbf{v}}$ are extracted for the individual A and V modalities respectively from pre-trained networks. Let ${\boldsymbol{X}}_{\mathbf{a}}=\{\boldsymbol{x}_{\mathbf{a}}^{1},\boldsymbol{x}_{\mathbf{a}}^{2},...,\boldsymbol{x}_{\mathbf{a}}^{L}\}\in\mathbb{R}^{d_{a}\times L}$ and ${\boldsymbol{X}}_{\mathbf{v}}=\{\boldsymbol{x}_{\mathbf{v}}^{1},\boldsymbol{x}_{\mathbf{v}}^{2},...,\boldsymbol{x}_{\mathbf{v}}^{L}\}\in\mathbb{R}^{d_{v}\times L}$ where ${d_{a}}$ and ${d_{v}}$ represent the dimensions of the A and V feature representations, respectively, and $\boldsymbol{x}_{\mathbf{a}}^{l}$ and $\boldsymbol{x}_{\mathbf{v}}^{l}$ denotes the A and V feature vectors of the video clips, respectively, for $l=1,2,...,L$ clips. The objective of the problem is to estimate the regression model $F:\boldsymbol{X}\to\boldsymbol{Y}$ from the training data $\boldsymbol{X}$, where $\boldsymbol{X}$ denotes the set of A and V feature vectors of the input video clips and $\boldsymbol{Y}$ represents the regression labels of the corresponding video clips.

B) Audio and Visual Networks: Spectrograms has been found to be promising with various 2D-CNNs (Resnet-18 [13]) for ER [14, 15]. Therefore, we have explored spectrograms in the proposed framework. In order to effectively leverage the temporal dynamics within A modality, we have also explored LSTMs across the temporal segments of the A sequences. Finally, the A feature vectors of $L$ video clips are shown as $\boldsymbol{X_{a}}=({\boldsymbol{x}_{\mathbf{a}}^{1},\boldsymbol{x}_{\mathbf{a}}^{2},...,\boldsymbol{x}_{\mathbf{a}}^{L}})\in\mathbb{R}^{{d_{a}}\times L}$.

C) Recursive Joint Attention Model: Given the A and V features, $\boldsymbol{X_{a}}$ and $\boldsymbol{X_{v}}$, the joint feature representation is obtained by concatenating the A and V feature vectors ${\boldsymbol{J}}=[{\boldsymbol{X}}_{\mathbf{a}};{\boldsymbol{X}}_{\mathbf{v}}]\in\mathbb{R}^{d\times L}$, where $d={d_{a}}+{d_{v}}$ denotes the dimensionality of concatenated features. The concatenated A-V feature representations ($\boldsymbol{J}$) of a video sub-sequence ($\boldsymbol{S}$) are now used to attend to unimodal feature representations ${\boldsymbol{X}}_{\mathbf{a}}$ and ${\boldsymbol{X}}_{\mathbf{v}}$. The joint correlation matrix $\boldsymbol{C}_{\mathbf{a}}$ across the A features ${\boldsymbol{X}}_{\mathbf{a}}$, and the combined A-V features $\boldsymbol{J}$ are given by:

$$\boldsymbol{C}_{\mathbf{a}}=\tanh\left(\frac{{\boldsymbol{X}}_{\mathbf{a}}^{T}{\boldsymbol{W}}_{\mathbf{j}\mathbf{a}}{\boldsymbol{J}}}{\sqrt{d}}\right)$$

(1)

where ${\boldsymbol{W}}_{\mathbf{j}\mathbf{a}}\in\mathbb{R}^{L\times L}$ represents learnable weight matrix across the A and combined A-V features, and $T$ denotes transpose operation. Similarly, the joint correlation matrix for V features are given by:

$$\boldsymbol{C}_{\mathbf{v}}=\tanh\left(\frac{{\boldsymbol{X}}_{\mathbf{v}}^{T}{\boldsymbol{W}}_{\mathbf{j}\mathbf{v}}{\boldsymbol{J}}}{\sqrt{d}}\right)$$

(2)

The joint correlation matrices capture the semantic relevance across the A and V modalities as well as within the same modalities among consecutive video clips, which helps in effectively leveraging intra- and inter-modal relationships. After computing the joint correlation matrices, the attention weights of the A and V modalities are estimated. For the A modality, the joint correlation matrix $\boldsymbol{C}_{\mathbf{a}}$ and the corresponding A features ${\boldsymbol{X}}_{\mathbf{a}}$ are combined using the learnable weight matrices $\boldsymbol{W}_{\mathbf{c}\mathbf{a}}$ to compute the attention weights of A modality, which is given by $\boldsymbol{H}_{\mathbf{a}}=ReLU(\boldsymbol{X}_{\mathbf{a}}\boldsymbol{W}_{\mathbf{c}\mathbf{a}}{\boldsymbol{C}}_{\mathbf{a}})$ where ${\boldsymbol{W}}_{\mathbf{c}\mathbf{a}}\in\mathbb{R}^{{d_{a}}\times{d_{a}}}$ and ${\boldsymbol{H}}_{\mathbf{a}}$ represents the attention maps of the A modality. Similarly, the attention maps ($\boldsymbol{H}_{\mathbf{v}}$) of V modality are obtained as $\boldsymbol{H}_{\mathbf{v}}=ReLU(\boldsymbol{X}_{\mathbf{v}}\boldsymbol{W}_{\mathbf{c}\mathbf{v}}{\boldsymbol{C}}_{\mathbf{v}})$ where ${\boldsymbol{W}}_{\mathbf{c}\mathbf{v}}\in\mathbb{R}^{{d_{v}}\times{d_{v}}}$. Then, the attention maps are used to compute the attended features of A and V modalities as:

$${\boldsymbol{X}}_{\mathbf{a}\mathbf{t}\mathbf{t},\mathbf{a}}=\boldsymbol{H}_{\mathbf{a}}\boldsymbol{W}_{\mathbf{h}\mathbf{a}}+\boldsymbol{X}_{\mathbf{a}}$$

(3)

$${\boldsymbol{X}}_{\mathbf{a}\mathbf{t}\mathbf{t},\mathbf{v}}=\boldsymbol{H}_{\mathbf{v}}\boldsymbol{W}_{\mathbf{h}\mathbf{v}}+\boldsymbol{X}_{\mathbf{v}}$$

(4)

where $\boldsymbol{W}_{\mathbf{h}\mathbf{a}}\in\mathbb{R}^{d\times{d_{a}}}$ and $\boldsymbol{W}_{\mathbf{h}\mathbf{v}}\in\mathbb{R}^{d\times{d_{v}}}$ denote the learnable weight matrices for A and V respectively. After obtaining the attended features they are fed again to the joint cross-attentional model to compute the new A and V feature representations as:

$${\boldsymbol{X}}_{\mathbf{a}\mathbf{t}\mathbf{t},\mathbf{a}}^{(t)}=\boldsymbol{H}_{\mathbf{a}}^{(t)}\boldsymbol{W}_{\mathbf{h}\mathbf{a}}^{(t)}+\boldsymbol{X}_{\mathbf{a}}^{(t-1)}$$

(5)

$${\boldsymbol{X}}_{\mathbf{a}\mathbf{t}\mathbf{t},\mathbf{v}}^{(t)}=\boldsymbol{H}_{\mathbf{v}}^{(t)}\boldsymbol{W}_{\mathbf{h}\mathbf{v}}^{(t)}+\boldsymbol{X}_{\mathbf{v}}^{(t-1)}$$

(6)

where $\boldsymbol{W}_{\mathbf{h}\mathbf{a}}^{(t)}\in\mathbb{R}^{d\times{d_{a}}}$ and $\boldsymbol{W}_{\mathbf{h}\mathbf{v}}^{(t)}\in\mathbb{R}^{d\times{d_{v}}}$ denote the learnable weight matrices of $t^{th}$ iteration for A and V respectively. Finally, the attended A and V features after $t$ iterations are further concatenated and fed to BLSTM to obtain the temporal dependencies within the refined A-V feature representations, which are fed to fully connected layers for final prediction.

A) Datasets: Affwild2 is among the largest public dataset in affective computing, consisting of $564$ videos collected from YouTube, all captured in-the-wild [19]. The annotations are provided by four experts using a joystick and the final annotations are obtained as the average of the four raters. In total, there are $2,816,832$ frames with $455$ subjects, out of which $277$ are male and $178$ female. The annotations for valence and arousal are provided continuously in the range of $[-1,1]$. The dataset is partitioned in a subject-independent manner so that every subject’s data will be present in only one subset. The partitioning produces 341, 71, and 152 videos for the training, validation, and test sets respectively.

B) Ablation Study: Table 1 presents the results of the experiments conducted on the validation set for the ablation study. The performance of the approach is evaluated using Concordance Correlation Coefficient (CCC). In this section, we have analyzed the contribution of BLSTMs, where we have performed experiments with and without BLSTMs. Firstly, we have conducted experiments without using BLSTM for both the individual A and V representations as well as the A-V feature representations. Then, we included BLSTMs only for the individual A and V modalities before feeding to the joint attention fusion model i.e., only Unimodal-BLSTMs (U-BLSTMs). By including U-BLSTMs to capture the temporal dependencies within the individual modalities, we can observe improvements in performance. Therefore, BLSTMs are found to be promising in capturing the intra-modal temporal dynamics better than that of correlation-based intra-modeling in the joint attention model. After that, we have also included joint BLSTM (J-BLSTM) in order to capture the temporal dynamics across the joint A-V feature representations, which further improved the performance of the system. Note that in the above experiments, we have not performed recursive attention. In addition to the impact of U-BLSTM and J-BLSTMs, we have also conducted a few more experiments to investigate the impact of the recursive behavior of the joint attention model. First, we did recursion without LSTMs and found some improvement due to recursion. Then we included LSTMs and conducted several experiments by varying the number of recursions (iterations) in the fusion model. As we increase the number of recursive times, the model performance increases and starts to decrease after a certain recursion number. A similar trend of the model performance is also observed in the test set. Therefore, this can be attributed to the fact that recursion also works as a regularizer which improves the generalization ability of the model. In our experiments, we found that $t=2$ gives the best performance i.e, we have achieved the best performance of our model with two recursive iterations.

C) Comparison to State-of-art: Table 2 shows our results against relevant state-of-the-art A-V fusion models on the Affwild2 dataset. The Affwild2 dataset has recently been widely used for Affective Behavior Analysis in-the-Wild (ABAW) challenges [20, 21]. Therefore, we compare our approach with relevant state-of-art approaches in the ABAW challenges. Kuhnke et al [2] used a simple feature concatenation using Resnet-18 for A and R3D for V modalities, showing better performance for arousal than valence. Zhang et al [22] proposed a leader-follower attention model for fusion, and improved the performance (arousal) of the model proposed by Kuhnke et al [2]. Rajasekhar et al. [4] explored the cross-attention model by leveraging only the inter-modal relationships of A and V, and showed improvement for valence but not for arousal. Rajasekhar et al. [8] further improved the performance of the model by introducing joint feature representation to the cross-attention model. The proposed model performs even better than that of vanilla JCA [8] by introducing LSTMs as well as a recursive attention mechanism.