Audio-Visual Fusion Layers for Event Type Aware Video Recognition

Given a video clip $\mathbf{V}$ with its corresponding audio $\mathbf{A}$, our backbone networks extract features for each modality. We use a two-stream architecture, that leverages each modality separately, similar to other existing audio-visual learning works. Our backbone networks take entire 10 sec. video and audio frames and extract features per-frame for each modality. The video network is a spatio-temporal network, similar to MC$x$ [45]. It takes a video $\mathbf{V}$ of $T$ frames as input and generates a video embedding $\mathbf{z}^{\mathbf{V}}$ with dimensions $T\times D$. The audio stream takes the log-mel spectrogram $\mathbf{A}$ of 10$T$ frames and passes it through a stack of 2D convolution layers to extract an audio embedding $\mathbf{z}^{\mathbf{A}}$ with dimensions $T\times D$ similar to video features. Thus, there is a corresponding audio feature for every video feature and we do not need any replication or tile operations to match audio and video feature dimensions.

This layer is designed to integrate audio and visual signals by performing temporal aggregation to each frame features from both modalities with the assumption of audio and visual signals are temporally correlated and aligned throughout the video. This temporal congruency between audio and visual signals play a key role for audio-visual sensory integration not only in cognitive science [40, 41] but also in audio-visual learning works [34, 44, 26, 19]. The integrated audio-visual feature $\mathbf{z}_{cont.}$ can be computed as follows:

where $\mathtt{concat}$ denotes the concatenation of two vectors and $t$ denotes time step. The continuous layer feature $\mathbf{z}_{cont.}$ is obtained by temporal aggregation over all time steps $T$ by average pooling.

Another type of audio-visual event that can occur is the sparse and isolated instant ones. These actual interesting actions happen when both audio and visual signals are semantically correlated and synced for a short period as a few important moments rather than long temporal correspondence. The assigned task for this layer is performed by finding the time steps (moments) that have the highest correlation scores between audio and visual features, $\mathbf{z}^{\mathbf{V}}$ and $\mathbf{z}^{\mathbf{A}}$ respectively. Figure 3 shows that moments with highest scores are located only in the last part of the video which can be easily understood since car appears in the scene and it is correlated with the car sound (visualized as colored frames). Remaining parts of the video are not useful for audio-visual integration as it only shows empty road like visual information. Correlation scores are computed by pairwise dot products between audio and visual embeddings [19, 1] at the same time step and audio-visual feature $\mathbf{z}_{inst.}$ is computed as follows:

where score list is defined as $\mathbf{S}_{av}[t]=\mathbf{z}_{t}^{V}{\cdot}\mathbf{z}_{t}^{A}$ and $\mathcal{K}=\mathtt{top\textrm{-}k}(\mathbf{S}_{av})$ represents $\mathtt{top\textrm{-}k}$ sorted time steps by ${S}_{av}$. The instant layer feature is obtained by averaging the features at top-$k$ time steps.

There is another type of audio-visual event that can be integrated on the event occurrences at regular points in time, i.e., rhythm [42, 8]. For example, sounds occur rhythmically and repetitively in dancing, musical instruments and bird calling events as they have prominent property in audio modality aligned with visual signals. Onset event layer is designed to leverage audio onsets. Audio onsets usually give information about the rhythms and beats [11, 48], musical notes and as well as beginning of the audio events [37, 25]. As can be seen in Figure 3, visual event (typebar hits the screen) occurs at the same time as onset moments which can be easily understood since this action makes rhythmic typebar sound. Furthermore, almost equal time gap between onset moments show that this event is rhythmic. Audio onsets are computed by using existing audio libraries [30] for the given audio $\mathbf{A}$ as $\mathcal{O}=\mathtt{onset}(\mathbf{A})$. This return a set of time stamp indices that onsets exist in the range of $[1,T]$. We compute $\mathbf{z}_{onset}$ as follows:

Our multisensory layers are inspired from human cognitive ability for multisensory integration as binding the multimodal signals if they are correlated and separating them otherwise. Considering some actions are soundless (“massaging legs”, “stretching leg”, etc.) or some scenes have irrelevant sounds, integrating this irrelevant sound signals to visual features acts as noise and affects the correct prediction ability. Thus, this visual event layer is designed to recognize the events only from visual perspective. It performs the task of assigning zero-valued audio features for each visual frame feature and applying Eq. (1) to output $\mathbf{z}_{visual}$.

Conversely to the visual event layer, scenes might have events that are outside of the field of view but still hearable or visual signal may be completely unrelated to the accompanying audio. Additionally, some videos might have poor visual signal. To make our network use audio only modality, this layer assigns zero-valued visual features to each audio frame feature and applies Eq. (1) to compute $\mathbf{z}_{audio}$.

Video classification is a task to classify a video by a single label. Since our model outputs multiple predictions from event-specific layers, we integrate them by majority voting to output a single prediction for a video-level classification task as follows:

where $I$ is an indication function, $p_{i}$ is a predicted label defined as $\operatornamewithlimits{\arg\max}_{j}O_{ij}$, and $j$ is an index of the vector $O_{i}$. In case of disagreement between the layers, that no majority consensus is obtained, the label from the most confident layer is selected.

We conduct a series of experiments to show how well our model predicts video-level labels. We compare the performance of our model with baselines on different datasets (Table 1). Note that our goal here is not to compete on classification accuracy with any other expensive video recognition models, but rather to show that our model analyzes a video from different modalities and different video characteristics which leads to improvement in classification.

Accuracies on uni-modal networks in Table 1 depicts the accuracy of the backbone networks trained with single stream modalities. Naïve audio-visual model represents audio-visual networks that leverage late fusion approach (simple concat.) for final representation as used in previous works [49, 26, 29]. As results show in Table 1, our approach offers improvement to overall performance in benchmark datasets. Our model is more effective on the datasets that are designed with audio-visual correspondence, e.g. VGGSound, Kinetics-Sound, and AVE, with the improvement around  $2\%$. Our performance improvement is less significant on Kinetics since it is a visual-dominant dataset, where many videos’ sound are not correlated to visual signals.

Our network contains multiple event-specific layers and each layer outputs a label prediction. The collection of each layer’s prediction form a multi-label prediction set. Let $p_{i}$ be a predicted label from the $i^{th}$ event-specific layer. Then a set of label predictions $\mathcal{P}$ can be defined as follows:

The function $f$ accepts a predicted label $p_{i}$ if and only if when the value $O_{ip_{i}}$ is higher than the other layer values $O_{lp_{i}}$. This process filters out label predictions with low confidence and only highly confident predictions are gathered in a multi-label prediction set. In our implementation, we utilize five types of layers, therefore, the cardinality of the prediction set is bounded by $1\leq|\mathcal{P}|\leq 5$. When our network is evaluated on the VGGSound dataset, the cardinality of $\mathcal{P}$, i.e. the average number of predicted labels is $2.21$.

This analysis leads to the question of “Are the multi-label predictions of our network are correct? or Does the VGGSound dataset contain multiple events but only annotated with a single label?”. To answer these questions, we need a multi-label evaluation. However, there is no multi-label ground truth for VGGSound dataset. Therefore, we make $12$ subjects annotate the multiple video-level predictions of our network that match with the given video for $\scriptstyle\sim$1200 videos. Note that candidate labels for annotation are limited to the labels predicted by our model. We check how many of the total predictions from our network actually do match with human answers. The user study shows that $62\%$ of all predicted labels match with human selections, which means our network on average outputs $1.4$ correct labels per sample. This study shows that VGGSound is indeed a multi-label dataset and single label prediction is insufficient to properly describe the videos.

To further evaluate the correctness of the multi-labels predicted by our network, we also conduct another experiment by using LLP as it contains multi-labels per video, i.e. the average number of labels is $1.81$. In this experiment, even though our network is trained with randomly selected single labels, it still has the ability to predict correct multi-labels for a given video. This is due to multiple layers that can output multiple contents by looking for different properties in the video. We compare our results with top-N results of Naive-AV model. The F1 metric is used to evaluate the performance of our model and Table 2 shows the results. Our model outputs multi-label predictions dynamically depending on whether the event-specific layers make consensus or not, while the top-N approach fixes the number of predictions. Although, the Top-2 performance is comparable to ours, without knowing the number of ground truth multi-labels in advance, we can not know the number of Top-N that should be selected. For example, the video shown in $2^{nd}$ row of Figure 5 contains people playing more than two instruments. In this case, Top-2 only outputs two predictions while our model outputs more than two predictions. In the other case, when there is only one dominant event, our model outputs only one prediction while the Top-2 baseline still outputs two predictions.

We apply our method on understanding the modality biases of the datasets [50]. Each dataset has different modality properties such as large portion of the videos may contain specific modality dominantly, i.e., Kinetics is visually heavy dataset. We pose the problem as finding the most dominant modality in a given dataset by analyzing every video. Our network outputs five different logits corresponding to the each event-specific layer. We use true class label $y$ to get confidence scores of the test video. Since this is ground truth class, the layer that is highly activated (highest confidence score for given $y$) reveals the interpretable modality information.

With this technique, we find the modality biases of the datasets as well as each category in the dataset. We apply majority voting rule within each category among all the videos and assign the modality label to the categories. See Supp. for the category-wise modality assignment. Table 3 shows the number of categories that are assigned to each layer for every dataset. Our analysis shows consistent results with the prior knowledge about these benchmark datasets. Results on Kinetics clearly shows that it is visual dominant dataset. AVE dataset is curated for audio-visual learning and our method validates this as one of the AV layers is dominant. LLP [43] reports that majority of the annotated events are audio events. Our analysis also confirms that LLP has tendency to the audio modality.

Additionally, we perform an experiment to see how many categories of Kinetics-Sound (created by selecting categories manually from Kinetics in [3]) matches with the categories that our audio-visual event-specific layers filter in Kinetics. This reveals that $66\%$ of the Kinetics-Sound categories are matched. Thus, our modality bias selection gives consistent results with human selections. Please see Supp. for details.

At first glance, the differences between each audio-visual event layer may not look significant. However, they indeed look for different audio-visual characteristics in the videos. We count the number of unique videos that one of these layers classifies correctly while the rest two layers fail, and also the samples that continuous layer fails but instant or onset layer predict correctly in Table 4. The number of uniquely correct samples for each layer is comparable to each other and the last column shows that the instant and onset layers capture significant amount of correct samples compared to the continuous layer. This shows that the proposed multisensory layers are complementary. As a consequence, the model enables a more in-depth interpretation of videos.

Continuous event layer pools information from every time step features of both modality without consideration of their correspondence. Therefore, number of the informative features may be less than uninformative ones for some videos, as in “car” example of Figure 3. Thus, using informative subset of these every time step features, i.e., instant or onset, may improve the accuracy for these videos since it will be less contaminated. Furthermore, the difference between instant and onset event layer can be also seen in the “car” example of Figure 3. Onset event layer uses the onset moments, i.e. pink dots, that are grouped in the part before the real action starts whereas instant event layer captures the instant, highly audio-visual correlated moments of the video, i.e. blue dotted box.

We qualitatively show the multi-labeling ability of our network even though the network is trained on single labels. Figure 5 shows predicted multi-labels are consistent with the human perception.

In order to have a better insight on what each event-specific layer learned, we visualize some of the videos that are maximally activated by a certain layer. Figure 6 displays some videos for each audio-visual layer. As it can be seen, instant event layer has short period high intensity-like patterns; onset event layer captures rhythmic-like patterns; and continuous event layer shows temporally constant-like events. Note that these patterns show similarities to abstract event visualizations in Figure 1. We also visualize that representations from these assigned layers may be more proper for given videos as only that certain layer makes correct prediction.