Accommodating Audio Modality in CLIP for Multimodal Processing

Text & Vision Encoder. We employ CLIP (Portillo-Quintero, Ortiz-Bayliss, and Terashima-Marín 2021) as our text and vision encoders to encode text input $T$ and vision input $V$. Each $t_{i}\in T$ is first tokenized into a token sequence and then added with a start token [SOS] and an end token [EOS], denoted as $\{t_{i}^{1},t_{i}^{2},\cdots,t_{i}^{L_{T}}\}$. After text encoding, outputs of each token are collected as word-level representations $\{x_{i}^{1},x_{i}^{2},\cdots,x_{i}^{L_{T}}\}$. Following CLIP, we choose the output of [EOS] token as the global text representation of $t_{i}$, denoted as $x_{i}^{g}$.

Audio Encoder Well-trained specialists could infer ambient events or human voice by watching spectrograms. Thus it is also possible for machine to encode audio information with visual spectrograms as inputs. To keep architecture consistency across different modalities, we design our audio encoder with the same model structure as the vision encoder. To process audios similarly as the visual signals, the first thing to do is to transfer the 1-dimensional long audio $a_{i}\in A$ into the image format, a matrix in the shape of 224$\times$224$\times$3 in this paper. To be specific, we convert the audio waveform into 224-dimensional log Mel filterbank (fbank) features with 32ms Hamming window every 8ms. In this way, a t-second audio stream will be transferred into a spectrogram in the shape of 125t $\times$ 224. We cut the spectrogram into k$\times$224$\times$224 along the temporal dimension without overlap, and pad with zeros if the last part is less than 224. Therefore, a t-second audio will be finally transferred into $\lceil\frac{t}{1.792}\rceil$ frames $\{a_{i}^{1},a_{i}^{2},\cdots,a_{i}^{L_{A}}\}$ with $a_{i}^{j}\in R^{224\times 224\times 3}$. Then the normalized segment frames can be encoded similarly with frame images. The final sequence of audio segment representations is denoted as $z_{i}=\{z_{i}^{1},z_{i}^{2},\cdots,z_{i}^{L_{A}}\}$. We also get a global audio embedding by average pooling of $z_{i}$, denoted as $z_{i}^{g}$.

Audio Type Token As we consider roughly two types of information in the audios of common videos (VerBal information and NonverBal information), we design audio type tokens to effectively control which type of features the audio encoder tends to generate. To be specific, after flattening patches of each audio segment, an audio type token [VB]/[NB] is added at the end of the patch sequence according to different application scenarios. For example, for dialogue or commentary, the audio type token [VB] could be used to encode the verbal information. While for natural activities/events, where the nonverbal information is more important, the [NB] token can be used as the control signal to extract the audio features from the nonverbal aspect. Furthermore, for the complex scenarios where both verbal information and nonverbal information are crucial, these two types of embeddings can also be combined for better video understanding. During pre-training, we set the audio type token according to the characteristics of audio pre-training datasets. During fine-tuning or testing, we add both audio type tokens at the end of the flatten patch sequences to flexibly extract both verbal and nonverbal information.

Inter-modal Learning During inter-modal learning, which learns cross-modal alignments between text, vision and audio, positive pairs of cross-modal representations should be closer than negative ones. In this work, we construct negative pairs of cross-modal representations within a mini-batch. With global embeddings of text, vision and audio modalities, we compute the cosine similarity matrix in $B\times B$ for text-audio pairs and vision-audio pairs within a mini-batch, where $B$ is the batch size. Since the vision and text encoders have been well pre-trained to learn vision-language alignment, we mainly train the audio encoder by maximizing the cosine similarity of $B$ positive pairs while minimizing the cosine similarity of the $B^{2}-B$ negative pairs. The symmetric cross entropy loss is calculated as follows:

	$\displaystyle{\rm NCE}_{\textit{at}}=\frac{1}{B}\sum_{i}^{B}{\log\frac{\exp(z_{i}^{g}\cdot x_{i}^{g})}{\sum_{j}^{B}{\exp(z_{i}^{g}\cdot x_{j}^{g})}}}$		(1)
	$\displaystyle{\rm NCE}_{\textit{av}}=\frac{1}{B}\sum_{i}^{B}{\log\frac{\exp(z_{i}^{g}\cdot y_{i}^{g})}{\sum_{j}^{B}{\exp(z_{i}^{g}\cdot y_{j}^{g})}}}$		(2)

where $x^{g}$, $y^{g}$, $z^{g}$ refer to global embbeddings of text, vision and audio modalities.

Intra-modal Learning To enhance the information representation ability of audio encoder, we further optimize it with intra-modal self-supervised learning. We first augment the audio $a_{i}$ to $\hat{a_{i}}$ by randomly masking the audio spectrograms along both channel and temporal dimension (Liu, Li, and Lee 2021). To be specific, we randomly sample the start step along the channel step and the time step with probability of 5% and 15% respectively, then we mask the subsequent 10 consecutive steps from the start step. Overlap is allowed in the masking. The original audio $a_{i}$ and its augmented version $\hat{a_{i}}$ then can be seen as a positive pair for the contrastive learning. Similar to the inter-modal NCE loss, other masked audios within a mini-batch are negative samples for $a_{i}$. The symmetric cross entropy loss is calculated as follows:

	$\displaystyle{\rm NCE}_{\textit{a}\hat{a}}$	$\displaystyle=\frac{1}{B}\sum_{i}^{B}{\log\frac{\exp(z_{i}^{g}\cdot\hat{z_{i}}^{g})}{S}}$		(3)
	$\displaystyle S$	$\displaystyle=\sum_{j}^{B}{\exp(z_{i}^{g}\cdot\hat{z_{j}}^{g})}+\sum_{k\neq i}^{B}{\exp(z_{i}^{g}\cdot z_{k}}^{g})$		(4)

where $\hat{z_{i}}^{g}$ is the global embedding of masked audio $\hat{a_{i}}$.

Fine-tuning for Video Retrieval Video Retrieval aims to search the target video based on a video caption as the retrieval query. Without encoding audio information, existing video retrieval works (Liu et al. 2019; Miech, Laptev, and Sivic 2018; Chen et al. 2020) only focus on the matching between text and vision modality. Benefiting from the tri-modality encoding ability of CLIP4VLA, we fully explore both vision and audio information in the video for text-video retrieval. Since there are three modalities involved in this task, effective multimodal fusion is important. In this paper, we explore three multimodal fusion approaches for text-video retrieval, including (1) Global to Global, (2) Global to Local, and (3) Local to Local. As illustrated in Figure 3, the Global to Global approach directly calculates similarity based on the global embeddings of vision and audio modalities via mean pooling. For the Global to Local approach, we apply a Video Temporal Encoding Module (N-layer transformer) to encode temporal relevance of vision and audio modalities, and calculate the similarity between text feature and fused video feature. For the Local to Local approach, we apply a Fine-grained Cross-modality Fusion Module (N-layer transformer) to further exploit the fine-grained correlation of text to vision and audio modalities. We analyze these multimodal fusion methods in the supplementary material.

Fine-tuning for Video Captioning Besides the video retrieval task, Video Captioning (Zhang et al. 2020; Lin, Gan, and Wang 2021; Wang et al. 2022) is another challenging task on video understanding, which aims to generate fluent natural language description of video contents. To conduct sentence generation, we introduce a Multimodal Caption Generator (N-layer transformer encoder) upon CLIP4VLA. At the $t^{th}$ decoding step, we feed previous generated words, vision frames and audio segments into CLIP4VLA. After intra-model encoding with three encoders, we concatenate the fine-grained features to construct multimodal sequence $U_{i}$. Input the sequence into Multimodal Caption Generator, the $t^{th}$ word is predicted as follows:

	$\displaystyle H_{i}$	$\displaystyle={\rm MCG}(U_{i}),$		(6)
	$\displaystyle p_{i}^{t}$	$\displaystyle={\rm softmax}(f(h_{i}^{t})),\quad h_{i}^{t}\in H_{i},$		(7)

where MCG refers to the Multimodal Caption Generator, $f(\cdot)$ is the linear output layer, $p_{i}^{t}$ is the predicted probabilities over the whole vocabulary size.

Pre-training Datasets Our pre-training data includes instructional video dataset Howto100M (Miech et al. 2019) and event video dataset Audioset (Gemmeke et al. 2017). To enable our audio encoder to distinguish verbal and nonverbal audio information, we choose [VB] as the audio type token for Howto100M and [NB] for Audioset, respectively. More details of data processing can be found in the supplementary material.

Fine-tuning Datasets We evaluate the pre-trained CLIP4VLA on retrieval and captioning benchmarks, including MSR-VTT (Xu et al. 2016), VATEX (Wang et al. 2019) and Audiocaps (Kim et al. 2019). After filtering out the silent videos, MSR-VTT remains 7867 and 884 videos for training and testing on the retrieval task, and 5867, 448, and 2617 videos for training, validation, and testing on the captioning task. VATEX remains 24667, 1427, and 1421 videos for training, validation, and testing on retrieval, and 24667, 2845, and 5698 videos for training, validation, and testing on captioning. Audiocaps keeps 49712, 495, and 967 videos for training, validation, and testing on the retrieval task. For training cost comparison with previous work, we further measure our model on event classification datasets of UCF101 (Soomro, Zamir, and Shah 2012) and ESC50 (Piczak 2015). The former one contains 13K videos of 101 action classes, and the latter one contains 2K audio clips of 50 classes.

Video Retrieval To demonstrate the effectiveness of our proposed CLIP4VLA, we first evaluate it for video retrieval on three benchmarks. Baselines could be grouped into three categories, corresponding to the three blocks in the table: 1) Classical Retrieval Methods: W2VV++ (Li et al. 2019), CE (Liu et al. 2019), HGR (Chen et al. 2020), MMT (Gabeur et al. 2020), SSB (Patrick et al. 2021), MoEE (Miech, Laptev, and Sivic 2018); 2) Pre-training based Methods: UniVL (Luo et al. 2020), ClipBERT (Lei et al. 2021), VLM (Xu et al. 2021); 3) CLIP-based Methods: CLIP (Portillo-Quintero, Ortiz-Bayliss, and Terashima-Marín 2021), CLIP-FRL (Chen et al. 2021), CLIP4Clip (Luo et al. 2021), CLIP2Video (Fang et al. 2021). Our model understands videos according to both vision and audio information. However, on visual-centric video datasets MSR-VTT and VATEX, not all videos contain audio information. To deal with the missing modality problem for each silent video during testing, we pair it with the audio of the most similar video, which is chosen from corresponding training set according to cosine similarity of global vision features. As shown in Table 1, firstly, our model CLIP4VLA achieves state-of-the-art performance on both MSR-VTT and VATEX datasets. Secondly, CLIP-based methods significantly outperform other baselines, which shows video understanding can benefit a lot from large-scale image-text pre-training. Thirdly, with audio content as extra input, our CLIP4VLA achieves better performance than other CLIP-based methods. This indicates that our model could well encode the correlation across text, vision and audio modality.

Video Captioning We further validate the adaptability of CLIP4VLA to video captioning task on MSRVTT and VATEXT. As shown in Table 3 and Table 4, our model achieves state-of-the-art captioning performance on both datasets as well. This indicates that our model also possesses good caption generation capability by leveraging well-aligned multimodal representations.

Audio Type Token To verify the validity of our proposed audio type token for different kinds of audio information encoding, we conduct the experiment to compare the audio retrieval performance when adjusting the mixing ratio of the two type embeddings of [NB] and [VB]. As shown in Figure 4, with the mixing ratio of [NB] embedding increased, the audio retrieval result on the Audiocaps dataset is significantly improved, because most of the audios in Audiocaps dataset are ambient sound. However, for the video datasets MSR-VTT and VATEX, the best results are yielded when the [NB] and [VB] embeddings are mixed with a ratio of 1:1, which further demonstrates that videos usually contain complex audios with both verbal and nonverbal information, while previous multimodal pre-training works have not specifically considered handling them simultaneously. The results on the three datasets show that our audio type token can effectively control the information aspect of encoded audio features for different application scenarios.

Key Components Table 5 ablates the contributions from key components of our model. The text-audio retrieval results on MSR-VTT and VATEX datasets consistently demonstrate the effectiveness of each proposed component. Especially, compared with row 1, directly initializing the audio backbone with vision backbone (row 2) has brought obvious gains, which further demonstrates that the audio information learning can benefit from existing visual knowledge.

Training Cost Fully exploiting the existing vision-text knowledge for audio pre-training can not only help the audio representation learning, but also reduce the training cost. In this section we compare the training cost and the classification performance on ECS50 and UCF101 with VATT (Akbari et al. 2021), which is a vision-text-audio model pre-trained from scratch. For fair comparison, we follow the VATT to train a linear classifier on top of the frozen multimodal backbones, and report the mean accuracy over official splits (5-fold and 3-fold cross validation for ESC50 and UCF101 respectively) . As the results shown in Table 6, our CLIP4VLA model achieves better downstream results with much less training cost, which demonstrates the advantages of learning audio from the existing visual-text knowledge.