M²-CLIP: A Multimodal, Multi-task Adapting Framework
for Video Action Recognition

Early action recognition algorithms mostly relied on end-to-end finetuning of models pretrained on ImageNet (Deng et al. 2009) with 2D CNNs (Wang et al. 2016; Lin, Gan, and Han 2019; Jiang et al. 2019), 3D CNNs (Feichtenhofer et al. 2019; Feichtenhofer 2020) and Transformers (Arnab et al. 2021; Li et al. 2022b; Liu et al. 2022; Bertasius, Wang, and Torresani 2021). ImageNet pretrained models were primarily used to initialize these networks’ backbones and expand to initialize some 3D convolutions (Li et al. 2022a; Carreira and Zisserman 2017). Recently, the advent of large-scale image-language pretrained models like CLIP brought about significant changes. Given CLIP’s strong performance on image-related tasks and remarkable generalization, researchers began full-finetuning from image to video based on CLIP. ActionCLIP (Wang et al. 2023) was an early work that transferred CLIP to video action recognition by adding a temporal modeling module to CLIP’s image branch, achieving competitive performance while keeping video generalization. X-CLIP (Ni et al. 2022) proposed frame-level temporal attention to reduce computation. These methods have achieved impressive results, but all require full finetuning on video data, making the training cost unaffordable for most researchers and practitioners.

The PEFT technique initially emerged in the NLP field (Houlsby et al. 2019) to address the challenges of full finetuning for large-scale language models. In video action recognition, this technique has also become a research hotspot in recent years (Ju et al. 2022; Lin et al. 2022b; Yang et al. 2023; Xing et al. 2023; Liu et al. 2023). EVL (Lin et al. 2022b) first proposed to leverage frozen CLIP image features with a lightweight spatiotemporal Transformer decoder to enhance video recognition tasks. ST-Adapter (Pan et al. 2022) introduced a parameter-efficient spatiotemporal adapter, which effectively harnesses the power of CLIP’s image models for video understanding. AIM (Yang et al. 2023) presented spatial, temporal, and joint adaptations to finetune pretrained image transformer models. These cost-effective methods have achieved SOTA performance, but they are all single-modal transfers, neglecting the text branch and thereby losing CLIP’s generalization ability. Vita-CLIP (Wasim et al. 2023) attempted to address this issue by adding prompts to both branches for transfer. However, we observed that its performance on temporally strong-correlated datasets was suboptimal, and their additional summary attention layers introduced in its visual branch increased the number of learnable parameters. In this work, we aim to apply PEFT to multi-modal frameworks to ensure competitive supervised performance with minimal increase in learnable parameters while maintaining strong generalization capabilities.

As illustrated in Fig. 3a, our framework comprises three key components: a video encoder, a text encoder, and a multi-task decoder. In this section, we will introduce the overview of the whole architecture and leave the details of the proposed multimodal adapters and the multi-task decoder in the following two sections.

Formally, the input to the framework is given as a video $\textbf{V}\in\mathbb{R}^{T\times H\times W\times 3}$ of spatial size $H\times W$ with $T$ sampled frames, and a text label y from a predefined label set $\mathcal{Y}$.

To better transfer CLIP to this task and enhance the semantic representation of action verbs in the labels, we introduce adapters for both the visual and text branches to improve their respective representation capabilities.

As shown in Fig. 3b, we first adopt a 1D temporal convolution for temporal feature enhancement. For the input of a TED-Adapter layer including the class token and patch tokens $\textbf{Z}=\{[\textbf{c}_{t},{\textbf{z}}_{v,t}]\}_{t=1}^{T}\in\mathbb{R}^{T\times(1+M)\times d_{v}}$, we perform the following operations:

where $\textbf{W}_{dn}$ and $\textbf{W}_{up}$ are the down-projection and up-projection weights. Conv1D represents the 1D-convolution for spatiotemporal modeling operating on the temporal dimension. Note that the reshape operations are omitted in this section for simplicity but are shown in Fig. 3b.

Next, for the temporal difference modeling, we subtract the previous frame’s feature from the current frame and then employ a 2D spatial convolution to learn useful information from the adjacent feature differences automatically. Formally, when given the input patch tokens $\textbf{z}_{v,t}$ the of the $t$-th frame,

where Conv2D represents the 2D spatial convolution. For the first frame, we set its feature differences to zeros.

Finally, the output of TED-Adapter can be obtained by fusing the two kinds of temporal features together. Moreover, a residual summation is applied to preserve the information in the input:

where $\textbf{Z}_{D}=\{[\textbf{O},\textbf{z}_{D,t}]\}_{t=1}^{T}$, and O is a zero matrix which has the same shape as $\textbf{c}_{t}$.

The TED-Adapter is simply placed before the Multi-Head Self-Attention (MHSA) by default unless otherwise specified. By incorporating the temporal enhancement and temporal difference operations, the proposed TED-Adapter can capture spatiotemporal features and local finer motion patterns, which are both crucial for this task.

The text adapters are inserted before the Feed-Forward Networks (FFN) of the transformer layer by default. By incorporating the text adapter, the model can enhance its understanding of the action labels, capturing more discriminative semantic information. This allows for improved alignment between the textual and visual representations, resulting in more accurate and effective video action recognition.

As previously described, we observed that when utilizing CLIP’s multimodal framework, relying solely on contrastive learning did not perform as well as the equivalently configured unimodal framework. To address this, we propose a multi-task decoder equipped with four distinct learning tasks, each corresponding to a separate head, as shown in the right part of Fig. 3a. This approach aims to leverage multiple task constraints to improve the joint representation power of the multimodal framework.

When a model is trained, it will be ready for zero-shot classification. In practice, the text input can be prompted like “a video of $<\hat{\textbf{y}}>$”, where $\hat{\textbf{y}}$ is a category name of $C$ classes. The process of predicting $\hat{\textbf{y}}$ of a certain video V is to find the highest similarity score calculated by:

In summary, by introducing these four learning tasks, we tap into a richer set of supervisory signals, guiding the model to better align the visual and textual modalities while simultaneously capturing various aspects of semantic information. This multi-task approach not only mitigates the performance disparity of supervised learning but also preserves CLIP’s remarkable generalization capabilities.

We evaluate our M²-CLIP for supervised learning in two primary datasets: Kinetics-400 (K400) (Kay et al. 2017) and Something-Something-V2 (SSv2) (Goyal et al. 2017). For the generalization evaluation, we test our model on UCF101 (Soomro, Zamir, and Shah 2012) and HMDB51 (Kuehne et al. 2011). We employ ViT-B/16 based CLIP as our backbone and use a sparse frame sampling strategy with 8, 16, or 32 frames during training and inference.

We present our results of K400 and SSV2 in Tab. 1 and Tab. 2, respectively, comparing our approach with SOTAs trained under various transfer methods, including full finetuning, unimodal and multimodal PEFT from frozen CLIP.

On K400, our 8-frame M²-CLIP-B/16 model surpasses models pretrained by ImageNet (Deng et al. 2009), achieving higher performance with fewer learnable parameters and computational requirements. Compared to end-to-end finetuned CLIP models with the same ViT-B/16 backbones, our approach demonstrates comparable results. With just 11% of the adjustable parameters, we exceed ActionCLIP (Wang et al. 2023)’s results (84.1% vs. 83.8%). Furthermore, our method is competitive with the latest approaches like X-CLIP (Ni et al. 2022) and S-ViT (Zhao et al. 2023), yet with much fewer learnable parameters. In addition, while our results fall slightly short when compared to the leader performances achieved by BIKE (Wu et al. 2023) and ILA (Tu et al. 2023), it’s important to note that they employed a much larger network architecture (ViT-L) and had 14 times the number of tunable parameters as our model. Compared with the unimodal PEFT approaches, our method achieves comparable or even superior results. For instance, our 8-frame M²-CLIP-B/16 model outperforms 8-frame ST-Adapter-B/16 (Pan et al. 2022) by 1.4%. It is worth noting that while unimodal methods exhibit high performance in supervised settings, they lack support for zero-shot generalization. In contrast, our approach achieves competitive results with them and demonstrates strong generalizations. Lastly, compared with multimodal PEFT approaches, our method achieves superior results. Note that Vita-CLIP (Wasim et al. 2023) is a multimodal Prompt-based method while we use adapters. It is evident that we achieve higher performance with only 41% trainable parameters.

As for SSv2, our approach achieves comparable performance and even surpasses several full-finetuned methods with similar configurations, such as Mformer-B (Patrick et al. 2021) and ILA-ViT-B (Tu et al. 2023), while utilizing fewer trainable parameters and computational resources. Compared with unimodal PEFT methods, our 8-frame M²-CLIP model surpasses AIM-ViT-B/16 (Yang et al. 2023) and EVL-ViT-B/16 (Lin et al. 2022b) in Top-1 performance and maintains a competitive position compared to other methods. In the domain of multimodal PEFT approaches, our method outperforms the recent method Vita-CLIP (Wasim et al. 2023) by a large margin of over 18%. The results demonstrate that our proposed multimodal adapters and the multi-task decoder are helpful strategies for efficient multimodal CLIP-based image-to-video knowledge transfer.

In this experiment, we employ the M²-CLIP-B/16 model pre-trained on K400, with 8 frames as input, for conducting generalization experiments on UCF101 and HMDB51. We use the outputs of the contrastive learning head for classification. It’s worth noting that the model used for testing is consistent with the one mentioned in the supervised experiments in Tab. 1, as Vita-CLIP(Wasim et al. 2023).

Our approach demonstrates impressive generalization capabilities as shown in Tab. 3. All methods in this table, except ResT (Lin et al. 2022a), are transferred from CLIP. In comparison to methods that require full finetuning, our results on both datasets outperform them by a significant margin. For instance, we surpass X-CLIP (Ni et al. 2022) by 2.5% on HMDB51 and 6.7% on UCF101. Moreover, our method requires far fewer trainable parameters than these approaches. Among other methods that do not require full finetuning and are based on Prompts, our method also outperforms the majority. While we may have a 1.5% lower performance than Vita-CLIP on HMDB51, our model uses only 41% of the trainable parameters of them and has better supervised results. Furthermore, our accuracy on UCF101 is 2.7% higher than theirs, making our approach the leader among all the methods on this dataset.

In this section, unless otherwise specified, we use ViT-B/16 as the backbone and 8 input frames in all ablation experiments on K400.
Effectiveness of Components. In Tab. 4, we first construct a CLIP frozen baseline and use its zero-shot performance as our baseline. Then, we gradually introduced our contributions based on this baseline. We have observed a significant performance boost when we introduce the learnable temporal modeling module, TED-Adapter, demonstrating its effectiveness. Subsequently, we add the Text-Adapter, which formulates the multimodal adapters with TED-Adapter to CLIP and further improves the performance. Finally, by adding the Multi-task decoder, we form the final M²-CLIP, incorporating multiple learning objectives and resulting in a substantial improvement. In summary, our proposed multimodal adapter and multi-task decoder are both highly effective and modular components that can be easily integrated into CLIP transfer frameworks as plug-and-play modules.
Ablations for Video TED-Adapter. We next conducted ablation experiments on Video TED-Adapter in Fig. 4a. We use one TED-Adapter with bottleneck width 384 as ST-Adapter (Pan et al. 2022). First, we attempt to add the TED-Adapter to the front half and the back half of the network. We have observed that although more TED-Adapters generally lead to better results, the benefits gained from adding them to deeper layers outweigh those from shallow layers. Secondly, as our TED-Adapter includes temporal enhancement (TE) and temporal difference (TD), we conducted separate experiments, revealing that the improvement from TE is more pronounced, but the combination of both yields the best performance. Finally, we found that the parallel addition of components is slightly better than the sequential one.
Ablations for Text-Adapter. We progressively added a varying number of Text-Adapters from deep to shallow. We show the performance changes in Fig. 4b, along with the corresponding zero-shot transfer performance on UCF101. It can be observed that as the number of Text-Adapters increases, the model’s performance first improves and then starts to decline slightly, with the best performance achieved when adding just one Text-Adapter. The zero-shot results exhibit a similar trend of variation. We believe the reason is that the text data contains only label information, and having too many Text-Adapters may lead the model to overfit on these labels, ultimately affecting overall performance and generalization. Therefore, our final model includes one Text-Adapter, balancing performance and generalization.
Ablations for Multi-Task Decoder. In Fig. 4c, we evaluate the impact of the individual heads in the decoder. It is evident that each head in our model contributes positively to the results. By incorporating CMC on top of the original CLIP’s contrastive learning, we achieve additional 0.7% improvements in performance, and notably, this enhancement is parameter-free. Furthermore, the inclusion of CMLM further boosted the results. Lastly, the addition of the VC head elevated the performance to 83.4%. This comprehensive analysis demonstrates our multi-task decoder’s effectiveness in enhancing the multimodal framework’s overall learning.