Multimodal Frame-Scoring Transformer for Video Summarization

Our Intuition about Video Summarization. When humans watch a video on YouTube or Netflix, they perceive visual, linguistic, and audio information through various sense organs and know which parts of the video are interesting. To consider whether a scene in a movie is absorbing, for example, we observe characters’ facial expressions and actions, recognize background and situation with language, and listen to the characters’ utterances and sound effects. Intuitively, humans have access to well-defined scoring function in their mind, using versatile sensory systems, while video summarization models from previous studies did not.

Two Underlying Research challenges. Video summarization aims to capture key frames using predicted frame-wise importance scores, given datasets as shown in Figure 1. Despite its importance and convenience, video summarization has two inherent challenges: (i) Most previous approaches exploit just visual features, leaving other modality features behind Zhao et al. (2018); Rochan et al. (2018); Zhang et al. (2018); Zhou et al. (2018); Jung et al. (2019); Rochan and Wang (2019); Park et al. (2020); Jung et al. (2020); Ghauri et al. (2021). (ii) Since the existing datasets Gygli et al. (2014); Song et al. (2015), which consist only of videos and frame-level ground truths, relatively insufficient to train a caption generator used for extracting text information from the datasets and to train audio-visual-text feature extractors. Note that frame-wise human-scored video dataset is expensive to obtain compared to other common video datasets.

Language-attended methods exploit extracted text information from videos and predict importance score based on visual and text features. Bor-Chun Chen and Chen (2017) jointly combines video summarization and video caption model and trains the recurrent network in end-to-end manner. Narasimhan et al. (2021) leverages language-guided video summarization model given videos and corresponding user query or automatically generated video captions. Haopeng et al. (2022) collects video titles and descriptions for pre-training the language-attended self-supervised learning model.

Though language-guided approaches alleviate modality issue somewhat, there are underlying constraints in conveying vivid audio features into a video summarization model (e.g., we know there are limits to expressing a beautiful song just with the text “beautiful song”).

Our Solutions. This paper proposes Multimodal Frame-Scoring Transfomer (MFST) to handle the modality combinations and frame-level scoring for video summarization. Note that existing datasets for generic video summarization Gygli et al. (2014); Song et al. (2015) are relatively insufficient to pretrain a dense caption generator and audio, visual, and text feature extractors. We investigate a new multimodal setting where it can mitigate the lack of human-scored videos used for training video summarization model.

To this end, our framework consists of three stages: (i) Generating dense video captions using learned caption generator and extracting each modality features (audio-visual-text) using feature encoders. (ii) Multimodal representation using caption-guided attention mechanism and coarse-grained projection based on modality fusion (iii) Frame-scoring transformer that takes audio-visual-text multimodal representations as input and predicts frame-level scores.

Our extensive experiments with previous models and ablation studies on TVSum and SumMe datasets demonstrate the effectiveness and superiority of our proposed method by a large margin in both F1 score and Rank-based evaluation.

Our Contributions. The main contributions of this work are summarized as below:

• •

  To the best of our knowledge, Our MFST is the first to introduce frame-scoring transformer exploiting multimodal features (audio-visual-text) for generic video summarization task.

• •

  We investigate a new multimodal setting where it can mitigate the lack of human-scored videos used for training generic video summarization model exploiting pretrained modules.

• •

  Our empirical study on generic video summarization datasets (TVSum and SumMe) demonstrates that MFST can surpass its all counterparts by nontrivial margins, and attests the effectiveness and superiority of our approach.

Two broad categories of Video Summarization. In video summarization task, there are two broad categories of methods: (i) generic video summarization Park et al. (2020); Jung et al. (2020); Ghauri et al. (2021); Narasimhan et al. (2021); Haopeng et al. (2022) and (ii) query-guided video summarization Narasimhan et al. (2021); Wu et al. (2022); Liu et al. (2022); Jiang and Mu (2022).

Generic Video Summarization. The First category of methods aim to extract representative frames from original videos using well-defined frame-wise scoring function. Existing models focus on both supervised learning and unsupervised learning. Zhou et al. (2018) designed a reward function which determines diversity and representativeness of generated summaries based on end-to-end reinforcement learning framework. Rochan et al. (2018) tried to solve video summarization as a sequence labeling problem based on fully convolutional sequence models. Zhang et al. (2018) proposed the retrospective encoder to embed both predicted summary and original video. Zhao et al. (2018) integrated shot-level segmentation and video summary into a hierarchical RNN. Jung et al. (2019) proposed variance loss with variational autoencoder and generative adversarial networks. Rochan and Wang (2019) learned a mapping function between raw videos and summarized videos because this kind of dataset is much easier to gain. Yuan et al. (2019) proposed a cycle-consistent adversarial networks which consist of frame selector and evaluator. Jung et al. (2020) exploited global and local input decomposition to capture the interdependencies of video frames. To represent a relation graph, Park et al. (2020) leveraged recursive graph modeling networks.

Noted that, these methods just employs visual features, leaving other modality features behind. In this paper, we investigate a new multimodal setting for video summarization including audio-visual-text features.

Query-Guided Video Summarization. The second category of methods find relevant moments according to user-defined query. Unlike generic video summarization, most query-guided models take Query-Focused Video Summarization Sharghi et al. (2016) dataset, UT Egocentric Lee et al. (2012) dataset, and QVHighlights Lei et al. (2021) dataset as input. Sharghi et al. (2017) introduced a parametrized memory network into query-focuesd video summarization. Wei et al. (2018) proposed a semantic attended network which consists of frame selector and video descriptor. Kanehira et al. (2018) investigated how to divide videos into groups under the assumption that video summaries extracted from similar videos should be similar. Narasimhan et al. (2021) leverages a framework for handling both generic model and query-guided model given videos and corresponding user query or automatically generated video captions. To effectively address generic queries from different modalities Wu et al. (2022) introduced a graph convolutional networks which is used for both summary module and intent module. Liu et al. (2022) proposed the unified multimodal transformer to cover different input modality combinations. Jiang and Mu (2022) jointly leveraged video summarization model and moment localization model.

Note that though query-driven approach is necessary because defining salient scenes is often subjective, it is difficult to apply if we do not know the contents of the video or if we do not need a subjective summary (e.g., YouTube video previews). In this paper, we aim to tackle the first category based on a novel multimodal frame-scoring framework.

Frame-Scoring Transformer. Some recent works apply transformer Vaswani et al. (2017) for generating video summaries. Narasimhan et al. (2021) employed transformer for predicting importance score using language-attended representation. Liu et al. (2022) leveraged modality encoder, query generator, and query decoder using transformer framework. Compared to Narasimhan et al. (2021), we propose frame-scoring transformer exploiting audio-visual-text modality represenatation.

Our Goal. In this section, we present the proposed MFST framework (as shown in Figure 2) with extracted modality features. we consider the standard generic video summarization setting, where given a set of videos $V$ and ground truth frame scores $S_{gt}$, the goal is to minimize the loss $\mathcal{L}_{\theta}$ with respect to predicted frame scores $S$:

Multimodal Feature Extraction. Note that existing datasets for generic video summarization Gygli et al. (2014); Song et al. (2015) are relatively insufficient to train a caption generator and audio, visual, and text feature extractors. Considering that large-scale videos scored by humans are not available, we co-opt a pretrained caption generator and feature extractors for each modality:

, where $C$ denotes a set of dense video captions, $g_{c}$ is a dense video caption generator Iashin and Rahtu (2020a), $f_{t}$ is a CLIP-based text feature extractor Xu et al. (2021), $f_{v}$ is a CLIP-based visual feature extractor Xu et al. (2021), and $f_{a}$ is a learned audio feature extractor in Baevski et al. (2020). Here, we use audio-visual-text features at the same time like human to make it easier for our model in understanding video context and demoting peripheral parts.

Fine and Coarse Spaces. Since our model handles three modality features, $\mathcal{T},\mathcal{V}$, and $\mathcal{A}$, configuration of modality space should be considered. Fine and coarse spaces (FAC) Alayrac et al. (2020) combines the modalities into a common embedding, while preserving fine-grained information and integrating modality features. Inspired by Alayrac et al. (2020), but unlike the study, we first propose to learn visual-text granularities in the FAC modality embedding graphs.

Though video caption methods based on automatic speech recognition are useful Hessel et al. (2019); Alayrac et al. (2020); Iashin and Rahtu (2020b), most videos in generic video summarization rarely have human dialogues. Note that in real-world applications, not all videos contain human voice, nor we always have to figure out exactly what people are saying for highlight detection (e.g., croud cheers at a soccer game).

In this paper, we co-opt the feature-based video cation generator $g_{c}$ Iashin and Rahtu (2020a) to extract dense video captions $C=\{C_{1},C_{2},...,C_{N}\}$. Then, we leverage a fine-grained embedding space where visual features $\mathcal{V}=\{\mathcal{V}_{1},\mathcal{V}_{2},...,\mathcal{V}_{N}\}$ and text features $\mathcal{T}=\{\mathcal{T}_{1},\mathcal{T}_{2},...,\mathcal{T}_{N}\}$ lie. Note that Liang et al. (2022) demonstrates ”multimodal video-text pretraining” paradigm can not solve the modality gap phenomenon completely which causes performance degradation. In this work, we compute text-attended visual representation using attention layer:

, where $h$ stands for feature embedding space. Intuitively, in the fine-grained embedding space, each $v_{i,l}$ from $\mathcal{V}_{i}$ chooses the most relevant caption ${t_{i}^{*}}$ in the $\mathcal{T}_{i}$ using attention mechanism.

Lastly, we project the fine-grained embedding space into the coarse-grained embedding space by modality fusion:

, where $\mathcal{F}_{M}$ denotes function of fusion and $h_{\mathcal{A}\mathcal{V}\mathcal{T}}$ represents coarse-grained feature representation. The complete process is summarized in Algorithm 1. An important note is though extracting the feature of three modalities has a cost of time, due to the transformer architecture and fusion operation, the cost of training and model size are not increased, compared to existing models using single or bimodal features.

Multimodal Frame-Scoring Transformer. Frame-scoring transformer Narasimhan et al. (2021) takes feature representation as input and predict importance scores. Note that previous approach exploits just text-visual representation, leaving audio features behind. MFST introduces frame-scoring transformer to video summarization, which is modified to predict frame-level importance scores $S$, based on coarse-grained feature representation $h_{\mathcal{A}\mathcal{V}\mathcal{T}}$:

, where $W^{Q_{\mathcal{V}\mathcal{T}\mathcal{A}}}$, $W^{K_{\mathcal{V}\mathcal{T}\mathcal{A}}}$, and $W^{V_{\mathcal{V}\mathcal{T}\mathcal{A}}}$ denote parameter matrices and $d_{k}$ is the dimension of $K_{\mathcal{A}\mathcal{V}\mathcal{T}}$.

Finally, we feed $h_{\mathcal{A}\mathcal{V}\mathcal{T}}$ to frame-scoring transformer with positional encodings at the bottom of transformer encoder and decoder stacks. Given ground truth frame scores $S_{gt}$ of N frames from a video, we train MFST using the mean square error:

The complete process is summarized in Algorithm 2.

The conclusion of this paper is threefold:

• •

  We propose MFST, a simple and effective frame-scoring framework given videos. MFST exploits audio-visual-text features using learned feature extractors and frame-scoring multimodal transformer. MFST first generates caption-attended representation on fine-grained embedding space using attention mechanism. Then, our model project fine-grained space to coarse-grained space based on modality fusion.

• •

  We wrestle with two underlying research questions for generic video summarization: (1) How to alleviate data sparsity problem? and (2) How to learn diverse modality features to predict importance score better? The first question arose because summarization datasets consist of frame-level human annotations and they are hard to collect. The second question should be solved in order to create a video summarization model considering multimodal information as people do.

• •

  Our comprehensive comparisons with previous approaches and ablation studies on generic video summarization datasets (TVSum and SumMe) evince that MFST can surpass its all counterparts by nontrivial margins, and attests the effectiveness and superiority of our approach.

Limitations. Though we demonstrate the effectiveness and superiority of our proposed method, lack of deep insights about modality representations and utilizing large-scale models are biggest limitations of our work. For example, we could not discover precise reasons why our framework is superior with respect to modality representations.

Additionally, This work also lacks the scrutiny to represent the modality well in the common space. In other words, we recognize that this work doesn’t provide experiments using various model architectures to reflect modality information to our model.

Despite these limitations, it is clear that our approach handle three different modalities well and achieves state-of-the-art performance by a large margin in F1 score and Rank-based evaluation by nontrivial margins.

Future Works. Our direction of future work is threefold:

• •

  Proving that our framework can circumvent data-hungry and sensitive to the overfitting issues, caused by the transformer structure, both theoretically and empirically.

• •

  Further study to express each modality information well in common space (e.g., contrastive learning).

• •

  Further research and ablations using existing approaches and various model architectures to better reflect modality information.