TCR: Short Video Title Generation and Cover Selection with Attention Refinement

Text generation mainly includes text summarization and video captioning. The former aims to generate a short summary for a long document while preserving the main idea of the document. The latter focuses on describing the visual content of a video using condensed text.

Conventional text summarization utilizes a single text modality to generate summaries [16, 10]. Recent works [7, 6] show promising results on summarization with the help of other modalities, e.g., the visual modality. For instance, Liu et al. [9] propose both RNN-based and Transformer-based encoder-decoder models with the input of videos and its ground-truth/ASR transcripts. Encoder-decoder networks are widely used in video captioning where the encoder converts the input into high-level contextual representations before the decoder takes the representations and generates words as the caption. For example, Tan et al. [15] propose a RNN-based encoder-decoder model with three visual reasoning modules to generate video captions.

The above methods perform well on generating factual descriptions of a video scene given the video and the corresponding well-edited transcripts, yet fail when they intend to generate titles beyond factual descriptions based on noisy multimodal information. In contrast, our method can handle noisy modality data and capture the subtle relations between modalities, thus being able to generate both factual descriptions and appealing titles.

Cover selection aims to find a representative frame that conveys the main idea of a video. Song et al. [14] propose a thumbnail selection system using clustering methods. Ren et al. [12] select the best frame based on the prediction scores. They utilize only the visual modality, which is insufficient for real-world short videos where the keyframe should be determined by multiple modalities. Though Li et al. [7] propose VMSMO, to train a cover selection model based on multimodal information, i.e., news video and news article pairs, their model’s performance is highly dependent on the quality of news articles that are well-edited by humans. However, the SVTG dataset does not provide such manually-created articles. Therefore, it is inappropriate to apply their method to our task for comparison. Moreover, we extract covers using a unified model instead of an independent cover selection model, which is more straightforward.

Since the frames and the text inputs are both temporally ordered, we can directly combine them to explicitly learn the dependency between all pairs of text tokens and frames. We first preprocess the training data into the following formats:

• •

  Text Representation: given the transcripts extracted by ASR and OCR from a video, the combined ASR and OCR text is tokenized into a sequence of WordPieces [17], i.e, $T=(t_{1},t_{2},...,t_{n})$, where $n$ is the number of tokens in the sequence.

• •

  Title Representation: the title is represented as a sequence of tokens $Y=(y_{1},y_{2},...,y_{m})$, where $m$ denotes the number of tokens in $Y$.

• •

  Video Representation: given a sequence of video frames of length $L$, we feed it into a pre-trained ResNet-101 2D convolutional neural network [3] to obtain visual features $V=\{v_{l}\}_{l=1}^{L}\in\mathbb{R}^{L\times d_{v}}$, where $d_{v}$ is the feature dimension of the pretrained ResNet model.

The title-cover generator is composed of three embedding layers, 12-layer transformer encoders, and a language modeling (LM) head on top as the decoder. On the one hand, we regard the frames as $L$ virtual tokens $\{v_{i}\}_{i=1}^{L}$ and concatenate them with text tokens $T$ to form a new sequence of length $L+n$. We also add a special token [CLS] at the start of the sequence and another special token [SEP] at the sequence end. These tokens, including Y, are embedded by three embedding layers that are trainable: token-level embedding, position-level embedding, and segment-level embedding.

On the other hand, we feed the frame features $V$ into a fully connected (FC) layer to project it into the same lower-dimensional space as the text token embeddings and regard the projected frame features as the frame token embeddings. To establish a connection between the text tokens and frames so that they could attend to each other, we combine their respective embeddings by simply replacing the virtual token embeddings with the frame embeddings. Therefore, the final representations for text tokens and video frames can be obtained by summing up their corresponding three embeddings. We then feed these representations as the input to the transformer encoders.

Assume the input is $X_{VT}\in\mathbb{R}^{L^{\prime}\times d_{h}}$ where $d_{h}$ is the hidden size of our model, the output of the i-th transformer encoder is denoted as $O_{i}\in\mathbb{R}^{L^{\prime}\times d_{h}}$. Each encoder aggregates the output of the previous encoder using multiple self-attention heads, thus $O_{i}$ is computed by:

In this way, we can obtain visual-language fusion representations at different encoder layers. Let $A_{i}$ denotes the attention score of each head in the i-th encoder.

$Q$ and $K$ are the queries and keys linearly transformed from the input of each encoder. $\text{Attention Mask}\in\mathbb{R}^{L^{\prime}\times L^{\prime}}$ ($\text{elements}\in\{0,-\infty\}$) is used to make sure the target token to be generated only attending to the leftward information including itself, the generated tokens, frames and text tokens [2]. The LM head consists of two dense layers and a layer normalization (LN) layer in between to calculate the vocabulary distribution for decoding. Once the generator is well-trained, we select a frame with the highest attention score obtained in Equation 1 as the video cover.

Considering redundant noise may exist in the ASR/OCR transcripts and continuous frames, noise filtering is intuitive to help improve the quality of the generated titles. To this end, we propose an attention-based refinement module, which will automatically select not only the key parts of the multimodal input within each sample but also select higher-quality samples. Moreover, the module can interact with the above generator to find the best combination of different input tokens for progressively refining the generator.

Token-level refinement. Given a title-cover generator, we could first generate titles for the training data. We then acquire the cross-attention scores between the generated titles and the inputs, i.e., the ASR/OCR tokens and the frame tokens, through Equation 1. We assume the tokens that the generated titles attend to a lot are critical for building a generator with good performance. Therefore, we extract them as refined sentences/frames. For sentence refinement, we first locate the sentence that has the highest attention score with each generated token. Then, the top $u$ sentences which the generated title attends to frequently are naturally selected as the key sentences.

Video titles always contain some words (e.g., “alley-oop” in a basketball game video) that frequently appear in the ASR/OCR text but are rarely related to video frames. Therefore, we need to select the related frames at a higher granularity than the way we select related sentences. To achieve this, we first assign a $weight\in[0,L]$ to each frame according to its attention scores with each generated title token, then calculate the total weights for each frame. Higher weights indicate greater contributions to title generation. We finally pick out the top $v$ frames with the highest weights as the refined frames or keyframes for the video.

Sample-level refinement. In addition to refining the tokens within each sample, we denoise the data by selecting higher-quality data at the sample level. A training data will be deemed high-quality if the generated title by the generator $M$ is similar to its ground truth title. The similarity is computed by the Rouge score [8].

Apparently, with refinement at both token-level and sample-level, we could refine the title-cover generator multiple times until we get the best test results. See the detailed refinement algorithm in Supplementary C.

We compare our proposed method with the following baseline models:

Lead-3 directly selects the first three sentences from the text as the title [10].

HSG constructs a heterogeneous graph with sentence nodes and word nodes, then selects sentences as the summary for a document by node classification [16].

MAST is a model consists of encoder layers, a hierarchical attention layer, and a decoder that generates a textual summary on multimodal inputs [6].

NMT is an RNN-based model [1] designed for sequence-to-sequence tasks and modified for video description tasks.

MFN generates a summary based on video and its ASR or ground-truth transcript through a multistage fusion model with a forget gate module to remove the redundant information [9].

Bert2Bert contains a text BERT encoder, a video transformer encoder and a BERT decoder, which is a popular framework used in recent video/image + language works [4].

We evaluate the above baseline models and our TCR model on the SVTG dataset following the split criteria in Table 1. We implement our experiments using PyTorch [11] on an NVIDIA V100 GPU. To preprocess the videos, we sample 25 frames for each video and extract visual features from these sampled frames.

Our multimodal generator is adapted from a text pretrained model architecture [2]. The vocabulary size is 21,128. We use AdamW optimizer with a learning rate of $1e-5$. Linear warmup schedule is set for the first $10\%$ of the total training steps with linear decay. Dropout is adopted for regularization. The batch size is set to 16, and the maximum input sequence length is set to 512. During training, we only mask $20\%$ of the title tokens, and the maximum number of masked tokens is set to 20. We train up to 20 epochs with the cross-entropy loss computed by the predicted title tokens and the ground truth tokens. The validation set is used to select the best model with the highest mean score of the evaluation metric. For testing, we use beam search with beam size 5 to report the final test results. For refinement, we select the top 3 sentences/frames and perform one iteration.

In addition, we evaluate the TCR on the publicly available How2 dataset [13] to further validate its generation ability. The How2 dataset consists of 79,114 videos accompanied by corresponding user-generated descriptions and summaries. It has a train set of 73,993 samples, a validation set of 2,965 samples, and a test set of 2,156 samples. Since the TCR method is designed for handling real-world short videos with noisy multimodal information, we use the ASR transcripts instead of the provided descriptions following the work of Liu et al. [9] for a fair comparison.

Here we first report the performance of the baseline models and our proposed TCR for title generation. Then we show the evaluation results of cover selection. Later, we present the ablation tests from different aspects.

Title generation performance. We evaluate the quality of the generated titles by the standard Rouge F1 matric [8] following previous works [16, 6, 9, 10, 7]. R-1, R-2, and R-L refer to the unigram, bigram, and longest common sub-sequence overlap with the ground truth title, respectively.

We first compare the results of our method and the baseline models on SVTG, as listed in Table 2. We can observe from the table that TCR outperforms all the baseline models in terms of all Rouge scores. Specifically, TCR outperforms the best baseline model by 6.42 R-1 points, 5.24 R-2 points, and 5.17 R-L points, indicating its superior ability to learn video titles when given noisy ASR/OCR text and frames. Though these baseline models perform well in generating summaries and captions given user-annotated text descriptions, they are incapable of generating titles in social media language and handling noisy text information. In contrast, TCR can generate more accurate titles by fully exploiting the dependency between all text/frame tokens and denoised training.

To further examine the generation ability of TCR, we apply it to a public dataset, How2, that is designated to generate a factual summary for videos. Table 3 presents the results on How2. We use the dataset and baseline models in the work [9] for a fair comparison. See Supplementary D for model details about these baselines. We can observe from Table 3 that TCR outperforms the baseline models in all Rouge scores. Notably, it outperforms the best baseline MFN-rnn model by 4.5 R-L points. This finding again indicates that TCR is excellent at generation both within factual descriptions (e.g., summaries in How2) and beyond factual descriptions (e.g., titles in SVTG). Moreover, by comparing the performance of MFN in Table 2 and Table 3, we can conclude that MFN is unable to generalize to SVTG that is distinct from traditional video summary/caption datasets.

Cover selection performance. We compare the selected cover images with the default cover that comes with the videos by human judgments from the following aspects : Informativeness (which picture corresponds to the topic conveyed by the video title), and Superior (which picture is better to be a video cover that attracts you). See Supplementary E for more details about the human evaluation.

The results are reported in Table 4. We can observe from the table that participants cannot tell which one is better in most cases because the two pictures are almost identical. Moreover, our selected covers outperform the default covers by 23.5% in terms of Superior while they are equivalent in Informativeness, which demonstrates the effectiveness of our method to automatically select the cover image from a series of frames.

Ablation studies. All ablation studies are performed on the SVTG dataset. The last three rows in Table 2 show the ablation results on the effect of different modalities. From this table, we can see that TCR outperforms TCR w/o Visual and TCR w/o Text, which demonstrates that multimodal information can benefit title generation more than a single modality. It can also be easily observed that the text modality is dominant while the visual modality has less contribution in title generation because ASR- and OCR-detected text usually contains words that describe the highlights of stories in short video cultures. Supplementary F further provides the ablation tests on the number of selected sentences/frames and refinement iterations.

Fig. 3 shows examples including the generated titles by baseline models and our proposed model, as well as the cover selection comparison. The results from Fig. 3(a) demonstrate that our proposed model could generate a more accurate title than the baselines. Furthermore, as shown in Fig. 3(b), the selected frame is better than the default cover that comes with the video since the selected frame captures the jumping moment of the defence player while the default cover does not. To better understand the refinement process and what our model has learned, we also visualize the attention scores in Supplementary G.