Contrastive Graph Multimodal Model for Text Classification in Videos

As mentioned in section 1, multimodal signals can provide abundant discrimination information so that we need to extract the multimodal representation of the texts. In this paper, we mainly extract visual, textual and coordinate features.

For visual modality, in contrast to conventional approaches which usually used classical VGG [15] or ResNet-based [16] network, we construct a shallow neural network as the backbone. In fact, from Figure 1, we observe that texts differ in low-level features of colors, fonts and sizes, thus above deeper networks are abandoned as high-level semantic information is extracted which is not our need. Correspondingly, BERT [17] is used as the backbone of textual extraction. Upper-left and bottom-right coordinates represent the position of text in the frame.

The fusion of multimodal features is a critical step to get the multimodal representation of video text. The process of visual and positional modalities is shown as bellow:

where $f_{fra}$ and $f_{pos}$ as raw visual and positional features respectively. Firstly, ROIAlign [18] is utilized to extract patch feature using $f_{pos}$ and $f_{fra}$ as its inputs. Then, we use a transformer [19] to learn implicit relation between a patch box and its corresponding full frame. Finally, the visual feature $f_{vis}$ and textual feature $f_{text}$ are simply concatenated.

Due to limited labeled data, we elaborately utilize a contrastive learning [20] strategy to fully explore inherent connections between samples using plentiful unlabeled videos. Fig.3 illustrates the overview of the contrastive learning framework. It learns a representation which embed positive input pairs nearby, while pushing negative pairs far apart.

Given a batch of $N$ samples $x_{1},x_{2},\dots,x_{N}$, the positive sample $\hat{x_{i}}$ of an anchor sample $x_{i}$ is established by disturbing its coordinate, changing image color, or replacing text with synonyms in lexical combinations. Thus a positive input pair is combined as $<x_{i},\hat{x_{i}}>$. The negative pair is formed as $<x_{i},x_{j}>$, where $j\in\{1,\dots,N\},j\neq i$. Positive and negative sample pairs are fed to contrastive learning frameworks.

Based on the trained backbone of unimodal by contrastive learning, we attempt to train video text classification model using the labeled videos by two strategies: fine-tuning and joint learning. Fine-tuning only uses traditional supervised learning to deal with our task. Joint learning combines the self-supervised contrastive learning and original supervised learning. The joint loss is denoted as follows:

Edited texts from news videos, such as texts of caption and subtitle, usually have regular layout. Figure 1 shows examples of caption and subtitle on some video frames. It is observed that the text of caption appears above that of person information, displayed in Fig.1(a), while the text of person information appears above that of subtitle, shown in Fig.1(b). This phenomenon inspires us to utilize the structural information to reinforce the feature representation. Motivated by Graph Neural Network [21], CorrelationNet is proposed to model the relations among adjacent texts.

For a certain text box $a_{j}$, consider its adjacent neighbors $a_{1},\dots,a_{N}$. The total $N$ text boxes are used to generate the aggregated feature of $f^{(b)}(a_{j})$. Features of $N$ neighboring text boxes are weighted by a attention mechanism and then summed to get the fused feature. The graph constructed by the $N$ text boxes is illustrated in Fig.4.

$w(a_{j},a_{k})$ is denoted as the weight between two text boxes $a_{j}$ and $a_{k}$. Concretely, features are transformed by a fully connected network and then do subtraction to get a representation measuring the difference. Finally, a multilayer perceptron (MLP) is applied to get the weight.

After calculating all the weights between $a_{j}$ and its neighboring texts, the aggregated feature is obtained:

By using weighted aggregation strategy, CorrelationNet can learn the contribution of each modality by highlighting its importance. The final multimodal feature $a_{j}$, which is fed to classification head, is:

There are no publicly available datasets with unified annotations for video text recognition and classification. To promote this new area, we construct a new well-defined dataset of news videos, called TI-News, which is dedicated especially to studying video text recognition and classification tasks jointly. Over 450000 text-box samples are annotated in TI-News, which are extracted from over 100 videos in 23 news programs. In training phase, 350000 samples come from 90 videos related to 8 programs. In testing phase, 100000 samples are selected for evaluation, from which 50000 samples are collected from the same programs with the training dataset to construct TI-News standard dataset, while 50000 samples are collected from extra 15 distinct videos to construct TI-News generalization dataset for generalization testing.

We use a general OCR engine to locate and recognize all the texts in videos. Then, an annotation system for video text classification task is designed. Three representative categories are defined. They are Caption, Subtitle and Person information. In addition, class Others is used to denote the text boxes which not belong to above-mentioned categories. Besides, abundant kinds of pre-trained models involved in visual, audio and text features extraction are presented, which are pre-trained by millions of short videos using contrastive learning. Therefore, the pre-trained models have strong ability of feature representation.

For visual modality, video frame is first resized to the shape of $384\times 480\times 3$. A simple model building with 3-layers CNN is used as feature extractor. Correspondingly, a 4-TransformerLayers BERT with embedding dimension of 768 is used to extract textual features. A stacked transformer block with 2 TransformerEncoderLayers (d_model=256, nhead=8, dim_feedforward=256) is used to implicitly learn the relation between patch box and full image. Adam is chosen as the optimizer with a learning rate of 0.0002.

We use standard precision, recall and F1 score to evaluate the performance of CGMM. To exploit the effectiveness of several designed modules, ablation study is applied. Experiments on TI-News standard set and generalization set are conducted.

Results on TI-News standard dataset and TI-News generalization dataset are shown in Table 1 and Table 2 respectively. In Table 1, the precision and recall of CGMM exceed 89.05% and 92.34% in standard set, with 90.67% as its F1 score, which lead to a splendid performance of video-text classification task. Moreover, it is easy to find that we can’t get good results with single modality. In Table 2, the precision and recall of CGMM exceed 84.33% and 87.44% in generalization set, with 85.86% as its F1 score, which shows that our method has good generalization. If only consider the visual modality, the results are shown to be an random distribution. By incorporating text modality, precision has increased a lot in the category of caption and subtitle. Furthermore, CorrelationNet can aggregate features of neighboring text boxes, which gives great help to the classification model. Also, contrastive learning is effective to enhance the modality representations.