CBVS: A Large-Scale Chinese Image-Text Benchmark for Real-World Short Video Search Scenarios

Compared to English multi-modal pre-training, the Chinese community is lagging behind. Yang et al. (2022) introduces translated versions of the English multi-modal datasets Chen et al. (2015); Krishna et al. (2017) to support Chinese multi-modal pre-training. Gu et al. (2022) releases a large-scale Chinese dataset Wukong containing 100 million image-text pairs collected from the web to bridge the language gap. Xie et al. (2023) further establishes large-scale Chinese cross-modal benchmarks by releasing two pre-training datasets and five fine-tuning datasets. Besides, Product1M Zhan et al. (2021) provides additions to the e-commerce domain. Zhang et al. (2022b); Nie et al. (2022); Xu et al. (2023) supplements the Chinese video-text data, and visual modalities are provided in the form of video frames. However, large-scale video cover data is scarce.

The image-text matching task aims to measure the semantic similarity of different modalities in the same embedding spaceAbdullah and Rangarajan (2021). Existing implementations fall into two categories: the first is embedding-based, i.e., encoding global representations of vision and text separately, and then performing similarity computation Chen et al. (2021); Faghri et al. (2017); Qu et al. (2020). The second is score-based, i.e., performing cross-modal interactions locally and calculating cumulative scores Chen et al. (2020); Diao et al. (2021); Liu et al. (2020). Due to the advantages of performance and efficiency, embedding-based methods have attracted the attention of researchers Fu et al. (2023). In particular, CLIP Radford et al. (2021) provides new ideas for multi-modal representation learning. A series of studies following CLIP have been applied to downstream tasks including image text matching. For example, GCLIPLi et al. (2022), CLIP-AdapterZhang et al. (2021) and SLIPMu et al. (2022) raise the upper performance limit of CLIP, AltCLIPChen et al. (2022), CN-CLIPYang et al. (2022), and TaiyiCLIPZhang et al. (2022a) expand the CLIP language domain.

A comparison with other Chinese image-text/video-text datasets is shown in Tab. 1. On the one hand, CBVS provides cover images, user queries, and is larger in size compared to publicly available video datasets such as VATEX Wang et al. (2019), BFVD/FFVD Zhang et al. (2020), and CNVid-3.5M Gan et al. (2023). CREATE Zhang et al. (2022b), Kwai-SVC Nie et al. (2022), ALIVOL-10M Lei et al. (2021) with similar scale are access restricted. On the other hand, compared to image-text datasets such as Wukong Gu et al. (2022), Product1M Zhan et al. (2021), M6-Corpus Lin et al. (2021), ZERO-Corpus/R2D2 Xie et al. (2023), etc., the biggest advantage of CBVS lies in the uniqueness of the video cover image and the cover text specific to the cover image. To the best of our knowledge, CBVS is the largest publicly available Chinese image-text dataset providing cover images.

In order to provide real video search data, we capture the user query logs of QQ Browser and divide user queries into two parts. We retrieve more than 8M videos from the Chinese video website BiliBili³³3https://www.bilibili.com through the first part of user queries. To avoid data distribution bias due to a single platform, we retrieve more than 5M videos from Tencent Video⁴⁴4https://v.qq.com as a supplement in the same way, and finally obtain more than 13M $<$cover-title$>$ pairs as a data source for CBVS-5M/10M. Besides, we manually select more than 2K high-quality user queries from the second part, and collect 20K high-quality $<$user query-cover image$>$ pairs in the same way, as the data source of CBVS-20K. The number of cover images under each user query is controlled in $[5,30)$.

We design the data cleaning program from two aspects: data quality and image-text relevance. First, we filter the video covers with low resolution and scale disproportion, and eliminate the dead links. After that, we score the relevance of video covers and titles in 13M data with the open-source Chinese image-text model QA-CLIP⁵⁵5https://github.com/TencentARC-QQ/QA-CLIP, and filter the trailing 3M data to obtain CBVS-10M. We randomly sample 1K data from 13M and 10M data, respectively, and human experts evaluate whether the video cover is relevant to the title. The evaluation conclusions show that the relevance is improved from 75.6% to 93.0% after data cleaning. Finally, CBVS-5M is obtained by sampling in CBVS-10M.

The data annotation of CBVS-20K is performed by trained experts in the field of video search in a two-stage, cross-validated approach. First, they annotate whether the user query reveals a clear need for video, and filter out query terms related to pornography and violence, and finally take the queries with a need for video as candidates for annotation in the second stage. After that, the annotators mark the degree of relevance for each $<$user query-cover image$>$ pair, which is categorized into three grades: strongly relevant, weakly relevant, and irrelevant. The semantics of cover images and cover texts are required to be considered together. Meanwhile, the annotators correct the OCR text extracted by the machine.

We exclude the controversial data, and end up with 20,001 image-text pairs consisting of 2,486 unique queries and 19,648 unique images. The percentage of strongly relevant, weakly relevant and irrelevant data is 29.74%, 30.80% and 39.46% respectively. The average length of user queries is 7.0 and the average length of OCR texts is 14.5. 33.41% of cover images come with OCR texts. Detailed data distribution is provided in the supplementary material.

We follow CLIP Radford et al. (2021) to co-train the image encoder with the text encoder, taking InfoNCE Loss Oord et al. (2018) as the Image-Text Contrastive (ITC) loss $L_{ITC}$, as shown in Fig. 3. Specifically, we maximize the similarity scores of the matched image and text embeddings in terms of batch. We adopt ViT Dosovitskiy et al. (2020) and RoBERTa Liu et al. (2019) as the visual and textual skeletons, respectively, and introduce the weight initialization of QA-CLIP.

In particular, to bridge the gap between the data morphology of the title and the user query, we employ a Chinese word-splitting component, Lexical Analysis of Chinese (LAC)⁶⁶6https://github.com/baidu/lac. In order to simulate the morphological distribution of the user query, the result of the lexical segmentation is composed into a string using spaces as the splice character. For the case of failed word splitting, the original title is employed. This setting takes effect for all fine-tuning tasks unless otherwise specified.

Video cover images differ from open domain images in that they partially carry cover texts. One option is to outsource the cover text understanding task to an external OCR engine and fuse the cover image with the cover text on the image side in an ALBEF Li et al. (2021) manner. However, the cost of image-text similarity inference becomes expensive. In addition, the image-text contrastive learning becomes highly dependent on the accuracy of the OCR engine, which creates an obstacle for model generalization.

Differently, we are inspired by Kim et al. (2022); Davis et al. (2022) to design UniCLIP in an OCR-free form. One idea is to guide the ViT to perceive the cover texts during the training process through agent tasks, so that it relies on no module related to the OCR function in the inference process. Since the presence of cover text is uncertain, we propose the presence-guided encoder, where the first agent task of UniCLIP is set as an Image Classification (IC) task: ”To determine whether an image carries cover texts”.

Specifically, as shown in Fig. 3, the presence-guided encoder takes the output tokens from the last layer of ViT as input. These tokens go through a 3-layer, 8-header Transformer Vaswani et al. (2017) structure, after which they are fed into a MLP layer for predicting the presence or absence of cover texts. The loss in this part is $L_{IC}$.

The presence-guided encoder directs the cover image encoder to focus on the cover texts, but does not involve semantic information. We further propose the semantic-guided encoder, which sets the second agent task as an Image-Text Matching (ITM) task: ”To determine whether the specified text is consistent with the text on the cover image”, encouraging the ViT to incorporate gains from the semantics of the cover texts. This design is motivated by the notion that visual tokens from the ViT contain the semantic information of cover texts, if they are successfully employed to discriminate the consistency of the cover text with a given text.

We design negative samples by nearest neighbor lookup. Take the training on CBVS-5M dataset as an example. First, we adopt RoBERTa-wwm-Base as the encoder and load the checkpoints released by QA-CLIP to encode all the 2.0M valid OCR texts. The Hierarchical Navigable Small World Algorithm (HNSW) Malkov and Yashunin (2018) is applied to retrieve the Top-$K$ OCR texts that are most semantically similar but not identical to the anchor, and one of them is randomly selected as the negative sample. For covers without OCR texts, only negative samples are considered. For covers with OCR texts, the percentage of positive samples is set to 70%.

The semantic-guided encoder accepts two inputs, i.e., tokens from the last layer of the ViT, and the embeddings of positive or negative samples. As shown in Fig. 3, the module is a 3-layer structure, where each layer consists of a self-attention, a cross-attention with an MLP, and includes residual connections. The embeddings of the samples are updated layer by layer. The loss of this process is $L_{ITM}$.

Pre-training for UniCLIP starts with the checkpoints released by QA-CLIP. Fine-tuning is then performed on the CBVS-5M/10M dataset with a total loss of:

where $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ are hyperparameters.

It is worth noting that the fine-tuning of UniCLIP relies on positive samples, negative samples and ground truths related to OCR texts, but the inference process does not rely on any OCR-related components. As shown in Fig. 3, the presence-guided encoder and semantic-guided encoder guide the training of the image-text alignment task in UniCLIP, but do not participate in the inference process. Therefore, UniCLIP infers in a manner consistent with CLIP.

We follow the model architecture setup in OpenAI CLIP with ViT-B/16 as the visual backbone network for UniCLIP. The text encoder is the 12-layer architecture of RoBERTa-wwm-Base. Both are implemented by 12-layer 12-head Transformers with 768 encoding dimensions and eventually mapped linearly to 512 dimensions. The vocabulary size of text tokenizer is consistent with CN-CLIP. The initialized weights of the visual encoder and text encoder are from QA-CLIP as described in Sec. 4.4. In addition, the weights of presence-guided encoder and semantic-guided encoder are randomly initialized with normal distribution, and $K$ for the semantic-guided encoder is set to 10.

In the fine-tuning stage, we perform random-size cropping and AutoAugment Cubuk et al. (2019) on the input image. All parameters of the image encoder and text encoder are allowed to be updated. Training is executed on 8 NVIDIA A100 GPUs for 20 epochs, with a learning rate of 2e-5. The maximum length of the text encoder is 12 and the weight decay is 1e-3. We leverage all-gather communications across GPU workers to compute $L_{ITC}$ on the global batch. The batch size is set to 1,520 due to GPU memory limitations. Mixed-precision training is activated. We save checkpoints for each epoch and report results with the highest Positive-to-Negative Ratio (PNR) on CBVS-20K. Due to computational resource constraints, we report the results of training on the CBVS-5M dataset. We simultaneously release the CBVS-10M dataset for subsequent studies.

To demonstrate the performance of our proposal and the value of the CBVS dataset, we extensively evaluate advanced Chinese image-text models on CBVS-20K. The competing models include CN-CLIP Yang et al. (2022), R2D2 Xie et al. (2023), Wukong Gu et al. (2022), TaiyiCLIP Zhang et al. (2022a), Ernie-ViL2.0 Shan et al. (2022) and AltCLIP Chen et al. (2022). The results are shown in Tab. 3.

The results show that, for example, WuKong, although it achieves competitive performance on other public datasets, it lacks the ability of image-text matching on CBVS-20K. The same conclusion appears in most of the other competitors, e.g., the MR of Taiyi-CLIP_ViT-B is only 0.407, which much lower than UniCLIP’s 0.692. The generalization performance of models trained on large-scale open domain images is generally low on the CBVS-20K, which demonstrates the domain uniqueness of video cover images. Vision-Language Models that perform well in the open domain do not migrate to the video cover domain as expected.

It is worth noting that although R2D2-250M_ViT-L’s recall metrics are significantly behind UniCLIP, its ranking metrics are close to those of UniCLIP. In particular, NDCG@1 slightly outperforms UniCLIP with a maximum of 0.789. We infer that the experimental result is due to the fact that R2D2-250M_ViT-L employs a more powerful visual architecture and enjoys a training corpus size of 250M. We encourage the incorporation of CBVS-10M into the training corpus on the one hand, and the adoption of ViT-L as a visual skeleton on the other hand, to facilitate further improvement of UniCLIP performance in subsequent studies.

Compared to the pre-trained QA-CLIP, the fine-tuning on the CBVS-5M dataset comprehensively improves the metrics, especially the PNR by 3.67%, and R@1 by 18.25%. Performing fine-tuning on the publicly released CN-CLIP_ViT-B, consistent findings are observed with a 7.21% improvement in PNR and a 22.66% improvement in R@1. Performing fine-tuning on the R2D2-250M_ViT-L, significantly higher recall metrics are observed, as well as largely comparable rank metrics. These results demonstrate that fine-tuning on a large-scale cover dataset can improve the performance of the model in the video search domain. In addition, UniCLIP achieves state-of-the-art performance with the highest metrics compared to its competitors and does not introduce additional inference cost compared to the simple and efficient CN-CLIP.

In order to evaluate the proposed presence-guided encoder and semantic-guided encoder, we implement versions of the model with and without $L_{IC}$ and $L_{ITM}$, respectively. The weights of $L_{ITC}$ and $L_{IC}$ (or $L_{ITM}$) are set to 0.8 and 0.2. Tab. 4 shows the results of the ablation study of UniCLIP.

If both the presence-guided encoder and the semantic-guided encoder are removed and the cover text is discarded, the model degenerates into a fine-tuned version of QA-CLIP. The PNR is reduced from 3.069 to 2.907 and the MR from 0.692 to 0.656. In addition, removing any of the two encoders results in varying degrees of degradation in model performance. Removing the presence-guided encoder reduces the PNR of UniCLIP by 2.05%. Besides, removing the semantic-guided encoder reduces the PNR by 2.54%. Interestingly, when only the semantic-guided encoder is employed, the NDCG@1/5/10, MAP, and R@1/5 of the model are basically the same as the final scheme, which indicates that the gain of cover text is mainly in the semantic information. If two encoders are employed at the same time, i.e., the proposed two agent tasks are considered at the same time, the highest metrics are achieved in all aspects. This result is ample proof of the validity of our proposal. We encourage follow-up studies to further generalise our ideas.

Since the training process of UniCLIP relies on the cover text modality, for a fair comparison, we implement an explicit OCR text fusion scheme, which is denoted as ALBEF-CLIP. Compared to CLIP, we replace ViT with an ALBEF structure, where the cover image and the cover text go through their respective encoders before passing through a 6-layer Attention-based fusion structure. For the case of missing OCR texts, the prompt word is employed. The results of ALBEF-CLIP are shown in Tab. 3. If the cover text is introduced, but with ALBEF-CLIP rather than our proposal to exploit this modality, the metrics are much lower than UniCLIP in all aspects. We hypothesize that the reason for this is that UniCLIP guides the semantic training of ViT and handles the modality missing problem more consistently, reducing information confusion.

To further evaluate the cover text capability, we categorize the data in CBVS-20K into two main categories according to the presence or absence of cover texts, which are denoted as $S_{T}$ and $S_{F}$, respectively. Tab. 5 demonstrates the PNR metrics for different combinations. Compared to the scheme without cover texts (QA-CLIP), ALBEF-CLIP significantly improves the matching ability for covers with cover texts, increasing the PNR from 3.203 to 3.375. However, for covers without cover texts, the scheme degrades the performance, which may be due to semantic confusions brought about by prompt words.

In comparison, UniCLIP is basically comparable to ALBEF-CLIP for matching between covers with cover texts. This is in line with expectations, as we discard cover texts modalities in our inference, however the very close results are a good indication of the promise of UniCLIP. Besides, UniCLIP performs much better for both other cases than QA-CLIP and ALBEF-CLIP. For the matching between covers without cover texts, which is most likely to happen, UniCLIP’s PNR exceeds that of the fusion scheme by 8.00% and has a lower inference cost. For the hybrid cases, UniCLIP achieves the PNR of 3.194. This suggests that UniCLIP is able to overcome the modality missing problem to some extent and handle cover images with or without cover texts uniformly. Thanks to this, UniCLIP shows the best performance on the full range of data.