TokenBinder: Text-Video Retrieval with One-to-Many Alignment Paradigm

In the field of Text-Video Retrieval (TVR), we primarily engage with two subtasks: text-to-video retrieval (t2v) and video-to-text retrieval (v2t). The goal of t2v is to identify the video within a database that most accurately aligns with a specified text query. Conversely, v2t retrieval focuses on finding the textual description that best matches a given video. Although these tasks are symmetrical in nature, t2v has historically garnered more research focus. Therefore, our discussion and subsequent results predominantly address the t2v task.

In the CLIP-based framework, a text sequence $\mathcal{T}$ is composed of $M$ tokens, expressed as $\mathcal{T}=\{t_{1},t_{2},...,t_{M}\ |t_{i}\in\mathbb{R}^{C}\}$. Similarly, a video $\mathcal{V}$ is represented by a sequence of frames $\mathcal{V}=\{f_{[cls]},f_{1},f_{2},...,f_{T}\}$, $f_{[cls]}$ denotes the global feature of the video $\mathcal{V}$, $T$ is the number of frame, each frame $f_{i}$ being a tensor in $\mathbb{R}^{H\times W\times C}$ with $H$, $W$, and $C$ representing height, width, and channel count, respectively. To facilitate feature extraction, the CLIP framework dissects each frame $f_{i}$ into $P^{2}$ patches of size $P\times P$, thus described by $\{f_{i}^{1},f_{i}^{2},...,f_{i}^{P^{2}}|f_{i}^{j}\in\mathbb{R}^{C}\}$.

As depicted in Figure 2, our proposed framework, TokenBinder, introduces a novel two-stage CLIP-based system designed to optimize text-video retrieval. The TokenBinder incorporates a three-step re-ranking process: query-binding, broad-view retrieval (stage 1), and focused-view retrieval (stage 2). The process begins with an advanced query encoding mechanism, wherein special tokens, termed query indicators, are embedded within the query encoder. These indicators are crucial for capturing key information, thereby dynamically enhancing the query’s ability to interact more effectively with video content across different stages. This enhancement not only facilitates a nuanced interaction with video features but also lays the groundwork for a sophisticated retrieval process.

In the broad-view retrieval stage, TokenBinder leverages the enriched queries to conduct a global analysis of the video database. The first query indicator serves as an anchor, gathering global features from videos and establishing preliminary rankings based on cosine similarities. This initial phase effectively filters out less relevant video candidates, streamlining the selection process. The subsequent focused-view retrieval stage involves a deeper examination of a select group of top-ranked videos. Here, a Focused-view Fusion Network, equipped with a cross-attention Transformer mechanism, precisely aligns the remaining query indicators with the local features of these video candidates. The integration of detailed alignment with initial global insights is accomplished through a MLP, which calculates the final similarity scores, ultimately determining the most relevant video outputs.

This layered approach ensures that TokenBinder captures not only the broad thematic elements of the query but also explores the finer textual and visual nuances of the video content. The framework’s capacity to dynamically adjust its focus during the retrieval process marks a significant advancement in text-video retrieval technology, resembling a conditional attention mechanism that optimizes for relevance and precision. Through these innovative features, TokenBinder sets a new benchmark in the field, offering a comprehensive method for engaging with complex multimedia content.

Effectively representing text and video features is foundational in our TokenBinder framework. Utilizing the CLIP architecture as our base, we handle text and video feature representation distinctly. In line with leading studies such as [24, 50, 13], video features are extracted using a Vision Transformer (ViT) [9]. Videos are decomposed into a series of frames, each enriched with three types of embedding (i.e. type embedding, video number embedding, and position embedding) to distinguish different tokens. Additionally, a [CLS] token is integrated to capture global video information comprehensively. Following the final ViT layer, a temporal mean pooling strategy [36, 25] is employed to significantly reduce the volume of local tokens while preserving critical video information. Formally, for a series of $i$-th frame patch features $\{f^{1}_{i},f^{2}_{i},\ldots,f^{P^{2}}_{i}\}$, after temporal mean pooling we have $\{f^{1}_{i},f^{2}_{i},\ldots,f^{n}_{i}\}$, where $n<<P^{2}$ and the number is re-indexed.

Text features are enhanced by query indicators, which are initialized randomly and processed with text tokens. At each text encoder layer, tokens undergo a self-attention module from the CLIP encoder. The query binding process then aggregates information from these tokens into the query indicators. For query indicators at layer $l$: ${I_{1}^{l},I_{2}^{l},\ldots,I_{m}^{l}}$, the update mechanism is defined as:

Here, $\theta$ represents the cross-attention layer within the text encoder, inserted after each self-attention module to dynamically capture relevant information, improving the query’s interaction with video content.

Following the extraction of textual and visual representations, the broad-view retrieval phase employs global features from the text query and videos to compute cosine similarity scores rapidly. The query indicators, which bind all key information during the text encoding process, serve as the global features for the text query. Specifically, we utilize the first query indicator after binding $i$-th query information $I_{1,i}$ to represent global text features. These representations facilitate the calculation of similarity scores across the dataset. The broad-view similarity between a text query $t_{i}$ and all videos in the dataset $N$ is calculated as follows:

Each similarity score $s^{{}^{\prime}}_{i,j}$ within the vector $\mathcal{S}^{{}^{\prime}}_{i}$ represents the dot product between the global text representation $I_{1,i}^{\top}$ of query $i$ and the global video representation $f_{j,[cls]}$ of each video. This approach quantifies the degree of relevance or similarity between the textual query and each video in the dataset, a crucial metric for effective retrieval.

Following the initial phase of Broad-view Retrieval, we obtain preliminary retrieval scores. These initial scores, while effective for a broad matching, often do not capture the intricate details within videos, necessitating further refinement. This need leads to the Focused-view Retrieval.

In this stage, the scores $\mathcal{S^{{}^{\prime}}}_{i}$ are used to select the top-$k$ videos as focused candidates for more precise optimization. This selection is facilitated by the focused-view fusion module, which comprises a cross-attention Transformer and a Multilayer Perceptron (MLP) network. The cross-attention Transformer specifically targets the binding of local features from the top-$k$ video candidates into the remaining query indicators post the Broad-view Retrieval. These enriched query indicators now embody comprehensive information necessary for precise alignment between the query and video candidates. Consequently, the MLP network projects these indicators into an alignment space, treating the projection as a classification problem to directly output the classification scores for these k videos with a Softmax layer. The formal notation of the cross-attention Transformer process $\eta(\cdot)$ is expressed as:

To facilitate differentiable sampling from a discrete distribution, we employ the Gumbel-Softmax [16, 26] to replace the traditional Softmax in the $\eta(\cdot)$ operation. Following this, an MLP is used to project the integrated information:

Ultimately, the refined similarity scores for a query $i$ in the stage-2 retrieval are calculated as:

During the training phase, we tailor specific optimization targets for the two stages of retrieval to supervise different query indicators, enhancing distinct aspects’ performance.

In the first stage, we utilize a Contrastive Loss to align the first query indicator $I_{1}$ in text with the global video token $f^{[CLS]}$. Formally, for a batch of size $B$, the contrastive loss functions for the t2v and v2t alignments are defined as follows:

Here, $\tau$ is the temperature parameter that scales the logits before applying the softmax function, effectively minimizing the distances between aligned pairs relative to other pairs in the batch.

For the focused view, the aim is to refine discrimination among closely related video candidates. This is achieved through a cross-entropy loss, which treats the scores projected by an MLP with a Softmax layer as class probabilities for a classification task. Given the distinct roles of video and sentence indicators, we apply supervision separately to each token type, denoted as $\ell_{\text{focus},v}$ and $\ell_{\text{focus},t}$.

The combined loss function, integrating both broad and focused supervision, is calculated as follows:

This formulation ensures that both the broad and focused objectives contribute equally to the overall training process, promoting robust learning across both modalities and enhancing detail within the representations.

We evaluate our proposed model, TokenBinder, on 6 widely recognized video text retrieval datasets: MSRVTT, DiDeMo, ActivityNet, VATEX, MSVD and LSMDC.

MSR-VTT [43] is one of the most extensively utilized in the field. The more widely adopted split in recent studies is known as MSRVTT-1K, which consists of 9,000 videos for training and 1,000 videos for validation. Our main experiments and ablation studies are conducted on this split.

DiDeMo [15] dataset includes 10,000 videos. It allocates 8,395, 1,065 and 1,004 videos to training, validation and test set correspondingly.

ActivityNet [10] contains 20,000 videos. Following the settings in [36, 44, 48], we report the results on the validation set 1, which consists of 4,900 videos.

VATEX [41] is another critical dataset in our study including 34,991 videos. Adhering to the division methods in [4, 22], we use 25,991 videos for training, 1,500 for validation, and another 1,500 for testing.

MSVD [3] dataset encompasses 1,970 video clips. For experimental consistency, we divided the dataset into 1,200 videos for training, 100 for validation, and 670 for testing.

LSMDC  [33] dataset is sourced from movie clips, consisting of 101,079, 7,408 and 1,000 videos for training, validation and test set correspondingly.

Our experimental results are positioned to demonstrate the robustness and effectiveness of TokenBinder across diverse datasets and retrieval challenges.

In our evaluation, we employ the most widely used metrics in video text retrieval to assess the performance of our method on the aforementioned datasets. These metrics include Recall at 1 (R@1), Recall at 5 (R@5), and Recall at 10 (R@10). Higher values in these metrics indicate better performance of the retrieval system, as they reflect the percentage of correct items found in the top 1, 5, and 10 positions, respectively. Additionally, we report on the Median Recall (MdR) and the Mean Recall (MeanR). Median Recall provides the median rank position of the first relevant document in the result list, which serves as a robust indicator of the central tendency of retrieval effectiveness, minimizing the impact of outliers. Mean Recall calculates the average rank position of all relevant documents across the test queries, offering a comprehensive measure of the overall retrieval performance.

Our model is built upon the CLIP-VIP [44] framework, and we utilize Pytorch version 1.8.0. Consistent with the practices observed in [24] [22], we begin by loading pretrained weights from CLIP to initialize our vision and text encoders. The query indicators are initialized using Kaiming normalization. For optimization, we employ the AdamW optimizer with a total batch size of 128. The learning rate is set to 1e-4 for the Focused-view fusion module and 1e-6 for other components, with a weight decay factor of 0.2. Given the distinct characteristics of each dataset, we set the training epochs for MSR-VTT, DiDeMo, ActivityNet, MSVD, VATEX and LSMDC at 5, 20, 20, 20, 20 and 10, respectively. Additionally, we conducted an ablation study on the MSR-VTT-1K-Test split to further assess the impact of various components and settings on the performance of our retrieval system.

In our experimental evaluation on the MSRVTT-1K-Test dataset, as summarized in Table  1, we segmented the analysis into three distinct sections. In the non-CLIP segment, our model demonstrates competitive performance. For the CLIP-based models, we specifically highlight the results obtained with the ViT-B/32 and ViT-B/16 as backbone architectures. Notably, the results for the CLIP-ViP [44] model were reproduced by its open-source codebase. Our approach, underpinned by a ViT-B/32 backbone, surpasses the existing methods across the majority of the evaluated metrics. The implementation with a ViT-B/16 backbone further amplifies this superiority. This superiority is attributed to our method’s robust feature extraction and cross-modality aggregation mechanism, which collectively enhance the alignment of textual and visual representations for more accurate retrieval performance.

Our extensive experimental evaluation extends beyond the predominant MSRVTT-1K dataset, encompassing a variety of datasets, as detailed in Table 2. The results on ActivityNet, DiDeMo, VATEX, MSVD alongside the LSMDC dataset, underscore the versatility and consistent performance of our TokenBinder model.

As demonstrated in Figure 3, upon examining the provided case study, we analyze contrasting instances to delineate the efficacy of the TokenBinder in discerning complex video-text correlations. For instance, our model accurately matches the query ”an Ethiopian woman asks a child what she is good at” with the corresponding video, showcasing the model’s nuanced understanding of the scene, as opposed to CLIP-ViP’s erroneous retrieval. In another case, TokenBinder effectively identifies the correct video for ”a team with blue uniforms are playing badminton with a team in white”, which demonstrates the model’s capability to recognize and differentiate between subtle visual details and activities. Furthermore, our framework successfully disambiguates the challenging query ”man pretends to be two different people”, illustrating its adeptness at interpreting complex human behaviors and interactions within videos. These case studies exemplify the advanced interpretative performance of TokenBinder.