Modality-Balanced Embedding for Video Retrieval

Our model architecture follows the popular two-tower (i.e., bi-encoder) formulation, as (Huang et al., 2013; Li et al., 2021; Zhang et al., 2020; Liu et al., 2021b; Zhan et al., 2021; Liu et al., 2021c), with a transformer-based text encoder of query and a multi-modal encoder of video.

Most existing works, e.g., (Huang et al., 2013; Liu et al., 2021b; Huang et al., 2020a), train bi-encoders with the approximated query-to-document retrieval objective. Specifically, given a query $q$, its relevant videos $\mathcal{M}^{+}_{q}$, and its irrelevant videos $\mathcal{M}^{-}_{q}$, the query-to-document objective is as below:

where $\tau$ is a temperature hyper-parameter set to 0.07, and $s(q,m)$ is the cosine similarity of a query-video pair $(q,m)$ (i.e., $s(q,m)=cos<\mathcal{F}_{q}(q),\mathcal{F}_{m}(m)>$). Notably, here we adopt in-batch random negative sampling, which means $\mathcal{M}^{-}_{q}$ is all the videos in current mini-batch except the positive sample $m$.

As recent works (Liu et al., 2021c; Liu et al., 2021a) added a conversed document-to-query retrieval loss, we formulate a corresponding $\mathcal{L}_{mq}$ as below:

where $\mathcal{Q}^{-}_{m}$ denotes the irrelevant queries of the video $m$,  i.e., all the queries in current mini-batch except $q$.

The sum of the query-to-document loss and the reversed document-to-query loss results in the main bidirectional objective:

Beside the above bidirectional objective optimizing the relevance between the query embedding and the video’s multimodal embedding, we also add a auxiliary task $\mathcal{L}_{t}$ ($\mathcal{L}_{v}$) to optimize the relevance between the query and the video’s text modality (the vision modality) with similar formulations. The whole objective $\mathcal{L}_{base}$ for the base model is:

where $\alpha=\beta=0.1$, are the weight hyper-parameters.

To identify the modality imbalance, we define an indicator ${R}_{vt}$ as below,

where the definitions of $\mathcal{F}_{m},\mathcal{F}_{v},\mathcal{F}_{t}$ are given in Eq. (1). $R_{vt}$ is the ratio between the cosine similarity of vision-video and that of text-video and measures the extent of modality bias of the multi-modal encoder $\mathcal{F}_{m}$.

For the base model in Eq. (5), we compute $R_{vt}$ for a randomly sampled set of videos and plot the density histogram graph in Fig. 2(a). As observed, most $R_{vt}<0.3$, indicating the model suffer from the modality imbalance problem and the multimodal embeddings are heavily biased to the text contents. Consequently, when retrieving videos with the base model, visually irrelevant videos can be recalled falsely with even higher similarities than videos relevant to the queries textually and visually. The fundamental cause of modality imbalance is that text matching provides a shortcut for the bi-encoder: the query-text relation is easier to grasp, and most samples in training set can be solved by lexical relevance.

To eliminate the shortcut, we generate novel Modality-Shuffled (MS for short) negative samples, whose vision modality is irrelevant with the query, while the text is relevant to the query. As illustrated in fig. 2(b), such MS negatives are serious adversarial attacks for the base model, which cannot distinguish MS negatives with the real positives. This inspires our loss design of MS negatives as below:

where $\mathcal{M}_{ms}$ denotes the set of generated MS negatives.

$\mathcal{L}_{ms}$ straightly promotes the model disentangle the MS negatives from the real positives as shown in Fig. 2(c). By the way, it is not hard to find that when both $R_{vt}$ and $R_{\hat{v}t}$ are close to 0, $\mathcal{F}_{m}(t,v)$ will be close to its MS negative $\mathcal{F}_{m}(t,\hat{v})$. Thus, MBVR  with $\mathcal{L}_{ms}$ also pushes $R_{vt}$ faraway from 0 as in Fig. 2(a), which indicates that $\mathcal{L}_{ms}$ effectively alleviates the modality imbalance problem. Consequently, the information of both text and vision can be well-preserved in the final video embedding.

How to generate MS negatives efficiently? We design to re-combine text embeddings and vision embeddings in the mini-batch as in Fig. 1. For a mini-batch of size $n$, the vision, text embeddings of the $k$-th video are $\mathcal{F}_{v}(v_{k})$, $\mathcal{F}_{t}(t_{k})$. Then the MS negatives of the k-th video can be computed as $\mathcal{H}(\mathcal{F}_{v}(v_{l}),\mathcal{F}_{t}(t_{k}))$, where $l$ is a randomly selected integer from $1$ to $n$ except $k$. Such design can generate one MS negative for each video with only one MSA operation, which is extremely efficient. By repeating the above process $M$ times, we can generate $M$ MS negatives for each video. Empirically, more MS negatives can result in better performance, we set M as 32 to balance the effectiveness and efficiency.

To further address the modality bias, we apply a dynamic margin $\lambda$ on the positive pair $(q,m)$ of $\mathcal{L}_{qm}$ as below:

$\lambda$ is computed from the visual relevance of $(q,m)$ through a scale and shift transformation:

where $w=0.3,b=-0.1$, and $\sigma$ denotes the sigmoid function, i.e., $\sigma(x)=\frac{1}{1+e^{-x}}$. Then the margin $\lambda$ varies in $(-0.1,0.2)$ and monotonically increases w.r.t.  the visual relevance (i.e., $cos<\mathcal{F}_{v}(v),\mathcal{F}_{q}(q)>$). We also do the same modification for the video-to-query loss $\mathcal{L}_{mq}$ and the MS loss $\mathcal{L}_{ms}$. Then the main objective $\mathcal{L}_{bi}$ in Eq. 4 with dynamic margin is referred as $\widetilde{\mathcal{L}}_{bi}$ and $\mathcal{L}_{ms}$ with dynamic margin as $\widetilde{\mathcal{L}}_{ms}$. Note that the gradient of the margin $\lambda$ is detached during model training.

To understand the effect of the dynamic margin easily, when $\lambda>0$, it can be considered as moving the positive video $m$ a bit faraway from the query before computing the loss and results in a larger loss value as in Fig. 1. Therefore, the dynamic margin encourages the model to produce even higher similarities for the vision related query-video pairs.

Finally, the overall learning objective of our MBVR  framework is as follows,

where $\gamma$ is a weight hyper-parameter and set as $0.01$. In summary, MBVR  solves the modality imbalance issue from two collaborative aspects: punishing on videos with unrelated vision contents with the MS component, and enhancing query-video pairs of related vision contents with the dynamic margin.

In this section, we describe our datasets, evaluation metrics.

For all models, we compute their Precision@K and MRR@K on Auto and PNR on Manual, which are defined as below:

Table 1 illustrates experimental results of compared methods on both Auto and Manual test sets. The drastic performance difference between Vision and Text results from the dominated role of text modality in video’s modalities. +IAT and +CDF bring marginal improvements over Base. Both +MS and +DM boost significantly over the strong baseline Base. MS is extremely effective, which brings nearly 4% absolute boost over Base ($46.53\%\longrightarrow 50.52\%$). Furthermore, the full model MBVR, with both MS and DM, achieves the best performance. The offline evaluations verify the effectiveness and compatibility of both components of  MBVR (MS and DM).

For the online test, the control baseline is current online search engine, which is a highly optimized system with multiple retrieval routes of text embedding based ANN retrieval, text matching with inverted indexing, etc., to provide thousands of candidates, and several pre-ranking and ranking models to rank these candidates. And the variant experiment adds our MBVR  multimodal embedding based retrieval as an additional route.