LGDN: Language-Guided Denoising Network
for Video-Language Modeling

Note that our LGDN is designed to filter out the unmatched/redundant frames for better token-level alignment, without leveraging the temporal information of the videos explicitly. Therefore, we firstly introduce a Momentum Video-Level Contrastive Learning (MVCL) module to address this problem.

The MVCL module utilizes a temporal module (e.g., Transformer block) to aggregate the frame embeddings to obtain the video embedding. Contrastive learning is then applied for holistic video-text alignment. However, video data takes up large GPU memory and the mini-batch size tends to be small with strict resource, which brings harm to contrastive learning. Inspired by MoCo [14], we introduce the momentum mechanism to maintain massive negative samples in memory bank for contrastive learning. Concretely, We firstly maintain video memory bank $\mathcal{M}^{v}=\{\mathbf{\hat{q}}^{v}_{j}\}_{j=1}^{N_{m}}$ and text memory bank $\mathcal{M}^{l}=\{\mathbf{\hat{q}}^{l}_{j}\}_{j=1}^{N_{m}}$ to store video/text features, where $N_{m}$ denotes the memory bank size and $\mathbf{\hat{q}}^{v}_{j}$/ $\mathbf{\hat{q}}^{l}_{j}$ denotes the $j$-th stored video/text feature vector. Let $f^{v}$ (with parameters $\theta^{v}$) and $\hat{f}^{v}$ (with parameters $\hat{\theta}^{v}$) denote vision encoder and vision momentum encoder, respectively. Similarly, let $f^{l}$ (with parameters $\theta^{l}$) and $\hat{f}^{l}$ (with parameters $\hat{\theta}^{l}$) denote language encoder and language momentum encoder, respectively. The parameters of momentum encoders are updated by:

$\displaystyle\hat{\theta}^{v}$

$\displaystyle=m\cdot\hat{\theta}^{v}+(1-m)\cdot\theta^{v},~{}~{}~{}~{}\hat{\theta}^{l}=m\cdot\hat{\theta}^{l}+(1-m)\cdot\theta^{l},$

(2)

where $m$ is the momentum coefficient hyper-parameter.

The loss function is thus constructed as follow: for each video $V_{i}$ in mini-batch $\mathcal{B}$, we define the video-to-text contrastive loss between its paired text $L_{i}$ and all negative samples in the text memory bank $\mathcal{M}^{l}$, resulting in an InfoNCE loss (with $\tau$ being the temperature hyper-parameter):

$$\mathcal{L}_{\text{V2T}}=-\frac{1}{|\mathcal{B}|}\sum_{(V_{i},L_{i})\in\mathcal{B}}\log\frac{\exp(\cos(\mathbf{f}^{v}_{i},\mathbf{\hat{f}}^{l}_{i})/\tau)}{\exp(\cos(\mathbf{f}^{v}_{i},\mathbf{\hat{f}}^{l}_{i})/\tau)+\sum_{\mathbf{\hat{q}}^{l}_{j}\in\mathcal{M}^{l}}\exp(\cos(\mathbf{f}^{v}_{i},\mathbf{\hat{q}}^{l}_{j})/\tau)},$$

(3)

where $\mathbf{\hat{f}}^{l}_{i}=\hat{f}^{v}(l_{i})$, and the similarity of two features is measured by the cosine similarity. Similarly, given each text description $L_{i}$ in mini-batch $\mathcal{B}$, we define the text-to-video contrastive loss as:

$$\mathcal{L}_{\text{T2V}}=-\frac{1}{|\mathcal{B}|}\sum_{(V_{i},L_{i})\in\mathcal{B}}\log\frac{\exp(\cos(\mathbf{f}^{l}_{i},\mathbf{\hat{f}}^{v}_{i})/\tau)}{\exp(\cos(\mathbf{f}^{l}_{i},\mathbf{\hat{f}}^{v}_{i})/\tau)+\sum_{\mathbf{\hat{q}}^{v}_{j}\in\mathcal{M}^{v}}\exp(\cos(\mathbf{f}^{l}_{i},\mathbf{\hat{q}}^{v}_{j})/\tau)},$$

(4)

where $\mathbf{\hat{f}}^{v}_{i}=\hat{f}^{v}(V_{i})$. Finally, the objective function for MVCL is defined as follows:

$$\mathcal{L}_{\text{MVCL}}=\mathcal{L}_{\text{V2T}}+\mathcal{L}_{\text{T2V}}.$$

(5)

As shown in Figure 1, video-text data inevitably contains misaligned frame-text pairs. Although an attention mechanism has been applied in Eq. (1), the irrelevant and noisy information would still mislead the cross-modal alignment in our model. To alleviate this problem, we thus propose a Salient Frame Proposal (SFP) mechanism for video-language modeling.

The core idea of our SFP mechanism is to dynamically filter out misaligned or redundant frames and maintain only a few important frames to represent the video well, which are called as salient frames. Formally, for each video-text pair, we first identify the relevance score $R(j|i)$ between the text $L_{i}$ and the $j$-th frame $E_{i,j}$ of the video $V_{i}$. Further, we perform language-guided denoising to retain only top-$N_{salient}$ salient frames by filtering out the unmatched/redundant frames from each video.

Since only video-level annotations are provided, we need to estimate the relevance scores $R$ automatically. As shown in Table 1, we introduce four strategies for estimating relevance scores. (1) SimDot prediction relies on the output of two separate encoders (i.e., frame encoder $f^{e}$ and language encoder $f^{l}$) to model the relevance score $R(j|i)$ by computing the dot product of the frame embedding $\mathbf{{f}}^{e}_{i,j}=f^{e}(E_{i,j})$ and the text embedding $\mathbf{{f}}^{l}_{i}=f^{l}(L_{i})$. However, since video-text data is noisy, only utilizing single-modality encoders may result in incorrect salient frames. (2) Momentum prediction improves SimDot prediction by introducing the supervision of momentum encoders (i.e., momentum frame encoder $\hat{f}^{e}$ and momentum language encoder $\hat{f}^{l}$), where the momentum frame embedding $\mathbf{\hat{f}}^{e}_{i,j}=\hat{f}^{e}(E_{i,j})$ and the momentum text embedding $\mathbf{\hat{f}}^{l}_{i}=\hat{f}^{l}(L_{i})$. (3) CrossMom prediction considers the frame-text alignment that is directly built on the interaction between one modality encoder and another modality’s momentum encoder. (4) Collaborative prediction combines Momentum prediction and CrossMom prediction for better performance.

Although the frame embeddings and text embeddings can be (roughly) aligned through applying video-text contrastive learning in Sec. 3.2, it is vital to precisely establish frame-text alignment for proposing/selecting salient frames. To this end, we introduce the MFCL module below.

To dynamically filter out the unmatched/redundant frames, we propose to adopt frame-level contrastive learning to directly measure the relevance scores $R$ between video frames and paired text. However, video data often contains misaligned frame-text pairs. Simply applying standard NCE-based contrastive learning would force the misaligned frame-text pairs to be pulled closer, which inevitably has negative effect on learning high-quality frame-text representation. Inspired by MIL-NCE [34], we thus propose a Momentum Frame-Level Multiple Salient-instance Learning (MSL) Contrastive Learning (MFCL) module to assist in alleviating the noise problem. The core idea is to use the salient frames filtered by the SFP Mechanism in each video to form a set of positive candidate pairs, instead of considering each positive pair independently. In this work, we suppose that MFCL and SFP have mutual interdependence so that they can bring boost to each other during training.

Similar to MVCL, we additionally maintain a frame-level memory bank $\mathcal{M}^{e}=\{\mathbf{\hat{q}}^{e}_{j^{\prime}}\}_{{j^{\prime}}=1}^{N_{m}*N}$ to store frame features, where $N_{m}$ is the memory bank size, $N$ is the number of sampled frames per video, and $\mathbf{\hat{q}}^{e}_{j^{\prime}}$ is a stored frame feature vector.

Given each text description $L_{i}$ in mini-batch $\mathcal{B}$, we select salient frames filtered by the SFP Mechanism in the paired video $V_{i}$ to form a set of positive candidate (frame-text) pairs $S_{i}$ and all frame samples in $\mathcal{M}^{e}$ to form the negative ones. We then define the text-to-frame contrastive loss as:

$$\mathcal{L}_{\text{T2E}}=-\frac{1}{|\mathcal{B}|}\sum_{(S_{i},L_{i})\in\mathcal{B}}\log\frac{\sum_{\mathbf{\hat{f}}^{e}_{ij}\in\mathbf{\hat{f}}^{s}_{i}}\exp(\cos(\mathbf{f}^{l}_{i},\mathbf{\hat{f}}^{e}_{ij})/\tau)}{\sum_{\mathbf{\hat{f}}^{e}_{ij}\in\mathbf{\hat{f}}^{s}_{i}}\exp(\cos(\mathbf{f}^{l}_{i},\mathbf{\hat{f}}^{e}_{ij})/\tau)+\sum_{\mathbf{\hat{q}}^{e}_{j^{\prime}}\in\mathcal{M}^{e}}\exp(\cos(\mathbf{f}^{l}_{i},\mathbf{\hat{q}}^{e}_{j^{\prime}})/\tau)},$$

(6)

where $S_{i}=\{E_{i,j}\}_{j=1}^{N}$ is the positive frame set of the video $V_{i}$, $N$ is the frame sequence length of the video, $\mathbf{f}^{l}_{i}=f^{l}(L_{i})$, and $\mathbf{\hat{f}}^{s}_{i}=\{\mathbf{\hat{f}}^{e}_{ij}\}_{j=1}^{N}=\{\hat{f}^{e}(E_{i,j})\}_{j=1}^{N}$.

Similarly, given each positive frame set $S_{i}$, we define the frame-to-text contrastive loss as:

$$\mathcal{L}_{\text{E2T}}=-\frac{1}{|\mathcal{B}|}\sum_{(S_{i},L_{i})\in\mathcal{B}}\log\frac{\sum_{\mathbf{f}^{e}_{ij}\in\mathbf{f}^{s}_{i}}\exp(\cos(\mathbf{f}^{e}_{ij},\mathbf{\hat{f}}^{l}_{i})/\tau)}{\sum_{\mathbf{f}^{e}_{ij}\in\mathbf{f}^{s}_{i}}\exp(\cos(\mathbf{f}^{e}_{ij},\mathbf{\hat{f}}^{l}_{i})/\tau)+\sum_{\mathbf{f}^{e}_{ij}\in\mathbf{f}^{s}_{i}}\sum_{\mathbf{\hat{q}}^{l}_{j^{\prime}}\in\mathcal{M}^{l}}\exp(\cos(\mathbf{f}^{e}_{ij},\mathbf{\hat{q}}^{l}_{j^{\prime}})/\tau)},$$

(7)

where $\mathbf{\hat{f}}^{l}_{i}=\hat{f}^{l}(L_{i})$ and $\mathbf{f}^{s}_{i}=\{\mathbf{f}^{e}_{ij}\}_{j=1}^{N}=\{f^{e}(E_{i,j})\}_{j=1}^{N}$ (text memory bank $\mathcal{M}^{l}=\{\mathbf{\hat{q}}^{l}_{j^{\prime}}\}_{{j^{\prime}}=1}^{N_{m}}$ is defined in Sec. 3.2). As a result, by combining the text-to-frame and frame-to-text contrastive losses, the objective function for MFCL is given by:

$$\mathcal{L}_{\text{MFCL}}=\mathcal{L}_{\text{E2T}}+\mathcal{L}_{\text{T2E}}.$$

(8)

After obtaining language-guided salient frames, we utilize a multi-modal cross-attention fusion Transformer (see Figure 2) to capture token-level semantic alignment between visual patches and words for better performance (see the design details of this Transformer in the supp. material).

Further, we take the [CLS] token embedding outputted by the multi-modal fusion Transformer as the joint representation of a frame-text pair ($V_{i},L_{i}$), and deploy a fully-connected layer to predict the matched probability, which is similar to the sentence pair classification task in BERT’s pre-training phase. The matching loss is defined as:

$$\mathcal{L}_{\text{LSFM}}=-\mathbb{E}_{(\mathbf{E}_{i,j},\mathbf{L}_{i})\sim\mathcal{D}_{salient}}\log P(y_{i,j}|\mathbf{E}_{i,j},\mathbf{L}_{i}),$$

(9)

where $\mathbf{E}_{i,j}$ denotes $j$-th frame feature of video $V_{i}$, $\mathbf{L}_{i}$ denotes text feature, $\mathcal{D}_{salient}$ is the set of salient frame-text pairs obtained by applying the SFP mechanism to the mini-batch, and $y_{i,j}$ is the ground-truth matching label (0 or 1) of the frame-text pair $(\mathbf{E}_{i,j},\mathbf{L}_{i})$. During inference, we use a mean pooling layer to aggregate all salient frame scores as the video-level prediction score.

Finally, by combining all the proposed modules for video-language modeling at three levels, we train our LGDN model via minimizing the total objective function:

$$\mathcal{L}_{\text{LGDN}}=\mathcal{L}_{\text{MVCL}}+\mathcal{L}_{\text{MFCL}}+\mathcal{L}_{\text{LSFM}}.$$

(10)

In this subsection, we conduct comprehensive ablation study to investigate the contributions of different components of our full model. If not specifically indicated, we set $N=16$ for global alignment and $N_{salient}=2$ for token-level alignment as the default setting.

We first report the text-video retrieval results on MSR-VTT with three data partitions in Table 3. It can be observed that: (1) our LGDN outperforms all previous works by large margins. Particularly, as compared with the most recent model Frozen in Time [2], our LGDN achieves an improvement of 7.9% (38.9% vs. 31.0%) for Text-to-Video R@1 on the MSR-VTT 1k-A test set. (2) Our LGDN also outperforms methods utilizing extra modalities (e.g., motion and audio) or those pre-trained on extremely-large video data (e.g., HowTo100M). (3) When leveraging a much larger pre-training (image-text) dataset, our LGDN (marked with ^†) achieves significant improvements.

To demonstrate the robustness of our model, we also evaluate it on VATEX, MSVD, and Didemo in Tables 4.2–4.2, respectively. Due to limited space, only text-to-video retrieval is considered here. For VATEX (Table 4.2), our LGDN significantly outperforms the state-of-the-art method Support Set which is trained on an order of magnitude more data. Our LGDN still performs the best on MSVD (Table 4.2) and Didemo (Table 4.2). Particularly, in the Didemo dataset, each description is annotated with localization information, in other words, annotations may only be aligned with the localized moments, thus causing the noise problem as many methods utilize all frames as the input. Recent works exploit temporal labels of captions to alleviate the noise problem and achieve higher performance. However, even without considering this, our LGDN still largely outperforms the most recent method Frozen [2], further demonstrating the effectiveness of our LGDN.

To show the general applicability of our LGDN, we evaluate our LGDN on the VideoQA task in Table 4.2. Even without utilizing large-scale video datasets devoted to the VideoQA task, our LGDN outperforms all competitors, validating the effectiveness of our LGDN in VideoQA. In addition, to reveal the critical importance of solving the noise issue for video-language modeling, we directly apply the SFP mechanism to the latest model CLIP4Clip [33] in Table 8. We find that applying the SFP mechanism brings boost to Clip4CLIP. The ensemble mechanism further improves the results, indicating that the proposed SFP mechanism is complementary to the baseline.

We provide visualization of our LGDN in Figure 4. We uniformly sample 5 frames from each video and provide relevance scores of 5 frames on the left, and the red ones denote salient frames selected by the SFP mechanism. It can be seen that: (1) Although the holistic video is semantically related to the paired text, there still exist noisy frames (e.g., the transition in Frame 1 and Frame 3 of Query7500) and unrelated frames (e.g., in Frame 4 and Frame 5 of Query7544, a man is rolling while the paired text is ‘a car goes racing down the road’). (2) The relevance scores obtained from the SFP mechanism correctly measure the consistency between each frame and the paired text, which indeed helps our LGDN to precisely filter out noisy information for better video-language modeling.