FlashVTG: Feature Layering and Adaptive Score Handling Network for Video Temporal Grounding

In Video Temporal Grounding (VTG) tasks, we represent the input video and the extracted features as $\mathcal{V}$ and $\mathbb{V}=\{v_{i}\}_{i=1}^{L_{v}}$, respectively, where each $v_{i}\in\mathbb{R}^{D_{v}}$. Here, $L_{v}$ denotes the number of video clips and $D_{v}$ represents the feature dimension of each clip. For the natural language query $\mathcal{Q}$, we represent it as a set $\mathbb{Q}=\{q_{i}\}_{i=1}^{L_{q}}$, where each $q_{i}\in\mathbb{R}^{D_{q}}$ is the $i$-th word token in the query. Here, $L_{q}$ denotes the word number of query and $D_{q}$ indicates the feature dimension of each word token.

For Moment Retrieval (MR), the objective of the model is to predict the start point $b_{s}$ and the end point $b_{e}$ of the moment based on the input $V$ and the query. In the common practices, the output is a set of tuples $(b_{s},b_{e})$ representing the temporal boundaries of these moments. Additionally, the model generates confidence scores $c$ to rank these predictions, although these scores are not required by the task itself. Thus, the model produces: $\{(b_{s,i},b_{e,i},c_{i})\mid i=1,2,\ldots,n\}$, with confidence scores $\{c_{1},c_{2},\dots,c_{n}\}$ used for ranking.

In the Highlight Detection task, the model outputs a sequence of clip-wise saliency scores $S=\{s_{i}\}_{i=1}^{L_{v}}$, where each score $s_{i}\in[0,1]$ quantifies the relevance of the respective clip to the query.

As illustrated in Figure 2, the input video and query are initially encoded into video and query features using frozen Feature Encoders. In addition to extracting the original video and text features, FlashVTG incorporates a dummy token to supplement semantic information beyond the query, enhancing alignment between the video and text.

Following the encoding process, the video and query features are integrated within the Feature Fusion Module, enabling precise alignment of the two modalities. The fused features are subsequently processed in two separate streams. For Highlight Detection, the HD head transforms the fused features into saliency scores, while the Temporal Feature Layering Module expands the features across multiple granularities.

The Adaptive Score Refinement Module then evaluates these multi-scale features to generate a confidence score, $s$. Concurrently, the Moment Retrieval prediction head outputs predictions across different scales, producing a series of tuples $(b_{s},b_{e})$. Finally, the predicted moments and confidence scores are combined into ${(b_{s,i},b_{e,i},c_{i})\mid i=1,2,\ldots,n}$, which specify the predicted start and end points of moments, along with their associated confidence scores.

Feature Extraction. Consistent with previous methodologies, we employ the CLIP [36] image encoder and SlowFast [6] to extract clip-level video features $\mathbb{V}$, and use the CLIP text encoder along with GloVe [35] to derive word-level text features $\mathbb{Q}$. Specifically, raw videos are segmented into clips at predetermined FPS, such as 0.5 or 1, using frozen pretrained models as feature extractors. This process transforms each clip into a distinct video feature $v_{i}$. Similarly, each word in the query is encoded into corresponding text features $q_{i}$.

Feature Fusion. To align video and query features for subsequent processing, we project them into the same dimensional space $d$ using two MLPs before fusion. We adopt the Adaptive Cross Attention (ACA) module from CG-DETR [32], which extends the traditional Cross Attention by incorporating the ability to supplement query information. A known limitation of standard Cross Attention, particularly with the softmax operation, is that it may not perfectly align all video clips with the query, as the query cannot fully encapsulate the semantic scope of an untrimmed video. In ACA module, we use learnable dummy tokens, $D=[D_{1},...,D_{L_{d}}]$, where $L_{d}$ is a hyperparameter. These tokens capture semantic information beyond the original query, complementing its content. The encoded dummy tokens $\tilde{D}$ are concatenated with the query features $\mathbb{Q}$ to form the Key, with the video features $\mathbb{V}$ as the Query and the query features $\mathbb{Q}$ as the Value, as shown in Equation 1.

	Query	$\displaystyle=[p_{Q}(v_{1}),\ldots,p_{Q}(v_{L_{v}})]$		(1)
	Key	$\displaystyle=[p_{K}(q_{1}),\dots,p_{K}(q_{L_{q}}),p_{K}(\tilde{D_{1}}),\dots,p_{K}(\tilde{D_{L_{d}}})]$
	Value	$\displaystyle=[p_{V}(q_{1}),\dots,p_{V}(q_{L_{q}})]$

Here, $p_{Q}(\cdot)$, $p_{K}(\cdot)$, and $p_{V}(\cdot)$ denote the projection functions used to transform inputs into the Query, Key, and Value formats, respectively. These Query, Key, Value are input into the ACA module to obtain the fused features $F\in\mathbb{R}^{L_{v}\times d}$, as shown in Equations 2 and 3.

	$\displaystyle\textit{F}=\text{ACA}(v_{i})=\sum_{j=1}^{L_{q}}W_{i,j}\odot\textit{V}_{j};$		(2)
	$\displaystyle W_{i,j}=\frac{\text{exp}\left(\frac{\textit{Q}_{i}\odot\textit{K}_{j}}{\sqrt{d}}\right)}{\sum_{k=1}^{L_{q}+L_{d}}\text{exp}\left(\frac{\textit{Q}_{i}\odot\textit{K}_{k}}{\sqrt{d}}\right)},$		(3)

where $F$ denotes the fused feature and $\odot$ stands for the dot product. While $Q$, $K$, and $V$ represent the Query, Key, and Value, respectively. Through this adaptive cross-attention mechanism, the model more effectively aligns the query with the relevant video segments.

Following the ACA module, the fused features $F$ are passed through a Transformer Encoder to further refine the multi-modal representations. This additional processing step enables the model to enhance the interaction between the video and query features, allowing for a more comprehensive understanding of the temporal and semantic relationships across the entire sequence.

Highlight Detection Head. The refined features output from the Transformer Encoder are further processed to generate clip-level saliency scores $s$ for highlight detection. We aggregate the fused features to form a global contextual representation, which is then combined with the original fused features through a linear projection. The operation can be expressed as:

$$s=\frac{\sum_{i}\left(\textbf{W}_{1}\textit{f}_{i}\circ\textbf{W}_{2}\textit{g}\right)}{\sqrt{d}},$$

(4)

where $\textbf{W}_{1}$ and $\textbf{W}_{2}$ represent two linear projection functions, $\textit{f}_{i}$ denotes the $i$-th clip-wise fused features, and g represents the global contextual features. The symbol $\circ$ represents the Hadamard operation. This operation identifies the key clips in the video, yielding a sequence of scores that quantify the relevance of each clip to the overall content.

Temporal Feature Layering. This module transforms the fused features into a feature pyramid to enhance the model’s capability to process features at various granularities. The feature pyramid is a commonly used structure in computer vision for extracting and utilizing vision features across multiple scales [47, 9]. This architecture emulates the hierarchical processing of the human visual system, while enabling the network to effectively perceive and process video information of varying lengths.

Building on this insight, we incorporate layered feature implementations into DETR-based models to overcome their limitations in short moment retrieval. This module separates moments of varying lengths, enabling the subsequent ASR module (Sec. 3.5) to provide more detailed supervision for short moments prediction. Specifically, we apply 1D convolution operations with various strides to the fused features $F$, constructing a feature pyramid composed of features at different granularities. Predictions of moment locations are then made at these various granularities using a uniform prediction head.

Let $F\in\mathbb{R}^{L_{v}\times d}$ represent the input fused feature, where $L_{v}$ is the length of the features and $d$ is the dimensionality of the features. The process of temporal feature layering can be represented as:

$\displaystyle F_{k}=\begin{cases}F,&\text{if }k=1,\\ \text{Conv1D}^{k-1}(F,\text{stride}=2),&\text{if }k=2,3,\dots,K.\end{cases}$

By applying the temporal feature layering, we obtain a sets of features at different granularities, which can be represented as:

$\displaystyle F_{k}\in\mathbb{R}^{\frac{L_{v}}{2^{k-1}}\times d},\quad k=1,2,\dots,K.$

Here, $\{F_{k}|k=1,2,\dots,K\}$ represent the feature pyramid obtained from the original fused feature $F$ through multiple convolution operations. This multi-scale processing approach enables the model to capture and process information at different temporal resolutions, thereby adapting to scene changes at various scales. As referenced in Sec. 4.5, this module significantly improves the model’s ability to retrieve moments of varying lengths. This operation paves the way for enhanced supervision of short moment retrieval in subsequent steps.

Moment Prediction Head. This module primarily adjusts and reduces the feature dimension to 2, and through a series of transformations, it yields the predicted start and end points of moments. The specific process can be expressed as:

$$B_{k}=\left(\sigma\left(\text{Conv1D}\left(\sigma\left(\text{Conv1D}(F_{k})\right)\right)^{\top}\right)\right)^{\top}\times C_{k}.$$

(5)

Here, $B_{k}\in\mathbb{R}^{\frac{L_{v}}{2^{k-1}}\times 2}$ represents a set of boundaries at scale $k$, $\sigma(\cdot)$ denotes the ReLU activation function. The term $C_{k}$ refers to a learnable parameter corresponding to each scale, which is used to adjust the influence of different scales on boundary prediction.

Adaptive Score Refinemen Module used to assign a confidence score $c\in[0,1]$ to each predicted moment in MR. This score indicates the extent to which the given query is relevant to the predicted boundary. Compared with the previous method of generating scores on a single scale feature, we leverage both intra-scale and inter-scale scores to improve the final predictions.

For each level of the feature pyramid $F_{k}$, intra-scale scores are first generated through a score head, producing outputs corresponding to the varying dimensions of the pyramid levels. These outputs are then concatenated into a unified tensor along the length dimension. These steps are shown in Equation 6 and Equation 7, respectively.

	$\displaystyle c_{k}=\text{ScoreHead}_{1}(F_{k})\in\mathbb{R}^{\frac{L_{v}}{2^{k-1}}\times 1},k=1,2,\dots,K.$		(6)
	$\displaystyle c_{\text{intra}}=\text{Concat}(c_{1},c_{2},\dots,c_{K}).$		(7)

Our score head utilizes a 2D convolutional network with a kernel size of $1\times 5$, which is effectively equivalent to a 1D convolution. Simultaneously, as shown in Equation 8, inter-scale scores are computed by concatenating the features from all pyramid levels and passing them through another score head, resulting in a tensor that matches the dimensions of the intra-scale output.

$\displaystyle c_{\text{inter}}=\text{ScoreHead}_{2}(\text{Concat}(F_{1},F_{2},\dots,F_{K})).$

(8)

The final prediction is obtained by a weighted combination of the intra-scale and inter-scale scores:

$$c_{\text{final}}=x\cdot c_{\text{intra}}+(1-x)\cdot c_{\text{inter}}.$$

(9)

Here, the learnable weighting factor $x$ enables adaptive adjustment between $c_{\text{intra}}$ and $c_{\text{inter}}$, thereby resulting in a more comprehensive score prediction.

In FlashVTG, we employ a series of loss functions to ensure the model converges towards the desired objectives. For MR, we use Focal Loss [21], L1 Loss, and Clip-Aware Score Loss to respectively optimize the classification labels, boundaries, and clip-level confidence scores of the predicted moments. For HD, we utilize SampledNCE Loss [29] and Saliency Loss to optimize the saliency scores for each clip. The overall loss can be expressed as:

	$\displaystyle\mathcal{L}_{\text{overall}}$	$\displaystyle=\lambda_{\text{Reg}}\mathcal{L}_{\text{L1}}+\lambda_{\text{Cls}}\mathcal{L}_{\text{Focal}}+\lambda_{\text{CAS}}\mathcal{L}_{\text{CAS}}$
		$\displaystyle\quad+\lambda_{\text{SNEC}}\mathcal{L}_{\text{SNCE}}+\lambda_{\text{Sal}}\mathcal{L}_{\text{Sal}},$

where each $\lambda_{*}$ represents the corresponding weight for each loss component. Due to the space limitation, we will focus on the Clip-Aware Score Loss, further details on the other loss functions can be found in the supplementary materials.

Clip-Aware Score Loss. This Loss is designed to align the predicted confidence scores with the target saliency labels. Given the predicted clip-wise moment confidence score $c_{\text{final}}$ and the target saliency scores $s_{\text{gt}}$, we first normalize both sets of scores using min-max normalization to get $\hat{c}_{\text{final}}$ and $\hat{s}_{\text{gt}}$. The loss is then computed as the mean squared error between $\hat{c}_{\text{final}}$ and $\hat{s}_{\text{gt}}$:

$$\mathcal{L}_{\text{CAS}}=\text{MSE}(\hat{c}_{\text{final}},\hat{s}_{\text{gt}}).$$

(10)

This loss encourages the model to produce confidence scores that not only match the target labels but also adhere to the relative distribution of saliency across clip-level, thereby specifically enhancing the model’s performance when predicting short moments.

We evaluated our model on five VTG task datasets, including QVHighlights, TACoS, Charades-STA, TVSum, and YouTube-HL.

QVHighlights [16] is the most widely used dataset for MR and HD tasks, as it provides annotations for both tasks. This dataset marked the beginning of a trend where these two tasks are increasingly studied together. It includes more than 10,000 daily vlogs and news videos with text queries. Our main experiments were conducted on this dataset, and we provide comprehensive comparisons with other methods on it. Charades-STA [7] and TACoS [37] were used to evaluate the model’s performance on MR, containing daily activities and cooking-related content, respectively. The other two datasets, TVSum [41] and YouTube-HL [43], are sports-related and were used for HD evaluation.

We follow previous works [16, 29, 15] and adopt consistent evaluation metrics: R1@X, mAP, and mIoU for MR, and mAP and Hit@1 for HD. Specifically, R1@X stands for “Recall 1 at X”, which refers to selecting the predicted moment with the highest confidence score and checking whether its IoU with any ground truth moment exceeds the threshold X. If it does, the prediction is considered positive, and R@X is then calculated at thresholds $X\in\{0.3,0.5,0.7\}$. For MR, mAP serves as the primary metric, representing the mean of the average precision (AP) across all queries at thresholds [0.5:0.05:0.95]. We also use mean Intersection over Union (mIoU) to evaluate the average overlap between predicted moments and ground truth segments. For HD, mAP remains the key evaluation metric, with Hit@1 used to assess the hit ratio for the highest-scored clip.

As in previous methods [33, 30, 16], we primarily used video and text features extracted by CLIP [36] and SlowFast [6] across all five datasets for a fair comparison. To further verify the generalizability of our model, we incorporated video and text features extracted by InternVideo2 [48] and LLaMA [44] for QVHighlights and Charades-STA, as well as features extracted by VGG [40] and GloVe [35] for Charades-STA. All feature dimensions were set to 256. The number of attention heads in the Feature Fusion Module was set to 8, with $K=4,5$ layers in temporal feature layering, and the number of 2D convolutional layers in the score head was set to 2. AdamW was used as the optimizer, and the NMS threshold during inference was set to 0.7. Training was conducted on a single Nvidia RTX 4090 GPU, taking approximately one and a half hours for 150 epochs on QVHighlights.

The comparison of MR experimental results is shown in Tables 1, 2, 3, 7, where FlashVTG achieved state-of-the-art (SOTA) performance on nearly all metrics. Table 1, 7 presents the MR results on the QVHighlights [16]. Table 1 demonstrates that combining FlashVTG with a newly selected backbone [48] resulted in a significant performance improvement, evidenced by a 5.8% increase in mAP on the test set. Even when using the same backbone, mAP improved by 1.4%. Table 7 shows that FlashVTG improved the mAP of short moment retrieval to 125% of the previous SOTA method.These performance gains on one of the most widely used datasets demonstrate the effectiveness of FlashVTG compared to contemporaneous approaches.

Tables 2 and 3 present experimental results on two other MR datasets, TaCos and Charades-STA. Similarly, FlashVTG significantly outperformed previous methods on almost all metrics, achieving either SOTA or the second-highest performance. On Charades-STA, performance improvements were observed across all three different backbones, further validating the robustness of FlashVTG.

The comparison of HD experimental results is shown in Tables 4, 6, 6. FlashVTG achieved SOTA performance on both QVHighlights and YouTube-HL, and outperformed all methods that used additional audio features on TVSum, reaching performance comparable to the SOTA. This indicates that FlashVTG can deliver excellent performance even with relatively smaller datasets.

We conducted an ablation study on the QVHighlights validation set to verify the effectiveness of FlashVTG. This dataset is currently the most widely used VTG benchmark and supports both MR and HD tasks, making it the most suitable for our research. We used the original single-layer feature and a method that generates confidence scores based solely on single predicted moments as our baseline.

Effect of different components. As shown in Table 8, we studied the effects of the Temporal Feature Layering and Adaptive Score Refinement modules. It can be observed that adding the TFL module to the baseline improved performance in both the MR and HD tasks, with the mAP for MR increasing by nearly 6%. Building on this, we further incorporated the ASR module to refine the confidence scores of the predicted moments. The improvements in MR-mAP, [email protected], and [email protected] indicate that the overall precision was maintained while enhancing Recall at 1, meaning the recall for the first predicted moment improved. Additionally, using the saliency score label from the HD task as a supervision signal indirectly boosted HD performance, with mAP increasing by 0.5% and Hit@1 improving by 1%.