A Survey on Video Moment Localization

The work of Hendricks et al. (Anne Hendricks et al., 2017) is generally regarded as the pioneer in this branch. It treats the video moment localization task as the moment retrieval problem and presents Moment Context Network (MCN)¹¹1https://github.com/LisaAnne/LocalizingMoments.. To train and evaluate the proposed model, (Anne Hendricks et al., 2017) collects the Distinct Describable Moments (DiDeMo) dataset, which consists of over 40,000 pairs of natural language queries and localized moments in unedited videos. According to the characteristic of this dataset, (Anne Hendricks et al., 2017) generates moment candidates via hand-crafted heuristics. Concretely, a video is first segmented into six five-second clips, and then any contiguous set of clips are utilized to construct a possible moment. Having obtained 21 moment candidates, it learns a shared embedding space for both video temporal context features and natural language queries, adopting the following inter-intra video ranking loss,

where $L_{i}^{intra}(\theta)$ is the intra-video ranking loss that encourages queries to be closer to the corresponding video moment than all other possible moments from the same video, $L_{i}^{inter}(\theta)$ is the inter-video ranking loss that encourages queries to be closer to corresponding video moments than moments with the same endpoints outside the video, $L_{R}(x,y)=max(0,x-y+b)$ is the ranking loss, $D$ is the square distance metric, $\theta$ refers to the learnable parameters of the model, and $\lambda$ is the weighting parameter. Building on top of MCN, several researchers promoted this branch from different angles, such as visual and query modeling. In the following, we would detail the corresponding studies in sequence.

Visual Modeling. (Anne Hendricks et al., 2017) considers the entire video as the visual context, which may induce inappropriate information in the context feature, influencing the localization performance. Inspired by this, Hendricks et al. (Hendricks et al., 2018) proposed the Moment Localization with Latent Context (MLLC) network²²2https://people.eecs.berkeley.edu/., which models visual context as a latent variable. Therefore, MLLC could attend to different video contexts conditioned on the specific query-video pair, overcoming the limitation of query-independent contextual reasoning. Besides, (Hendricks et al., 2018) builds the temporal reasoning in video and language dataset (TEMPO) based on the DiDeMo dataset (Anne Hendricks et al., 2017), which comprises two parts: TEMPO-TL (Template Language) and TEMPO-HL (Human Language). The former is constructed by the original DiDeMo sentences with language templates, while the latter is built with real videos and newly collected user-provided temporal annotations.

Query Modeling. A common limitation of previous approaches is that they utilize a single textual embedding to represent the query, which may not be precise. To fully exploit temporal dependencies between events within the query, Zhang et al. (Zhang et al., 2019) proposed a Temporal Compositional Modular Network (TCMN)³³3https://github.com/Sy-Zhang/TCMN-Release., which leverages a tree attention network to parse the given query into three components concerning the main event, context event, and temporal signal. Besides, a temporal relationship module is designed to measure the similarity between the phrase embedding for the temporal signal and the location feature, and a temporal localization module is designed to measure the similarity between the event-related phrase embedding and the visual feature. The overall matching score between each moment candidate and the given query is calculated by the late fusion strategy.

The work of Gao et al. (Gao et al., 2017) is the pioneer in this branch. They proposed a Cross-modal Temporal Regression Localizer (CTRL) to tackle the video moment localization task⁴⁴4https://github.com/jiyanggao/TALL.. To collect moment candidates for training, CTRL adopts multi-scale temporal sliding windows. In the framework of CTRL, the pre-trained C3D model (Tran et al., [n.d.]) is first utilized to extract visual features for moment candidates. Meanwhile, a Long Short-term Memory (LSTM) (Hu et al., 2016) network or Skip-thought (Kiros et al., 2015) is adopted to extract query representations. Afterwards, a cross-modal processing module is designed to jointly model the query and visual features, which calculates element-wise addition, multiplication, and direct concatenation. Finally, the multi-layer perception network (MLP) is designed for visual semantic alignment and moment location regression. To jointly train for visual-semantic alignment and moment location regression, CTRL utilizes a multi-task loss as follows,

where $N$ is the batch size, $cs_{i,j}$ is the matching score between query $s_{j}$ and moment candidate $m_{i}$, $\alpha_{c}$ and $\alpha_{w}$ are the hyper parameters, $R(t)$ is the $L_{1}$ function, $t_{s,i}^{*}$ and $t_{e,i}^{*}$ is the ground truth offsets, as well as $t_{s,i}$ and $t_{e,i}$ is predicted offsets. More importantly, for evaluation, CTRL adopts the TACoS dataset, and builds a new dataset on top of Charades by adding sentence temporal annotations, called Charades-STA. Hereafter, to further improve the localization performance from different aspects (e.g., visual, query, and interaction modeling), several methods are proposed in the past few years. We will elaborate them in the following paragraphs.

Visual Modeling. Although CTRL considers visual context information, it ignores the complex interactions among contexts and fails to identify the importance of each moment, therefore some important cues are missing. Inspired by this, Liu et al. (Liu et al., 2018a) extended the work of (Gao et al., 2017) and developed an Attentive Cross-Modal Retrieval Network (ACRN)⁵⁵5https://sigir2018.wixsite.com/acrn.. In particular, they designed a memory attention mechanism to emphasize the visual features mentioned in the query, obtaining the augmented moment representations.

As visual feature extracted from the $Fc6$ layer of C3D may weaken or ignore the critical visual cues, a Visual-attention Action Localizer (VAL) is introduced in (Song and Han, 2018), which extracts visual features from the $CONV5\_3$ layer of C3D. Therefore, it could capture more completed visual information. Similarly, Jiang et al. (Jiang et al., 2019) proposed a Spatial and Language-Temporal Attention (SLTA) method, which takes advantage of object-level local features to enhance the visual representations⁶⁶6https://github.com/BonnieHuangxin/SLTA.. To be specific, SLTA extracts local features on each frame by Faster R-CNN (Ren et al., 2015) and introduces the spatial attention to selectively attend to the most relevant local features mentioned in the query. And then it utilizes LSTM to encode the local feature sequence, obtaining the local interaction feature. Finally, SLTA integrates the global motion features extracted by pre-trained C3D with local interaction features as the final visual representations. Recently, to identify the fine-grained differences among similar video moment candidates, Zeng et al. (Zeng et al., 2021) proposed a Multi-Modal Relational Graph (MMRG) framework⁷⁷7https://cvpr-2021.wixsite.com/mmrg.. It develops a duel-channel relational graph to capture object relations and the phrase relations. Moreover, it considers two self-supervised pre-training tasks: attribute masking and context prediction, to enhance the visual representation and alleviate semantic gap across modalities.

Query Modeling. CTRL simply treats queries holistically as one feature vector, which may obfuscate the keywords that have rich temporal and semantic cues. Inspired by this, Liu et al. (Liu et al., 2018b) proposed a cRoss-modal mOment Localization nEtwork (ROLE)⁸⁸8https://acmmm18.wixsite.com/role.. It designs a language-temporal attention module, which adaptively reweighs each word’s features according to the textual query information and moment context information, therefore deriving useful query representations.

Interaction Modeling. Existing scheme CTRL adopts the straightforward multi-modal fusion method, i.e., fusing element-wise addition, multiplication, and concatenation, lacking in-depth analysis. To overcome this drawback, Wu et al. (Wu and Han, 2018) put forward a new multi-modal fusion approach, i.e., Multi-modal Circulant Fusion (MCF)⁹⁹9https://github.com/AmingWu/Multi-modal-Circulant-Fusion.. Particularly, after reshaping feature vectors into circulant matrices, it defines two types of interaction operations between vectors and matrices. The first one is the matrix multiplication between circulant matrix and projection vector, while the second is element-wise product between projection vector and each row vector of circulant matrix. More importantly, the proposed MCF can be integrated into the existing video moment localization models, to further improve the localization accuracy. To adequately exploit rich semantic cues about activities in videos and queries, Ge et al. (Ge et al., 2019) proposed an Activity Concepts based Localizer (ACL)¹⁰¹⁰10https://github.com/runzhouge/MAC., which is the first work to mine the activity concepts from both the videos and the sentence queries to facilitate the localization. To explore the interactions between two modalities, ACL separately conducts the multi-modal processing to the pair of activity concepts and the pair of visual features and sentence embeddings. Besides, ACL designs an actionness score generator to calculate the likelihood of the moment candidate containing meaningful activities. Eventually, the alignment score multiplied by the actionness score is set as the final score to predict the alignment confidence between each candidate and query.

Different from aforementioned two methods that focus on inter-modal interaction modeling, Zhang et al. (Zhang et al., 2021a) proposed a Multi-modal Interaction Graph Convolutional Network (MIGCN)¹¹¹¹11https://github.com/zmzhang2000/MIGCN/., which simultaneously explores the complex intra-modal relations and inter-modal interactions residing in the video and sentence query. Particularly, it introduces the multi-modal interaction graph, where two types of nodes are considered (i.e., clip nodes and word nodes) and edges compile both intra-modal relations and inter-modal interactions. Therefore, the graph convolution could refine node representations by jointly considering intra- and inter-modal interaction information.

Instead of using hand-crafted heuristics or multi-scale sliding windows to generate moment candidates, Xu et al. (Xu et al., 2019) advanced a Query-guided Segment Proposal Network (QSPN) to generate moment candidates¹²¹²12https://github.com/VisionLearningGroup/Text-to-Clip_Retrieval.. Concretely, QSPN generalizes the SPN from R-C3D model (Xu et al., 2017) by introducing query representations as the guidance information to generate moment candidates. And an early fusion retrieval model, instantiated as a two-layer LSTM, is introduced to find the moment that best matches the given query. Besides, a captioning loss is considered to enforce the LSTM to re-generate the query sentence, achieving improved retrieval performance. The combination of retrieval loss and captioning loss is defined as follows,

where $\sigma(s_{j},m_{j})$ is the matching score predicted by the LSTM, $m_{j}^{{}^{\prime}}$ is the negative moment either comes from the same video or from a different video, $K$ is the number of queries, $T_{k}$ is the number of words in the $k$-th query, $f(m_{k})$ represents the visual feature of the moment aligned with the $k$-th query. Similarly, Chen et al. (Chen and Jiang, 2019) proposed a Semantic Activity Proposal (SAP) framework, which also integrates the semantic information in queries into the moment candidate generation process. To be specific, it first trains a visual concept detection CNN with paired query-clip training data. Subsequently, the visual concepts extracted from the query and video frames are utilized to calculate the visual-semantic correlation score for every frame. By grouping frames with high visual-semantic correlation scores, moment candidates could be generated. Different from the above two methods, Xiao et al. (Xiao et al., 2021b) proposed a Boundary Proposal Network (BPNet), which utilizes VSLNet (Zhang et al., 2020c) to generate several high-quality moment proposals, avoiding redundant candidates.

Summarization. In what follows, we summarize the two-stage supervised video moment retrieval methods.

• •

  Hand-crafted heuristics: In this branch, the pioneer work MCN focuses on learning the shared embedding space for the moment representation and the query representation. To further improve the performance of MCN, MLLC enforces the model to attend to different video contexts, while TCMN exploits temporal dependencies between events in the query.

• •

  Multi-scale sliding windows: To enhance the performance of the pioneer work CTRL in this branch, ACRN, VAL, SLTA, and MMRG focus on the visual modeling. Specifically, ACRN emphasizes the visual features mentioned in the query, VAL captures more completed visual information from the feature map of C3D, SLTA takes advantage of object-level local features, and MMRG considers self-supervised pre-training tasks. Differently, ROLE focuses on query modeling and designs a language-temporal attention module to derive useful query representations. Moreover, MCF, ACL, and MIGCN pay attention to the interaction modeling. Particularly, MCF and ACL are devoted to modeling inter-modal interactions, while MIGCN jointly considers the inter-modal and intra-modal interaction modeling.

• •

  Moment generation networks: To generate moment candidates, QSPN generalizes the SPN by introducing query representations as the guidance information, while SAP generates moment candidates by grouping frames with high visual-semantic correlation scores. Differently, BPNet directly utilizes the VSLNet (Zhang et al., 2020c) to generate moment proposals.

Chen et al. (Chen et al., 2018) proposed the first dynamic single-stream end-to-end Temporal GroundNet (TGN) model for video moment localization¹³¹³13https://github.com/JaywongWang/TGN.. It sets each time step as an anchor and generates $K$ moment candidates that end at the current time step. Moreover, it utilizes the hidden states generated by the interaction module to yield confidence scores for moments ending at the corresponding time step. The objective of TGN is defined as,

where $(V,S)$ denotes a training pair from the training set, $y_{t}^{k}$ is interpreted as whether the $k$-th moment candidate at time step $t$ corresponds to the given query, $c_{t}^{k}$ denotes the prediction result, $w_{0}^{k}$ and $w_{1}^{k}$ are calculated according to the frequencies of positive and negative samples in the training set with length $k\delta$.

According to the primary contributions of existing anchor-based one-stage approaches, they can be further classified into two categories, i.e., focusing on localization module building and interaction modeling.

Localization Module. To yield more precise localization, Wang et al. (Wang et al., 2020a) proposed an end-to-end Contextual Boundary-aware Prediction (CBP) model¹⁴¹⁴14https://github.com/JaywongWang/CBP., which jointly predicts temporal anchors and boundaries at each time step. Its localization module consists of an anchor sub-module as (Chen et al., 2018) did and a boundary sub-module that is utilized to decide whether the current step is a semantic boundary corresponding to the start/end time of the moment.

Interaction Modeling. As simply combining the video and query information is not expressive enough to explore the interaction information, Li et al. (Li et al., 2019b) proposed a Bidirectional Single-Stream Temporal Localization (BSSTL) model. It introduces an attentive cross-modal fusion model, consisting of dynamic attention weighted query, joint gating, and fusion, to capture the interaction between the queries and videos. Liu et al. (Liu et al., 2020a) designed a deep Rectification-Modulation Network (RMN), which adopts a multi-step reasoning framework to gradually capture the higher-order multi-modal interaction. To be specific, it utilizes multi-step rectification-modulation layers to progressively refine video and query interactions, improving localization accuracy. Different from previous approaches that merely focus on exploiting unidirectional interaction information from the video to query, Qu et al. (Qu et al., 2020) proposed a Fine-grained Iterative Attention Network (FIAN) to capture bilateral query-video interaction information. Specifically, it leverages a symmetrical iterative attention module to generate video-aware query and query-aware video representations, where two attention branches share the same architecture but opposite input. To model multi-level vision-language cross-modal relation and long-range context, Ning et al. (Ning et al., 2021) proposed an Interaction-Integrated Network (I2N). To be specific, it introduces an interaction-integrated cell, which integrates both cross-modal and contextual interactions between vision and language. By stacking a few interaction-integrated cells in a chain, the network could explore more detailed vision-language relations on multiple granularities, therefore achieving more accurate semantics understanding and more accurate localization results.

The aforementioned single-pass methods mainly focus on exploring the cross-modal relations between the video and query, ignoring the importance of jointly investigating the cross- and self-modal relations. In light of this, Zhang et al. (Zhang et al., 2019b) introduced a Cross-Modal Interaction Network (CMIN)¹⁵¹⁵15https://github.com/ikuinen/CMIN., which jointly considers multiple crucial factors. Specifically, the syntactic graph convolution network is adopted to enhance the query understanding. And the multi-head self-attention module is built to capture long-range semantic dependencies from the video context. Moreover, a multi-stage cross-modal interaction module is presented to exploit the potential relations of video and query contents. Recently, inspired by the fact that captioning can improve the performance of image-based grounding of textual phrases (Rohrbach et al., 2016), Zhang et al. further improved the model CMIN by integrating a query reconstruction module (Lin et al., 2020), dubbed as CMIN-R. Likewise, Liu et al. (Liu et al., 2020b) presented a Cross- and Self-Modal Graph Attention Network (CSMGAN)¹⁶¹⁶16https://github.com/liudaizong/CSMGAN.. It introduces a cross-modal relation graph to highlight relevant instances across the video and query. Meanwhile, a self-modal relation graph is utilized to model the pairwise relation inside each modality. In the same year, Chen et al. (Chen et al., 2020) proposed a Pairwise Modality Interaction module (dubbed PMI-LOC), which feeds multimodal features to a channel-gated modality interaction module, to exploit both intra- and inter-modality information.

Different from anchor-based approaches that generate moment candidates via multi-scale windows at each time step, sampler-based ones output moment candidates via different sampler networks, such as temporal convolution, segment-tree, and enumeration.

Temporal Convolution. Yuan et al. (Yuan et al., 2019a) proposed a Semantic Conditioned Dynamic Modulation (SCDM) mechanism based approach¹⁷¹⁷17https://github.com/yytzsy/SCDM., which leverages query semantic information to modulate the temporal convolution processes in a hierarchical temporal convolutional network. Particularly, similar to (Lin et al., 2017)(Liu et al., 2016) for object/action detection, the position prediction module is introduced to output location offsets and overlap scores of moment candidates based on the modulated features. Similarly, Zhang et al. (Zhang et al., 2019a) proposed Moment Alignment Network (MAN)¹⁸¹⁸18https://github.com/dazhang-cv/MAN. to deal with semantic misalignment. In particular, it treats the encoded word features as efficient dynamic filters to convolve with input visual representations, and then a hierarchical convolutional network is applied to directly produce multi-scale moment candidates. Different from MAN, Hu et al. (Hu et al., 2021b) presented an end-to-end Coarse-to-fine cross-modaL sEmantic Alignment netwoRk (CLEAR)¹⁹¹⁹19https://github.com/Huyp777/CSUN., which utilizes bi-directional temporal convolution network followed by multi-scale temporal pooling to generate moment candidates. And to explore cross-modal semantic correlation and improve the localization efficiency, it respectively advances a multi-granularity interaction module and a semantic pruning strategy.

Different from that existing methods mainly leverage vanilla soft attention to perform the alignment in a single-step process, Liu et al. (Liu et al., 2021c) presented an Iterative Alignment Network (IA-Net), which captures complicated relations between inter- and intra-modality through multi-step reasoning. To be specific, it proposes an improved co-attention mechanism that utilizes learnable paddings to address nowhere-to-attend problem with deep latent clues, and a calibration module to refine the alignment knowledge of inter- and intra-modal relations.

To boost the efficiency of moment localization, Hu et al. (Hu et al., 2021a) proposed an end-to-end Cross-Modal Hashing Network (CMHN)²⁰²⁰20https://github.com/Huyp777/CMHN.. It designs a cross-model hashing module to project cross-modal heterogeneous representations into a shared isomorphic Hamming space for compact hash code learning. Therefore, with the well-trained model at hand, the hash codes of any upcoming videos could be obtained offline and independently, improving the localization efficiency and scalability.

Segment-tree. Unlike the aforementioned sampler-based approaches, Zhao et al. (Zhao et al., 2021) formulated the video moment localization task as a multi-step decision problem and proposed a Cascaded Prediction Network (CPN). To be specific, it adopts a segment tree-based structure to generate moment candidates in different temporal scales and refine the representation of them in a message-passing way via graph neural network.

Proposal Generation Network. To well exploit moment-level interaction and speed up the localization efficiency, Liu et al. (Liu et al., 2021d) proposed an Adaptive Proposal Generation Network (APGN). It first leverages a binary classification module to predict foreground frames, and then it utilizes the boundary regression module to generate proposals on each foreground frame. In this way, the redundant proposals are decreased. Meanwhile, Xiao et al. (Xiao et al., 2021a) proposed a Learnable Proposal Network (LP-Net) for video moment localization²¹²¹21https://github.com/xiaoneil/LPNet/.. In the LP-Net, proposal boxes are represented by 2-d parameters ranging from 0 to 1, denoting normalized center coordinates and lengths, which are randomly initialized. And these learnable proposal boxes are updated by dynamic adjustor during training.

Enumeration. Given that the input video has temporal length $n$, the Temporal Modular Network (TMN) proposed in (Liu et al., 2018c) will regress $n$ correspondence scores for each temporal segment, and then combine the scores of consecutive segments to produce $\frac{n(n+1)}{2}$ scores for all possible moment candidates. Finally, the sub-video with the maximum score is predicted as the golden moment.

To model temporal relations between video moments, Zhang et al. (Zhang et al., 2020b) presented a 2D temporal map, where one dimension indicates the start time of a moment and the other indicates the end time. Based on the 2D map, (Zhang et al., 2020b) introduces a Temporal Adjacent Network (2D-TAN)²²²²22https://github.com/microsoft/2D-TAN. to generate the 2D score map, i.e., the matching scores of moment candidates on the 2D temporal map with the given query. Recently, Zhang et al. (Zhang et al., 2021b) extended the 2D-TAN to a multi-scale version and proposed a Multi-Scale Temporal Adjacency Network (MS-2D-TAN), which models the temporal context between video moments by a set of predefined two-dimensional maps under different temporal scales²³²³23https://github.com/microsoft/2D-TAN..

Different from aforementioned methods that use alignment information to find out the best-matching candidate, Wang et al. (Wang et al., 2020b) proposed a unified Dual Path Interaction Network (DPIN). It jointly considers the alignment and discrimination information to make the prediction. Particularly, the frame-level representation path extracts the discriminative boundary information from the fused features, while the candidate-level path arranges the moment candidate features in a 2D temporal map as (Zhang et al., 2020b) did and extract the alignment information. Finally, DPIN fuses the two kinds of representations to make prediction. Likewise, Wang et al. (Wang et al., 2021c) developed a Structured Multi-level Interaction Network (SMIN) by jointly considering multiple levels of visual-textual interaction and moment structured interaction. (Zhang et al., 2021d) presents the Multi-stage Aggregated Transformer Network (MATN) to enhance the cross-modal alignment and localization accuracy. Particularly, it designs a new visual-language transformer backbone by using different parameters to process different modality contents, and introduces a multi-stage aggregation module to calculate the moment representation via considering three stage-specific representations. Wu et al. (Wu et al., 2021) proposed a video moment retrieval model, named SV-VMR, which jointly models the fine-grained and comprehensive relations by using both semantic and visual structures. Unlike previous work, Gao et al. (Gao et al., 2021) formulated the video moment localization task as the video reading comprehension and presented a Relation-aware Network (RaNet)²⁴²⁴24https://github.com/Huntersxsx/RaNet.. To distinguish similar moment candidates in the visual modality, it introduces a coarse-and-fine cross-modal interaction module to simultaneously capture the sentence-moment and token-moment level interaction, obtaining sentence-aware and token-aware moment representations. Meanwhile, it leverages graph convolutional network to capture moment-moment relations, which further strengthens the discriminative of moment representations.

To against the temporal location biases of video moment localization, Yang et al. (Yang et al., 2021a) advanced the Deconfounded Cross-modal Matching (DCM) method²⁵²⁵25https://github.com/Xun-Yang/Causal_Video_Moment_Retrieval., which considers the moment temporal location as a hidden confounding variable. To remove the confounding effects of moment location, it disentangles the moment representation to infer the core feature of visual content, and then applies causal intervention on the multi-modal input based on backdoor adjustment. To well balance the localization accuracy and speed, Gao et al. (Gao and Xu, 2021a) proposed the Fast Video Moment Retrieval (FVMR) model, which replaces the complex cross-modal interaction module in existing methods with a cross-modal common space.

Unlike the anchor- and sampler-based methods that depend on moment candidates, the proposal-free ones directly predict the start and end time of the target moment. This removes the need to retrieve and re-rank multiple moment candidates.

As the first work in this subbranch, Chen et al. (Chen et al., 2019a) proposed a localizing network (LNet), which works in an end-to-end fashion, to tackle the video moment localization task. It first matches the query and video sequence by cross-gated attended recurrent networks, to exploit their fine-grained interactions and generate a query-aware video representation. Afterwards, a self interactor is designed to perform cross-frame matching, which dynamically encodes and aggregates the matching evidences. Finally, a boundary model is introduced to locate the positions of video moments corresponding to the query by directly predicting the start and end points. To further improve the localization accuracy of LNet, the follow-up research of this branch is dedicated to exploring how to well exploit interactive information between the video and the given query.

To adequately explore the cross-modal interactions between the query and video, Yuan et al. (Yuan et al., 2019b) proposed an end-to-end Attention Based Location Regression (ABLR) model ²⁶²⁶26https://github.com/yytzsy/ABLR_code.. It leverages a multi-modal co-attention mechanism to learn both video and query attentions. Moreover, a multi-layer attention-based location prediction network is proposed to regress the temporal coordinates for the target video moment. Similarly, Ghosh et al. (Ghosh et al., 2019) proposed an Extractive Clip Localization (ExCL) approach to predict the start and end frames of the target moment. Particularly, they compared three variants of span predictor, i.e., MLP, Tied-LSTM, and Conditioned-LSTM, to predict the start and end probabilities for each frame. Liu et al. (Liu et al., 2021b) designed a Single-shot Semantic Matching Network (SSMN), which properly explores the cross-modal relationship and the temporal information via an enhanced cross-modal attention module. For in-depth relationship modeling between semantic phrases and video segments, Mun (Mun et al., 2020) proposed a Sequential Query Attention Network (SQAN) based method by performing local-global video-query interactions²⁷²⁷27https://github.com/JonghwanMun/LGI4temporalgrounding.. Tang et al. (Tang et al., 2021a) proposed an Attentive Cross-modal Relevance Matching (ACRM) model²⁸²⁸28https://github.com/tanghaoyu258/ACRM-for-moment-retrieval., which uses element-wise multiplication and subtraction functions to model the interaction between the frame and frame-specific query features. Liang et al. (Liang et al., 2021) proposed a Multi-branches Interaction Model (MIM), which re-weights the video features according to relevance of query and video in multiple sub-spaces. By treating the video as a text passage and the target moment as the answer span, Zhang et al. (Zhang et al., 2020c) proposed a video span localizing network (VSLNet) on top of the standard span-based QA framework²⁹²⁹29https://github.com/IsaacChanghau/VSLNet.. In particular, a context-query attention (CQA) (Xiong et al., 2017) is designed to capture the cross-modal interactions between visual and textural features. Recently, to address the performance degradation on long videos, Zhang et al. (Zhang et al., 2022) further extended VSLNet to VSLNet-L by introducing a multi-scale split-and-concatenation strategy. It first partition long video into clips of different lengths, and then locate the target moment in the clips that are more likely to contain it. Unlike the above methods, Opazo et al. (Opazo et al., 2020) introduced a Proposal-free Temporal Moment Localization model using Guided Attention (PFGA)³⁰³⁰30https://github.com/crodriguezo/TMLGA., which utilizes an attention-based dynamic filter to transfer query information to the video. Moreover, a new loss is designed to enforce the model to focus on the most relevant part of the video. Since humans have difficulty agreeing on the start and end time of an action inside a video (Alwassel et al., 2018)(Sigurdsson et al., 2017), PFGA uses soft-labels (Szegedy et al., 2016) to model the uncertainty associated to the labels. The final loss for training PFGA method is defined as,

where $L_{KL}$ loss aims to minimize the Kullback-Leibler divergence between the predicted and ground truth probability distributions, $L_{att}$ loss is designed to encourage the model to attend relevant features, $\delta$ is the Kronecker delta, $n$ denotes the length of visual sequence, and $a_{i}$ is the attention of the corresponding location.

To further explore fine-grained interaction information within the video and query sentence, Chen et al. (Chen and Jiang, 2020) advanced a Hierarchical Visual-Textual Graph (HVTG) model for video moment localization³¹³¹31https://github.com/forwchen/HVTG.. To be specific, it first builds an object-sentence subgraph to obtain sentence-aware object features for each frame. And then it feeds sentence-aware object features into an object-object subgraph, to capture object-object interactions inside each frame. Afterwards, a sentence-guided attention followed by a Bi-LSTM is introduced to establish temporal relations among frames, outputting final visual representations. To explore multi-level query semantics (both word- and phrase-level) as well as model multi-level interactions between two modalities, Tang et al. (Tang et al., 2021b) proposed a Multi-level Query Exploration and Interaction (MQEI) model. To be specific, it leverages a stack of syntactic GCNs to model the syntactic dependencies of the query, obtaining the word-level query features. And a sequential attention module is adopted to learn phrase-level query representations. With such multi-level query representations, the context-query attention is adopted for the word-level and phrase-level cross-modal feature fusion.

Different from most proposal-free methods that are devoted to multi-modal modeling (i.e., fusion or interaction), Li et al. (Li et al., 2021a) focused on exploiting the fine-grained temporal clues in videos and presented a Contextual Pyramid Network (CP-Net) to mine rich temporal contexts through fine-grained hierarchical correlation at different 2D temporal scales. Similarly, a language-conditioned message-passing algorithm is proposed in (Rodriguez-Opazo et al., 2021) to well learn the video feature embedding, which could capture the relationships between humans, objects and activities in the video. Inspired by the successful application of biaffine mechanism in dependency parsing, Liu et al. (Liu et al., 2021e) applied the biaffine mechanism to video moment localization and presented the Context-aware Biaffine Localizing Network (CBLN)³²³²32https://github.com/liudaizong/CBLN.. It leverages a multi-context biaffine localization module to aggregate both multi-scale local and global contexts for each frame representation, and then scores all possible pairs of start and end frames for moment localization.

To alleviate the imbalance problem between positive and negative samples to some extent, Lu et al. (Lu et al., 2019) proposed a DEnse Bottom-Up Grounding (DEBUG) framework by regarding all frames in the ground truth moment as positive samples. For each positive frame, DEBUG has a classification subnet to predict its relatedness with the query, and a boundary regression subnet to regress the unique distances from its location to bi-directional ground truth boundaries. This helps to avoid falling into the local optimum caused by independent predictions since each pair of boundary predictions is based on the same frame feature. Note that DEBUG can be seamlessly incorporated into any backbone to boost performance.

Through utilizing the distances between the frame within the ground truth and the start (end) frame as dense supervisions, Zeng et al. (Zeng et al., 2020) designed a dense regression network (DRN)³³³³33https://github.com/Alvin-Zeng/DRN.. To be specific, DRN first forwards the video frames and the query to the video-query interaction module for extracting the multi-scale feature maps. Afterwards, each feature map is processed by the grounding module to predict a temporal bounding box, a semantic matching score, and an IoU score at each temporal location for ranking. Finally, combining the matching score and the IoU score, DRN could find the best grounding result.

Despite the enormous success of the aforementioned models, they may suffer from the spurious correlations between textual and visual features due to the selection bias of the dataset. To address this problem,  (Nan et al., 2021) first presents the Interventional Video Grounding (IVG) by introducing the causal interventions, as shown in Fig. 5. Moreover, it introduces a dual contrastive learning to learn more informative visual and textual representations. To be specific, the loss $L_{vv}$ is expressed as follows,

where $I_{\theta}^{vv(s)}$ and $I_{\theta}^{vv(e)}$ respectively denote the mutual information between the start and end boundaries of the video and the other clips. And the contrastive loss $L_{vq}$ is defined as follow,

where $V_{a}^{{}^{\prime}}$ denote the features that reside within a target moment, $V_{b}^{{}^{\prime}}$ denote the features are located outside of the target moment, $C$ refers to the mutual information discriminator, and $sp(z)=log(1+e^{z})$.

Different from most existing methods that only consider RGB images as visual features, Chen et al. (Chen et al., 2021) proposed a multi-modal learning framework for video temporal grounding using (D)epth, (R)GB, and optical (F)low with the (T)ext as the query (DRFT). To model the interactions between modalities, this paper develops a dynamic fusion mechanism across modalities via co-attentional transformer. Moreover, to enhance intra-modal feature representations, it leverages self-supervised contrastive learning across videos for each modality.

Summarization. Below, we summarize the one-stage video moment localization methods.

• •

  Anchor-based models: In this branch, on top of TGN, CBP designs a boundary prediction module to predict temporal boundaries at each time step. To capture the interaction information, BSSTL, RMN, FIAN, and I2N focus on exploring the inter-modal interactions, while CMIN, CSMGAN, PMI-LOC and CMIN-R focus on exploring both intra- and inter-modal interactions.

• •

  Sampler-based models: In this branch, SCDM, MAN, CLEAR, IA-NEt, and CMHN adopt the temporal convolution network to generate moment candidates. Specifically, SCDM designs semantic modulated temporal convolution, MAN utilizes hierarchical convolution, and CLEAR leverages bi-directional temporal convolution. Although IA-Net and CMHN also utilizes temporal convolution to generate moment candidates, they respectively are devoted to tackling the interaction modeling and location efficiency issue. CPN innovatively integrates the segment-tree into the video moment localization for moment generation. Differently, APGN and Lp-Net introduce different proposal generation networks for moment candidates generation. TMN, 2D-TAN, MS-2D-TAN, DPIN, MATN, SMIN, SV-VMR, RaNet, DCM, and FVMR enumerate all possible moment candidates. More specifically, 2D-TAN and MS-2D-TAN respectively utilize the temporal adjacent network and multi-scale temporal adjacent network to gradually perceive more context of adjacent moment candidates. DPIN, MATN, SMIN, SV-VMR, and RaNet focus on developing different mechanisms to capturing interaction information. Differently, DCM is devoted to addressing the location bias issue, while FVMR aims to well balance the location accuracy and efficiency.

• •

  Proposal-free models: In this branch, ABLR, ExCL, SQAN, PFGA, VSLNet, VSLNet-L, ACRM, MIM, and SSMN focus on exploring cross-modal interaction information between videos and queries. Thereinto, VSLNet and VSLNet-L tackle video moment localization by using the standard span-based QA framework. HVTG and MQEI are devoted to modeling both intra- and inter-modal interactions, while CBLN, CP-Net, and DORi merely pay attention to enhance the visual representation. DEBUG aims to tackle the imbalance problem between positive and negative samples, while DRN aims to promote the accuracy of localization by considering frame-level dense supervisions. IVG targets to address the spurious correlations between textual and visual features, while DRFT tackles the video moment localization by adopting multi-modal visual features.

Among two-stage supervised video moment localization models, the hand-crafted heuristics based ones (i.e., MCN, MLLC, and TCMN) are mainly evaluated on DiDeMo and TEMPO. Their experimental results are reported in Table 4, 5, and 6. As one of the earliest introduced methods, MCN is considered the de facto baseline result. MLLC outperforms the baseline model MCN, suggesting that learning to reason about which context moment is correct (as opposed to considering global video context) is beneficial. TCMN exhibits the promising performance across all the metrics of the complex sentence and comparative results in simple sentences on Tempo-HL. The results show that the compositional modeling of complex queries can improve the localization performance.

As to video moment localization approaches that utilize multi-scale sliding windows to generate moment candidates (i.e., CTRL, ACRN, VAL, SLTA, MMRG, ROLE, MCF, ACL, and MIGCN), experiments are primarily executed on TACoS and Charades-STA. The experimental results are reported in Table 7 and 8. We can see that: 1) ROLE achieves better performance as compared to CTRL on Charades-STA, verifying the importance of visual-aware query modeling. 2) Although ACRN, VAL, SLTA, and MMRG all focus on improving visual modeling, MMRG achieves superior performance on both TACoS and Charades-STA. This mainly because it jointly considers the object relations and the phrase relations modeling for video moment localization, meanwhile, it introduces pre-training tasks to enhance the visual representation. 3) Compared with MCF that also focus on intra-modal interaction modeling, ACL achieves remarkable performance improvements on both TACoS and Charades-STA. This reflects that the pair of activity concepts extracted from both videos and queries play a vital role in improving the cross-modal alignment. 4) MIGCN outperforms ACL on Charades-STA, justifying the necessity of capturing both intra- and inter-modal interaction information. And 5) MMRG achieves superior performance, and the results are competitive to other multi-scale sliding windows based methods. One possible reason is that it adopts two self-supervised pre-training tasks: attribute masking and context prediction to alleviate semantic gap across modalities.

In contrast, the methods that leverages moment generation networks are mainly evaluated on Charades-STA and ActivityNet-Captions. As reported in Table 8 and 9, BPNet outperforms both QSPN and SAP. Because it utilizes VSLNet to generate high-quality moment candidates, therefore boosting the localization accuracy. However, the localization accuracy of BPNet is significantly worse than the MMRG. The reasons may be that 1) Compared with VSLNet, the multi-scale sliding windows generate higher quality moment candidates. And 2) MMRG could identify the fine-grained differences among similar video moment candidates. Despite achieving promising performance, these two-stage approaches should pre-process the untrimmed videos to obtain moment candidates, therefore they are inferior in efficiency.

We summarized experimental results of all discussed one-stage methods in Table 6-9. From these results, we could find that: 1) Among anchor-based one-stage methods, FIAN achieves the best performance on all datasets. Particular, the performance of FIAN significantly surpasses TGN, demonstrating the importance of inter-modal interaction modeling. Moreover, compared with other inter-modal interaction modeling methods, FIAN achieves better performance. This verifies the effectiveness of the iterative attention mechanism. More importantly, FIAN even achieves superior performance as compared to methods that jointly consider intra- and inter-modal interaction modeling. This may be because it iteratively captures bilateral query-video interaction information. Furthermore, CMIN-R performs better than CMIN, verifying that reconstructing the natural language queries could indeed enhance the cross-modal representations. 2) For sampler-based localization methods, SMIN achieves the best performance on Charades-STA. However, MATN achieves the best performance on TACoS and ActivityNet-Captions and significantly surpasses the work SMIN. On one hand, this reflects that the enumeration strategy could generate higher quality moment candidates as compared to other strategies. On the other hand, it demonstrates the powerful ability of the visual-language transformer for learning modality representations. 3) For proposal-free methods, CP-Net achieves the best performance on TACoS, but its performance is slightly worse on the other two datasets. This is mainly because the video length of TACoS is longer than the other two datasets, namely there are lots of visual similar moment candidates. CP-Net focuses on exploiting the fine-grained temporal clues to enhance the discriminative of different moments, therefore achieving the better performance on TACoS. As DORi could capture complex relationships between humans, objects and activities in the video, it outperforms CP-Net on Charades-STA which is collected for activity understanding. Compared to TACoS and Charades-STA datasets, ActivityNet-Captions is collected from Youtube, of which the videos contain multi-modal information. Thereby, DRFT achieves the best performance on ActivityNet-Captions since it utilizes multimodal information and adequately exploit interactions between modalities. And 4) among all one-stage methods, MATN achieves superior performance as compared to FIAN, CP-Net, DRFT, and DORi. This reflects that the performance gap between moment generation based methods and proposal-free methods is still large. Thereby, for proposal-free methods, it is worth to explore more effective interaction strategies to further improve localization accuracy.

Reinforcement learning based localization models alleviate the efficiency issue to a certain extent, yet their performance is inferior as reported in Table 6-9. The main reason may be that they mostly focus on the design of policy and rewards, ignoring the importance of multiple crucial factors, such as query representation learning, video context modeling, and multimodal fusion.