Text-based Localization of Moments in a
Video Corpus

The main contributions of the proposed work are as follows:

• •

  We explore an important, yet under-explored, problem of text query-based localization of moments in a video corpus.

• •

  We propose a novel framework, HMAN, that uses stacked temporal convolutional layers in a hierarchical structure to represent video moments and texts jointly in an embedding space. Combined with the proposed learning objective, the model is able to align moment and sentence representations by distinguishing both local subtle differences of the moments as well as global semantics of the videos simultaneously.

• •

  Towards solving the problem, we propose a novel learning objective that utilizes text-guided global semantics of the videos to distinguish moments from different videos.

• •

  We empirically show the efficacy of our proposed approach on DiDeMo, Charades-STA, and ActivityNet Captions dataset and study the significance of our proposed learning objective.

Consider that we have a set of $N$ long and untrimmed videos $\mathcal{V}=\{v_{i}\}_{i=1}^{N}$, where a video $v$ is associated with $m_{v}$ temporal sentence annotations $\mathcal{T}=\{(s_{j},\tau_{j}^{s},\tau_{j}^{e})\}_{j=1}^{m_{v}}$. Here, $s_{j}$ is the sentence annotation and $\tau_{j}^{s},\tau_{j}^{e}$ are the starting time and ending time of the moment in the video that corresponds with the sentence annotation $s_{j}$. The set of all temporal sentence annotations is $\mathcal{S}=\{\mathcal{T}_{i}\}_{i=1}^{N}$. Given a natural language query $s$, our task is to predict a set $s_{det}=\{v,\tau^{s},\tau^{e}\}$ where, $v$ is the video that contains the relevant moment and $\tau^{s},\tau^{e}$ are the temporal information of that moment.

Our goal is to learn representations for candidate moments and sentences in such a way that the related moment-sentence pairs are aligned in the joint embedding space. Towards this goal, we propose HMAN, which is illustrated in Figure 2. First, we employ a feature extraction unit to extract clip level features $\{\boldsymbol{c}_{i}\}_{i=1}^{l}$ from a video and sentence features $\hat{\boldsymbol{s}}$ from a sentence. Clip representations and sentence representations are used to learn the semantic alignment between sentences and candidate moments. To project the moment representations and sentence representations in the joint embedding space, we use a hierarchical moment encoder module and a sentence encoder module respectively. The moment encoder module is inspired by single shot temporal action detection approach [4] where temporal convolutional layers are stacked in a hierarchical structure to obtain multi-scale moment features representing video segments of different duration. For the sentence encoder module, we use a two-layer feedforward neural network. Based on text queries, we derive the learning objective to explicitly focus on distinguishing intra-video moments and inter-video global semantics. We adopted sum-margin based triplet loss [59] and max-margin based triplet loss [59] separately in two different settings to train the model in an end-to-end fashion and gained performance improvement over baseline approaches in both setups. In the inference stage, for a query sentence, the candidate moment with the most similar representation is retrieved from the corpus of videos.

To work with data from different modalities, we extract feature representations using modality specific pretrained models.

Video Feature Extraction. We extract high level video features using a deep convolutional neural network. Each video $v$ is divided into a set of $l$ non-overlapping clips and we extract features for each clip. As a result, the video is represented by a set of features $\{\boldsymbol{c}_{i}\}_{i=1}^{l}$, where $\boldsymbol{c}_{i}$ is the feature representation of the $i^{th}$ clip. To generate representations for all the candidate moments of a video in a single shot approach [4], we keep the input video length, i.e., number of clips, $l$, fixed. A video longer than the fixed length is truncated and a video shorter than the fixed length is padded with zeros.

Sentence Feature Extraction. To represent sentences, we use GloVe word embedding [60] for each word in a sentence. Then these word embedding sequences are encoded using a Bi-directional Gated Recurrent Unit (GRU) [61] with $512$ hidden states. Here, words in a sentence are represented by a $512$-dimensional vector, corresponding to their GRU hidden states. So, we can have a set of word-by-word representations of a sentence $S=\{\boldsymbol{h}_{i}\}_{i=1}^{n}$, where $n$ is the number of words present in the sentence. The average of the word representations is used as the sentence representation $\boldsymbol{\hat{s}}$.

Existing approaches for moment localization based on learning joint visual-semantic embedding space either use a temporal sliding window with multiple scales [6] or optimize over a predefined set of consecutive clips based on clip-sentence similarity [23] to generate candidate segments. However, sliding over a video with different scales or optimizing for all possible combinations of clips is computationally heavy. Again, in both cases, extracted candidate moments or predefined clips are projected in the joint embedding space independent of neighboring or overlapping moments/clips of the same video. Consequently, while learning the moment-sentence or clip-sentence semantic alignment, representations for neighboring or overlapping moments are not constrained to be well clustered to preserve the semantic similarity. Therefore, instead of projecting representations for candidate moments independently and inefficiently in the joint embedding space, inspired by the single shot activity detection [4], we use temporal convolutional layers [62] in a hierarchical setup to project representations for all candidate moments of a video simultaneously. We use a stack of $1D$ convolutional layers where the convolution operation can be denoted as $Conv(\sigma_{k},\sigma_{s},d)$. Here, $\sigma_{k}$, $\sigma_{s}$, and $d$ indicate the kernel size, stride size, and filter numbers, respectively. The set of moment representations generated for $K$ layers of hierarchical structure is $\{\{\boldsymbol{m}_{i}^{k}\}_{i=1}^{T_{k}}\}_{k=1}^{K}$. Here, $T_{k}$ is the temporal dimension of the $k^{th}$ layer, which decreases in the following layers. $\boldsymbol{m}^{k}_{i}\in\mathcal{R}^{d}$ is the $i^{th}$ moment representation of the $k^{th}$ layer and $k^{th}$ layer generates $T_{k}$ moment representations. Feature representations in the top layers of the hierarchy correspond to moments with shorter temporal duration, while the feature representations in the bottom layers correspond to moments with longer duration in a video. We keep the feature dimension of each moment representation fixed to $d$ for all the layers of the temporal convolutional network.

We learn to project the textual representations in the joint embedding space keeping the inputs from different modalities with similar semantics close to each other. We use two layers of feedforward neural network with learnable parameters $\boldsymbol{W}^{s}_{1}$, $\boldsymbol{W}^{s}_{2}$, $\boldsymbol{b}^{s}_{1}$, and $\boldsymbol{b}^{s}_{2}$ to project the sentence representation $\boldsymbol{\hat{s}}$ in the joint embedding space, which can be defined as,

Here, the dimension of the projected sentence representation $\boldsymbol{s}$ is kept consistent with the projected moment representation $\boldsymbol{m}$ ($\boldsymbol{m},\boldsymbol{s}\in\mathcal{R}^{d}$).

Projected representations in the joint embedding space from different modalities need to be close to each other if they are semantically related. Training procedures to learn projecting representations in the joint embedding space mostly adopts two common loss functions: sum-margin based triplet ranking loss [59] and max-margin based triplet ranking loss [63]. We consider both of these loss functions separately. As illustrated in Figure 3, we focus on distinguishing intra-video moments and inter-video global semantic concepts. In this section, we discuss our approach to learn projecting representations from different modalities in the joint embedding space for multimodal data.

Similarity Measure. We use the cosine similarity of projected representations from two modalities in the joint embedding space to infer their semantic relatedness. So, the similarity between a candidate moment $m$ and a sentence $s$ is,

where $\boldsymbol{m}$ and $\boldsymbol{s}$ are the projected moment representation and sentence representation in the joint embedding space.

Learning for Intra-video Moments. To localize a sentence query in a video, the model needs to identify the subtle differences of the candidate moments from the same video and distinguish them. Among the candidate segments of a video, one or few of the moments can be considered related to the query sentence based on some IoU threshold. While training the network, we consider related moments with the queried sentence as the positive pairs and non-corresponding moments with the queried sentence as the negative pairs. Suppose, for a pair of video-sentence $(v,s)$, we consider the set of positive moment-sentence pairs $\{(m,s)\}$ and the set of negative moment-sentence pairs $\{(m^{-},s)\}$. We compute the intra-video ranking loss for all video-sentence pairs $\{(v,s)\}$. Using the sum-margin setup, the intra-video triplet loss is:

Similarly, using the max-margin setup, we calculate the intra-video triplet loss by,

Here, $[f]_{+}=max(0,f)$ and $\alpha_{intra}$ is the ranking loss margin for intra-video moments.

Learning for Videos. Learning to distinguish intra-video moments only allows the model to learn subtle changes in the video. It does not allow the model to distinguish moments from different videos. However, learning to differentiate moments from different videos is important as we need to localize the correct moment in the video corpus. Hence, we also learn to distinguish moments from different videos by capitalizing on the text-guided global semantics of videos. As the global semantics varies across videos we try to distinguish videos based on these global semantics. To do so, we learn to maximize the relevance of correct video-sentence pairs. Video-sentence relevance is computed in terms of moment-sentence relevance. As a result, learning to align video-sentence pairs enforces constraints on the representation of moments from different videos to be dissimilar. Inspired by the work of [27], we compute the relevance of a video and a sentence by,

where $\beta$ is a factor that determines how much to magnify the importance of the most relevant moment-sentence pair and $\{m\}$ is the set of all the moments in video $v$. As $\beta\rightarrow\infty$, $R(v,s)$ approximates $\max_{m_{i}\in v}S(m_{i},s)$. This is necessary because all the segments of the video do not correspond to the sentence.

For each positive video-sentence pair $(v,s)$ where the sentence $s$ relates to a segment of the video $v$, we can consider two sets of negative pairs $\{(v^{-},s)\}$ and $\{(v,s^{-})\}$. Using the sum-margin setup, we calculate the triplet loss for video-sentence alignment of all the positive video-sentence pairs $\{(v,s)\}$ by,

Similarly, using the max-margin setup, we compute the triplet loss for video-sentence alignment by,

Here, $\alpha_{video}$ is the ranking loss margin for learning inter-video global semantic concepts.

Learning Objective. We combine the calculated loss for intra-video case and video-sentence alignment case and try to minimize it as our final objective. For the sum-margin setup, the final objective is,

Similarly, for the max-margin setup, the final objective is,

Here, $\theta$ represents the network weights and all the learnable weights are lumped together in $\mathcal{W}$. $\lambda_{1}$ balances the contribution between learning to distinguish intra-video moments and learning to distinguish videos based on a text query. $\alpha$ is the weight on the regularization loss. Our objective is to optimize $\theta$ to generate a proper representation for candidate moments and sentences to minimize these combined losses. During training, these losses are computed for a mini-batch where the mini-batches are sampled randomly from the entire training set. This stochastic approach yields the advantage of reducing the probability of selecting instances with high semantic relation as the negative samples.

Inference. In the inference step, for a query sentence, we compute the similarity of candidate moment representations with the query sentence representation using Eqn. 2. We retrieve the candidate moment from the video corpus that results in the highest similarity.

We conduct experiments and evaluate the performance on three benchmark text-based video moment retrieval datasets, namely DiDeMo [6], Charades-STA [5], and ActivityNet Captions [66]. All of these datasets contain unsegmented and untrimmed videos with natural language sentence annotations with temporal information. Table I summarizes the details of the contents of three datasets.

DiDeMo. The Distinct Describable Moments (DiDeMo) dataset [6] is one of the most diverse datasets for the temporal localization of moments in videos given natural language descriptions. The videos are collected from Flickr and each video is trimmed to a maximum of 30 seconds. The videos in the dataset are divided into 5-second segments to reduce the complexity of annotation. The dataset is split into training, validation, and test sets containing 8,395, 1,065, and 1,004 videos respectively. The dataset contains a total of 26,892 moment-sentence pairs and each natural language description is temporally grounded by multiple annotators.

Charades-STA. Charades-STA dataset is introduced in [5] to address the task of temporal localization of moments in untrimmed videos. The dataset contains a total of $6{,}670$ videos with $16{,}128$ moment-sentence pairs. We have used the published split of videos during training and testing (train-$5{,}336$, test-$1{,}334$). As a result, the training set and the testing set contain $12{,}408$ and $3{,}720$ moment-sentence pairs respectively. This dataset is originally built on the Charades [67] activity dataset with temporal activity annotation and video-level description. Authors in [5] adopted a keyword matching strategy to generate clip-level sentence annotation.

ActivityNet Captions. ActivityNet Captions [66] dataset, which is proposed for dense video captioning task, is built on the ActivityNet dataset [68]. It consists of YouTube video footage where each video contains at least two ground truth segments and each segment is paired with one ground truth caption [11]. This dataset contains around 20k videos which are split into training, validation, and testing set. We use the published splits over videos (train set – $10{,}009$ videos, validation set – $4{,}917$ videos), where the evaluation is done on the validation set. Videos are typically longer in length than DiDeMo and Charades-STA datasets.

We use the standard evaluation criteria adopted by various previous temporal moment localization works [5, 13, 12]. These works use $R@k,IoU{=}m$ metric, which reports the percentage of cases where at least one of the top-$k$ results have Intersection-over-Union $(IoU)$ larger than $m$ [5]. For a sentence query, $R@k,IoU{=}m$ reflects if one of the top-$k$ retrieved moments has Intersection-over-Union with the ground truth moment larger than the specified threshold $m$. So, for each query sentence, $R@k,IoU{=}m$ is either 1 or 0. As this metric is associated with a queried sentence, we compute it for all the sentence queries in the testing set (DiDeMo, Charades-STA) or in the validation set (ActivityNet Captions) and report the average results. We report $R@k,IoU{=}m$ over all queried sentences for $k\in\{10,100\}$ and $m\in\{0.50,0.70\}$. We also use median retrieval rank (MR) as an evaluation metric. MR computes the median of the rank of the correct moment for each query. Lower values of MR indicate good performance. We compute MR for $IoU\in\{0.50,0.70\}$. Note that DiDeMo dataset provides multiple temporal annotations for each sentence. We consider a detection is correct if it overlaps with a minimum of two temporal annotations with a specified $IoU$.

For DiDeMo dataset, we use ResNet-152 features [69], where pool5 features are extracted at $5$ fps over the video frames. Then the features are max-pooled over $2.5s$ clips. Also, we extract optical flow features from the penultimate layer from a competitive activity recognition model [70]. We use Kinetics pretrained I3D network [71] to extract per second clip features for the Charades-STA dataset. For ActivityNet Captions dataset, we use extracted C3D features [72]. We set the number of input clips of a video, $l=12$ for DiDeMo dataset, $l=64$ for Charades-STA dataset, and $l=512$ for ActivityNet Captions dataset. Here, per unit length of input video represents non-overlapping clip of $2.5s$ duration for DiDeMo and non-overlapping clip of $1s$ duration for both Charades-STA and ActivityNet Captions dataset. For DiDeMo dataset, we use a fully connected layer followed by max-pool to generate representations with temporal dimension $6$ for each video. Then we use $6$ temporal convolutional layers to generate representations with temporal dimensions of $\{6,5,4,3,2,1\}$ resulting in representations for $21$ candidate moments. Similarly for Charades-STA, we use a fully connected layer followed by max-pool to generate representations with temporal dimension $32$ for each video. Then we use $6$ temporal convolutional layers with the temporal dimension of $\{32,16,8,4,2,1\}$ where we use the $31$ candidate moment representations from the last $5$ layers. Additionally, we use a branch temporal convolutional layer to generate representations of $30$ overlapping candidate moments, each with $6s$ duration and $2s$ stride. Combining these, we consider $61$ candidate moments for each video of Charades-STA dataset. For ActivityNet Captions dataset, we use a feedforward network followed by $10$ convolutional layers to generate representations with temporal dimension of $\{512,256,128,64,32,16,8,4,2,1\}$, resulting in $1023$ candidate moment representations. Table II illustrates the implementation details for video processing for all three datasets. we consider sentences with maximum of 15 words in length. If a sentence contains more than 15 words, the tailing words are truncated.

The proposed network is implemented in TensorFlow and trained using a single RTX 2080 GPU. To train the HMAN network, we use mini-batches containing $64$ video-sentence pairs for DiDeMo and Charades-STA and $32$ video-sentence pairs for ActivityNet Captions. We use the learning rate with exponential decay initializing from $10^{-3}$ for all three datasets. ADAM optimizer is used to train the network. We use 0.9 as the exponential decay rate for the first moment estimates and 0.999 as the exponential decay rate for the second-moment estimates. We set $\alpha_{intra}$ and $\alpha_{video}$ to 0.05 and 0.20, respectively for all three datasets. $\lambda_{1}$ is empirically set to $5$, $1$, and $1.5$, respectively for DiDeMo, Charades-STA, and ActivityNet Captions. $\alpha$ is set to $5\times 10^{-5}$ for all three datasets.

We conduct the following experiments to evaluate the performance of our proposed method:

• •

  Comparison of the performance of proposed HMAN for the task of temporal localization of moments in video corpus with different baseline approaches and a concurrent work.

• •

  Evaluation of the effectiveness of utilizing hierarchical moment encoder module.

• •

  Investigation of the impact of learning joint embedding space by utilizing different components of the loss function (learning for intra-video moments ($\mathcal{L}^{intra}$) and learning for videos ($\mathcal{L}^{video}$).

• •

  Evaluation of the effectiveness of utilizing global semantics to identify the correct video.

• •

  Analyzing the effectiveness of video relevance computation (Eqn. 6) for the task of temporal localization of moments in a video corpus.

• •

  Studying the performance of proposed HMAN for different visual features.

• •

  Performance comparison of HMAN with decreasing number of test set moment-sentence pairs.

• •

  Evaluation of the run time efficiency.

• •

  Analysis of the $\lambda_{1}$ parameter sensitivity.

Temporal Localization of Moments in Video Corpus. Table III, Table IV, and Table V illustrate the quantitative performance of our framework for the task of temporal localization of moments in the video corpus. The evaluation setup considers $IoU\in\{0.50,0.70\}$ and for each $IoU$ threshold, we report $R@10$, $R@100$ and MR. For a query sentence, the task requires to search over all the videos and retrieve the relevant moment. For example, in the DiDeMo dataset, the test set consists of $1{,}004$ videos totaling $4{,}016$ moment-sentence pairs. Again, we consider $21$ candidate moments for each video. So, for each query sentence, we need to search over $21\times 1{,}004=21{,}084$ moment instances and retrieve the correct moment. This is itself a difficult task and the addition of ambiguity of similar kinds of activities in different videos makes the problem even harder. We compare the proposed method with the following baselines:

• •

  Moment Frequency Prior: We use Moment Frequency Prior baseline from [6], which selects moments that correspond to gifs most frequently described by the annotators.

• •

  MCN: The Moment Context Network [6] for temporal localization of moments in a given video is scaled up to search moment from the entire video corpus.

• •

  SCDM: The state-of-the-art Semantic Conditioned Dynamic Modulation (SCDM) network [13] for temporal localization of moments in a video is scaled up to search over the entire video corpus.

• •

  VSE++ + SCDM: We use joint embedding based retrieval approach (VSE++ [63]) combined with SCDM as a baseline. In this setup, the framework first retrieves a few relevant videos (top $5\%$) and then localize moments on those retrieved videos using SCDM approach.

• •

  CAL: We compare with Clip Alignment of Language [23]. It is a concurrent work that addresses the task of localizing moments in a video corpus by aligning clip representation with language representation in the embedding space.

We observe that MCN and CAL perform better than the state-of-the-art SCDM approach in DiDeMo dataset but perform poorly compared to the SCDM approach in Charades-STA dataset. This is due to the fact that the video contents and language queries differ a lot among different datasets [12]. MCN and CAL learn to distinguish both intra-video moments and inter-video moments locally while SCDM only learns to distinguish intra-video moments. As DiDeMo dataset contains diverse videos of different concepts and relatively less number of candidate moments, learning to differentiate inter-video moments locally improves performance significantly. However, learning to differentiate inter-video moments locally does not have much impact on Charades-STA dataset. This also indicates the importance of distinguishing moments from different videos based on global semantics for a diverse set of video datasets. We also observe that in some of the cases, VSE++ + SCDM scores drop compared to the SCDM approach. Since the performance of VSE++ + SCDM depends on retrieving correct video, the localization performance drops if the retrieval approach fails to retrieve correct videos with higher accuracy.

For HMAN, we report the performance for both sum-margin and max-margin based triplet loss setups. Additionally, for DiDeMo and Charades-STA dataset, we report the performance of HMAN for two different loss calculation setups: TripSiam [64] and DSLT [65]. In Table III, Compared to baseline approaches, the performance of our proposed approach is better for all metrics and outperforms other approaches with a maximum of $11.88\%$ absolute improvement in DiDeMo dataset. We observe that the sum-margin based triplet loss setup outperforms the max-margin setup, while both of these setups perform better than other baselines in DiDeMo dataset. For a fair comparison with CAL and MCN, we report the performance of HMAN with the ResNet-152 feature computed from RGB frames only. This setup also outperforms CAL and MCN. We also conduct experiment incorporating temporal end point feature in HMAN for DiDeMo dataset. It results in $\sim 0.5\%-1\%$ improvement over HMAN (sum-margin) in $R@k$ metrics. It indicates the bias in the dataset where different types of events are correlated with different time frames of the video. In Table IV, for the Charades-STA dataset, the performance of HMAN is better for all metrics and the max-margin based triplet loss setup outperforms other baseline approaches with a maximum of $3.4\%$ absolute improvement. In Table V, for ActivityNet Captions dataset, the HMAN max-margin setup outperforms other baselines with a maximum of $3.17\%$ absolute improvement. We do not compute SCDM and VSE++ + SCDM baselines for ActivityNet Captions dataset. Moment representations in SCDM and VSE++ + SCDM approaches are conditioned on sentence queries. For each query sentence, we need to compute moment representations from all the videos, resulting in $O(n^{2})$ complexity. So testing on a set of $34{,}160$ query sentences and $4{,}917\times 1{,}023=5{,}030{,}091$ moment representations is impractical using these approaches.

TripSiam [64] and DSLT [65] are two different variants of triplet loss which are used in object tracking. TripSiam defines a matching probability for each triplet to measure the possibility of assigning the positive instance to exemplar compared with the negative instance and tries to maximize the joint probability among all triplets during training. DSLT [65] utilizes modulating function to minimize the contribution of easy samples in the total loss. While both setups perform better than baseline approaches, we observe that there is a significant improvement in median retrieval rank (MR). This indicates that even if TripSiam and DSLT can not retrieve the correct moment, they are robust in terms of the semantic association between moments and sentences.

Effectiveness of Hierarchical Moment Encoder. HMAN utilizes stacked temporal convolutional layers in a hierarchical structure to represent video moments. We conduct experiments to analyze the effects of using the hierarchical moment encoder module in our proposed model. We consider two setups, i) w/ TCN: the hierarchical moment encoder module built using temporal convolutional network is present in the model and ii) w/o TCN: the hierarchical moment encoder module is replaced with a simple feedforward network to project the candidate moment representations in the joint embedding space. We consider both sum-margin based and max-margin based triplet loss to train the networks. Table VI illustrates the effect of utilizing hierarchical moment encoder module. We observe that for both the learning approaches and for both datasets, there is a significant improvement in performance when the hierarchical moment encoder module is used. For example, in DiDeMo dataset, we observe $\sim 14\%$ (sum-margin) and $\sim 8\%$ (max-margin) absolute improvement in performance for $R@100,IoU=0.50$.

Ablation Study of Learning Joint Embedding Space. We conduct experiments to analyze the impact of different components of the loss function to learn the joint embedding space for our targeted task in DiDeMo dataset and reported the results in Table VII. We use three setups to learn the joint embedding space:

• •

  HMAN (intra): Only uses $\mathcal{L}^{intra}$. So the network only learns to distinguish intra-video moments.

• •

  HMAN (video): Only uses $\mathcal{L}^{video}$. So the network only learns to disntinguish moments from different videos based on global semantics.

• •

  HMAN (proposed): Our proposed approach, combination of $\mathcal{L}^{intra}$ and $\mathcal{L}^{video}$.

In Table VII, we observe that the performance of HMAN is poor for both the case of HMAN (intra) and HMAN (video). Performance of HMAN (intra) is better compared to HMAN (video) in Table VII when higher $IoU$ threshold requirement is considered ($R@k,IoU=0.7$). This indicates that HMAN (intra) learns to better identify temporal boundaries in a video compared to HMAN (video), while HMAN (video) is better at distinguishing moments from different videos compared to HMAN (intra). However, when we combine both of these criteria, there is a significant performance boost as the model is able to effectively learn to identify both the correct video and the temporal boundary. All the results in Table VII are reported for sum-margin based triplet loss setup.

Effectiveness of Utilizing Global Semantics. Our proposed learning objective utilizes global semantics to distinguish moments from different videos. To do so, we learn to align corresponding video-sentence pairs, where the video-sentence relevance $R(v,s)$ in the embedding space is computed in terms of moment-sentence similarity $S(m,s)$. So we use this video-sentence relevance score $R(v,s)$ to analyse the models performance to identify or retrieve the correct video given a text query and report the results in Table VIII. We use the standard evaluation criteria $R@k$ for video retrieval task and report $R@10$, $R@100$, and $R@200$ scores for DiDeMo and Charades-STA dataset. Here, $R@K$ calculates the percentage of query sentences for which the correct video is found in the top-K retrieved videos to the query sentence. In DiDeMo test set, there are $1{,}004$ videos with $4{,}016$ moment-sentence pairs ($\sim$ $4$ sentences per video) and in Charades-STA testset, there are $1{,}334$ videos with $3{,}720$ moment-sentence pairs ($\sim$ 2.8 sentences per video). Due to the one-to-many correspondences, we consider $4{,}016$ and $3{,}720$ video-sentence pairs respectively for DiDeMo and Charades-STA datasets for the video retrieval task, where a single video can pair up with multiple sentences. Table VIII shows that both sum-margin (HMAN (sum)) and max-margin (HMAN (max)) based triplet loss setups of our proposed approach outperforms standard Visual Semantic Embedding based retrieval approach (VSE++) for the task of retrieving the correct video. Along with the consistent improvement of performance in all metrics for both datasets, We observe $\sim 40\%$ absolute improvement of retrieval performance for the metric R@200 for DiDeMo dataset. As the video-sentence relevance is computed in terms of moment-sentence similarity, this experiment validates the models capability to distinguish videos as well as moments from different videos utilizing global semantics.

Analysis of Video Relevance Computation Approach. In an untrimmed video with temporal language annotation, the segment/portion of the video mostly matches with the sentence semantics. So to compute the video-sentence relevance, it needs to focus on the moments that have higher similarity with the query sentence semantics. To tackle this issue, we compute the video-sentence relevance using LogSumExp pooling (Eqn. 6) of the moment-sentence similarity. In Table IX, we analyze the significance of the LogSumExp pooling compared to average pooling for both sum-margin and max margin based triplet loss setups. In Table IX, ‘ave’ and ‘log’ indicates average and LogSumExp pooling respectively, while ‘sum’ and ‘max’ indicates sum-margin based and max-margin based triplet loss respectively. For both DiDeMo and Charades-STA datasets, we observe that LogSumExp pooling performs better than average pooling for the target task of temporal localization of moments in video corpus in both sum-margin based and max-margin based triplet loss setups.

Ablation Study of Different Visual Features. We conduct experiments to study the performance of HMAN for different visual features for DiDeMo dataset and reported the results in Table X. We use extracted features from VGGNet [73], ResNet-152 [69] for RGB frames and optical flow features from [70]. In Table X, we observe that a combination of RGB and optical flow features perform better than using only an RGB stream. It indicates the models increased capacity due to the increase in the number of learnable weights. As a result, HMAN is suitable to work with multiple encodings of the same data together compared to the shallow embedding networks [6, 23]. We have reported the results for sum-margin based triplet loss setup.

Performance of HMAN on Decreased Number of Moment-sentence Pairs. Since HMAN searches for the correct candidate moment across all the videos in the test set during inference, the temporal localization performance of HMAN is expected to improve by decreasing the number of moment-sentence pairs in the test set. We conduct experiments on DiDeMo dataset to evaluate the performance of HMAN (learned using sum-margin based triplet loss) on the decreased number of moment-sentence pairs in the test phase. We consider four setups: HMAN (100%): Model searches over the full test set during inference, HMAN (50%): Model searches over each 50% of the test set separately and take the average of the scores, HMAN (25%): Model searches over each 25% of test set separately and take the average of the scores, HMAN (10%): Model searches over each 10% of test set separately and take the average of the scores. Table XI illustrates the performance for all four setups. We observe that with decreased number of test set moment-sentence pairs, the performance of HMAN improves.

Evaluation of Run Time Efficiency. We conduct experiments on the Charades-STA dataset to compare the run time of HMAN with the sliding window-based approaches. The differences in the sliding-based approach compared to the setup of HMAN is that: i) the moment encoder module with temporal convolutional network of HMAN is replaced by a simple single layer feedforward network, ii) instead of generating candidate moment representations directly from the video, we slide over the video to extract features of different temporal durations, then use extracted features to generate candidate moment representations. Table XII illustrates that for both training case and inference case, the sliding-based approach takes longer than HMAN per epoch, even though the network is much smaller in the sliding-based approach compared to HMAN. For a fair comparison, we keep the number of candidate moments the same, and similar computations (apart from hierarchical moment encoder module replaced by single layer feed forward network) are done for both the approaches. We have computed the run time for five epochs and reported the average results. Here, the inference time is higher due to the added requirement of computing the cosine distance between each text query and all the candidate moment representations.

$\lambda_{1}$ Parameter Sensitivity Analysis. In our framework, $\lambda_{1}$ balances the contribution of $\mathcal{L}^{intra}$ and $\mathcal{L}^{video}$ for both sum-margin and max-margin case. We choose the value of $\lambda_{1}$ empirically. We conduct an experiment to check the sensitivity of HMAN performance based on a set of values for $\lambda_{1}$ in the DiDeMo dataset where $\lambda_{1}\in\{3,4,5,6,7\}$. We observe that for this set of values of $\lambda_{1}$, the performance is stable.

t-SNE Visualization. We provide t-SNE visualization of embedding representations of text query and candidate moments in Fig. 5. For a text query, we consider embedding representation of the text query, representations of candidate moments from the correct video, and representations of candidate moments from randomly picked nine other videos and visualize the distribution of representations. In Fig. 5, different color represents different videos. Each video has $21$ candidate moments. We keep the color of the text query representation the same as the color of candidate moments representation from the correct video and use separate markers for correct candidate moment and text query representation. We observe that representations of the text query and the correct candidate moment coincide. Also, the representations of candidate moments from the same video are clustered together.

Example Illustration. In Figure 6, we illustrate some qualitative results for our proposed approach. The two examples in the top row are for the DiDeMo dataset and the two examples in the bottom row are for the Charades-STA dataset. For each query sentence, we demonstrate the examples where the network is able to retrieve the correct moment as the rank-1 from the test set videos. We also display rank-2 and rank-3 moments retrieved by the model for each query sentence. Figure 6(a) shows that for the query ‘The baby falls down’, the model was able to retrieve the correct moment with the highest matching. However, the interesting fact lies in the retrieved rank-2 and rank-3 moments. For the query ‘The baby falls down’, the retrieved rank-2 and rank-3 moments also contain activity of a baby, including a baby falling down. Similar results are observed for other examples for both datasets. For example, in Figure 6(b), for the query sentence ‘A person opens the door’, the model was able to retrieve the correct moment with the highest matching. However, all top-3 ranked moments contain activity related to a door. In the rank-2 moment, a person is opening a door and in the rank-3 moment, a person is fixing a door. Similarly, the top retrieved moments for a query of a dog running and hiding contain activities of a dog (Figure 6(b)) and top retrieved moments for a query of a person standing and sneezing contain standing activity and sneezing activity (Figure 6(d)). These results indicate the model’s capability of retrieving moments with similar semantic concepts from the corpus of videos.