Reinforcement Learning for Weakly Supervised Temporal Grounding of Natural Language in Untrimmed Videos

Following the common-used formulation (Gao et al., 2017), we represent a video $V$ by $N$ clips $\{V_{1},V_{2},...,V_{N}\}$, each clip corresponds to a small chunk of sequential frames. Taking $V$ and a text query $T$ as inputs, the studied task aims to output a video segment $[j,k]$ ($j$ and $k$ indicate the start and end clip indices respectively) that semantically matches the query description. Our work focuses on the weakly supervised setting of this task. Specifically, only a set of $V$-$T$ pairs are provided but the video segment annotation for each pair is not available. Inspired by the observation that humans usually locate interest events during a long video with a heuristic search strategy, we propose to formulate this task as a Markov Decision Process. A Boundary Adaptive Refinement (BAR) framework is thus designed: starting from an initial segment, the reinforcement learning technique is utilized to refine its temporal boundary progressively. The overall architecture of the proposed BAR framework is depicted in Figure 2. As illustrated, this modular framework employs a context-aware feature extractor to encode the environment state into cross-modal contextual concepts. The cross-modal alignment evaluator is crafted to provide a tailor-designed reward and the termination signal for the iterative refinement process. An adaptive action planner is designed to reason the direction and amplitude of the action from contextualized observation adaptively, instead of shifting a fixed amplitude every step (He et al., 2019). The details of these modularized components will be described in the following sections.

The context-aware feature extractor takes a video-query pair ($V$-$T$) from the external environment and encodes it into the context-aware cross-modal concepts. Each word in query $T$ is firstly encoded using GloVe (Pennington et al., 2014) embeddings and then fed into the GRU (Chung et al., 2014) to capture long-range dependencies. The summarized query representation $\bm{\mathrm{E}}$ is obtained from the last hidden state of the GRU. A pre-trained video feature extractor (C3D (Tran et al., 2015) or TSN (Wang et al., 2016)) is used to extract the clip-level feature for each video clip. A video segment is represented as a set of clip features, i.e., $\bm{\mathrm{F}}=\{\bm{\mathrm{F_{1}}};...;\bm{\mathrm{F_{i}}};...;\bm{\mathrm{F_{M}}}\}\in\mathbb{R}^{d_{k}\times M}$. $\bm{\mathrm{F_{i}}}\in\mathbb{R}^{d_{k}}$ denotes the clip-level feature for the video clip $V_{i}$ and $M$ is the number of clips in the corresponding video segment. At each time step, the updated boundary divides the whole video into three parts: left segment, current segment and right segment. And we collect all clip-level features within the corresponding boundary into a set to obtain the three corresponding segment-level features. Rather than directly taking the current segment’s feature as independent inputs (He et al., 2019), this extractor also leverages context-aware contextual cues derived from other segments in the video (i.e., left and right segment feature) for state encoding. Furthermore, the extractor explicitly involves the normalized boundary location $\bm{\mathrm{L_{t-1}}}$ into the encoded features to provide some notion of relative position:

where $l^{s}_{t-1}$ and $l^{e}_{t-1}$ denote the start and end clip indices of the boundary, respectively. $t=\{1,\cdots,T_{max}\}$ and $T_{max}$ indicates the maximum iterations in the refinement process.

The cross-modal alignment evaluator is designed specially to address two critical issues in our RL-based approach. On the one hand, this evaluator is crafted to assign a target-oriented reward to address the difficulty that the adaptive action planner can not directly obtain a reliable reward function without granular boundary annotations. On the other hand, this alignment evaluator manages to determine an accurate stop signal to terminate the refinement process. Given a video segment, the dimension of each clip feature is reduced to the same as the summarized query representation $\bm{\mathrm{E}}$ via a filter function $\theta$, which consists of a fully-connected layer followed by ReLU (Krizhevsky et al., 2012) and Dropout (Srivastava et al., 2014) function. $\bm{\mathrm{E}}$ is taken to create a temporal attention over all video clips, which manages to emphasize crucial video clips and weaken inessential parts. Concretely, the scaled dot-product attention mechanism (Luong et al., 2015) is utilized to obtain attention weight $a_{i}$ and the segment attention feature $\bm{\mathrm{A}}$:

where $\odot$ indicates the dot product operation between two vectors. $k$ is the dimension of $\bm{\mathrm{E}}$. Then the segment attention feature and query representation are mapped to a joint embedding space to compute the alignment score $S$:

The alignment score can be regarded as a reward estimate to provide reliable reward. Specifically, the evaluator measures the alignment score of the consecutive segment-query pairs, and assigns the corresponding reward $r_{t}$:

where $S^{c}_{t}$ denotes the alignment score of the current segment and sentence query at time step $t$. This reward function returns +1 or -1. Basically, if the next boundary has a higher alignment score than the current one, the reward $r_{t}$ of the action $a_{t}$ moving from the current window to the next one is +1, and -1 otherwise. Such binary rewards reflect more clearly which action can drive the boundary towards the ground-truth and thus facilitate the agent’s learning.

The adaptive action planner is designed to infer action sequences to refine the temporal boundary. To get a fixed-length visual representation, we utilize a mean pooling layer over feature set $\bm{\mathrm{F}}$ of the global, current, left and right segment, obtaining the pooling features $\bm{\mathrm{F^{g}}},\bm{\mathrm{f^{c}_{t-1}}},\bm{\mathrm{f^{l}_{t-1}}},\bm{\mathrm{f^{r}_{t-1}}}$ respectively. Then the cross-gated interaction method (Feng et al., 2018) is further adopted to enhance the effects of the relevant segment-query pairs. Concretely, the current pooling feature $\bm{\mathrm{f^{c}_{t-1}}}$ is gated by query representation $\bm{\mathrm{E}}$, and meanwhile the gate of $\bm{\mathrm{E}}$ depends on $\bm{\mathrm{f^{c}_{t-1}}}$:

where $\bm{\mathrm{W}^{s}}$ and $\bm{\mathrm{W}^{v}}$ are parameter matrices and $\sigma$ denotes the sigmoid function. These cross-modal features are then concatenated and fed into two cascaded fully-connected layers $\phi$ to get the state activation representation $s_{t}$:

Such contextual features encourage the planner to perform a left-right tradeoff on the video contents and infer a more accurate action. $s_{t}$ is further fed into a GRU cell to enable the agent to incorporate the memory information about the video segments that have been explored. Then the output state of the GRU is followed by two separate fully-connected layers (i.e., actor and critic) to respectively estimate a policy function $\pi(a_{t}|s_{t})$ and a value approximator $v^{\pi}(s_{t})$. A primitive action $a_{t}\in\mathcal{A}$ is sampled from the policy function $\pi(a_{t}|s_{t})$ in the training procedure. In our work, the action space $\mathcal{A}$ is composed of four primitive actions: shifting the start/end point backward/forward $N/\nu$ clips. $\nu$ is an amplitude factor that empirically sets as:

where $\lfloor\rfloor_{+}$ denotes the lower bound of a positive integer. $S^{g}$ and $S^{c}_{t}$ denotes the global and current alignment score estimated by the alignment evaluator. $\mathrm{tanh}$ is used to constrain the action amplitude to fluctuate around $N/10$ (an empirical number used in  (He et al., 2019)). $S^{g}$ plays as a baseline of the alignment degree to determine $\nu$: when $S^{c}_{t}$ is lower, $\nu$ becomes smaller and the agent markedly shifts the boundary; when $S^{c}_{t}$ becomes higher, $\nu$ is larger and the boundary is marginally refined. This adaptive setting enables the agent to determine the action amplitude based on the current observation, which is also in line with human habits.

The state-value $v^{\pi}(s_{t})$ predicted by the critic is the value estimation of the current state. Under the assumption that the critic produces the exact values, the actor is trained based on an unbiased estimation of the gradient.

Due to its efficiency, the advantage actor-critic (A2C) (Sutton and Barto, 2018) algorithm is chosen to train our adaptive action planner. Multiple instance learning algorithm with a combined ranking loss $\mathcal{L}_{rank}$ is designed to train the cross-modal alignment evaluator and context-aware feature extractor. The total loss in BAR is summarized as:

where $\mathcal{L}_{A2C}$ denotes the loss function in the A2C algorithm. $\eta$ is a trade-off factor between the two losses.

Inter-videos generally include substantially broad semantic abstractions that are hard to distinguish similar contents in a specific video. To this end, we design the intra-video ranking loss $\mathcal{L}_{intra}$ to capture more subtle concepts in the intra-video to further optimize the network. Expressly, if the score of any one of left, current and right segment-query pairs surpasses the global one during the refinement process, we assume this pair should have higher alignment score than the other two pairs:

where $S^{l}_{t}$ and $S^{r}_{t}$ are the alignment scores of the left segment-query pair and the right segment-query pair at the time step $t$, respectively. $\psi$() is a binary indicator function. If the inequality in parentheses holds, $\psi$() will output 1, otherwise 0. Specifically, when the score of a segment-query pair, say $S^{c}_{t}$, surpasses $S_{g}$, the optimization target is to increase the gap between $S^{c}_{t}$ and the other two ($S^{l}_{t}$ and $S^{r}_{t}$ ) by increasing $S^{c}_{t}$ or decreasing $S^{l}_{t}$ and $S^{r}_{t}$ . Noted that by lowering $S^{c}_{t}$ below $S_{g}$ might be another option, but this usually becomes increasingly impractical with the progress of inter-video training. In addition, when there exist more than one segment-query pairs of score surpass $S_{g}$, the optimization target of $\mathcal{L}_{intra}$ will usually guide the alignment evaluator to suppress the score of the sub-optimal matching pair(s) to be lower than $S_{g}$ and at the same time drive the action planner to adjust the boundary. Intuitively, $\mathcal{L}_{intra}$ encourages the text query to be closer to a semantically matched video moment than other possible moments from the same video, which contributes to obtaining a content-aware alignment score.

$\mathcal{L}_{intra}$ manages to i) widen the score gap between matched and unmatched segment-query pair to increase the confidence of the alignment evaluation; ii) improve the reward calculation by affecting the alignment evaluator to drive the action planner to achieve better temporal boundary adjustments. To sum up, the combined ranking loss $\mathcal{L}_{rank}$ is defined as:

where $\lambda$ is a weighting parameter to achieve a ranking loss trade-off between the intra-video and the inter-video. In the early stage of this collaborative training scheme, it is very unlikely that the score of a segment-query pair exceeds $S_{g}$ and $\mathcal{L}_{intra}$ tends to 0, hence $\mathcal{L}_{inter}$ plays a dominant role that learns to transfer the matching between video-query pair to segment-query pair. As the training progresses, $\mathcal{L}_{inter}$ converge gradually and it is more common for the score of segment-query pair to exceed $S_{g}$, $\mathcal{L}_{intra}$ begins to play a critical role.

At each time step, BAR executes an action $\hat{a_{t}}$ via greedy decoding algorithm to adaptively adjust the temporal boundary. And the cross-modal alignment evaluator computes a score $S^{c}_{t}$ to provide confidence for alignment degree and termination. Empirically, the final grounding result corresponding to the query usually occupies a reasonable and appropriate video length. Hence to penalize the video segment with abnormal lengths, we propose to update the confidence score with a Gaussian penalty function as follows:

where $\delta$ denotes the penalty factor corresponds to abnormal lengths. $\tau$ is a modulating factor that as $\tau$ increases the effect of the penalty degree is likewise decreased. The segment with the max $\hat{S^{c}_{t}}$ during testing is regarded as the final grounding result.

We leverage C3D and the TSN model to encode video representation. The initial boundary is set to $L_{0}=[N/4;3N/4]$. $N/4$ and $3N/4$ denote the start and end clip indices of the boundary respectively. $T_{max}$ is set to 12 and the size of the hidden state in GRU is 1024. The batch size is 12 and the total loss is optimized via the Adam optimizer with the learning rate of 0.001. The margin $\epsilon$ in ranking loss is 0.2. The hyper-parameters $\alpha$ and $\gamma$ is fixed to 0.1 and 0.4, receptively. The factor $\eta$ and $\lambda$ are empirically set to 1 and 0.1. The modulating factor $\tau$ is set to 0.5 by cross validation. And penalty baseline factor $\delta$ is fixed to 0.35 and 1.0 receptively on Charades-STA and ActivityNet. We use $K$ = 500 in the alternating update procedure.

We compare the proposed BAR with several state-of-the-art models based on the weakly-supervised and fully-supervised settings in Table 1. On the one hand, BAR significantly outperforms the weakly-supervised method and establishes new state-of-the-art performance on both datasets. Employing the C3D based video feature, BAR boosts the [email protected] to 27.04% and 30.73%, with an improvement of 3.46%, 1.51% compared with SCN (Lin et al., 2020) on the two datasets, receptively. Furthermore, it manages to achieve 33.98% (33.12%) in [email protected] via more powerful TSN feature. It reveals that our approach helps to better obtain accurate video segments. On the other hand, BAR even achieves better or comparable results than some fully-supervised methods. For instance, BAR outperforms QSPN (Xu et al., 2019) by 3.03% w.r.t [email protected] on the ActivityNet dataset. This is an inspiring result as it reveals that our model can get impressive results via learning from massive coarse video-level annotations, which is of great benefit to practical application.

We perform extensive ablation studies and demonstrate the effectiveness of several essential components in BAR. The experiments are conducted on the Charades-STA with the TSN feature. The results are reported in Table 2.

$\bullet$ Effectiveness of Reinforcement Learning. More accurate measurement of the factual RL contribution is to directly remove it and use the generated proposals of an off-the-shelf weakly-supervised action localization method (Mithun et al., 2019). Hence we design a variant (abbreviated as “Ours w/o RL”) to follow the above setting. We can observe that removing RL from BAR will lead to a noticeable drop in performance. For example, [email protected] declines from 33.98% to 25.89%. It reveals that the introduction of RL is fundamental and can bring more flexible and adaptable temporal proposals, this alone is an advantage that cannot be achieved with traditional two-stage frameworks, not to mention its high efficiency.

$\bullet$ Effectiveness of Tailor-designed Reward. In order to validate that a target-oriented reward is essential for this task, we design a baseline (abbreviated as “Ours w/ random reward”) that samples a random scalar value from the uniform distribution of [-1,1] as the reward for optimization. Table 2 shows that this baseline obtains an exceedingly inferior result, which is approximate to a stochastic one. It indicates that a tailor-designed reward is definitely necessary for the RL setting.

$\bullet$ Effectiveness of Boundary Initialization. The initial boundary in this paper is fixed to $L_{0}=[N/4;3N/4]$. To compare different boundary initializations, we design two baselines (denoted as “Initial boundary [N/3; 2N/3]” and “Initial boundary [N/5; 4N/5]”) that sets the initial boundary as $[N/3;2N/3]$ and $[N/5;4N/5]$, respectively. As reported in Table 2, different boundary initialization is not sensitive to the performance of the algorithm, and all can obtain competitive experimental results, which reflects the robustness of BAR.

$\bullet$ Effectiveness of Adaptive Setting. Rather than shifting a fixed distance for each action, BAR can adaptively adjust its action amplitude according to the current state. To demonstrate the superiority of this adaptive setting, we design three variants (named as “Ours w/ N/5 amplitude”, “Ours w/ N/10 amplitude” and “Ours w/ N/15 amplitude”) that the agent shifts $N/5$, $N/10$ and $N/15$ clips at each step, respectively. As summarized in Table 2, “Ours w/ N/10 amplitude” when set with fixed adjustment strategy. However, our approach with the adaptive setting manages to achieve more impressive performance, which reveals that this adaptive setting is more flexible and effective in our proposed framework.

$\bullet$ Effectiveness of Context Information. BAR additionally builds contextualized video representations for action decisions. To investigate the effectiveness of the context information, we design a baseline that removes the context concepts ($f^{l}_{t-1}$, $f^{r}_{t-1}$) from $s_{t}$ in Equation 6, abbreviated as “Ours w/o context”. From Table 2, we can see that although the model without context representation can still achieve promising results, our model with context involved gains 2.35% and 2.53% improvement w.r.t [email protected] and [email protected] respectively, which demonstrates that contextual concepts helps for obtaining more content-aware results.

$\bullet$ Effectiveness of Intra-video Ranking Loss. To verify the effectiveness of the $\mathcal{L}_{intra}$, we construct a comparison variant that merely uses $\mathcal{L}_{inter}$ to optimize the evaluator, named as “Ours w/o $\mathcal{L}_{intra}$”. Table 2 reveals that the grounding result suffers from an obvious drop without $\mathcal{L}_{intra}$. For example, [email protected] declines from 51.64% to 46.82%. Our approach with intra-video ranking loss manages to achieve more precise alignment scores and more accurate grounding results. To further demonstrate the effectiveness of the alignment score $S$ obtained by our model, we additionally calculate the correlation coefficient (CC) between $S$ and ground-truth IoU. It shows that CC can reach 0.79, which reveals the obtained $S$ is reliable enough to correctly reflect the matching degree and infer the target-oriented rewards.

$\bullet$ Analysis of Stop Signal. We did not include an ending signal in the action space (He et al., 2019) as there is no absolutely reliable and stable internal segment-query matching that can help to effectively terminate the iteration. We further introduce an alignment threshold as a stopping signal (abbreviated as“Ours w/ stop”), which led to inferior results. In order to validate the significance of the length penalty strategy, we design a baseline that directly takes the score $S^{c}_{t}$ to determine the termination time, denoted as “Ours w/o penalty”. The results indicate that this baseline suffers from performance degradation. It may be due to the fact that “Ours w/o penalty” tends to provide an excessive score when the length of the video segment is too long or too short.

Figure 3 depicts the performance curves with varying $\delta$ or $\tau$ respectively in the procedure of cross-validation. We can see that a factor $\delta$ with too large or too small value will lead to obvious performance decline, which reveals that a video with suitable length is more likely to produce impressive results. A similar changing trend can be observed with varying $\tau$. It demonstrates that an appropriate gaussian penalty encourages the model to perform better. we empirically observed that $\delta$=0.35 and $\tau$=0.5 contribute to obtaining the most promising performance in different levels of tIoU.

To further investigate the efficiency of this boundary adaptive refinement process, we compare BAR with TGA (Mithun et al., 2019) in terms of average running time and number of candidate segments. As summarized in Table 3, BAR reduces the localization time and candidate boundaries by a sizeable margin. Please notice that the boundary in BAR is equivalent to the temporal proposal number to some extent, but the “boundary” here is more flexible and adaptable. BAR merely needs to refine an initial temporal boundary progressively, which manages to avoid redundant computations and employ a time-efficient and space-efficient manner. Based on the above discussion, we can conclude that BAR is better than the previous competitive methods in both accuracy and efficiency.

We illustrate two qualitative results in Figure 4, 5 to show the whole process of how BAR obtains the described event location. We observe that our algorithm mainly performs optimization from coarse to fine. The agent will choose a larger movement adjustment at the initial stage of the iteration to quickly narrow the semantic difference between language and vision, and as the iteration progresses, the adjustment range of the movement will change rapidly to achieve local fine-tuning, this is also more consistent with humans performing cross-modal target retrieval.