Learning Temporal Dynamics from Cycles in Narrated Video

Let $V_{t_{0}:t_{1}}$ and $T_{t_{0}:t_{1}}$ be sequences of video and text, respectively, drawn from some temporal interval $[t_{0},t_{1}]$. These sequences can be discretized into frames $\{V_{i}\}_{i=1}^{N_{V}}$ and utterances $\{T_{i}\}_{i=1}^{N_{T}}$, where $N_{V},N_{T}$ are the number of instances the sequence is split into. We refer to each instance as a node, which allows viewing the training goal as learning a cyclic path through a graph, as depicted in Figure 2.

In order to differentiate through the cycle generation process, let $uv$ be an edge in the graph shown in Figure 2. We implement edges as soft retrievals of $v$ given $u$, as shown in Figure 3. This soft retrieval operation can be viewed as an application of the well-known attention mechanism [52].

We start by running all visual and textual nodes through embedding networks $\Phi_{T}$ and $\Phi_{V}$ initialized with random weights. We use the architecture from [36] for embedding text nodes and a ResNet-18 [17] for visual nodes. This operation yields series of embeddings $\{z_{V_{i}}\}_{i=1}^{N_{V}}$ and $\{z_{T_{i}}\}_{i=1}^{N_{T}}$.

We then compute the cycle edges, where each edge is an instance of soft attention as described above. The attention operation $\mathbf{Attn}(Q,K,V)$ accepts sets of query, key, and value vectors $Q\in\mathbb{R}^{N_{Q}\times d},K,V\in\mathbb{R}^{N_{K}\times d}$ and returns a set of new values $Z\in\mathbb{R}^{N_{Q}\times d}$, computed (with $\tau$-temperature softmax along the second dimension) as

$$Z=\mathbf{Attn}\left(Q,K,V\right)=\text{Softmax}\left(\frac{QK^{T}}{\tau}\right)V.$$

(1)

To cycle through narrated video, we first select a modality $M\in\{T,V\}$ and a start node $M_{\alpha}\in M_{t_{0}:t_{1}}$ (we describe the process for selecting $M_{\alpha}$ in Section 3.3). We find the representation of the corresponding node in the other modality $M^{\prime}$ with a cross-modal attention edge:

	$$z_{M^{\prime}_{\alpha}}=\mathbf{Attn}\left(\pi_{M_{\alpha}},\{\pi_{M^{\prime}_{i}}\}_{i=1}^{N_{M^{\prime}}},\{z_{M^{\prime}_{i}}\}_{i=1}^{N_{M^{\prime}}}\right).$$		(2a)
We learn to project representations $z$ into a shared semantic space, using a modality-specific projector $\Pi_{M}:\mathbb{R}^{d}\to\mathbb{R}^{d}$ that outputs vectors $\pi$. For notational convenience, we denote by $\mathbf{Attn}_{n_{0}:n_{1}}$ the same attention operation which considers keys (and values) at indices $\{n_{0},\ldots,n_{1}\}$. In this notation we can rewrite the above as:
	$$z_{M^{\prime}_{\alpha}}=\mathbf{Attn}_{1:N_{M^{\prime}}}\left(\pi_{M_{\alpha}},\{\pi_{M^{\prime}_{i}}\},\{z_{M^{\prime}_{i}}\}\right).$$		(2b)

The representations from both modalities are concatenated and run through a multi-layer perceptron $f:\mathbb{R}^{2d}\to\mathbb{R}^{d}$. This operation yields $z_{\alpha}=f({z}_{M_{\alpha}},z_{M^{\prime}_{\alpha}})$, embedding the joint information back into the shared semantic space. This choice also allows us to train our temporal edges without cross-modal information and accept input from only one modality with some probability $p_{\text{unimodal}}$, since $\Phi_{V}$ and $\Phi_{T}$ also map to $z$-space, in $\mathbb{R}^{d}$.

Our model must now go from this multi-modal state representation $z_{\alpha}$ to a future state representation $z_{\beta}$. First, we predict an estimated representation of the future in projection ($\pi$) space, with an MLP $g_{\text{fwd}}$. We then retrieve the node in modality $M$ corresponding to this future state with a forward-in-time attention edge:

$$z_{M_{\beta}}=\mathbf{Attn}_{1:N_{M}}\left(g_{\text{fwd}}(z_{\alpha}),\{\pi_{M_{i}}\},\{z_{M_{i}}\}\right).$$

(3)

It is important to note that attention is order-invariant in $K$ and $V$, i.e. shuffling rows of $K$ and $V$ yields the same output $Z$, since the individual matrix-row multiplications are agnostic to row index. This means that, importantly, the model is not given temporal information about input nodes, which could be used as a shortcut in learning²²2E.g., the model could cycle by selecting the node it knows is at $t+1$ or $t-1$.. We then retrieve the corresponding node in $M^{\prime}$, $z_{M^{\prime}_{\beta}}$, with another cross-modal edge, projecting queries and keys into $\pi$-space:

$$z_{M^{\prime}_{\beta}}=\mathbf{Attn}_{1:N_{M^{\prime}}}\left(\pi_{M_{\beta}},\{\pi_{M^{\prime}_{i}}\},\{z_{M^{\prime}_{i}}\}\right).$$

(4)

As before, these vectors are combined to yield a future state representation $z_{\beta}=f(z_{M_{\beta}},z_{M^{\prime}_{\beta}})$.

This process is repeated to predict backward in time. We compute $g_{\text{back}}(z_{\beta})$, where $g_{\text{back}}$ shares its first few layers with $g_{\text{fwd}}$ to allow learning features useful for dynamics in either direction (see Section 3.5 for more details). To close the cycle, we compute the normalized similarity scores between $g_{\text{back}}(z_{\beta})$ and the $\pi$-space nodes in $M$:

$$\mathbf{p}=\mathbf{Attn}_{1:N_{M}}\left(g_{\text{back}}(z_{\beta}),\{\pi_{M_{i}}\},\{1\}\right).$$

(5)

We train our system with the negative log likelihood loss on the score vector cycling back to the location of $M_{\alpha}$, which we denote $i_{\alpha}$:

$$\mathcal{L}_{\text{cycle}}=-\log\mathbf{p}^{(i_{\alpha})}.$$

(6)

A key component of our cycle model is the ability to find correspondences between vision and language. Eqs. 2b and 4 crucially rely on this ability in order to incorporate multi-modal information into temporal edges. Recent work has demonstrated remarkable progress in training models on massive datasets for this cross-modal retrieval task: given a moment at time $t$ in one modality of a video - $M_{t}$ - find the matching moment in the other modality $M^{\prime}$.

We build on the approach presented in [36] which uses a contrastive loss to train representations of vision and language, where temporally co-occurring information is considered ground truth ($M_{t}$ should retrieve $M^{\prime}_{t}$), and other vision-language pairs are used as negatives. To handle the common misalignment intrinsic to real-world video, [36] allows for representations within $k$ nodes of the ground truth node $M^{\prime}_{t}$ to be considered as positives. We adopt this approach to learn cross-modal correspondence, training it for finer-grained discrimination among a set of candidate moments drawn from the same video as opposed to randomly across the entire dataset. We denote the loss used to train cross-modal correspondence $\mathcal{L}_{\text{cross-modal}}$. For the full cross-modal formulation, please see Supplementary Material.

Our model will be unable to learn semantic transitions between states if the initial input node depicts noisy or unclear data. This is especially probable when training on unconstrained, real-world video datasets. Therefore, instead of randomly sampling start nodes, we sample from a distribution defined by a “concreteness” score $s_{i}$. We calculate this score for each node $M_{i}\in M$ as the highest cross-modal similarity between $M_{i}$ and some node $M^{\prime}_{j}$ in the other modality. Intuitively, this score captures concreteness since frames and utterances that align strongly tend to contain objects or actions which are salient in both modalities:

$$s_{i}=\max_{j}\left(\pi_{M_{i}}\cdot\pi_{M^{\prime}_{j}}\right)$$

(7)

We run the above scores through a softmax with $\tau=0.1$, yielding a distribution from which we sample $M_{\alpha}$.

Training on the above formulation of $\mathcal{L}_{\text{cycle}}$ in practice may lead to fast collapse to a simple “looping in place” solution, where temporal edges always point to the current node. We propose two strategies to prevent this collapse:

In practice, we combine both the above strategies for the strongest results.

We combine $\mathcal{L}_{\text{cycle}}$, $\mathcal{L}_{\text{cross-modal}}$, and $\mathcal{L}_{\text{sim}}$ in our final loss:

$$\mathcal{L}=\lambda_{1}\mathcal{L}_{\text{cycle}}+\lambda_{2}\mathcal{L}_{\text{cross-modal}}+\lambda_{3}\mathcal{L}_{\text{sim}}.$$

(9)

We embed images into $\mathbb{R}^{d}$ ($d=512$) using a ResNet-18 [17], and embed text using a word embedding matrix followed an MLP and global pooling, as in [36].

We implement all modules ($\Pi_{M}$, $f$, $g_{\text{fwd}}$, $g_{\text{back}}$) as MLPs, where each layer is followed by ReLU and a LayerNorm except for the final layer, which is followed by $l_{2}$ normalization if its output is in $\pi$-space. $\Pi_{M}$ and $f$ are one-layer MLPs, and $g_{\text{fwd}},g_{\text{back}}$ are four-layer MLPs, with weights of the first two layers shared. We randomly sample batches of video segments of maximum duration $t_{1}-t_{0}=64\text{sec}$. The sparsity at which data is sampled affects the time elapsed by input videos in a batch as well as the granularity of visual information provided to the model. Denser data is less likely to miss key moments, but more likely to contain redundant information. We therefore train models on various image sampling frame rates $r\in\{0.25\text{fps},0.5\text{fps},1\text{fps}\}$.

Because good cross-modal correspondence is necessary to learn strong, semantic cycles, we initialize $\lambda_{2}=1$ and exponentially increase $\lambda_{1}$ from some small value $\epsilon$ up to $1$, across 30 epochs. We peg $\lambda_{3}=3\lambda_{1}$ when using the similarity loss. For further details on training and architecture, please see Supplementary Material.

We train our model on unconstrained real-world video data. Specifically, we use a subset of the HowTo100M dataset [38], which contains around 1.23 million videos and their automatically extracted audio transcripts. Videos in this dataset are roughly categorized by subject area, and we use only the videos categorized “Recipe”, around a quarter of the dataset. We build a train-validation-test split such that of 338,033 total recipe videos, 80% are in train, 15% in validation, and 5% in test. Recipe videos are rich in complex objects, actions, and state transitions, and the subset allows us to train models faster.

For more controlled testing, we use the CrossTask dataset [70], which contains similar videos along with task-specific annotations. Videos are associated with tasks (e.g., “making pancakes”), where each task has a predefined sequence of high-level subtasks with rich long-term temporal inter-dependencies (e.g., [“pour flour into bowl”, “crack egg into bowl”, …, “drizzle maple syrup”]). Video segments that depict one of these subtasks are annotated as such.

Central to our formulation is the model’s ability to learn dynamic predictions of the future and past that undo each other, as well as finding strong cross-modal correspondences. Thus, we begin by evaluating how well different model variants are able to solve our self-supervised objective on the Recipes test set. We ablate various design choices, including multi-modal information usage, cycle edge order, and temporal constraints on edges.

We compute this probability efficiently for all $n^{2}$ pairs in long clips of continuously sampled video ($n\approx 600$). We then look at the top temporal transitions discovered by the model in each video. We show results on the Recipes test set in Figure 5. The top transitions show clear state transitions such as adding chocolate chips to dough, segmenting an orange, and baking a loaf. These predictions could not be made by a model trained to predict a fixed future, since they occur at varied temporal offsets.

For a video $V$ belonging to task $\tau$ (with $N_{\tau}$ predefined subtasks), let $v\in V$ be a clip with subtask label $T_{\tau,i}$ ($i^{\text{th}}$ in the predefined sequence). We would like to predict future actions $T_{\text{future}}=\{T_{\tau,j}\}_{j=i+1}^{N_{\tau}}$ from $v$. For example, given a short clip of eggs being added to a bowl with flour, the model should assign high likelihoods to subtasks such as “mix batter” and low likelihoods to “crack eggs” or “season steak”. Formally, we define a future likelihood score given a video segment $v$ and candidate future subtask $T_{j}$. We first sample frames from the video segment and compute their average embedding $\bar{z}_{v}$. The likelihood score uses our learned representations and predictive model to define a score $s_{v\to j}=g_{\text{fwd}}(\bar{z}_{v})\cdot\pi_{T_{j}}$. We compute likelihood scores for all subtask descriptions in the CrossTask validation set, and consider the model’s prediction correct if any of the future actions in $T_{\text{future}}$ are predicted, since not all future subtasks are necessarily related to the given visual state.

Table 2 shows recall and percentile rank statistics for this task. We compare our model to [16, 21, 55], replacing $g_{\text{fwd}}$ with each method’s predictive model. Since [16] is vision-only, we set $\pi_{T_{j}}$ to the average visual representation of all video segments with a given subtask label $T_{j}$. We also define a cross-modal similarity score $s_{v\to j}=\bar{\pi}_{v}\cdot\pi_{T_{j}}$ as a strong baseline, taking advantage of contextual similarities in video and text. Our model outperforms all baselines and self-supervised state of the art on detecting the temporal relationships between visual states and future actions.

Given $P(u\to v)$ scores computed by Eq. 12, we induce a fully-connected directed graph with sampled frames as nodes and edge weights given by $\texttt{weight}(\vec{uv})\equiv-\log P(u\to v)$. Adding a special null node connected to all other nodes with edge weight 0 allows running this graph through an off-the-shelf traveling salesperson problem (TSP) solver³³3https://pypi.org/project/elkai/. The optimal TSP solution then represents the lowest-cost (ordered) path through all video clips, effectively unshuffling the input.

We run this experiment on CrossTask, where videos are annotated with ordered steps and their associated temporal segments. We treat each segment as a node by computing its average visual representation, as before. We then use these representations to find $P(u\to v)$ scores between labeled segments and solve an optimal path. We run this experiment both in vision only (by passing projected visual representations directly into $g$ in Eq. 12) as well as with ground truth vision-text pairings, and show results in Table 3. We show example predicted orderings in Figure 6. Again, we can replace $g_{\text{fwd}}$ with the future prediction model in other methods and run the same algorithm. Our model outperforms previous work on all evaluation metrics.

We consider the more challenging task of predicting the temporal relationship between two far-apart input frames – which one comes first? For frames which depict unrelated moments, this task is perhaps near-impossible, even for humans. But if frames show semantically related states, the direction of likely transition provides a useful signal for solving the arrow-of-time task.

We train linear classifiers on top of frozen features as well as a full network from scratch to solve the arrow of time task on randomly shuffled pairs of frames. We sample pairs of frames using our learned predictive model by selecting the highest-probability futures of start frames selected with the concreteness prior (Eq. 7). We demonstrate in Table 4 that the temporal ordering of frames mined by our model is much more classifiable than that of frames sampled using predictive models in previous work. Further, our learned features are much more able to classify a given pair of frames, since they must capture temporal information in training. This confirms that a strong understanding of dynamics that emerges from the cycle consistency task.

Dissecting neuron activations: To solve our cycle consistency problem, our model must be able to attend to relevant parts of the visual input which inform predictions of the future and past, and lead to strong cross-modal correspondences. We probe our learned representation for discovered objects and actions and visualize this in Figure 7.

First, we compute the correlation between the activation of each neuron in the visual representation given by $\Phi_{V}$ and the presence of words in corresponding text, using the Spearman rank correlation test run for all $d\cdot|\mathcal{V}|$ neuron-word pairs, where $\mathcal{V}$ is the vocabulary and $d$ the dimension of the embedding. We select neuron-word pairs with the highest correlation score and visualize examples that maximally activate these neurons, along with a GradCAM [47] localization of image regions that caused the activation. We generate the visualization by randomly selecting among images that yield the top 0.01% of neuron activation values. Note that the images in Figure 7 have selected without considering their corresponding textual information, and are filtered only by how much they excite the neuron in question. Our model appears to learn localization of common actions and objects in the training data, despite training without any supervision.

Our model relies on cross-modal correspondence to use information from both modalities, but this can be learned in separate pre-training stages, which allows $\lambda_{2}=0$. This decreases cycle accuracy by 7%. Co-training with this loss term (and annealing its weight, Sup. Mat. L152-156) prevents catastrophic forgetting of cross-modal correspondences, which are used to learn cycles. The visual similarity loss is optional, since the constrained temporal attention functions well on its own, so we can set $\lambda_{3}=0$, slightly impacting performance (e.g., -1% on cycle rank). The model does not learn temporal dynamics at all with $\lambda_{1}=0$; this can be thought of as the cross-modal baseline.