Action Duration Prediction for Segment-Level Alignment of Weakly-Labeled Videos

Our method takes two inputs. The first input is a video of $T$ frames, represented by $\mathbf{x}_{1}^{T}$, which is the sequence of per-frame features. Feature extraction is a black box, our method is not concerned with how those features have been extracted from each frame. The second input is an ordered sequence $\boldsymbol{\tau}=(\tau_{1},\tau_{2},...,\tau_{M})$ of $M$ action labels, that list the sequence of all actions taking place in the video.

A partitioning of the video into $N$ consecutive segments is specified using a sequence $\mathbf{c}_{1}^{N}$ of action labels ($c_{n}$ specifies the action label for the $n$-th segment) and a sequence $\mathbf{l}_{1}^{N}$ of corresponding segment lengths ($l_{n}$ specifies the number of frames of the $n$-th segment). Given such a partition, we use notation $\pi_{n}$ for the first frame of the $n$-th segment.

Given inputs $\mathbf{x}_{1}^{T}$ and $\boldsymbol{\tau}$, the goal of our method is to identify the most likely sequence $\overline{\mathbf{c}}_{1}^{N}$ of action labels $c_{n}$ and corresponding sequence $\overline{\mathbf{l}}_{1}^{N}$ of durations $l_{n}$:

We note that $N$ (the number of segments identified by our method) can be different than $M$ (the number of action labels in input $\boldsymbol{\tau}$). This happens because our method may output the same action label for two or more consecutive segments, and all consecutive identical labels correspond to a single element of $\boldsymbol{\tau}$. We use $\Omega_{n}$ to denote the earliest segment number such that all segments from segment $\Omega_{n}$ up to and including segment $n$ have the same action label. For example, in Fig. 2, $\Omega_{4}=\Omega_{3}=\Omega_{2}=2$.

Consider a frame $\pi_{n}$, that is the starting frame of the $n$-th segment. We assume that the remaining duration of an action at frame $\pi_{n}$ depends on the type of action $c_{n}$, the elapsed duration $\mathbf{l}_{\Omega_{n}}^{n-1}$ of $c_{n}$ up to frame $\pi_{n}$, and the visual features of a window of $\alpha$ frames starting at frame $\pi_{n}$. We denote this window as $\mathbf{w}_{n}=\mathbf{x}_{\pi_{n}}^{\pi_{n}+\alpha-1}$. Also, we decompose each action label $c_{n}$ into a corresponding verb $v_{n}$ and object $o_{n}$. For example the action “take cup” can be represented by the $(take,cup)$ pair, where $take$ and $cup$ are the verb and object respectively. Working with “verbs” instead of “actions” lets us benefit from the shared information among “actions” with the same “verb”. This specifically helps in analyzing any weakly-labeled video where the frame-level pseudo ground-truth is inaccurate. Based on the above, we rewrite $p(\mathbf{c}_{1}^{N},\mathbf{l}_{1}^{N}|\mathbf{x}_{1}^{T},\boldsymbol{\tau})$ as:

We should note that, in the above equations, in the boundary case where $\Omega_{n}=n$, we define $\mathbf{l}_{\Omega_{n}}^{n-1}$ to be 0. The Duration and Action Selector Network, described next, will be used to compute the probability terms in Eq. 4. Then, using our Segment-Level Beam Search, the most likely segment alignment will be identified.

Previous work [19, 31, 22] has tried to model the duration of actions. Richard et al. [31] have used a class-dependent Poisson distribution to model action duration, assuming that the duration of an action only depends on the type of that action. In contrast, we propose a richer duration model, where the length of an action segment depends not only on the type of that action, but also on the local visual features of the video, as well as on the length of the immediately preceding segments if they had the same action label as the current segment (Eq. 4).

The proposed model allows the estimate of the remaining length of an action to change based on video features. For example, our model can potentially predict a longer remaining duration for the action “squeeze orange” if the local visual cues correspond to a person just picking up the orange, compared to a person squeezing the orange.

In our method, the range of possible durations of a given action depends on the $verb$ of that action. For example, one second could be half of a short action associated with verb “take” and only one-hundredth of a longer action associated with verb “frying”. We model this dependency by mapping time length to progress units for each verb. We denote by $\gamma_{v}$ the median length of verb $v$ across all training videos, and by $L$ the number of time duration bins. We should note that the system cannot know the true value of $\gamma_{v}$, since frame-level annotations are not part of the ground truth. Instead, our system estimates $\gamma_{v}$ based on pseudo-ground truth that is provided using an existing weakly supervised action alignment method, such as [8, 31]. Given this estimated $\gamma_{v}$, we discretize the elapsed and remaining time lengths into verb-dependent bins; i.e.  the bin width ${b}_{v}$ is calculated based on the type of each verb:

The above equation assures that the median length of a verb falls on or around the middle bin, which creates a more balanced distribution for learning.

In our method, $p(l_{n}|\mathbf{w}_{n},\mathbf{l}_{\Omega_{n}}^{n-1},v_{n})$ is modeled by a Bi-LSTM network preceded by a fully-connected layer and followed by fully connected layers and a softmax function $\sigma$ as shown in Fig. 3. The input to this network, for any segment $n$ at a given time $\pi_{n}$ , is the one-hot vector representation of the verb $\mathbf{{v}}_{n}\in\mathbb{R}^{V}$ of a given action $c_{n}$ and its discretized elapsed duration $\mathbf{{d}}_{c_{n}}\in\mathbb{R}^{L}$ as well as the local visual features $\mathbf{w}_{n}\in\mathbb{R}^{\Gamma\times F}$. Here, $V$ is the total number of verbs, $F$ is the input feature dimension, and $\Gamma$ is the number of temporally sampled features over $\alpha$ frames starting with frame $\pi_{n}$. At the end, this network outputs the corresponding verb-dependent future progress probability corresponding to each bin. This probability is expressed as an $L$-dimensional vector $\mathbf{k}_{v_{n}}$, whose $i$-th dimension is the probability that the duration of action $c_{n}$ falls in the $i$-th progress unit for verb $v_{n}$, given the inputs described above.

During training, we used a Gaussian to represent the progress probability labels as soft one-hot vectors. This representation considers the bins that are closer to the true bin more correct than the further ones. The resulting labels are used to compute the standard cross-entropy loss, as the DurNet loss function.

Finally, we translate this progress indicator back to time expressed as number of frames, according to verb-dependent steps ${s}_{v}$:

Thus, the i-th discretized duration $l_{v,i}$ for verb $v$ corresponds to the i-th dimension of vector $\mathbf{k}_{v_{n}}$, and the value of $\mathbf{k}_{v_{n}}$ in the i-th dimension gives the probability of discretized duration $l_{v_{n},i}$.

This network selects the label of the action occurring at any time in the video. Each action is decomposed as a $(verb,object)$ pair. The importance of objects and verbs in action recognition has been studied before [36, 44]. For example, the verb “take” in both “take bowl” and “take cup” is expected to visually look the same way. These two actions only differ in their corresponding objects. This approach has the advantage that not only the network can access more samples per class (verb/object), but also classification is done over fewer number of classes, because several actions share the same verb/object. This is specifically helpful in weakly-labeled data as the frame-level ground truth is not reliable. The probability of the selected action is obtained by the factorized equation below:

$\eta[\,]$ is a normalization function that assures :

The Action Selector Network consists of three components: i) The verb selector network. ii) The object selector network. iii) The main action recognizer (Fig. 1). The influence of each network is adjusted by the $\zeta,\beta$ and $\lambda$ hyper parameters.

i) The Verb Selector Network(VSNet): It focuses only on the local temporal features during the given time frame [$\pi_{n}$, $\pi_{n}+\alpha-1$] to select the correct verb $v_{n}$ for segment $n$. The video-level verb labels $\mathbf{v}_{\boldsymbol{\tau}}\in\{0,1\}^{V}$ are also given as input to the network, where for every $i\in\{0,1,...,V-1\}$, $v_{\boldsymbol{\tau}i}=1$ if $v_{\boldsymbol{\tau}i}$ is present in the video-level verbs, otherwise $v_{\boldsymbol{\tau}i}=0$.

ii) The Object Selector Network(OSNet): Similar to the VSNet, using the local temporal features, this module selects the correct segment object $o_{n}$ from the set of video-level objects $\mathbf{o}_{\boldsymbol{\tau}}\in\{0,1\}^{O}$, where $O$ is the number of available objects in the dataset. Selecting the target object is also influenced by the type of the verb for a given action according to Eq. 8. In order to model this dependency, latent information from the VSNet flows into the OSNet (Fig. 3).

iii) The Main Action Recognizer(MAR): Unlike the other two components, this module produces frame-level probability distribution for the main actions. This network is more discriminative than the other two and particularly helpful in videos with repetitive verbs and objects. Note that the MAR module can be replaced by any baseline neural network architecture like CNNs or RNNs.

Finally, as shown in Eq. 8, the probability of a segment action is defined by fusing the output of the three above-mentioned networks. In the special case of $\zeta,\beta=1$ and $\lambda=0$, the definition of Eq. 8 would be truly probabilistic, and there would be no need for the normalization function $\eta$. The contribution of each network is quantitatively shown in Sec. 4.2.3. It is noteworthy to mention that our method is equally applicable without the verb-object decomposition assumption. In case there is no specific object associated with actions, our formulation still stands by setting $\zeta=0$ and working with the actions as our set of verbs.

We introduce a beam search algorithm with beam size $B$ to find the most likely sequence of segments, as specified by a sequence of labels $\mathbf{c}_{1}^{N}$ and a sequence of lengths $\mathbf{l}_{1}^{N}$. By combining Eq. 1 with Eq. 4 we obtain:

In frame-level beam search, different sequences of action classes are considered at every single frame until the end of the video. In contrast, our Segment-Level Beam Search allows the algorithm to consider such sequences only at the beginning of every segment. This technique is inspired by the fact that actions do not change rapidly from one frame to another.

We introduce the notation $A_{i}(c,l,t_{i})$ to represent the probability of segment-level alignment $i$ until frame $t_{i}$ for each video, where $c$ and $l$ are the action class and length of the last segment. We also define $\max^{B}\{a_{1},a_{2},...,a_{n}\}$ as the set of $B$ greatest $a_{i}$, and calculate $A_{i}(c,l,t_{i})$ of alignment $i$ recursively for every action $c_{n}$ and length $l_{n}$ of segment $n$. Then, the $B$ most probable alignments with $n$ segments are selected over all combinations of $c_{n}$ and $l_{n}$. Algorithm 1 summarizes the procedure for our proposed Segment-Level Beam Search with the following constraints:

• •

  $c_{1}=\tau_{1}$, $c_{N}=\tau_{M}$

• •

  $t_{i}\leq T,\,\forall i\in\{1,2,...,B\}$

$\phi(c_{n-1})$ refers to the set of possible actions for segment $n$. $\phi(c_{n-1})$ is either a repetition of the action $c_{n-1}$ of the previous segment or the start of the next action in $\boldsymbol{\tau}$. The final segment labels $\mathbf{c}_{1}^{N}$ and $\mathbf{l}_{1}^{N}$ are derived by keeping track of the maximizing arguments $c_{n}$ and $l_{n}$ in the maximization steps.

Note that $p(c_{n}|\mathbf{x}_{1}^{T},\boldsymbol{\tau})$ in Algorithm 1 is factorized according to Eq. 8, and every $c_{n}\in\phi(c_{n-1})$ is broken down to its corresponding $(v_{n},o_{n})$ pair. This factorization approach encourages segments that cover the whole duration of an action to avoid the penalty each time a new segment is added. This results in faster alignments with a smaller number of unreasonably short segments.

Time complexity of our Segment-Level Beam Search, for each video, depends on the beam size $B$, number of segments $N$ and number of length bins $L$. As $B$ and $L$ are constant values, the time complexity for the algorithm above would be $O(N)$, and only limited to the number of segments per video. Based on our experiments, for the current public action alignment datasets, $N_{max}\approx 70$ is two orders of magnitude less than $T_{max}\approx 9700$. This makes the proposed beam search more efficient than the Viterbi algorithms used in [31, 22] and [19], which have the complexity of $O(T^{2})$ and $O(T)$ respectively.

Comparison Settings. In addition to evaluating existing methods, we also evaluate some combinations of existing methods, as follows: 1,2) Ours/ [31] pg and Ours/ [22] pg: Ours initialized with NNviterbi [31] and CDFL [22] pseudo-ground truth respectively, and a single layer GRU as the MAR. 3) Ours/ [8] pg: Ours initialized with the training results of [8] as our pseudo-ground truth, and the TCFPN [8] network as the MAR. 4)  [8]/ [31] pg: The ISBA+TCFPN method [8] initialized with NNViterbi [31] pseudo-ground truth. 5)  [31]/ [8] pg: The NNViterbi method [31] initialized with [8] pseudo-ground truth.

Action Alignment Results. Table 1 shows results for weakly-supervised action alignment. Our method produces better or competitive results for most cases on both datasets. Initialized with CDFL, our method achieves state-of-the-art in two of the three metrics for the Breakfast dataset and in one metric on the Hollywood. We compare our method with CDFL [22], NNViterbi [31] and TCFPN [8] more extensively, because they are the best open source methods that follow a similar pseudo-ground approach for training. Also for better comparison, in Table 1 we present the results of training NNViterbi on the pseudo ground-truth from TCFPN and vice versa.

In direct head-to-head comparisons with CDFL, NNViterbi and TCFPN, the proposed method often outperforms the respective competitor, and in some cases the head-to-head performance improvement by our method is quite significant. Our method improves action alignment results of TCFPN [8] and NNViterbi [31] in 5 (Table LABEL:head2head_tcfpn) and 4 (Table LABEL:head2head_nnviterbi) out of 6 metrics respectively. In addition, we outperform CDFL in frame-level accuracy with and without background on the Breakfast dataset, and when tested on the Hollywood dataset, CDFL accuracy without background is improved while the inference complexity is decreased to $O(N)$ from CDFL’s $O(T^{2})$ (Table LABEL:head2head_cdfl).

In Table 2, our Segment-Level Beam Search achieves consistent improved results in frame accuracy for both datasets when the background frames are excluded. Considering acc-bg is essential especially for the Hollywood dataset as on average around 60% of the video frames are background, so acc values can be misleading.

There are two plausible explanations on why the performance of our method for non-background actions is not equally repeated for the background segments. First, there is a lack of defined structure in what background can be, which makes it harder to learn. Second, there are cases where background depicts scenes where a person is still or no movement is happening. It is a tough task for even humans to predict how long that motionless scene would last, so the DurNet can easily make confident wrong predictions resulting in inaccurate alignments of background segments.

Fig. 5 shows how alignment results vary with video length on the Breakfast Dataset. The performance of our method compared to NNViterbi and TCFPN improves as video length increases. In longer videos, the DurNet can maintain the same action longer depending on the context, while in [31] any duration longer than the action average length gets penalized.

All analysis and ablation study is done using the TCFPN [8] pseudo ground-truth initialization. We also ran our ablation study experiments on the Breakfast dataset mainly, because it consists of videos with many actions and high duration variance, so the impact of learning duration can be measured more effectively.

We set 19 and 14 to be the number of objects and verbs (including background as a separate object/verb) in the Breakfast dataset. $\zeta$, $\beta$, and $\lambda$ were adjusted to 1, 30, and 5 respectively for our selector network using [31] and [22] as the baseline. In experiments where TCFPN results [8] were used as the initial pseudo ground-truth, the aforementioned parameters were slightly changed to 1, 40, and 1.

There are 17 actions (including the background) in the Hollywood Extended dataset, and most of these actions do not share verbs or objects with each other. Hence, it would be redundant to decompose the main actions into their verb and object attributes. As a result, for this dataset, we removed the object selector component and used the 17 main actions as our verbs. $\beta$, and $\lambda$ were set to 3 and 1, and 20 and 1 for the TCFPN [8] and NNViterbi [31] baselines respectively. In cases where CDFL [22] were used, $\beta$ was increased to 50.

Around 60% of the frames are background in this dataset. Therefore, it is worth mentioning that a naive classifier, that outputs “background” for every single frame, can achieve results competitive to the state-of-the-art on the acc metric. This is why we emphasize that, specifically for the Hollywood Extended dataset, evaluation using acc-bg is more informative. Our method outperforms existing models on this metric while producing better or competitive results on IoU.

During our observations, we realized that the provided frame-level features are missing for a significant amount of frames in four videos²²21-P34_cam01_P34_friedegg, 2-P51_webcam01_P51_coffee, 3- P52_stereo01_P52_sandwich, 4-P54_cam01_P54_pancake in the Breakfast dataset. While TCFPN [8], NNViterbi [31] and CDFL [22] originally trimmed those videos, we decided to remove them for all experiments including our method as well as all baselines [8, 31, 22]. In Tables 1 and 2 of the main paper, we denote with symbol ${\dagger}$ the best results that we obtained after running the authors’ source code for multiple times. The reason we ran the code multiple times is that each training process is randomly initialized and leads to different final result.

For CDFL [22] in Table 1 and 2, the alignment acc-bg on the Hollywood dataset is somewhat different than the one mentioned in the referenced paper. Similarly, for TCFPN [8], in some cases, our reproduced results are not the same as the ones mentioned in [8]. In this case, we reported the results after contacting the authors and having their approval. For a fair comparison in both baselines, we reported the results, that represent the initial pseudo ground-truth in our method.

Without loss of generality, our final accuracy depends on the quality of the initial pseudo ground-truth, so we have provided the initial pseudo ground-truth and pre-trained main action recognizer models (for TCFPN and NNViterbi on the Breakfast dataset) that we used as supplementary material so our results can be reproduced precisely. All the code and pre-trained models that we provide in supplementary material will be publicly available upon publication of this paper.