Key Frame Proposal Network for Efficient Pose Estimation in Videos

We will represent the dynamics of the input data by using the dynamics-based atomic (DYAN) autoencoder introduced in [19], where the atoms are the impulse response $y(k)=cp^{k-1}$ of linear time invariant (LTI) systems with a pole¹¹1Poles are in general complex numbers. Systems with real outputs with a non real pole $p$ must also have its conjugate pole $p^{*}$: $y(k)=cp^{k-1}+c^{*}.{p^{*}}^{(k-1)}$. $p$, $c$ is a constant and $k$ indicates time. The model uses $N\gg T$ atoms, collected as columns of a dictionary matrix $\mathbf{D}\in\mathbb{R}^{T\times N}$:

Let $\mathbf{Y}\in\mathbb{R}^{T\times M}$ be the input data matrix, where each column has the temporal evolution of a datapoint (i.e. one coordinate of a human joint or the value of a feature, from time 1 to time $T$). Then, we represent $\mathbf{Y}$ by a matrix $\mathbf{C}\in\mathbb{R}^{N\times M}$ such that ${\mathbf{Y}}=\mathbf{D}\mathbf{C}$, where the element $\mathbf{C}(i,j)$ indicates how much of the output of the $i^{th}$ atom is used to recover the $j^{th}$ input data in $\mathbf{Y}$:

In [19], the dictionary $\mathbf{D}$ was learned from training data to predict future frames by minimizing a loss function that penalized the reconstruction error of the input and the $\ell_{1}$ norm of $\mathbf{C}$ to promote the sparsity of $\mathbf{C}$ (i.e. using as few atoms/pixel as possible):

In this paper, we propose a different loss function to learn $\mathbf{D}$, which is better suited to the task of key frame selection. Furthermore, the learning procedure in [19] requires solving a Lasso optimization problem for each input before it can evaluate the loss (2). In contrast, the loss function we derive in section 3.2 is computationally very efficient, since it does not require such optimization step.

Given an input video $\mathcal{V}$ with $T$ frames, consider a tensor of its deep features $\mathcal{Y}\in\mathbb{R}^{T\times c\times w\times h}$ with $c$ channels of width $w$ and height $h$, reshaped into a matrix $\mathbf{Y}\in\mathbb{R}^{T\times M}$. That is, the element $\mathbf{Y}(k,j)$ has the value of the feature $j$, $j=1,\dots,M=cwh$, at time $k$. Then, our goal is to select a subset of key frames, as small as possible, that captures the content of all the frames. Thus, we propose to cast this problem as finding a minimal subset of rows of $\mathbf{Y}$ (the key frames), such that it would be possible to recover the left out frames (the other rows of $\mathbf{Y}$) by using these few frames and their atomic dynamics-based representation.

Problem 1 can be written as the following optimization problem:

The first term in the objective (3) minimizes the recovery error while the second term penalizes the number of frames selected. The constraint (4) establishes that $\mathbf{C}_{r}$ should be the atomic dynamics-based representation of the key frames and the constraints (5) force the binary selection matrix $\mathbf{}P_{r}$ to select $r$ distinct frames. However, this problem is hard to solve since the optimization variables are integer ($r$) or binary (elements of $\mathbf{P}_{r}$).

Next, we show how we can obtain a relaxation of this problem, which is differentiable and suitable as a unsupervised loss function to train our key frame proposal network. The derivation has three main steps. First, we use the constraint (4) to replace $\mathbf{C}_{r}$ with an expression that depends on $\mathbf{P}_{r}$, $\mathbf{D}$ and $\mathbf{Y}$. Next, we make a change of variables so we do not have to minimize with respect to a matrix of unknown dimensions. Finally, in the last step we relax the constraint on the binary variables to be real between 0 and 1.

Given a video with $T$ frames, let $\mathbf{H}_{r}\in\mathbb{R}^{r\times 2J}$ be the 2D coordinates of $J$ human joints for $r$ key frames, $\mathbf{P}_{r}\in\mathbb{R}^{r\times T}$ be the associated selection matrix, and $\mathbf{D}^{(h)}$ be a dynamics-based dictionary trained on skeleton sequences using a DYAN autoencoder [19]. Then, the Human Pose Interpolation Module (HPIM) finds the skeletons $\mathbf{H}\in\mathbb{R}^{T\times 2J}$ for the entire sequence, which can be efficiently computed. Its expression can be derived as follows. First, use the reduced dictionary: $\mathbf{D}^{(h)}_{r}=\mathbf{P}_{r}\mathbf{D}^{(h)}$ and (9) to compute the minimum Frobenius norm atomic dynamics-based representation for the key frame skeletons $\mathbf{H}_{r}$: $\mathbf{C}_{r}={\mathbf{D}^{(h)}_{r}}^{T}(\mathbf{D}^{(h)}_{r}{\mathbf{D}^{(h)}_{r}}^{T})^{-1}\mathbf{H}_{r}.$ Then, using the complete dictionary $\mathbf{D}^{(h)}$, the entire skeleton sequence $\mathbf{H}=\mathbf{D}^{(h)}\mathbf{C}_{r}$ is given by:

where $\mathbf{D}^{(h)}{\mathbf{D}^{(h)}}^{T}$ can be computed ahead of time.

Fig. 3 shows the architecture for the K-FPN, which is trained completely unsupervised, by minimizing the loss (16). It consists of two Conv2D modules (Conv + BN + Relu) followed by a Fully Connected (FC) and a Sigmoid layers. The first Conv2D downsizes the input feature tensor from $(T\times 512\times 7\times 7)$ to $(T\times 64\times 3\times 3)$ while the second one uses the temporal dimension as input channels. The $T\times 1$ output of the FC layer is forced by the Sigmoid layer into logits close to either 0 or 1, where a ‘1’ indicates ‘key frame’ and its index which one. Inspired by [38], we utilized a control parameter $\alpha$ to form a customized classification layer, represented as $\sigma(\alpha x)=[1+exp(-\alpha x)]^{-1}$, where $\alpha$ is linearly increased with the training epoch. By controlling $\alpha$, the output from the K-FPN is nearly a binary indicator such that the sum of its elements is the total number of key frames. The training and inference proceduresare summarized in Algorithms 1 to 3. and code is available at https://github.com/Yuexiaoxi10/Key-Frame-Proposal-Network-for-Efficient-Pose-Estimation-in-Videos.

The proposed K-FPN can be modified to process incoming frames, after a minimum set of initial frames has been processed. To do this, we add a discriminator module as shown in Fig. 4, consisting of four (Conv2D + BN + Relu) blocks, which is used to decide if an incoming frame should be selected as a key frame or not. The discriminator is trained to distinguish between features of the incoming frame and features predicted from the set of key frames selected so far, which are easily generated by multiplying the atomic dynamics-based representation of the current key frames with the associated dynamics-based dictionary extended with an additional row (since the number of frames is increased by one) [19]. The reasoning behind this design is that when the features of the new frame cannot be predicted correctly, it must be because the frame brings novel information and hence it should be incorporated as a key frame.

We followed conventional data preprocessing strategies. Input images were resized to 3x224x224 and normalized using the parameters provided by [10]. After that, in order to capture a better pose estimation from the pose model, we utilized the person bounding box to crop each image and pad to 384x384 with a varying scaling factor from 0.8 to 1.4. The Penn Action dataset provides such an annotation, while JHMDB does not. Therefore, we generated the person bounding box on each image by using the person mask described in [20].

Following [23, 20, 31], we evaluated our performance using the PCK score [41]: a body joint is considered to be correct only if it falls within a range of ${\beta L}$ pixels, where $L$ is defined by $L=\max(H,W)$, where $H$ and $W$ denote the height and width of the person bounding box and ${\beta}$ controls the threshold to justify how precise the estimation is. We follow convention and set ${\beta=0.2}$.

Our full framework consists of three steps: given an input video of length $T$, K-FPN first samples $k$ key frames; then, pose estimation is done on these $k$ frames; and HPIM interpolates these results for the full sequence. The reported running times are the aggregated time for these three steps. All running times were computed on NVIDIA GTX 1080ti for all methods.

Figs. 1 and 5 show qualitative examples where it can be seen that the proposed approach can successfully recover the skeletons from a few key frames. Please see the supplemental material for more examples and videos.

In order to evaluate the effectiveness of our approach, we conducted ablation studies on the validation split for each dataset.

Comparisons against the state-of-art are reported in Table 4. We report our performance using Resnet34 for Penn Action and Resnet18 for Sub-JHMDB, and also using Resnet50, since it is the backbone used by [23]. Our approach achieves the best performance and is 1.6X faster (6.8ms v.s 11ms) than the previous state-of-art [23] for the Penn Action dataset, using an average of 17.5 key frames. Moreover, if we use our lightest model (Resnet34), our approach is 2X faster than [23] with a minor PCK degradation. For the sub-JHMDB dataset, [23] did not provide running time and it is not open-sourced. Thus, we compare time against the best available open sourced method [20]. Our approach performed the best of all methods, with a significant improvement on elbow (95.3%) and wrist (91.3%). For completeness, we also compared against the baseline [36], which is a frame-based method, on both datasets. We can observe that by applying our approach with the lightest model, we run more than 2X faster than [36] without any degradation in accuracy.

We hypothesize that our approach can achieve better performance than previous approaches using fewer input frames because the network selects “good” input frames, which are more robust when used with the frame-based method [36]. To better quantify this, we ran an experiment where we randomly partially occluded/blurred/changed illumination at random frames in the sub-JHMBD dataset. Table 5 shows that our approach (using ResNet18) is more robust to all of these perturbations when compared to [36].